AIREITERvLLM Humming is a strong candidate for high-concurrency quantized MoE serving on Hopper-class hardware, but public benchmark evidence does not support treating it as a universal backend upgrade. H100 group-32 scaling and Ampere batch-1 decode show why the exact model, GPU, format, and batch shape must be tested.
The short answer: Humming is a conditional upgrade
Humming is a JIT-compiled GEMM kernel library for quantized inference, released by the vLLM community with Ant Group. It targets dense and mixture-of-experts (MoE) workloads across multiple quantization formats and NVIDIA GPU generations. The official announcement reports kernel speedups of up to 5x and selected end-to-end MoE gains of 14%–43%, but those figures come from specific model, hardware, format, and concurrency combinations (vLLM announcement).
My adoption rule is simple:
- Test Humming first for large-batch or high-concurrency W4A8/W4A16 MoE serving on supported Hopper or newer configurations.
- Do not assume a gain for batch-1 decode, older Ampere GPUs, or unusual group-wise scaling.
- Keep a fallback because current issue reports include shape-routing bugs and backend-specific slow paths.
One independent DeepSeek-V4.1-Flash report reached 5,856 output tokens/s with a tuned vLLM baseline, 12,802 with data-parallel attention, and 17,634 with modified Humming W4A8 kernels on one four-GH200 node (engineering report). That result required a kernel fix and tuning fix, so it is evidence for a workload class, not an out-of-the-box promise.
What Humming changes in a vLLM deployment
Humming moves quantized matrix multiplication into a flexible JIT kernel layer for dense and MoE GEMM paths. The project targets NVIDIA architectures from Turing onward and multiple low-bit formats (Humming repository).
In the cited Hopper test, Marlin dequantized FP4 weights to BF16, while Humming upcast FP4 weights to FP8 inside the SM and used FP8 tensor-core multiplication. That distinction helped only after the decode workload became compute-bound. Memory traffic, shape tiling, scale handling, launch overhead, and register pressure can dominate instead.
Where the speedup is real, and where it disappears
The public evidence is best read as a map of performance boundaries rather than one headline percentage.
High-concurrency MoE: the clearest win
The official DeepSeek-V4.1-Flash example compares Marlin W4A16 with Humming W4A8 on eight H20 GPUs and reports 14%–43% higher end-to-end MoE throughput, with the larger improvement at higher concurrency. A user reposting the benchmark gave the concrete comparison as 442 to 634 output tokens/s at concurrency 64, while noting that the quantization format changed as well (user discussion).
The result supports Humming as a promising serving path, not a pure same-format kernel multiplier.
H100 group-32 W4A8: a serious warning
Humming issue #38 tested MXFP4 weights with MXFP8 activations and group-32 scales on an H100. All compared paths were numerically correct, but the fused path was dramatically slower in the reported shapes. At (4096,3072,4096), fused Humming took 4.969 ms versus 0.580 ms for the QDQ baseline; at (4096,14336,4096), it took 22.298 ms versus 1.256 ms (issue #38).
The issue later recorded local tuning above 500 TFLOPS, while an experimental implementation reached roughly 580–665 TFLOPS with extra preprocessing and weight storage. Scale granularity, tile choice, register pressure, and preprocessing can therefore change the result completely.
SM80 single-request decode: do not infer Ampere gains
Issue #72 reports weak INT3 utilization on NVIDIA SM80 for M=1 decode. On a CMP 170HX, the measured Humming kernel reached about 691 GB/s against approximately 1,591 GB/s peak HBM bandwidth, or 43.4% utilization. A comparable RTX 5070 Ti SM120 run reached about 753 GB/s against 827 GB/s peak, or roughly 91% utilization (issue #72).
The same report cites A800 repository results of 692.4 GB/s at M=1, only 34.0% of the stated 2,039.04 GB/s peak. Plain BF16 reached 1,232.3 GB/s in that comparison.
“Could the current SM80 INT3 Humming path be suboptimal for this workload?” —
yangfangyunyun, vLLM Humming issue #72
Installation and the safest first test
Official materials identify vLLM 0.29.0 as the point where CUDA installations include Humming as a dependency. Use --moe-backend humming for the MoE path, --linear-backend humming for supported linear layers, or the documented Humming quantization configuration for a model-specific route (vLLM quantization docs). Check the release and model documentation before copying a flag between architectures.
Start with a controlled test:
- Record vLLM, CUDA, PyTorch, GPU architecture, checkpoint quantization, and model shape.
- Hold prompt length, output length, concurrency, and repetition count constant.
- Compare the existing backend with Humming using the same serving command plus the relevant backend flag.
- Measure output tokens/s, TTFT, inter-token latency, GPU memory, and startup/JIT warm-up time.
- Repeat after the kernel cache is warm, then test batch 1 and production concurrency.
- Keep an automatic or manual fallback for unsupported shapes and failed selection.
A throughput win is not sufficient if TTFT, tail latency, warm-up cost, or memory use regresses. vLLM’s perf-eval project recommends YAML recipes, repeated runs, median aggregation, and fixed input/output lengths for comparable tests (perf-eval repository).
Backend routing can block a valid Humming configuration
A separate vLLM issue shows why fallback testing matters. In issue #60700, a compressed-tensors W4A8 MoE checkpoint had an intermediate dimension of 640. CUTLASS required 256-aligned dimensions, while Humming could handle the tested shape, but an unconditional assertion ran before the selected backend loaded. Explicit --moe-backend humming therefore failed too.
The reporter’s patch scoped the assertion to CUTLASS. With that change, all 62 shards loaded on two H100 NVL GPUs, and the test reported 12%–50% lower TTFT and 2%–13% higher system throughput than its reference CUTLASS setup (issue #60700). The issue was still open when checked, with PR #60711 proposed as the routing fix.
A failed Humming launch may therefore be a vLLM capability-check problem rather than a kernel problem. Check the selected backend, shape restrictions, and fix status before discarding the configuration.
A decision table for operators
| Your workload | Humming decision | Why |
|---|---|---|
| High-concurrency MoE on Hopper/GH200 | Test it early | Independent serving work reached 17,634 tok/s after fixes and tuning. |
| DeepSeek-style decode with large batches | Strong candidate | FP8 tensor-core execution can help once MoE GEMMs are compute-bound. |
| Batch-1 or low-batch decode on SM80 | Do not assume a gain | Issue #72 reports 34%–43% effective bandwidth in tested INT3 paths. |
| Group-32 activation scaling on H100 | Benchmark a narrow matrix of shapes | Issue #38 shows severe regressions in one fused path. |
| Non-256-aligned W4A8 MoE shapes | Check vLLM version and routing fixes | Shared assertions can block Humming before weights load. |
| Production with no source-patch budget | Use only after a representative bake-off | Some independent wins required kernel or heuristic changes. |
FAQ
Which GPUs does Humming support?
The project targets NVIDIA GPUs from Turing onward and documents multiple quantization paths. Support does not mean every format and workload has been tuned equally; SM80 and unusual scaling cases need their own benchmark.
Does Humming require a pre-quantized checkpoint?
Humming is an inference-kernel library, not a general-purpose quantizer. The cited workflows use checkpoints with compatible quantized weights and activation schemes, so verify the model’s format in the vLLM documentation and repository before serving.
Is Humming faster than every other vLLM backend?
No. It is faster in selected quantized workloads, especially the reported high-concurrency MoE cases. Results vary with architecture, format, dimensions, scale grouping, and batch size.
The practical trade-off is clear: Humming can materially speed up a well-matched quantized vLLM service, but a short, repeatable bake-off is part of adopting it because the public results are configuration-specific.