NVFP4 on Blackwell: Serving a 4-Bit Model End to End

Blackwell GPUs compute natively in a 4-bit floating-point format, NVFP4, and Hugging Face is filling up with NVFP4 checkpoints. They promise a model a quarter the size of BF16 and much faster. We served Qwen3-8B four ways on a Blackwell machine and measured: the 4-bit version decoded 2.7 times faster than BF16, read long prompts twice as fast, and lost about 1.5 points on a math benchmark. This piece shows what NVFP4 stores, how vLLM runs it, and what to check before you switch.

Four bits, three numbers

A 4-bit number has only 16 possible values. NVFP4's are those of E2M1: one sign bit, two exponent bits, one mantissa bit, which gives 0, 0.5, 1, 1.5, 2, 3, 4, and 6, and their negatives (NVIDIA's description). A model's weights are numbers like 0.0229 or −0.0693, so they cannot be stored as such: each group of weights gets a scale that maps them onto that grid.

NVFP4 uses two scales:

• A block scale for every 16 values, stored in FP8 (E4M3). Small blocks follow the local range of the data closely.
• A tensor scale, one FP32 number for the whole tensor, so the block scales fit in FP8's range.

A weight is its 4-bit value × its block scale × the tensor scale. The cost: 16 values of 4 bits plus one 8-bit scale, 4.5 bits per weight, against 16 for BF16 and 8 for FP8.

Two other 4-bit formats are worth knowing. MXFP4, from the Open Compute Project's microscaling standard, uses the same E2M1 values with one power-of-two scale per 32 values: cheaper to apply, coarser. INT4 uses integers from −8 to 7 with one scale per group, usually 64 or 128 values, and is what most weight-only methods (GPTQ, AWQ) produce. Here is what each does to real weights:

Two things to try:

1. Compare the three formats on these 64 weights from Qwen3-8B. NVFP4's error is the smallest (8.6% against 10.9% for MXFP4 and 11.2% for INT4 here): its blocks are smaller and its scales are not limited to powers of two. It also spends a little more on scales (4.5 bits against 4.25).
2. Turn weight 15 into an outlier. One large value stretches the scale of its whole block, and every small value next to it loses precision. With 16-value blocks, the damage stays local; with one scale for 64 values, it spreads to all of them.

Those errors look large, but a model is not a list of independent numbers: what matters is how much its answers change, which only a measurement can tell.

Serving it with vLLM

An NVFP4 checkpoint is served like any other model; vLLM reads the format from its config:

$ vllm serve RedHatAI/Qwen3-8B-NVFP4 --gpu-memory-utilization 0.85 --max-model-len 16384

The startup log says which kernel it picked:

Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
Model loading took 5.98 GiB memory and 61.302655 seconds
GPU KV cache size: 667,664 tokens, Maximum concurrency for 16,384 tokens per request: 40.75x

vLLM 0.31.0 tries its NVFP4 kernels in a fixed order and keeps the first one your GPU and checkpoint support (source). On Blackwell, that is a 4-bit tensor-core kernel: weights and activations in FP4 (W4A4). On older NVIDIA GPUs (from compute capability 7.5), it falls back to Marlin, which stores weights in FP4 but computes in 16 bits, and warns that "this may degrade performance for compute-heavy workloads" (source). If nothing fits, it falls back to a slow emulation path, again with only a warning (source). Read that log line before you trust any NVFP4 benchmark.

Measured: one model, four formats

We served Qwen3-8B on an NVIDIA GB10 (Blackwell, DGX Spark class) with vLLM 0.31.0, 2026-10-06, in four forms: the original BF16, Qwen's own FP8 release, and two NVFP4 checkpoints, RedHatAI/Qwen3-8B-NVFP4 (made with LLM Compressor) and nvidia/Qwen3-8B-NVFP4 (made with NVIDIA Model Optimizer). Speed comes from vllm bench serve (512-token prompts and 256-token answers at 1 and 32 requests; 8,000-token prompts for prompt reading), quality from 250 GSM8K math problems, answered greedily with thinking turned off.

What it shows:

1. Decoding follows the bytes. One request decodes at the speed the weights stream from memory: 15.3 GiB to 6.0 GiB is 2.6 times fewer bytes, and decoding got 2.7 times faster. That is the memory-bound rule from earlier pieces, measured.
2. Prompt reading got faster too, twice as fast as BF16. Reading a prompt is limited by computation, and W4A4 runs the matrix multiplications on 4-bit tensor cores. A weight-only format would not do that.
3. The weights are not a quarter of BF16. 5.98 GiB, not 15.27 / 3.5 ≈ 4.4: both checkpoints leave the embeddings and the output layer in BF16, and for Qwen3-8B, with its 151,936-token vocabulary and separate input and output matrices, those alone are 2.3 GiB.
4. Quality moved a little. 95.6% to 94.0% and 94.4% on GSM8K. With 250 problems, one standard error is about 1.4 points, so this is at the edge of the noise: a small loss, or none. Run your own evaluation on your own task before deciding; math with long reasoning chains is often where quantization shows first.
5. One checkpoint quietly changed the KV cache. NVIDIA's checkpoint declares kv_cache_quant_algo: FP8, and vLLM followed it: Using fp8_e4m3 data type to store kv cache. Twice the KV cache room, and 19% more total throughput at 32 requests than the other NVFP4 checkpoint. Good news, if you expected it. Check the log, because a checkpoint can change more than the weights.

Should you switch?

• On Blackwell, for latency and cost, yes, after an evaluation. NVFP4 gave 2.7 times faster decoding and twice faster prompt reading, with a small or no accuracy loss on this model. The case is strongest when memory is short or one user's speed matters.
• Prefer FP8 when quality must not move. It matched BF16 exactly on our evaluation, decoded 1.6 times faster, and it runs on Hopper too.
• On GPUs without FP4 tensor cores, NVFP4 is a memory saver, not a speed-up for prompts: vLLM runs it weight-only through Marlin.
• Pick checkpoints that say how they were made. Calibration data, which layers stay in high precision, and whether the KV cache is quantized all differ between producers, and all of them show up in your numbers.

Next

A smaller model per GPU lets one machine serve more. The next piece goes the other way, to many machines: disaggregated serving, where prompt reading and token generation run on separate GPUs, and the stacks (NVIDIA Dynamo, llm-d) that coordinate them.