Quantizing the KV Cache: FP8 and Below

The weights of Qwen3-8B take 15.3 GiB once; its KV cache takes 144 KiB for every token of every request, so at long context the cache, not the model, decides how many requests fit. vLLM 0.31.0 has nine ways to store it in fewer bits, from FP8 to 3-bit; seven ran on our GPU. Their capacity is what the bits predict, their quality loss shows up in long-context recall and not in perplexity, and their speed follows the attention kernel more than the bytes.

Why the cache, and what each format stores

Every token a model has read leaves a key and a value vector per layer and per KV head in the cache. For Qwen3-8B, that is 36 layers × 8 KV heads × 128 dimensions × 2 (key and value) × 2 bytes = 144 KiB per token in BF16 (the formula, with a calculator). A 32k-token conversation holds 4.5 GiB; ten of them hold 45 GiB, three times the weights. Halve the bytes per token and twice as many conversations fit, or each one can be twice as long.

--kv-cache-dtype picks the format. The ones vLLM 0.31.0 offers for a standard attention model, and what they store:

The modes are listed in vLLM's KVQuantMode; the TurboQuant presets in its config, which credits the rotation-plus-scalar-quantization design to the DRIVE and EDEN line of work. The backend column is what vLLM selected on our GPU, an NVIDIA GB10 (Blackwell, compute capability 12.1), with Qwen3-8B in BF16 and --gpu-memory-utilization 0.70, which left 67 to 68 GiB for the cache depending on the backend (vLLM also offers nvfp4_4over6, an NVFP4 variant, and we did not run it or turboquant_k3v4_nc):

The capacity is the arithmetic, plus two details. FP8 halves the bytes and holds 1.96 times as many tokens, not 2, because the FlashInfer backend left 1.5 GiB less memory for the cache than FlashAttention did. The per-token-head formats pay 2.25 KiB per token for their scales (one 4-byte scale per token, head, layer, key and value: 36 × 8 × 2 × 4 bytes), which is why they hold 950k tokens instead of 975k. INT4 per token and head holds 3.7 times as many tokens as BF16, the most of all.

TurboQuant gains less than its bits suggest because it protects the model's edges: for a dense model, vLLM keeps the first two and the last two layers in BF16 (source); its config notes that without this, the aggressive presets lose about 30 GSM8K points on Qwen3-4B. Those 4 layers cost 16 KiB per token, more than a third of the 3-bit preset's 41.5 KiB, which is why it holds 3.4 times as many tokens rather than about 5. It also means TurboQuant's quality results below come with 4 protected layers that the other formats don't have.

Where the quantized cache is read, and why perplexity misses it

Prefill writes the prompt's keys and values into the cache; decode reads them back for every new token (the two phases). Whether prefill itself reads the quantized values depends on the backend. TurboQuant's first prefill chunk attends over the keys and values it just computed, in full precision, and only stores them quantized (source).

That has a consequence for evaluation. The usual quick check, perplexity over a text, is computed from prompt log-probabilities, so it only exercises prefill. Our WikiText-2 perplexity (298,862 tokens, windows of 2,048) came out at 9.722 for BF16 and at the same value, 9.7218, for all three TurboQuant presets, the 3-bit one included: the quantized cache was never read. The other formats moved by less than 0.03, except INT4 per token and head (9.812). Those formats do read their cache during the test, but over windows of 2,048 tokens, too short for errors to accumulate the way they do at 30,000. TurboQuant's own config reports +20.59% perplexity for its 3-bit preset (source), measured on a path that reads the cache; a quick evaluation through the serving engine never takes that path. To see what a quantized cache costs, you need a test where the answer is generated while reading it.

Quality: what long-context recall loses

We used three tests that decode while reading the cache:

• Hard retrieval. About 30,000 tokens of WikiText-2 with 64 lines like "The access code for Juniper-24 is 951936." inserted at random depths, names that differ only by a number, and a question asking for four codes at once; a trial passes when all four are exact. 50 trials, the same for every format. (An easier version, one of 8 codes at 24,000 tokens, gave every format 60 out of 60.)
• GSM8K, all 1,319 problems, short answers.
• GSM8K with thinking on, 200 problems: answers of about 1,100 tokens that read their own cached reasoning.

Each trial is 2 points, and the 50 haystacks are the same for every format, so the fair comparison is trial by trial against BF16: how many trials each format failed that BF16 passed, and the reverse.

• Only 3-bit loses recall beyond noise. TurboQuant 3-bit failed 10 trials that BF16 passed and passed one that BF16 failed (p = 0.012), and missed 8.5% of individual codes against 3.5% for BF16.
• Everything down to 4 bits is within what 50 trials can resolve. Per-token-head INT8 and FP8 sit at BF16's level (90%, 88%); per-tensor FP8 and INT4 per token and head at the bottom of the band (82%, 80%), losing 3 and 4 trials without gaining any. That one-sided pattern hints at a small loss; 50 trials can't confirm it.
• GSM8K's accuracy hides it; its per-problem answers don't. Every format stays within 2 points of BF16 on all 1,319 problems. But compared problem by problem with BF16, the formats that match BF16 flip about as many answers each way (FP8: 18 lost, 21 gained), while the 3- and 4-bit TurboQuant presets lose twice as many as they gain (41 lost against 20 gained at 4 bits, 50 against 25 at 3 bits). With thinking on, the same two presets went from 96.5% to 94.0%, losing 5 and 6 problems that BF16 solved and gaining at most one.

Speed: the kernel decides

Capacity is guaranteed by the arithmetic; speed is not. We sent 32 requests of 16,384 prompt tokens and 256 answer tokens, 8 at a time, to servers started like this:

$ vllm serve Qwen/Qwen3-8B --gpu-memory-utilization 0.70 --max-model-len 32768 \
    --kv-cache-dtype fp8_per_token_head

The startup log says which attention kernel the format got:

Using TRITON_ATTN attention backend out of potential backends: ['TRITON_ATTN']
GPU KV cache size: 950,992 tokens, Maximum concurrency for 32,768 tokens per request: 29.02

At 16k tokens of context, each decode step reads 2.25 GiB of BF16 cache per request, so a smaller cache should mean faster steps. Per-tensor FP8 was the fastest format: 23% less time per token and 18% less time to the first token than BF16. But it also ran on a different kernel (FlashInfer, against FlashAttention for BF16), and reading a 16k-token prompt is limited by arithmetic, not by the cache, so we can't say how much of that gain is the bytes. The per-token-head formats show the kernel's weight from the other side: on vLLM's Triton attention backend they took a third more time per token and 1.8 times longer to the first token, whether their values are 8 or 4 bits wide. TurboQuant sits between, close to BF16, its smaller reads paid for by decoding a codebook. (Time per output token is high for every format because each step shares the GPU with other requests' 16k-token prefills; the comparison between formats is what matters.)

What didn't run

Two formats from the list did not survive the test on this Blackwell GPU:

• nvfp4 refused to start: No valid attention backend found for cuda with ... kv_cache_dtype=nvfp4. Only FlashInfer lists it, and only on compute capability 10.x (B200, GB200) through TensorRT-LLM kernels.
• fp8_e5m2 started, then crashed on the first long prompts with CUDA error: an illegal memory access was encountered inside FlashInfer.

Neither is a statement about the formats; both are about the kernels one GPU generation gets, which is worth checking before you plan around a format.

Choosing

• Start with FP8. On a GPU where it runs on a fast kernel (FlashInfer here), per-tensor fp8 doubles capacity and was the fastest format we measured; its recall stayed within noise of BF16, at the low end. If recall matters more than speed, per-token-head FP8 or INT8 matched BF16 at the same capacity, on a slower kernel here. On a model without QK-norm, give fp8 calibrated scales (see Gotchas).
• Go to 4 bits only when capacity is the bottleneck, and test the workload that reads long contexts. int4_per_token_head holds 3.7 times as many tokens and kept GSM8K, but lost recall; on our GPU it also ran on the slow Triton kernel.
• Treat 3-bit as experimental. TurboQuant 3-bit gives 3.4 times the capacity and the largest quality loss we saw.
• Measure speed on your GPU. The same format runs on different attention kernels on different GPUs, and the kernel decided speed more than the bytes did.

Gotchas

Per-tensor FP8 uses a scale of 1.0 unless the checkpoint brings one. A BF16 checkpoint has no KV scales, and vLLM defaults them to 1.0: keys and values are cast to FP8 as they are. E4M3 is a floating-point format, so its relative precision is the same from 2⁻⁶ to 448; a scale only matters for values outside that range, which lose precision below it and saturate above it. Qwen3 normalizes its queries and keys (QK-norm), which keeps keys in range; on a model without it, use calibrated scales (llm-compressor stores them in the checkpoint) or a per-token-head format.

Perplexity through the engine can't see the cache. For some backends it never reads it; for the others, its windows are too short. Test with long generations or long-context recall.

Check the backend before you plan on a format. The startup log names it; the same format can land on a different kernel on another GPU generation.

Headline accuracy is blind to it. On GSM8K, every format stayed within 2 points of BF16; only the problem-by-problem comparison showed the TurboQuant presets losing twice as many problems as they gained.

This is vLLM. SGLang 0.5.21 offers fp8_e4m3, fp8_e5m2, mxfp8, nvfp4, and an MX-style FP4 for its KV cache (source), not the per-token-head and TurboQuant formats; the measurements here don't transfer to its kernels.

Fewer bits per token is one way to shrink the cache; caching fewer layers is the other, and hybrid, sliding, and sparse attention sizes what that saves.