Quantization in Practice

What NVFP4 stores, how vLLM serves it on Blackwell, and what it changes in memory, speed, and quality.

What can be quantized in a model, what each saves (memory, bandwidth, compute), and what each risks, with Qwen3-8B's numbers.

Qwen3-8B in thirteen formats from BF16 to 3-bit GGUF, on one GPU with vLLM: memory, decode speed, prompt reading, perplexity, and GSM8K, plus the formats that would not run.

Why rounding each weight to the nearest value is not enough, what calibration data is for, and the two kinds of outliers in a real model's activations, measured on Qwen3-8B.

How GPTQ trades weight error for output error, in 30 lines of PyTorch on a real layer, then how to run it with llm-compressor and serve it with vLLM, and which settings actually change the result on Qwen3-8B.

Which 1% of weights matter, why scaling them protects them, and the search behind AWQ, in a few lines of PyTorch on a real layer, then AWQ on Qwen3-8B with llm-compressor and vLLM against plain rounding, GPTQ and a published checkpoint.

Why activation outliers break INT8, how SmoothQuant moves them into the weights, and what it changes for FP8 W8A8, measured on Qwen3-8B with vLLM.

A rotation flattens activation outliers and, on real layers, rescues INT4 activations. On Qwen3-8B in NVFP4, it cost 5 GSM8K points instead.

Why Kimi K2 Thinking ships in INT4 and gpt-oss and K3 in MXFP4: training with rounding in the loop, explained and measured on a small model.

Seven KV cache formats in vLLM, from FP8 to 3-bit, on one model: how many more tokens fit, what long-context recall loses, and why speed follows the kernel.

What Q4_K_M, IQ3_XXS, and the rest mean: the block layouts from llama.cpp's source, the real bits per weight of each file, and the weight error of each type on Qwen3-8B.

GSM8K said every 4-bit format matched BF16. A noise baseline, paired comparisons, token-level KL, and a code benchmark showed which ones did not.