llm-compressor

How GPTQ trades weight error for output error, in 30 lines of PyTorch on a real layer, then how to run it with llm-compressor and serve it with vLLM, and which settings actually change the result on Qwen3-8B.

Which 1% of weights matter, why scaling them protects them, and the search behind AWQ, in a few lines of PyTorch on a real layer, then AWQ on Qwen3-8B with llm-compressor and vLLM against plain rounding, GPTQ and a published checkpoint.

Why activation outliers break INT8, how SmoothQuant moves them into the weights, and what it changes for FP8 W8A8, measured on Qwen3-8B with vLLM.

A rotation flattens activation outliers and, on real layers, rescues INT4 activations. On Qwen3-8B in NVFP4, it cost 5 GSM8K points instead.