Articles

Why activation outliers break INT8, how SmoothQuant moves them into the weights, and what it changes for FP8 W8A8, measured on Qwen3-8B with vLLM.

Seven KV cache formats in vLLM, from FP8 to 3-bit, on one model: how many more tokens fit, what long-context recall loses, and why speed follows the kernel.

What TTFT, TPOT, ITL, end-to-end latency, and goodput each measure, how the tools compute them, and why peak throughput is the wrong target.

Why a prompt token costs hundreds of times less than an answer token on the same weights: prefill and decode, seen in a kernel trace.

One chat request followed through a real inference server, from chat template and tokens to prefill, decode, sampling, and streaming, each stage timed.

Which 1% of weights matter, why scaling them protects them, and the search behind AWQ, in a few lines of PyTorch on a real layer, then AWQ on Qwen3-8B with llm-compressor and vLLM against plain rounding, GPTQ and a published checkpoint.

How GPTQ trades weight error for output error, in 30 lines of PyTorch on a real layer, then how to run it with llm-compressor and serve it with vLLM, and which settings actually change the result on Qwen3-8B.

Why rounding each weight to the nearest value is not enough, what calibration data is for, and the two kinds of outliers in a real model's activations, measured on Qwen3-8B.

What Q4_K_M, IQ3_XXS, and the rest mean: the block layouts from llama.cpp's source, the real bits per weight of each file, and the weight error of each type on Qwen3-8B.

Qwen3-8B in thirteen formats from BF16 to 3-bit GGUF, on one GPU with vLLM: memory, decode speed, prompt reading, perplexity, and GSM8K, plus the formats that would not run.

What can be quantized in a model, what each saves (memory, bandwidth, compute), and what each risks, with Qwen3-8B's numbers.

Five places where low precision breaks real computations, each reproduced on Qwen3-8B's own tensors, and the fix every serious implementation applies.