SmoothQuant and W8A8: Quantizing Activations Too

Quantizing weights to 8 bits is easy; quantizing the activations that flow into them is not, because a handful of channels carry values 10 to 25 times larger than the median one. SmoothQuant fixes this with one algebraic trick: shrink those channels and grow the matching weights by the same factor, so the product is unchanged and both sides fit in 8 bits. On a real Qwen3-8B layer, it cut INT8's error sixfold; on FP8 it changed almost nothing, and on a Blackwell GPU in vLLM, FP8 is the only W8A8 there is.

Why activations are the hard part

W8A8 means 8-bit weights and 8-bit activations, so the matrix products run on 8-bit tensor cores: twice the arithmetic rate of BF16, which is what speeds up prefill. Weight-only formats (INT4 GPTQ or AWQ) shrink memory and speed up decode, but they dequantize to BF16 before multiplying, so prompt reading gets no faster.

The weights are well behaved. The activations are not: in every transformer large enough to matter, a few hidden dimensions carry values far larger than the others, consistently, token after token (measured here). We captured the inputs of one linear layer of Qwen3-8B, layer 10's query projection, on 8,192 tokens of WikiText-2:

INT8 has 255 levels. With one scale for the whole tensor, that scale must cover 17.5, so each level is 0.14 wide: the median channel, whose values stay under 0.68, gets about ten levels. A scale per token helps a little (each token gets its own range) but every token still carries channel 3828. A scale per channel would fix it, but a matrix product can't apply a different scale per input channel without breaking the tensor-core kernel, which wants one scale per row of X and one per column of W.

The trick: move the range into the weights

For a linear layer, Y = X·W. For any positive vector s, one value per input channel:

Y = (X · diag(s)⁻¹) · (diag(s) · W)

Divide input channel j by s_j, multiply the weights that read channel j by s_j, and the output is exactly the same. Pick s large where the activations have outliers and the activations become flat. The weights pay for it: multiplying row j by s_j raises the maximum of every output channel that reads it (0.41 to 3.02 here). They can afford it because they start small and flat, and their rounding error is no longer multiplied by an outlier input.

SmoothQuant chooses s from both sides, with one knob, the migration strength alpha:

s_j = max|X_j|^alpha / max|W_j|^(1 - alpha)

Alpha 1 gives s_j = max|X_j|: every activation channel peaks at 1 and the weights take the whole range. As alpha falls toward 0, the weight term takes over and s starts flattening the weights at the activations' expense (at exactly 0, s_j = 1 / max|W_j|, which is not "no smoothing"). Values around 0.5 split the range. The division by s doesn't cost anything at run time: X comes out of a normalization layer (RMSNorm before the attention projections and before the MLP), so s is folded into the norm's weights once, offline. That is also why SmoothQuant only smooths those layers: llm-compressor's default mappings for Llama-style models such as Qwen3 pair input_layernorm with the query, key, and value projections and post_attention_layernorm with the gate and up projections (source). The output and down projections, whose inputs don't come straight out of a norm, are left alone.

One layer, measured

We swept alpha from 0.1 to 1 on the same layer, plus a point without any smoothing (labelled alpha 0), quantizing weights per output channel and activations either with one static scale per tensor (from the calibration tokens' maximum) or one dynamic scale per token, in INT8 and in FP8 E4M3. The error is measured on the layer's output for 2,048 held-out tokens, ‖XW − Q(X)Q(W)‖ / ‖XW‖:

Four things stand out.

• For INT8, smoothing is the difference between unusable and fine. Static per-tensor INT8 goes from 8.1% to 1.4% error, sixfold; per-token INT8 from 4.8% to 1.1% at alpha 0.7.
• For FP8, it barely matters. 2.09% to 1.95%. FP8's exponent gives each value its own scale within the format: E4M3 keeps about the same relative precision from 0.68 to 17.5, so a big channel costs the small ones almost nothing. Uniform INT8 levels can't do that.
• Smoothed INT8 beats FP8 on this layer. 1.06% per token at alpha 0.7 against 1.77% for the best FP8 point. FP8's 3 mantissa bits leave a floor of about 2% error; INT8's 255 even steps are finer once the range is flat.
• Alpha has a sweet spot, and two traps. Too little (0.1) is worse than no smoothing (10.6% against 8.1% per tensor): near 0 the weight term dominates the scale, so it rescales channels by their weights' range instead of flattening the activations. Too much (1.0) moves so much range into the weights that their quantization error takes over. On this layer the best alpha was 0.7 to 0.8, the paper's recommendation for most models is 0.5, and llm-compressor's default is 0.5 (source). Sweep it on your model.

The full model, served

The layer tells you about error; the model tells you about quality and speed. We made three FP8 W8A8 checkpoints of Qwen3-8B with llm-compressor 0.14.0, each calibrated on the same 256 UltraChat conversations:

# abridged from scripts/gx10/sq_variants.py: model is Qwen3-8B in BF16, tok its tokenizer,
# calib 256 tokenized UltraChat samples
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.transform.smoothquant import SmoothQuantModifier

recipe = [
    SmoothQuantModifier(smoothing_strength=0.8),        # omit for the plain FP8 checkpoint
    QuantizationModifier(targets="Linear", scheme="FP8", ignore=["lm_head"]),
]
oneshot(model=model, dataset=calib, recipe=recipe, max_seq_length=2048, num_calibration_samples=256)
model.save_pretrained("qwen3-8b-fp8-static-sq", save_compressed=True)
tok.save_pretrained("qwen3-8b-fp8-static-sq")

scheme="FP8" stores FP8 weights with one scale per tensor and one static activation scale per layer, measured on the calibration data; scheme="FP8_DYNAMIC" (no calibration data needed) uses a scale per output channel for the weights and computes a scale per token at run time. vLLM 0.31.0 serves all three with the same command, vllm serve <checkpoint>, through its compressed-tensors FP8 path. The setup for everything below: an NVIDIA GB10 (Blackwell, compute capability 12.1), vLLM 0.31.0 with --gpu-memory-utilization 0.70 --max-model-len 16384; perplexity on WikiText-2 (298,862 tokens); all 1,319 GSM8K problems, greedy, thinking off; decode with 32 concurrent requests of 512 tokens in and 256 out; the first token of an 8,000-token prompt, median of 8. The BF16 row comes from our 13-format sweep, same harness:

• Smoothing brought static FP8 back to BF16's perplexity: 9.757 to 9.706, against 9.722 for BF16, a difference too small to call either way. GSM8K moved within its noise (problems flip between runs; one point is about 13 problems).
• Static scales took 9% less time to read the prompt than dynamic ones (1,096 against 1,205 ms for 8,000 tokens). A dynamic scale means finding each token's maximum before every matrix product; the dynamic checkpoint also has per-channel weight scales, so we can't attribute all of it to the activations. Decode speed is the same, set by the 8.8 GiB of weights.
• All three decode 1.6 times faster than BF16 and read prompts 1.5 to 1.6 times faster, the W8A8 payoff.
• Dynamic per-token FP8 had the worst perplexity of the three (9.801), although per-token scales did better on our single layer. The three FP8 checkpoints are within 0.1 of each other and of BF16, differences that come from many layers at once, including the output and down projections that smoothing doesn't touch; we read them as "all close to BF16", not as a ranking. Red Hat's published FP8-dynamic checkpoint of the same model scored 9.82.

INT8 W8A8 doesn't load on Blackwell in vLLM

We also made an INT8 W8A8 checkpoint, SmoothQuant 0.8 followed by GPTQ, llm-compressor's standard INT8 recipe. vLLM 0.31.0 refused to load it:

Error: dispatch_scaled_mm, .../w8a8/cutlass/c3x/scaled_mm_helper.hpp:34, Int8 not supported on SM121.
Use FP8 quantization instead, or run on older arch (SM < 100).

This is not specific to the GB10. vLLM picks its CUTLASS W8A8 kernels by default, and they pass no INT8 implementation for compute capability 10.x (SM100, B200 and B300) or 12.x (SM120, RTX 50-series, RTX PRO, GB10), and raise this error instead. vLLM also has a Triton INT8 kernel, reachable by disabling the CUTLASS one (VLLM_DISABLED_KERNELS=CutlassInt8ScaledMMLinearKernel); we didn't test it. The hardware gives INT8 no speed edge either: it runs at the FP8 rate on Hopper, Ada, and B200, and on Blackwell Ultra NVIDIA lists about 167 INT8 TOPS per GPU against 5,000 for FP8 (the numbers). On Blackwell, W8A8 in practice means FP8.

Choosing

• On Blackwell, use FP8 W8A8. INT8 activations have no CUTLASS kernel in vLLM there, and CUTLASS is the default. Start with static per-tensor FP8, add SmoothQuant at 0.5 to 0.8, and compare perplexity against BF16; on Qwen3-8B, smoothing closed the whole gap.
• On Hopper and Ada, start with FP8. INT8 runs at the same rate. FP8 needs little smoothing and covers the layers SmoothQuant leaves alone; INT8 with SmoothQuant can be more precise where it applies (1.1% against 1.8% on our layer). Try INT8 only if your evaluation shows an FP8 gap.
• On Ampere (A10, RTX 30-series, and the older A100), INT8 W8A8 is the option, since Ampere has no FP8 tensor cores. There, SmoothQuant is not optional: without it, our layer's INT8 error was 8.1% per tensor and 4.8% per token.
• Prefer static activation scales when your traffic resembles your calibration data: they are faster. Use dynamic ones when inputs vary widely (code, many languages, tool output), since a static scale clips anything larger than what calibration saw.

Gotchas

Smoothing only reaches layers whose input can absorb a scale. llm-compressor's default mappings for Qwen3-style models smooth the query, key, value, gate, and up projections, whose input comes out of an RMSNorm. The down projection could take its scale from the up projection, as AWQ's mappings do, but its outliers move from token to token (measured here), so one scale per channel can't fix them; the output projection can't take one from the value projection under grouped-query attention (1,024 channels against 4,096). Rotation methods spread both kinds across all channels; they are a later piece in this series.

Calibration sets the static scales. A static FP8 checkpoint calibrated on chat data meets a 20,000-token code file with activations it never saw; values past the calibrated maximum are clipped. Calibrate on what you serve, or use dynamic scales for inputs that vary widely.

The 8-bit speed-up is mostly a prefill speed-up. The first token of an 8,000-token prompt came 1.6 times sooner (1.8 s to 1.1 s). Weight-only 4-bit formats were slower than BF16 on that same prompt in our sweep (about 2.2 s against 1.8 s), because they dequantize to BF16 before every product. If prompts are long, quantizing activations is what pays.

Activations are not the only tensor that gets rounded at serving time: the KV cache is quantized the same way, with the same choice of scale granularity, and quantizing the KV cache measures what each format costs.