Rotations for 4-Bit Activations: QuaRot, SpinQuant, and Friends

SmoothQuant moves an activation outlier into the weights, one channel at a time, using a scale computed from calibration data. That works when the same channel is always the large one. In Qwen3-8B's down_proj it isn't: a different channel spikes on different tokens, so no fixed scale per channel can flatten it, and smoothing barely helps. A rotation does not care which channel the outlier is in. Multiply the activations and the weights by the same orthogonal matrix and the layer computes exactly the same thing, while every outlier is spread across all channels. That is the idea behind QuaRot, SpinQuant, and QuIP, and the reason 4-bit activations became possible. We measured it on two real layers and on the full model in NVFP4, served by vLLM, and the result on the full model went the other way.

A rotation changes nothing, and that is the point

A linear layer computes Y = X·Wᵀ. For any orthogonal matrix Q (Q·Qᵀ = I):

Y = X · Wᵀ = (X · Q) · (W · Q)ᵀ

So we can quantize X·Q and W·Q instead of X and W. If Q is a Hadamard matrix scaled by 1/√n, every entry is ±1/√n: each output channel of X·Q is a signed sum of all input channels. An outlier of size M in one channel contributes ±M/√n to every channel, and the rest is a sum of thousands of small values, which looks like a Gaussian. Gaussians are easy to quantize: no single value is far above the others, so a scale per token wastes few levels.

Two numbers show it. The ratio of the largest value to the median channel's maximum, which says how many quantization levels the typical channel loses to the outlier, and the kurtosis, which measures how heavy the tails are (3 for a Gaussian; hundreds to thousands when a few channels dominate).

On q_proj, whose outliers sit in fixed channels, smoothing cuts the ratio from 26.5 to 7.3 and the rotation to 1.7. On down_proj, smoothing goes from 29.8 to 23.6, nearly nothing, while the rotation reaches 2.1. Kurtosis falls to 2.8 and 3.8: after rotation, both layers' inputs are close to Gaussian.

What it buys in 4 bits, and what it doesn't

The measure that matters is the error of the quantized product, ‖X·Wᵀ − Q(X)·Q(W)ᵀ‖ / ‖X·Wᵀ‖, on tokens not used for calibration. Two 4-bit formats: INT4 with one scale per token for the activations and one per output channel for the weights, and NVFP4, with an FP8 scale for every block of 16 values (simulated in PyTorch).

For INT4, the rotation is the difference between unusable and usable: 46% error down to 11% on q_proj, 75% down to 22% on down_proj, where smoothing did almost nothing. For NVFP4, the starting error is already six to seven times lower, because a scale every 16 values isolates an outlier inside its block: only its 15 neighbors lose precision. The rotation then adds error, 6.6 to 7.2% and 11.9 to 12.4%. Our reading: it turns a block with one large value and fifteen small ones, which a per-block scale handles well, into blocks of Gaussian values, which E2M1's eight magnitudes fit less well. (The demo uses full-width random rotations; the checkpoints below use 128-wide Hadamard blocks.)

Where the rotations go

A model is not one layer. The SpinQuant paper names four places to put a rotation in a transformer block, and the cost depends on whether it can be folded into the weights:

QuaRot uses random Hadamard matrices in these places; SpinQuant learns R1 and R2 on calibration data; llm-compressor's QuIP modifier, borrowing from the weight-only QuIP methods, puts a Hadamard on the input of every linear layer, applied online. Offline rotations are free at run time; online ones cost a fast Hadamard transform per token, which is why kernels fuse it with the quantization of the activations.

The recipe

llm-compressor 0.14.0 adds the rotation as one more modifier before the quantization step:

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.modifiers.transform import SpinQuantModifier

recipe = [
    SpinQuantModifier(rotations=["R1", "R2"], transform_block_size=128, transform_type="hadamard"),
    QuantizationModifier(targets="Linear", scheme="NVFP4", ignore=["lm_head"]),
]
oneshot(model=model, dataset=calibration, recipe=recipe, max_seq_length=2048, num_calibration_samples=256)
model.save_pretrained("qwen3-8b-nvfp4-spin-r1r2", save_compressed=True)

With transform_type="hadamard", the rotations are fixed, non-random Hadamard matrices in blocks of 128, in SpinQuant's positions: neither learned (SpinQuant) nor randomized (QuaRot; "random-hadamard" gives that). Adding "R4" to the list adds the online rotation of down_proj's input; QuIPModifier(rotations=["v"], ...) rotates the input of every linear layer instead. The checkpoint records the transforms, and vLLM applies the online ones when it loads it.

The full model in NVFP4, on vLLM

We made four NVFP4 W4A4 checkpoints of Qwen3-8B, same calibration data, and served each with vLLM 0.31.0 on an NVIDIA GB10 (Blackwell, compute capability 12.1):

Plain NVFP4, rounded to nearest, is already close to BF16: +0.35 perplexity, and a GSM8K score inside the 92.7-93.1% that BF16 itself scored across our runs. The rotated checkpoint is worse on both: 11% higher perplexity and 72 fewer GSM8K problems solved, where BF16 run twice differs by a net 5 (measuring the damage). Note that R1 and R2 leave down_proj's input, the layer that needed a rotation most, unrotated; that is R4's job, and R4 did not load. Speed is unchanged, as it should be: R1 and R2 are folded into the weights, and the kernels are the same NVFP4 matrix products.

The layer measurements predicted this. NVFP4's blocks of 16 already do what the rotation was invented for, and the rotation makes the blocks harder for E2M1. An ICLR 2026 study of FP4 formats (Egiazarian et al.) reached the same conclusion across models: rotations help MXFP4, whose scales are powers of two shared by 32 values, and hurt NVFP4; its MR-GPTQ method uses block-wise rotations together with GPTQ and scale tuning to narrow the gap, mostly for MXFP4.

Gotchas

Online rotations with NVFP4 did not load on a GB10. Both checkpoints with online transforms failed at startup with RuntimeError: runGemmNv, .../qutlass/csrc/fused_quantize_nv_sm100.cu:187, Error Internal. vLLM sends NVFP4 layers with an input transform to a fused Hadamard-and-quantize kernel from QuTLASS when the GPU's compute capability is at least 10.0 (source); that test admits 12.x GPUs (RTX 50, RTX PRO, GB10), but a vLLM build for both data center and consumer Blackwell compiles QuTLASS for SM100 only (cmake), and that kernel fails on 12.x. Test online rotations on the GPU you will serve on.

Rotations are format-specific. The win for INT4 and MXFP4 is real; for NVFP4, measure before shipping. Perplexity alone saw the regression here, but check a task too.

Block size matters. transform_block_size=128 mixes channels in groups of 128; a full-size Hadamard spreads further, at higher cost for online rotations. With NVFP4, a 128-wide block spreads each outlier over eight 16-value groups, which undoes the isolation the format's small groups provide; the MR-GPTQ paper instead uses block-wise rotations sized to the format's groups.

A rotation does not replace a good rounding method. These checkpoints use round to nearest. Rotations combine with GPTQ (QuaRot and SpinQuant both pair them), and that combination is where the published results come from.

When to rotate

• 4-bit activations with a scale per token or per tensor (INT4 W4A4, MXFP4): rotate. On our layers it cut INT4 error from 46-75% to 11-22%, and R1 and R2 cost nothing at run time.
• NVFP4 W4A4: start without. Measure a rotated variant only with GPTQ-style rounding, and keep it only if a task benchmark improves.
• Weight-only 4-bit (W4A16): activations stay in 16 bits, so rotations matter far less; GPTQ or AWQ do the work.