AWQ: Protecting the Weights That Matter

Next to almost every GPTQ checkpoint on a model hub sits an AWQ one: same model, 4-bit integer weights, groups of 128, about the same size, served by the same kernels. The difference is in how the integers were chosen. GPTQ rounds one column at a time and repairs the error on the columns that remain. AWQ (Lin et al., 2023) repairs nothing: before rounding, it rescales the weights so that the few that matter most lose less precision. This piece reproduces its starting observation on one real Qwen3-8B layer, builds the method from it in a few lines of PyTorch, then runs it on the whole model with llm-compressor and measures it against plain rounding and GPTQ.

Which weights matter

AWQ starts from an experiment. Quantize a layer to low precision, but keep 1% of its weights in 16 bits. Which 1%? The paper keeps whole input channels (columns of W) and compares three choices: the channels whose weights are largest, random channels, and the channels whose inputs are largest on calibration data. Only the last one helps much.

I repeated it on layer 10's q_proj of Qwen3-8B (4,096 inputs, 4,096 outputs), the layer of the GPTQ demo, with the same quantizer (symmetric, groups of 128) and the same data: 8,192 tokens of WikiText-2 to measure the inputs, 2,048 later tokens to measure the error of the layer's output. Keeping 41 channels (1%) in 16 bits:

Random channels change nothing. Channels with the largest weights barely help (6.8% to 6.7% in INT4). Channels with the largest inputs cut the error by more than a third, in INT4 and in INT3. The two selections overlap little: only 13 of the 41 channels are in both lists.

The reason is the product. A weight's contribution to the output is the weight times its input. In this layer a few input channels are large on every token (one reaches 26 times a typical channel, see Post-Training Quantization), so the rounding error of their weights is multiplied by a large number every time. A large weight with a small input matters less than a modest weight with a large input. Which weights matter is decided by the activations, hence the name: activation-aware weight quantization.

Protecting without mixed precision

Keeping 1% of the channels in 16 bits works but is awkward to serve: the kernel would have to handle two formats in one matrix. AWQ gets most of the benefit with all weights in 4 bits, through a rescaling that leaves the layer's output unchanged before rounding.

For any positive number s, w · x = (w · s) · (x / s). Multiply one input channel's weights by s and divide that input by s: the output is the same. Now quantize the scaled weights. The rounding error of a weight is at most half a quantization step, and the step is set by the extremes of its group. If the scaled weight does not become one of them, the step does not change, so the absolute error on w · s stays the same size; once the input is divided by s, the error that reaches the output is s times smaller. The cost lands on the other weights of the group when the scaled one becomes an extreme: the step grows and every other weight in the group loses precision. A little scaling helps the busy channel; too much starves its neighbors.

The paper shows it on OPT-6.7B in INT3: scaling the 1% of salient channels by 2 takes perplexity from 23.54 to 11.92; scaling by 4 does worse than by 2. The division by s costs nothing at run time either: it is folded into the operation that produces the input, the RMSNorm before q_proj, k_proj and v_proj, or the previous linear layer, once, offline.

How much to scale: the search

Rather than choose channels and a factor by hand, AWQ scales every input channel according to its average magnitude, with one exponent for the whole layer:

s = mean(|x|) ^ alpha

At alpha = 0 every s is 1, plain rounding. At alpha = 1 each channel is scaled by its full input size. AWQ tries a grid of alphas between 0 and 1, quantizes W · diag(s), divides the input by s, and keeps the alpha with the smallest output error on the calibration data. On the same layer, in a few lines:

def awq(W, bits, xmean, alpha):
    s = xmean.pow(alpha).clamp(min=1e-4)
    s = s / (s.max() * s.min()).sqrt()           # center the scales around 1
    return quant_groups(W * s, bits) / s          # Q(W diag(s)) diag(s)^-1

xmean = calib.abs().mean(0)                       # mean |x| per input channel
grid = [(a / 20, out_err(W, awq(W, 4, xmean, a / 20), calib)) for a in range(21)]
alpha = min(grid, key=lambda g: g[1])[0]

The curve has the shape the argument predicts. Scaling with alpha up to about 0.25 lowers the error from 6.9% to 5.35%; beyond about 0.6 it is worse than not scaling at all, and at alpha = 1 the error is 11.7%. On held-out tokens, the best alpha gives 5.4% against 6.8% for plain rounding: a fifth less error, with every weight in 4 bits. Keeping the top 1% in 16 bits did better (4.4%), at the cost of a mixed format. In INT3 the best alpha is 0.3 and the error goes from 15.5% to 12.4%.

Two properties follow from the method. Its scales come from the average input magnitude per channel, not from GPTQ's 4,096 × 4,096 matrix per layer. And it fits nothing to the calibration text beyond one number per layer: there is no reconstruction and no update of individual weights, which the paper credits for AWQ's robustness to the choice of calibration data. The paper also searches a clipping range for the weights after scaling; the demo above does not, and neither does llm-compressor.

Running it with llm-compressor

In llm-compressor, AWQ is a transform that runs before quantization: AWQModifier computes and folds the scales, then QuantizationModifier rounds the scaled weights. The script behind the measurements below:

from llmcompressor import oneshot
from llmcompressor.modifiers.transform.awq import AWQModifier
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = [
    AWQModifier(duo_scaling=True),   # the default; False is the paper's search space
    QuantizationModifier(targets="Linear", scheme="W4A16_ASYM", ignore=["lm_head"]),
]
oneshot(model=model, dataset=ds, recipe=recipe, max_seq_length=2048, num_calibration_samples=512)
model.save_pretrained("qwen3-8b-awq", save_compressed=True)

model, tok and ds are prepared exactly as in the GPTQ script, here with 512 UltraChat samples of up to 2,048 tokens. Import AWQModifier from llmcompressor.modifiers.transform.awq. Version 0.14.0 still has the old path, llmcompressor.modifiers.awq, but there it is a compatibility wrapper that expects the quantization scheme in its own arguments; combined with a separate QuantizationModifier, as above, it fails with "QuantizationModifier requires that quantization fields be specified".

For Qwen3, the modifier knows which operation absorbs each scale: the attention's input norm for q_proj, k_proj and v_proj; the MLP's input norm for gate_proj and up_proj; up_proj for down_proj. A fourth pair, v_proj for o_proj, is skipped in all 36 layers ("36 mappings were skipped due to incompatible shapes"): with grouped-query attention, v_proj has 1,024 outputs and o_proj 4,096 inputs, so there is no one-to-one channel to scale, and o_proj is rounded as is. For each pair, the modifier runs the calibration set through the block, tries 20 points of the grid, and keeps the scales that minimize the error of the block's output, not just of one linear layer.

The default differs from the paper. With duo_scaling=True, the scale also divides by the weights' own average magnitude per input channel: s = mean|x|^alpha / mean|w|^(1 − alpha). Channels whose weights are already large are scaled less, since scaling them grows the step of their group. duo_scaling=False is the paper's mean|x|^alpha; "both" splits the grid between the two. I quantized Qwen3-8B both ways.

The output is a compressed-tensors checkpoint, served by vLLM like the GPTQ one:

vllm serve ./qwen3-8b-awq

Nothing in the stored format records that AWQ chose the weights: they are 4-bit integers with a scale and a zero point per group of 128, and vLLM runs them through the same Marlin kernel as any INT4 weight-only checkpoint. Checkpoints made with AutoAWQ, such as Qwen's own Qwen/Qwen3-8B-AWQ, use a different packing, which vLLM detects and runs through its AWQ variant of the Marlin kernel.

Measured on Qwen3-8B

I quantized Qwen3-8B with both scalings, INT4 with a zero point in groups of 128, calibrated on the 512 UltraChat samples of the GPTQ run, served each with vLLM 0.31.0 and measured perplexity on the same 298,862 tokens of WikiText-2 as before (lower is better; BF16 scores 9.72):

AWQ beat plain rounding and GPTQ on the same data. Round to nearest with a zero point scores 10.121 and GPTQ 10.132. AWQ scores 10.066 with llm-compressor's default and 10.061 with the paper's scaling: it removes about 14% of the loss that round to nearest leaves (0.34 points of perplexity above BF16 instead of 0.40), where GPTQ removed none. It lands next to Red Hat AI's published GPTQ checkpoint (10.054), which my GPTQ runs did not reproduce.

The two scalings tie. 10.066 against 10.061 is a difference too small to call on one run. On this model, dividing by the weights' magnitude neither helps nor hurts; the input magnitude does the work, as the one-layer experiment suggested.

The published AWQ checkpoint is the worst of the 4-bit set. Qwen's own Qwen/Qwen3-8B-AWQ scores 10.181, worse than plain rounding with a zero point. It was made with a different tool (its weights use AutoAWQ's packing). The method name on a checkpoint says how it was made, not how good it is. The same caution as before applies to the test: on GSM8K, this checkpoint scored 93.3% against 93.1% for BF16 (How Much Does Each Bit Cost?), so a gap of a few hundredths of perplexity on Wikipedia text may not show on the tasks you serve.

AWQ took longer to run. On a GB10 (DGX Spark), each AWQ checkpoint took 2 hours 4 minutes, against 29 minutes for GPTQ on the same samples. The grid search is the cost: for each of the three pairs in each of the 36 layers, llm-compressor runs the 512 samples through the block 20 times. The served model is the same size as any INT4 weight-only checkpoint: vLLM loads 5.71 GiB of weights and logs Using MarlinLinearKernel for CompressedTensorsWNA16, as for GPTQ.

AWQ or GPTQ

• Both produce the same kind of checkpoint. INT4 weights with a scale and a zero point per group of 128, the same kernel, the same speed. The choice affects only accuracy and the time to quantize.
• AWQ is the simpler method and was the better one here. It fits one exponent per pair of layers from average input magnitudes; on Qwen3-8B with 512 chat samples it beat both plain rounding and GPTQ on WikiText-2.
• GPTQ can win with the right recipe. Red Hat AI's GPTQ checkpoint matched my AWQ runs. If a well-made checkpoint of either kind exists, measuring it is faster than making your own.
• Keep the zero point and the defaults. Use W4A16_ASYM; duo_scaling made no difference here, so the default is fine.
• Budget the time. On an 8B model with 512 samples of up to 2,048 tokens, expect hours for AWQ, not minutes. The time grows with the calibration tokens, so fewer or shorter samples shorten it.

What to take away

• Which weights matter is decided by their inputs: keeping the 1% of channels with the largest activations in 16 bits cut one Qwen3-8B layer's INT4 output error from 6.8% to 4.4%; choosing them by weight size barely helped (6.7%).
• AWQ protects those channels with every weight in 4 bits: it multiplies their weights by s and divides their inputs by s, folded into the previous operation, with s = mean|x|^alpha and alpha found by a grid search on the output error. On the layer, the error fell to 5.4%.
• With llm-compressor it is AWQModifier followed by QuantizationModifier; the result serves in vLLM through the same Marlin kernel as GPTQ.
• On Qwen3-8B, AWQ (10.06 to 10.07) beat plain rounding with a zero point (10.12) and GPTQ (10.13) on the same calibration data, while Qwen's published AWQ checkpoint (10.18) did not. Measure the checkpoint you serve.

The next piece, SmoothQuant, uses the same rescaling for a different goal: moving activation outliers into the weights so that activations, too, can run in 8 bits.