GPTQ, Explained and Run

Search a model hub for "GPTQ" or "w4a16" and you find 4-bit versions of nearly every open model, most made with the same method and the same defaults. This piece opens it: what it minimizes, the algorithm in about 30 lines of PyTorch on a real layer, then a real run on Qwen3-8B with llm-compressor and vLLM, varying the settings that matter.

What GPTQ minimizes

A linear layer computes Y = XWᵀ, where X holds the inputs (one row per token) and W the weights. Quantizing W to Q changes the output by X(W − Q)ᵀ. Round to nearest makes each entry of W − Q as small as it can, independently. GPTQ (Frantar et al., 2022) aims directly at the output: it looks for the quantized Q that minimizes ‖XWᵀ − XQᵀ‖² over a set of calibration inputs X, one layer at a time.

Written per row of W (one output channel), that error is a quadratic form: for a change δ in the row, the output error is δ H δᵀ / 2, with H = 2XᵀX, the Hessian of the error. H only depends on the inputs. Its diagonal says how large each input channel is; its off-diagonal terms say which channels move together. That is everything the calibration data contributes: one 4,096 × 4,096 matrix per layer input in Qwen3-8B.

The algorithm

GPTQ comes from Optimal Brain Quantization (OBQ), which quantizes one weight at a time and, after each, updates all the remaining weights of the row to compensate the error it just made. If weight q is rounded with error e = w_q − quant(w_q), the best correction to the weights F not yet quantized is

δ_F = −e / [H_F⁻¹]_qq · (H_F⁻¹)_:,q

and H_F⁻¹ must then be updated to remove q. Done weight by weight, in a greedy order per row, this is far too slow for a model with billions of weights. GPTQ makes three changes:

1. One order for all rows. The authors found that the greedy order barely beats "quantizing the weights in arbitrary order", especially on large layers. If every row is quantized in the same column order, every row shares the same H_F⁻¹, so the inverse is updated once per column instead of once per weight: the cost drops by a factor of the number of rows.
2. Lazy batch updates. Rounding column i only depends on updates to column i, so updates to later columns can wait. GPTQ processes blocks of 128 columns, keeps updates inside the block, and applies them to the rest of the matrix once per block, as one matrix multiplication. This turns a memory-bound loop into GPU-friendly work: "an order of magnitude speedup for very large models".
3. Cholesky reformulation. Repeated updates of H⁻¹ accumulate numerical error. GPTQ computes once the rows of H⁻¹ it will need, through a Cholesky decomposition, and adds dampening, a small constant on H's diagonal (1% of its average), to keep the inversion stable.

Here is the core, from the script behind the measurements below. Hinv is the upper Cholesky factor of H⁻¹; S holds the scales, one per group of 128 inputs, computed from the original weights:

H = 2 * X.double().T @ X.double()
H += DAMP * torch.mean(torch.diag(H)) * torch.eye(n, dtype=H.dtype)
Hinv = torch.linalg.cholesky(torch.cholesky_inverse(torch.linalg.cholesky(H)), upper=True)
for i1 in range(0, n, block):                       # blocks of 128 columns
    i2 = min(i1 + block, n)
    W1, Hi = W[:, i1:i2].clone(), Hinv[i1:i2, i1:i2]
    Q1, E1 = torch.zeros_like(W1), torch.zeros_like(W1)
    for i in range(i2 - i1):
        w, d = W1[:, i], Hi[i, i]
        q = quant(w, S[:, (i1 + i) // GROUP].double())   # round this column
        Q1[:, i] = q
        err = (w - q) / d
        W1[:, i:] -= err.unsqueeze(1) @ Hi[i, i:].unsqueeze(0)   # compensate inside the block
        E1[:, i] = err
    Q[:, i1:i2] = Q1
    W[:, i2:] -= E1 @ Hinv[i1:i2, i2:]              # lazy update of the remaining columns

The inner loop rounds a column, then pushes its error onto the columns to its right, weighted by how strongly their inputs correlate with this one. The last line applies a whole block's corrections to the rest of the matrix at once.

On a real layer

I ran this on layer 10's q_proj of Qwen3-8B (4,096 inputs, 4,096 outputs), in INT4 with groups of 128 and the same scales for both methods. The calibration inputs are the layer's real inputs on WikiText-2 text; the error is measured on 2,048 other tokens from much later in the text, which neither method saw:

GPTQ's weights are further from the original than round to nearest's, by a third or more. Its outputs are closer: 4.5% error instead of 6.8% with 8,192 calibration tokens, 35% less. That is the whole idea in one picture. Weight error is not the target: errors that cancel in the dot product cost nothing, and GPTQ arranges for them to cancel.

More calibration tokens help, steadily: 128 tokens already beat round to nearest, 8,192 do much better. With few tokens, H describes the calibration text more than the layer's typical inputs, and the compensation fits that text.

Running it with llm-compressor

On a whole model, the same algorithm runs layer by layer: llm-compressor feeds the calibration set through one decoder layer at a time, accumulates H for each linear layer's input, quantizes the layer, then passes the quantized layer's outputs on to the next one (propagate_error, on by default), so each layer compensates for the errors of the layers before it. The full script:

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import GPTQModifier

MODEL = "Qwen/Qwen3-8B"
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype="auto")
tok = AutoTokenizer.from_pretrained(MODEL)

ds = load_dataset("neuralmagic/LLM_compression_calibration", split="train[:1024]").shuffle(seed=42)
ds = ds.map(lambda ex: {"text": tok.apply_chat_template(ex["messages"], tokenize=False)})
ds = ds.map(lambda ex: tok(ex["text"], max_length=8192, truncation=True, add_special_tokens=False),
            remove_columns=ds.column_names)

oneshot(model=model, dataset=ds,
        recipe=GPTQModifier(targets="Linear", scheme="W4A16_ASYM", ignore=["lm_head"]),
        max_seq_length=8192, num_calibration_samples=1024)
model.save_pretrained("qwen3-8b-gptq", save_compressed=True)
tok.save_pretrained("qwen3-8b-gptq")

scheme="W4A16_ASYM" means 4-bit integer weights with a zero point, one scale per group of 128, activations left in 16 bits; W4A16 is the same without a zero point. ignore=["lm_head"] keeps the output layer in BF16. The modifier's defaults are the paper's: dampening 1% (dampening_frac=0.01), blocks of 128 columns (block_size=128), plus activation ordering (actorder="static", below).

The result is a compressed-tensors checkpoint that vLLM loads as is:

vllm serve ./qwen3-8b-gptq

vLLM 0.31.0 reads the quantization config, logs Using MarlinLinearKernel for CompressedTensorsWNA16, and loads 5.71 GiB of weights instead of 15.27 GiB for BF16. Speed is the same as any INT4 weight-only checkpoint (How Much Does Each Bit Cost?): GPTQ changes which integers are stored, not how they are used.

On a GB10 (DGX Spark), quantizing Qwen3-8B took 44 seconds for round to nearest and, for GPTQ, 4 minutes with 32 calibration samples, 9 minutes with 128 and 29 minutes with 512. Time grows with the calibration tokens, since every layer runs the calibration set.

Measured on Qwen3-8B: what changes the result

I quantized Qwen3-8B several ways, all INT4 in groups of 128, served each with vLLM 0.31.0 and measured perplexity on the same 298,862 tokens of WikiText-2 (lower is better; BF16 scores 9.72):

Four things stand out.

Calibration size matters. Without a zero point, GPTQ with 32 chat samples (39,000 tokens) does no better than round to nearest: 10.288 against 10.286. With 128 samples it gains a little (10.23), with 512 (572,000 tokens) it removes a quarter of round to nearest's loss (10.14).

The zero point matters as much as GPTQ. Round to nearest with a zero point scores 10.121, better than symmetric GPTQ with 512 samples. A zero point lets each group's integer range follow its own minimum and maximum instead of being centered on zero, which on its own removes 29% of the symmetric loss.

With a zero point, my GPTQ runs gained nothing. GPTQ with the same 512 chat samples scores 10.132. I then followed the recipe Red Hat AI documents for its Qwen3-8B checkpoint: 1,024 samples of neuralmagic/LLM_compression_calibration (instruction data: math, code, reasoning, general questions), up to 8,192 tokens each, 26 minutes of quantization. It scores 10.124. Neither beats plain rounding with a zero point.

The published checkpoint does. Red Hat AI's own checkpoint scores 10.054 on the same test. Its shipped configuration matches mine (zero point, groups of 128, activation ordering; the card's text says groups of 64, the config says 128), but it was made in May 2025 with an older llm-compressor, and the card's code snippet asks for the symmetric scheme while the shipped model is asymmetric, so the snippet is not exactly what produced it. I could not reproduce its result from the published recipe.

A caution on the test itself. WikiText-2 is Wikipedia text and none of these calibration sets is: a method fitted to chat or instructions can help on the work it was calibrated for and look neutral on Wikipedia. Perplexity on one corpus is a quick, sensitive check, not a verdict; check the tasks you serve (on GSM8K, all the 4-bit checkpoints of How Much Does Each Bit Cost? were within about a point of BF16).

Settings that matter

• Scheme. Prefer a zero point (W4A16_ASYM) for INT4: it costs one small integer per group of 128 weights and gave the largest single gain here. Group size trades size for accuracy the same way as in Integers for AI: 128 is the common default.
• Calibration data. Use text like what the model will serve, with its chat template applied. On this model, neither chat nor instruction data made GPTQ beat plain rounding on WikiText-2; an effect may show on tasks like the calibration data instead, which I did not measure.
• Calibration amount. Dozens of samples are not enough; plan for a few hundred thousand tokens or more, and for minutes to tens of minutes per 8B model.
• Activation ordering. actorder quantizes the columns with the largest inputs first, while many columns remain to absorb their error. The original repository introduced it as --act-order ("quantizing columns in order of decreasing activation size"). llm-compressor's static (alias weight) keeps the groups fixed in the stored order, so the checkpoint loads and runs like one without it; it is on by default.
• Dampening. 1% of the average diagonal is the paper's choice and the default. If llm-compressor warns "Hessian inversion failed" for some modules, those were quantized with round to nearest instead; it suggests raising dampening_frac or adding calibration samples.

What to take away

• GPTQ minimizes each layer's output error on calibration data, not the weight error: it rounds one column at a time and pushes each column's error onto the columns not yet rounded, through the inverse of H = 2XᵀX.
• On one Qwen3-8B layer it moves the weights further from the original (17% error vs. 13%) and keeps the outputs closer (4.5% error vs. 6.8%).
• With llm-compressor it is one modifier; the checkpoint serves in vLLM through the same Marlin kernel as any INT4 weight-only model.
• On Qwen3-8B, the zero point changed the result more than GPTQ did: with one, none of my GPTQ runs beat plain rounding on WikiText-2, while Red Hat AI's published checkpoint did. Measure the checkpoint you serve against plain rounding with a zero point before trusting the method name.

The next piece covers AWQ, which uses the same calibration data differently: it scales the important input channels instead of compensating errors.