GGUF Quant Types Decoded: K-Quants and I-Quants

Open the files of any popular model in GGUF format, the format of llama.cpp and the tools built on it (Ollama, LM Studio), and you face a list like this one, for Qwen3-8B: Q2_K, Q3_K_M, Q4_0, Q4_K_M, Q5_K_M, Q6_K, Q8_0, IQ4_XS, UD-IQ3_XXS, UD-Q4_K_XL... This reference decodes the names from llama.cpp's own source code, gives the real size of each, and measures what each does to the weights of a real model.

This site's measurements focus on vLLM and SGLang; GGUF is presented here, not benchmarked for speed.

Reading a name

A GGUF type name has up to three parts:

• The bits: the number after Q (Q4, Q6...) is the size of each stored integer in the main block type.
• The family: nothing (Q4_0, Q8_0) for the original types, K (Q4_K) for k-quants, IQ (IQ3_XXS) for i-quants.
• The mix: for files, a suffix such as _S, _M, or _L (small, medium, large) says how many tensors are kept in larger types. Unsloth's UD- prefix marks its own per-model mixes.

The 0 and 1 of the original types say how the scale works: Q4_0 stores one scale per block (value = scale × q), Q4_1 a scale and a minimum (value = scale × q + min), like the symmetric and asymmetric integers of Integers for AI.

The original types: one scale per 32

The first GGUF types are the integer quantization of the integer piece, with blocks of 32 weights and a 16-bit scale each:

A 16-bit scale every 32 weights costs half a bit per weight. That overhead is what the next family reduces.

K-quants: super-blocks of 256

K-quants, added to llama.cpp in June 2023, group 256 weights into a super-block. Inside it, each small block of 16 or 32 weights gets a scale (and sometimes a minimum) of only 4 to 8 bits, and the super-block stores one or two FP16 numbers that scale those small scales.

It is the same idea as NVFP4's two levels of scale: fine block scales, cheap to store because they are themselves quantized against a coarser scale. The sizes, from the block definitions in ggml-common.h:

I-quants: codebooks and non-uniform grids

I-quants go further below 4 bits by changing what the stored bits mean.

Non-uniform values. IQ4_NL and IQ4_XS store 4-bit indices into a fixed table of 16 values that are denser near zero, where most weights are: −127, −104, −83, −65, −49, −35, −22, −10, 1, 13, 25, 38, 53, 69, 89, 113 (times the block scale). A uniform INT4 grid would space them evenly.

Codebooks. The 2- and 3-bit types store groups of 8 weights as an index into a codebook of allowed patterns. The pull request that introduced IQ2_XXS in January 2024 explains the design, borrowed in part from the QuIP# research method: the magnitudes of 8 weights are encoded as one of 256 points of the E8 lattice (8 bits), their signs in 7 bits by forcing an even number of negative signs, and a few spare bits hold a block scale. The result is "exactly 64 bits per block of 32", plus an FP16 scale per 256 weights: 2.0625 bits per weight.

Unpacking a codebook index takes more work than shifting a 4-bit integer, and what that costs depends on the backend: the IQ2_XXS pull request reported a 7B model decoding faster than Q4_0 with CUDA (155 against 130 tokens per second on an RTX 4080) and slower with Metal (54 against 63.5 on an M2 Max).

Mixes: what a file really contains

A file named Q4_K_M is not all Q4_K. When llama.cpp quantizes a model, llama-quant.cpp applies per-tensor rules: some tensors are kept in a larger type, chosen by name and by layer. For Q4_K_M, the attention value projection (attn_v) and the feed-forward down projection (ffn_down) go to Q6_K in the first eighth of the layers, the last eighth, and every third layer in between; the output layer goes to Q6_K. On Qwen3-8B, with 36 layers, that is 18 of 36 layers.

Reading the published Qwen3-8B files confirms the rules tensor by tensor, and gives their real size:

• Q4_K_M is 80% Q4_K and 20% Q6_K: 4.90 bits per weight, a 5.0 GB file.
• Q3_K_M keeps attention outputs, most attn_v and ffn_down in Q4_K: 4.02 bits per weight.
• Q2_K keeps ffn_down and attention outputs in Q3_K and attn_v in Q4_K: 3.20 bits per weight, not 2.6.
• In every mix, the output layer (7.6% of Qwen3-8B's parameters) stays in Q6_K.

Unsloth's UD files go further: per its documentation, it chooses "the quantization type of every possible layer" separately for each model, so a UD file's composition is specific to it and has to be read from the file.

The importance matrix

Most published GGUF files are made with an importance matrix (imatrix), and the Qwen3-8B files above record it in their metadata (Unsloth's imatrix_unsloth.dat, 685 chunks of calibration text). Its author explains the idea in the pull request that added it: when a weight row multiplies an activation vector, the error that matters is in the product, so weights that meet large activations should be rounded more carefully. llama.cpp runs the model on calibration text, records the average squared activation of each input column, and uses those numbers as weights in the rounding error it minimizes. It is a light cousin of the calibration that GPTQ and AWQ use, and it is why the calibration data matters: text unlike your use can steer the rounding the wrong way.

Measured: how much each file changes the weights

To compare the types on equal terms, I dequantized every linear-layer tensor of ten Qwen3-8B GGUF files and compared it with the original BF16 weights, the same measure used for the integer and float formats in this series:

• The error grows fast below about 4.5 bits per weight: 7% around 4.5 to 5 bits, 13% at 4, about 20% at 3.2 to 3.3.
• K-quants beat the original types: Q4_K_M has less error than Q4_1 with fewer bits (6.9% at 4.90 against 7.3% at 5.12), thanks to its finer, two-level scales.
• IQ4_XS gets within a point of Q4_K_M with half a bit less per weight.
• Unsloth's UD-Q4_K_XL spends 0.1 bit more than Q4_K_M for a little less error (6.5% against 6.9%).

Two cautions. Files made with an importance matrix are optimized for a weighted error, not this plain one, so this measure does not show what the imatrix buys. And weight error is not quality: served through vLLM's GGUF plugin, Q8_0 and Q4_K_M scored 93.0% and 92.4% on GSM8K against 93.1% for BF16, Q3_K_M 91.7%, and Q2_K 89.1% (How Much Does Each Bit Cost?).

Which to pick

• Q8_0: as close to the original as GGUF gets, at 8.5 bits per weight.
• Q6_K, Q5_K_M: 1.8% and 3.5% weight error, for when memory allows.
• Q4_K_M: a common default, really 4.9 bits per weight.
• IQ4_XS: close to Q4_K_M's weight error with half a bit less per weight.
• Below 4 bits: i-quants and imatrix files matter most here, and the quality cost grows quickly; check it on your own tasks.

How much each bit costs in answers, not just weights, is measured across formats in How Much Does Each Bit Cost?.