The Serving Stack

llama-cpp

2 posts

INFERENCE ENGINES · 01MANY USERSYOUR MACHINEONE TARGET
inference-enginesservingvllm

Choosing an Inference Server: The Landscape, from Data Centers to Laptops

Inference servers in three groups: production engines and their stacks, local apps built on llama.cpp, and enthusiasts' specialized engines, with how to pick.

by The Serving Stack
QUANTIZATION · 11Q4BITSKK-QUANTMMIX4.0 NOMINAL → 4.9 REALBITS PER WEIGHT, QWEN3-8B Q4_K_M
quantizationggufllama-cpp

GGUF Quant Types Decoded: K-Quants and I-Quants

What Q4_K_M, IQ3_XXS, and the rest mean: the block layouts from llama.cpp's source, the real bits per weight of each file, and the weight error of each type on Qwen3-8B.

by The Serving Stack