Prefill vs. Decode: Why the Second Token Is a Different Workload
Why a prompt token costs hundreds of times less than an answer token on the same weights: prefill and decode, seen in a kernel trace.
Articles
NVFP4: Two Levels of Scale
How NVFP4 differs from MXFP4: 16-value blocks, FP8 block scales, and a per-tensor scale, with what each part buys measured on Qwen3-8B.
DeepSeek: From V1 to V4, the Models and the Breakthroughs
DeepSeek's models in order, the idea behind each one (MoE, latent attention, FP8 training, RL reasoning, sparse attention), and what each changed for everyone else.
From Prompt to Token: The Life of an LLM Request
One chat request followed through a real inference server, from chat template and tokens to prefill, decode, sampling, and streaming, each stage timed.
Microscaling: MXFP8, MXFP6, MXFP4 and Block Scales
The OCP microscaling formats: 32 values sharing one power-of-two scale, what MXFP8, MXFP6, and MXFP4 cost in accuracy on a real model, and why the scale rule matters.
Choosing an Inference Server: The Landscape, from Data Centers to Laptops
Inference servers in three groups: production engines and their stacks, local apps built on llama.cpp, and enthusiasts' specialized engines, with how to pick.
Where Low Precision Bites: Rounding, Overflow, and Accumulation
Five places where low precision breaks real computations, each reproduced on Qwen3-8B's own tensors, and the fix every serious implementation applies.
Quantization-Aware Training: When the Model Learns Its Own Precision
Why Kimi K2 Thinking ships in INT4 and gpt-oss and K3 in MXFP4: training with rounding in the loop, explained and measured on a small model.
Post-Training Quantization: Rounding, Calibration, and the Outlier Problem
Why rounding each weight to the nearest value is not enough, what calibration data is for, and the two kinds of outliers in a real model's activations, measured on Qwen3-8B.
Quantization, Mapped: Weights, Activations, KV Cache
What can be quantized in a model, what each saves (memory, bandwidth, compute), and what each risks, with Qwen3-8B's numbers.
FP8: E4M3, E5M2, and Why There Are Two
The two FP8 formats, what each is for, and how FP8 is scaled, measured on Qwen3-8B's weights and activations against INT8.
GGUF Quant Types Decoded: K-Quants and I-Quants
What Q4_K_M, IQ3_XXS, and the rest mean: the block layouts from llama.cpp's source, the real bits per weight of each file, and the weight error of each type on Qwen3-8B.