What DeepGEMM computes for DeepSeek's models: block-scaled FP8 and FP4 matrix multiplications, grouped GEMMs for MoE experts, and the fused Mega MoE kernel.

Why serving a big MoE model means spreading its experts over many GPUs, what dispatch and combine cost, and how DeepEP and its V2 redesign make it fast.

DeepSeek's open-source engine runs one model on Blackwell and Ascend: why it came out, what the code shows, what was kept back, and what it means for vLLM users.

Qwen's preview of its next architecture, piece by piece: linear and sparse attention, a gated four-branch residual, 512 experts and an n-gram memory, and what each costs to serve.

DeepSeek's Engram adds a hashed n-gram memory that a model looks up instead of computing: how it works, why it helps, and how DeepSeek-V4.1 and Qwen serve it.

Common load-test mistakes made on purpose next to the correct run, on one vLLM server, so each rule comes with how far it moves the numbers.

Why Kimi K2 Thinking ships in INT4 and gpt-oss and K3 in MXFP4: training with rounding in the loop, explained and measured on a small model.

Write SAXPY in CUDA, compile it for a Blackwell GPU, check it, time it, and read what Nsight Systems and Nsight Compute say about it.

Real coding-agent turns replayed on a 27B model on one desk-side GPU: what the prefix cache saves, how many agents fit, how well it calls tools.

A rotation flattens activation outliers and, on real layers, rescues INT4 activations. On Qwen3-8B in NVFP4, it cost 5 GSM8K points instead.

The parts of a GPU that decide inference speed, from streaming multiprocessors to DRAM, with one real GPU's numbers and three measurements.

GSM8K said every 4-bit format matched BF16. A noise baseline, paired comparisons, token-level KL, and a code benchmark showed which ones did not.

Why activation outliers break INT8, how SmoothQuant moves them into the weights, and what it changes for FP8 W8A8, measured on Qwen3-8B with vLLM.

Seven KV cache formats in vLLM, from FP8 to 3-bit, on one model: how many more tokens fit, what long-context recall loses, and why speed follows the kernel.

What TTFT, TPOT, ITL, end-to-end latency, and goodput each measure, how the tools compute them, and why peak throughput is the wrong target.

Why a prompt token costs hundreds of times less than an answer token on the same weights: prefill and decode, seen in a kernel trace.

One chat request followed through a real inference server, from chat template and tokens to prefill, decode, sampling, and streaming, each stage timed.

Which 1% of weights matter, why scaling them protects them, and the search behind AWQ, in a few lines of PyTorch on a real layer, then AWQ on Qwen3-8B with llm-compressor and vLLM against plain rounding, GPTQ and a published checkpoint.

How GPTQ trades weight error for output error, in 30 lines of PyTorch on a real layer, then how to run it with llm-compressor and serve it with vLLM, and which settings actually change the result on Qwen3-8B.

What Q4_K_M, IQ3_XXS, and the rest mean: the block layouts from llama.cpp's source, the real bits per weight of each file, and the weight error of each type on Qwen3-8B.

Why rounding each weight to the nearest value is not enough, what calibration data is for, and the two kinds of outliers in a real model's activations, measured on Qwen3-8B.

Qwen3-8B in thirteen formats from BF16 to 3-bit GGUF, on one GPU with vLLM: memory, decode speed, prompt reading, perplexity, and GSM8K, plus the formats that would not run.

Five places where low precision breaks real computations, each reproduced on Qwen3-8B's own tensors, and the fix every serious implementation applies.

What can be quantized in a model, what each saves (memory, bandwidth, compute), and what each risks, with Qwen3-8B's numbers.

Which number formats each data-center GPU computes natively, at what rate, from the vendors' own pages: A100 to Rubin, MI300X to MI355X.

How NVFP4 differs from MXFP4: 16-value blocks, FP8 block scales, and a per-tensor scale, with what each part buys measured on Qwen3-8B.

The OCP microscaling formats: 32 values sharing one power-of-two scale, what MXFP8, MXFP6, and MXFP4 cost in accuracy on a real model, and why the scale rule matters.

How integers represent real weights: scales, zero points, per-tensor, per-channel, and per-group, measured on every weight of Qwen3-8B.

The two FP8 formats, what each is for, and how FP8 is scaled, measured on Qwen3-8B's weights and activations against INT8.

Inputs, products, and accumulators: the precision a matrix multiplication really uses, and why FP32 accumulation still matters.

Why training moved from FP16 to BF16, and why a BF16 model may still run fine in FP16: the same 16 bits split differently, measured on Qwen3-8B.

What the bits of a floating-point number mean, why range and precision trade against each other, with an interactive bit explorer.

What distillation is, the ways labs distill models, and how they use it on their own models and on others', from primary sources.

Z.ai's (Zhipu's) GLM models in order, from the GLM pretraining objective to agentic MoE models, the idea behind each step, and what each changed for the ecosystem.

What NVFP4 stores, how vLLM serves it on Blackwell, and what it changes in memory, speed, and quality.

How a cheap guess and one verification pass produce several tokens per step, when it speeds things up and when it does not, measured.

Why a repeated prompt can cost almost nothing, how vLLM and SGLang find shared prefixes, what breaks a hit, and how to structure prompts for agents and chat.

How engines split long prompts across steps, what the per-step token budget trades between time to first token and time per token, and how to set it.

How vLLM and SGLang guarantee valid JSON or a grammar: compiling a schema into a token mask, applying it at every step, and what it costs in latency and quality.

Where prefill and decode sit against a GPU's compute and memory limits, and why batching, quantization, and speculative decoding work, with an interactive roofline.

MiniMax's models in order: lightning (linear) attention at a 4-million-token context, reasoning, the return to full attention for agents, and what each choice taught the ecosystem.

DeepSeek's models in order, the idea behind each one (MoE, latent attention, FP8 training, RL reasoning, sparse attention), and what each changed for everyone else.

What NVIDIA's open speech models are for: Parakeet and Canary for recognition, Nemotron Speech for streaming, Magpie for synthesis, and how they fit a voice agent.

Why production stacks run prefill and decode on separate GPUs, how NVIDIA Dynamo and llm-d move the KV cache between them, and when the split pays.

Moonshot AI's Kimi models in order, from long-context chat to trillion-parameter agentic MoE models, the idea behind each step, and what each changed for the ecosystem.

How Whisper turns audio into text, from mel spectrograms to an encoder-decoder; serving it with vLLM, transcribing in real time, and labeling speakers.

How modern text-to-speech models turn text into audio tokens and waveforms, the open models that lead today, and how to choose one for latency, quality, and voices.

Inference servers in three groups: production engines and their stacks, local apps built on llama.cpp, and enthusiasts' specialized engines, with how to pick.

How serving engines hand out KV cache memory in 16-token blocks instead of reserving the maximum, how the same blocks let requests share a prompt, and what that saves in memory and time to first token.

Why serving engines re-form the batch at every decode step, what it buys in throughput and latency, and the settings that control it.

Why a 27B model holds 262k tokens of context on a 24 GB GPU: what hybrid, sliding-window, and sparse attention store and read per token, on real models.

Compute what each token of KV cache costs from a model's config, then predict the capacity vLLM reports at startup, and see where the two differ.