vllm

Qwen's preview of its next architecture, piece by piece: linear and sparse attention, a gated four-branch residual, 512 experts and an n-gram memory, and what each costs to serve.

How serving engines hand out KV cache memory in 16-token blocks instead of reserving the maximum, how the same blocks let requests share a prompt, and what that saves in memory and time to first token.

Inference servers in three groups: production engines and their stacks, local apps built on llama.cpp, and enthusiasts' specialized engines, with how to pick.

One chat request followed through a real inference server, from chat template and tokens to prefill, decode, sampling, and streaming, each stage timed.

How engines split long prompts across steps, what the per-step token budget trades between time to first token and time per token, and how to set it.

Why a prompt token costs hundreds of times less than an answer token on the same weights: prefill and decode, seen in a kernel trace.

Compute what each token of KV cache costs from a model's config, then predict the capacity vLLM reports at startup, and see where the two differ.

Why serving engines re-form the batch at every decode step, what it buys in throughput and latency, and the settings that control it.

How vLLM and SGLang guarantee valid JSON or a grammar: compiling a schema into a token mask, applying it at every step, and what it costs in latency and quality.

How a cheap guess and one verification pass produce several tokens per step, when it speeds things up and when it does not, measured.

Why a repeated prompt can cost almost nothing, how vLLM and SGLang find shared prefixes, what breaks a hit, and how to structure prompts for agents and chat.

What NVFP4 stores, how vLLM serves it on Blackwell, and what it changes in memory, speed, and quality.

How GPTQ trades weight error for output error, in 30 lines of PyTorch on a real layer, then how to run it with llm-compressor and serve it with vLLM, and which settings actually change the result on Qwen3-8B.

Which 1% of weights matter, why scaling them protects them, and the search behind AWQ, in a few lines of PyTorch on a real layer, then AWQ on Qwen3-8B with llm-compressor and vLLM against plain rounding, GPTQ and a published checkpoint.

Why activation outliers break INT8, how SmoothQuant moves them into the weights, and what it changes for FP8 W8A8, measured on Qwen3-8B with vLLM.

A rotation flattens activation outliers and, on real layers, rescues INT4 activations. On Qwen3-8B in NVFP4, it cost 5 GSM8K points instead.

Seven KV cache formats in vLLM, from FP8 to 3-bit, on one model: how many more tokens fit, what long-context recall loses, and why speed follows the kernel.

GSM8K said every 4-bit format matched BF16. A noise baseline, paired comparisons, token-level KL, and a code benchmark showed which ones did not.

Why production stacks run prefill and decode on separate GPUs, how NVIDIA Dynamo and llm-d move the KV cache between them, and when the split pays.

What TTFT, TPOT, ITL, end-to-end latency, and goodput each measure, how the tools compute them, and why peak throughput is the wrong target.

Common load-test mistakes made on purpose next to the correct run, on one vLLM server, so each rule comes with how far it moves the numbers.

How Whisper turns audio into text, from mel spectrograms to an encoder-decoder; serving it with vLLM, transcribing in real time, and labeling speakers.