serving

How serving engines hand out KV cache memory in 16-token blocks instead of reserving the maximum, how the same blocks let requests share a prompt, and what that saves in memory and time to first token.

Inference servers in three groups: production engines and their stacks, local apps built on llama.cpp, and enthusiasts' specialized engines, with how to pick.

One chat request followed through a real inference server, from chat template and tokens to prefill, decode, sampling, and streaming, each stage timed.

Why serving engines re-form the batch at every decode step, what it buys in throughput and latency, and the settings that control it.

Why production stacks run prefill and decode on separate GPUs, how NVIDIA Dynamo and llm-d move the KV cache between them, and when the split pays.