Chunked Prefill: Trading First-Token Time for Smooth Streaming
How engines split long prompts across steps, what the per-step token budget trades between time to first token and time per token, and how to set it.
scheduling
2 posts
Continuous Batching, Animated: How a Server Keeps the GPU Busy
Why serving engines re-form the batch at every decode step, what it buys in throughput and latency, and the settings that control it.