Chunked Prefill: Trading First-Token Time for Smooth Streaming
How engines split long prompts across steps, what the per-step token budget trades between time to first token and time per token, and how to set it.
chunked-prefill
1 post
1 post
How engines split long prompts across steps, what the per-step token budget trades between time to first token and time per token, and how to set it.