When Most Layers Stop Caching: Hybrid, Sliding, and Sparse Attention, Sized
Why a 27B model holds 262k tokens of context on a 24 GB GPU: what hybrid, sliding-window, and sparse attention store and read per token, on real models.
long-context
2 posts
Quantizing the KV Cache: FP8 and Below
Seven KV cache formats in vLLM, from FP8 to 3-bit, on one model: how many more tokens fit, what long-context recall loses, and why speed follows the kernel.