When Most Layers Stop Caching: Hybrid, Sliding, and Sparse Attention, Sized
Why a 27B model holds 262k tokens of context on a 24 GB GPU: what hybrid, sliding-window, and sparse attention store and read per token, on real models.
attention
1 post
1 post
Why a 27B model holds 262k tokens of context on a 24 GB GPU: what hybrid, sliding-window, and sparse attention store and read per token, on real models.