The Roofline for LLMs: Why Decode Is Memory-Bound
Where prefill and decode sit against a GPU's compute and memory limits, and why batching, quantization, and speculative decoding work, with an interactive roofline.
memory-bandwidth
1 post
1 post
Where prefill and decode sit against a GPU's compute and memory limits, and why batching, quantization, and speculative decoding work, with an interactive roofline.