DeepGEMM: The Matrix Multiplications Behind DeepSeek's Models
What DeepGEMM computes for DeepSeek's models: block-scaled FP8 and FP4 matrix multiplications, grouped GEMMs for MoE experts, and the fused Mega MoE kernel.
gpu
7 posts
Prefill vs. Decode: Why the Second Token Is a Different Workload
Why a prompt token costs hundreds of times less than an answer token on the same weights: prefill and decode, seen in a kernel trace.
The KV Cache, Sized: What the Formula Says and What Your Engine Gives You
Compute what each token of KV cache costs from a model's config, then predict the capacity vLLM reports at startup, and see where the two differ.
A GPU in One Picture: SMs, Warps, and Memory
The parts of a GPU that decide inference speed, from streaming multiprocessors to DRAM, with one real GPU's numbers and three measurements.
The Roofline for LLMs: Why Decode Is Memory-Bound
Where prefill and decode sit against a GPU's compute and memory limits, and why batching, quantization, and speculative decoding work, with an interactive roofline.
Which GPU Computes Which Format: Ampere to Rubin, MI300 to MI355
Which number formats each data-center GPU computes natively, at what rate, from the vendors' own pages: A100 to Rubin, MI300X to MI355X.
Your First CUDA Kernel, Compiled and Profiled
Write SAXPY in CUDA, compile it for a Blackwell GPU, check it, time it, and read what Nsight Systems and Nsight Compute say about it.