How Much Does Each Bit Cost? Size, Speed, and Quality, Measured
Qwen3-8B in thirteen formats from BF16 to 3-bit GGUF, on one GPU with vLLM: memory, decode speed, prompt reading, perplexity, and GSM8K, plus the formats that would not run.
benchmark
1 post