Speculative Decoding from First Principles
How a cheap guess and one verification pass produce several tokens per step, when it speeds things up and when it does not, measured.
speculative-decoding
1 post
1 post
How a cheap guess and one verification pass produce several tokens per step, when it speeds things up and when it does not, measured.