This Simple Trick Made ALL LLMs 2x Faster
bycloud · 12:18 · Watch on YouTube
Speculative decoding speeds up LLM inference by letting a cheap draft model guess several future tokens and having the large target model verify that draft in one forward pass, using a rejection rule that keeps the ta...