Speculative decoding
A model writes one token at a time, and each token requires re-reading everything before it: slow by construction. Speculative decoding flips the load. A small fast model drafts several tokens ahead, then the large model reads them all in a single pass and keeps the longest prefix it would have written itself. The rest is discarded. You get exactly the text the large model would have produced alone, with two to three times fewer round trips.
Strengths
- The output is identical to the large model alone: this mechanism accelerates without degrading
- Nothing to change on your side: the gain comes from the provider, with no prompt changes
Limitations
- The gain depends on how predictable the text is: on a very technical answer where the small model is often wrong, its drafts are discarded and the speedup melts away
- Two models run instead of one: latency drops, memory footprint rises
Best for
- Understanding where a provider-announced speed gain comes from, before switching models for the wrong reason