What is speculative decoding and how does it improve LLM inference speed?

Speculative decoding uses a smaller model to predict tokens ahead of a larger model to reduce latency.

A small draft model generates multiple candidate tokens quickly, and a larger model verifies them in parallel. Accepted tokens are committed, reducing expensive forward passes of the large model. This improves throughput while maintaining output quality guarantees.