How does speculative decoding improve LLM inference performance?

Speculative decoding accelerates inference by allowing smaller models to predict candidate tokens that are later verified by larger models.

Autoregressive decoding is inherently slow because LLMs generate tokens sequentially. Each token depends on all previously generated tokens.

Speculative decoding improves throughput by introducing a two-model architecture:

A small, fast model predicts multiple future tokens.

The larger primary model validates the draft predictions.

If predictions are accepted, multiple tokens are generated in a single step instead of one-by-one decoding.

Advantages include: