How does speculative execution style parallel decoding differ from standard autoregressive decoding?

Speculative decoding generates multiple tokens in parallel using a draft model, while autoregressive decoding generates one token sequentially.

Standard autoregressive decoding generates tokens one-by-one, where each token depends on all previous tokens. This is inherently sequential and limits throughput.

Speculative decoding introduces a smaller draft model that predicts multiple future tokens in parallel. A larger target model then verifies these tokens in fewer steps. If predictions are correct, multiple tokens are accepted at once, significantly improving throughput.

This architecture reduces decoding latency without sacrificing output quality by combining fast approximation with accurate verification.