How does speculative decoding improve ChatGPT inference speed?

Speculative decoding uses a smaller draft model to predict tokens and a larger model to verify them, speeding up generation.

Speculative decoding is an optimization technique where a small, fast “draft model” generates multiple candidate tokens. The larger ChatGPT model then verifies these tokens in parallel instead of generating one token at a time.

If the large model agrees with the draft tokens, multiple tokens are accepted at once, reducing sequential decoding steps. If not, fallback correction occurs.

This architecture significantly reduces latency while preserving output quality, especially for long-form generation tasks.