How does KV caching optimize autoregressive decoding in transformers?

KV caching stores past key/value tensors to avoid recomputation during token generation.

In autoregressive decoding, each new token attends to all previous tokens. Without caching, keys and values are recomputed at every step, leading to O(n²) redundancy. KV caching stores previous layer activations, reducing decoding complexity from quadratic recomputation to incremental linear updates. This is critical for latency-sensitive inference systems.