What is KV caching in transformer inference?

KV caching stores past key-value pairs to avoid recomputing attention during inference.

During autoregressive generation, previously computed keys and values are cached so only new tokens compute attention against past states, reducing complexity from O(n²) to O(n).