What is KV cache optimization in transformer-based inference?

KV caching stores attention keys and values to avoid recomputation during autoregressive decoding.

In transformer inference, each new token requires attention over previous tokens. KV cache stores computed key-value pairs so they are reused instead of recomputed at every step. This reduces computational complexity from O(n²) recomputation to incremental O(n). Efficient KV cache management is critical for long-context LLMs and high-throughput serving systems. However, it increases memory pressure and requires careful eviction or paging strategies.