How does caching architecture in ML inference systems influence variance and consistency?

Caching reduces system variance and latency but can introduce bias if stale predictions are reused.

Caching in ML inference systems stores previous predictions to reduce computation cost and latency. While this improves system stability and reduces observed variance, it can introduce bias if cached outputs are served for inputs that have since become outdated due to model updates or data drift.

Architecturally, caching layers must balance freshness and efficiency using TTL (time-to-live), invalidation strategies, and cache-aware routing. In high-frequency systems like ad serving, stale cache can significantly distort user experience.…