How does A/B testing infrastructure interact with bias and variance estimation in production ML systems?

A/B testing provides empirical bias and variance estimates, but can introduce variance due to sample noise and traffic allocation strategies.

A/B testing is a core evaluation mechanism in production ML systems used to compare model variants under real traffic. It provides a direct estimate of generalization performance, effectively measuring both bias (systematic performance difference between models) and variance (instability across user segments or time windows).

However, A/B testing itself introduces statistical variance due to sampling noise, traffic split randomness, and external confounders (seasonality, user behavior shifts). Small sample sizes exaggerate variance, while imbalanced traffic allocation can bias results.…