How does distributed training synchronization strategy affect bias and variance in large-scale ML systems?

Synchronous training improves stability and reduces variance, while asynchronous training improves throughput but can increase variance due to stale gradients.

In distributed ML systems, synchronization strategy determines how model updates are aggregated across workers. In synchronous training, all workers compute gradients and wait for a global aggregation step (e.g., all-reduce). This produces consistent updates, reducing variance in optimization and improving convergence stability.

In asynchronous training, workers update parameters independently, which improves hardware utilization but introduces stale gradients. These stale updates increase variance in the optimization trajectory and can lead to instability in convergence.…