Why does distributed TensorFlow training degrade when batch size increases beyond a threshold?

Large batch sizes reduce gradient noise but hurt generalization and increase synchronization overhead.

In distributed training, batch size is split across workers. Increasing global batch size reduces stochasticity in gradients, often leading to sharper minima and worse generalization. Additionally, synchronization cost (all-reduce communication) increases, making training slower despite more compute. There is a trade-off between convergence stability and communication efficiency.