Why do distributed TensorFlow systems suffer from straggler problems?

Stragglers are slow workers that delay synchronization in distributed training.

In synchronous distributed training, all workers must complete computation before gradients are aggregated. If one worker is slower due to hardware variance, network latency, or data imbalance, it becomes a bottleneck for the entire system. This is called the straggler problem and significantly reduces scaling efficiency.