How does gradient accumulation interact with distributed training?

Gradient accumulation simulates larger batches across distributed workers by delaying synchronization.

In DDP, gradients are typically synchronized every backward pass. With accumulation, synchronization is delayed for multiple micro-batches, reducing communication overhead and effectively increasing batch size.