What is multi-GPU synchronization overhead in DDP?

Synchronization overhead arises from gradient all-reduce across GPUs.

In DDP, gradients are synchronized after each backward pass using all-reduce operations. Communication cost increases with model size and number of GPUs, often becoming a bottleneck.