What is the difference between DataParallel and DistributedDataParallel (DDP) in PyTorch?

DataParallel uses a single process and splits batches, while DDP uses multiple processes with independent model replicas and is more scalable.

DataParallel (DP) replicates the model on one process and uses scatter-gather across GPUs, causing a Python bottleneck. DDP launches one process per GPU and synchronizes gradients via all-reduce, reducing overhead and improving scalability and performance significantly.