What is FSDP (Fully Sharded Data Parallel) in PyTorch?

FSDP shards model parameters, gradients, and optimizer states across GPUs.

Unlike DDP which replicates models, FSDP partitions parameters across devices. During forward and backward passes, parameters are gathered on-demand and then re-sharded, significantly reducing memory usage.