How does ZeRO optimization relate to PyTorch distributed training?

ZeRO partitions optimizer states, gradients, and parameters across GPUs to reduce memory.

ZeRO (Zero Redundancy Optimizer) eliminates redundant memory usage in distributed training by sharding model states across devices while maintaining correctness via communication steps.