What is distributed optimizer state sharding?

Optimizer state sharding splits optimizer states across GPUs to reduce memory usage.

Optimizers like Adam store momentum and variance buffers. In large models, these states are sharded across GPUs using ZeRO or FSDP, reducing per-device memory footprint significantly.