What is model sharding in distributed training systems?

Model sharding splits model parameters across devices to reduce memory usage.

Each GPU stores only a subset of parameters, gradients, or optimizer states. During computation, required shards are communicated on demand. This is foundational in FSDP and ZeRO optimizations.