How does model sharding architecture influence bias and variance in large neural networks?

Model sharding reduces memory constraints enabling low-bias models, but can introduce variance due to communication overhead and partition imbalance.

Model sharding splits a large neural network across multiple devices. This enables training and serving of extremely large models that would otherwise not fit in memory, reducing bias by increasing model capacity.

However, sharding introduces communication overhead and synchronization delays. If partitions are not balanced, certain shards become bottlenecks, leading to inconsistent execution timing and increased system-level variance.

Architectures like tensor parallelism, pipeline parallelism, and sequence parallelism are used to optimize tradeoffs between compute efficiency and stability.