How does model checkpointing strategy in distributed training influence bias and variance?

Checkpointing improves fault tolerance and training stability, but improper checkpoint frequency can increase variance or bias in recovered training states.

In distributed training systems, checkpointing periodically saves model weights, optimizer states, and sometimes RNG states. This ensures recovery after failures and supports reproducibility.

From a bias-variance perspective, checkpoint frequency matters. Too infrequent checkpointing risks losing long training trajectories, causing higher variance in recovered models after failure. Too frequent checkpointing can introduce overhead and encourage restarting from suboptimal intermediate states, potentially increasing bias if training is repeatedly resumed from earlier checkpoints.…