What is checkpoint inconsistency in distributed training?

Checkpoint inconsistency occurs when model states across GPUs are not synchronized during saving.

In DDP or FSDP, parameters may be sharded or replicated. Improper checkpointing can save partial or mismatched states, leading to incorrect model restoration unless full state consolidation is performed.