What is gradient checkpointing in deep NLP models?

Gradient checkpointing trades compute for memory by recomputing activations during backpropagation.

Instead of storing all intermediate activations during forward pass, only selected checkpoints are saved. During backpropagation, missing activations are recomputed, significantly reducing memory usage at the cost of additional compute.