What is Gradient Checkpointing and how does it reduce memory usage in Deep Learning?

Gradient checkpointing reduces GPU memory usage during training by selectively storing intermediate activations and recomputing them during backpropagation.

Training deep neural networks requires storing intermediate activations for backpropagation. As models grow larger (especially Transformers and LLMs), activation memory becomes a major bottleneck.

Gradient checkpointing solves this by trading compute for memory.

Core idea: Instead of storing all activations during forward pass:

Memory-compute tradeoff:

How it works:

2.…