What is activation checkpointing vs gradient checkpointing?

Activation checkpointing stores minimal forward activations; gradient checkpointing recomputes them during backprop.

Both techniques reduce memory usage by trading compute for storage. Activation checkpointing stores selected activations, while gradient checkpointing recomputes intermediate values during backward pass. These are essential for training large transformers on limited GPU memory.