What is gradient accumulation and why is it important in large model training?

Gradient accumulation simulates large batch sizes by accumulating gradients over multiple forward passes.

When GPU memory is limited, large batch training becomes infeasible. Gradient accumulation solves this by computing gradients over multiple mini-batches and updating weights only after several steps. This stabilizes training while maintaining effective batch size without requiring more memory.