What is gradient accumulation and when should you use it?

Gradient accumulation simulates large batch sizes by accumulating gradients over steps.

Instead of updating weights every batch, gradients are summed over multiple iterations before optimizer step.