What is gradient accumulation and how does it affect cost computation?

Gradient accumulation simulates large batch training by summing gradients over multiple steps before updating weights.

When memory constraints prevent large batch sizes, gradients are accumulated over multiple forward-backward passes. This effectively approximates the cost function gradient over a larger batch without increasing memory usage. However, it changes optimization dynamics slightly due to delayed updates and reduced noise.