What is the role of the replay buffer capacity in Deep Q-Learning?

Replay buffer capacity controls how much past experience the agent retains for training stability and diversity.

The replay buffer stores transitions (s, a, r, s'). Its capacity determines how diverse and relevant training samples are. A small buffer leads to overfitting recent experiences, while a very large buffer may dilute important recent learning signals. The buffer introduces decorrelation between samples, which is crucial for breaking temporal dependency in reinforcement learning updates. It also improves sample efficiency by reusing past experiences multiple times.