What is the impact of over-optimistic Q-value initialization in exploration behavior?

Optimistic initialization encourages exploration by making unexplored actions appear valuable.

When Q-values are initialized to high values, the agent is naturally incentivized to try all actions to confirm whether they truly yield high reward. As learning progresses, values are corrected downward. This creates a structured exploration strategy without explicit randomness. However, in deep Q-learning, naive optimistic initialization can destabilize gradients and is harder to control compared to tabular settings.