What is the impact of delayed policy improvement in Q-Learning?

Delayed policy improvement slows learning because Q-values take time to reflect better actions.

Q-Learning improves policy indirectly through value updates. Since updates propagate one step at a time, improvements in policy appear gradually. This delay can make training inefficient in sparse or long-horizon environments. Techniques like prioritized replay and multi-step bootstrapping reduce this delay.