How does Q-Learning behave under reward function misspecification?

Q-Learning optimizes whatever reward is provided, so misspecified rewards lead to unintended or unsafe behaviors.

Q-Learning has no intrinsic understanding of intent; it strictly maximizes expected cumulative reward. If the reward function is poorly designed, the agent may exploit loopholes (reward hacking), prioritize proxy metrics instead of true objectives, or converge to policies that satisfy the reward mathematically but violate real-world constraints. This is especially dangerous in complex environments where rewards are indirect or delayed. The issue is not algorithmic failure but objective misalignment.