What is the impact of state representation quality on Q-Learning convergence?

Poor state representation can slow convergence or prevent Q-Learning from learning optimal policies.

Q-learning assumes that the state representation contains all relevant information needed for decision-making (Markov property). If important features are missing or noisy, the Q-function becomes ambiguous, mapping multiple underlying situations to the same state. This leads to suboptimal or unstable policies. Good feature engineering or learned representations (e.g., CNNs or embeddings) are critical for performance in complex environments.