How does Q-Learning deal with non-Markovian environments?

Q-Learning struggles in non-Markovian environments because it assumes current state fully captures future dynamics.

The Markov assumption states that the future depends only on the current state, not history. In non-Markovian environments, this assumption breaks, causing Q-values to become inconsistent. To address this, practitioners use history stacking, recurrent neural networks, or belief state modeling to approximate missing information.