What are states, actions, rewards, and Q-values in Q-Learning?
Updated Feb 20, 2026
Short answer
The Q-value is the expected long-term reward of taking an action in a state and following the best policy afterward. In Q-Learning, an agent learns these values by exploring states, taking actions, receiving rewards, and updating its Q-table until it can choose better decisions.
Deep explanation
Q-Learning is a reinforcement learning algorithm where an agent learns what to do by interacting with an environment. The core idea is to estimate the value of every possible decision using a function called the Q-function.
A Q-Learning problem is described using four main concepts:
1. State (s)
A state represents the current situation the agent observes. It contains the information needed for making a decision.
Examples:
- A robot's current location in a warehouse.
- A game character's position, health, and nearby objects.
- A recommendation system's current user context.
A state answers:
"Where am I right now?"
The agent does not directly know the future. It only sees the current state and chooses an action.
2. Action (a)
An action is a choice the agent can make from a given state.
Examples:
- A robot can move
left,right, orforward. - A game agent can
attack,defend, orescape. - A recommendation system can show different content.
An action answers:
"What can I do from here?"
The available actions usually depend on the current state. A robot at a wall may not have the action move_forward.
3. Reward (r)
A reward is feedback from the environment after the agent takes an action. It tells the agent whether the decision was helpful or harmful.
Examples:
- Reaching a destination:
+100. - Hitting an obstacle:
-50. - Taking a normal step:
-1.
The reward does not necessarily describe the final goal directly. The agent tries to maximize the total reward collected over time.
The reward is immediate feedback, but Q-values estimate future consequences.
4. Q-value (Q(s, a))
A Q-value represents how good an action is when taken from a specific state.
Q(s, a) means:
s: the current state.a: the action chosen.Q: the expected total future reward.
A high Q-value means the action is likely to lead to better outcomes.
For example:
| State | Action | Q-value | Meaning |
|---|---|---|---|
| Near goal | Move forward | 90 | Likely good choice |
| Near goal | Move backward | 20 | Less useful |
| Near obstacle | Move forward | -80 | Dangerous |
The agent stores these values in a Q-table for small problems.
The learning process follows this loop:
Q-Learning improves decisions by repeatedly updating estimates, not by being given the correct answer.
The update rule is:
Q(s, a) = Q(s, a) + α * (r + γ * max Q(s', a') - Q(s, a))Where:
αis the learning rate. It controls how much new information changes the old Q-value.γis the discount factor. It controls how much the agent cares about future rewards.max Q(s', a')is the best estimated future value from the next state.
Exploration vs. Exploitation
A major challenge is deciding whether to:
| Strategy | Meaning | Benefit | Risk |
|---|---|---|---|
| Exploration | Try new actions | Finds better strategies | May make poor choices |
| Exploitation | Choose highest Q-value action | Uses learned knowledge | May miss better options |
A common approach is the epsilon-greedy strategy:
- With probability
ε, choose a random action. - Otherwise, choose the action with the highest Q-value.
A good Q-Learning agent balances discovering new options with using what it already learned.
Q-Learning vs. Traditional Programming
In traditional software:
- Developers define rules.
- The program follows those rules.
In Q-Learning:
- Developers define the environment and reward structure.
- The agent learns the strategy.
This makes Q-Learning useful when writing every possible rule is difficult, such as robotics, games, and scheduling.
Real-world example
Imagine training a delivery robot to navigate a building.
The robot's:
- State: current room and nearby obstacles.
- Action: move north, south, east, or west.
- Reward: positive reward for reaching the delivery location, negative reward for collisions or wasted movement.
- Q-value: the learned estimate of which movement is best from each location.
Initially, the Q-table contains poor guesses. After many attempts, the robot learns routes that maximize rewards.
A simplified update might look like this:
q_table[state][action] = q_table[state][action] + learning_rate * ( reward + discount * max(q_table[next_state]) - q_table[state][action])After enough training, the robot does not memorize one path. Instead, it has learned values that help it choose good actions in many situations.
Common mistakes
- * **Confusing rewards and Q-values** - A reward is immediate feedback
- a Q-value predicts the total future reward. Keep the distinction clear.
- * **Ignoring exploration** - Always choosing the current best action can prevent discovering better strategies. Use methods like `epsilon-greedy`.
- * **Assuming Q-values are guaranteed correct** - Q-values are estimates that improve through experience. They may be inaccurate early in training.
- * **Choosing poor rewards** - Bad reward design can teach unwanted behavior. Define rewards that match the actual goal.
- * **Using Q-tables for huge problems** - Large state spaces make tables impractical. Use function approximation methods such as neural networks.
Follow-up questions
- How does Q-Learning differ from supervised learning?
- What is the role of the discount factor γ in Q-Learning?
- Why does Q-Learning need exploration?
- What happens if the learning rate α is too high?
- Why is Q-Learning called a model-free algorithm?