Reinforcement learning (RL) studies how an agent can learn decisions from interactions with an environment. The agent observes a situation, selects an action, receives a reward and another observation, and uses those experiences to improve future decisions. Unlike a classifier trained on labeled examples, the agent often must discover which action sequences lead to useful outcomes over time.
This guide introduces the main concepts, the difference between model-free and model-based approaches, a practical Python example and the limits of applying RL outside simulation. An agent’s success depends on the reward design, environmental assumptions, training budget and independent evaluation. Interacting with an environment does not guarantee that a system will learn a safe or optimal policy.
Agent, environment, observation, action and reward
The agent is the decision-making system. The environment contains the conditions the agent acts within. In each step, the environment supplies an observation and the agent chooses an allowed action. The environment then produces a reward and the next observation. In an episodic task, the interaction eventually ends at a terminal condition or a time limit.
A policy describes how actions are chosen. It can map a state to one selected action or assign probabilities to several actions. A reward is an immediate numerical signal specified by the environment designer. The return is a measure of cumulative future rewards, often discounted so that near-term outcomes contribute more strongly than distant ones.
For example, an agent learning to navigate a small maze may receive a positive reward for reaching the goal and penalties for undesirable transitions. But the exact reward design matters. Rewarding rapid movement alone might teach an agent to move repeatedly without reaching the destination. A high reward in training can therefore reflect a badly specified objective rather than useful behavior.
In a Markov decision process (MDP), the environment is described through states, actions, transition behavior and rewards, usually with the assumption that the current state contains sufficient information to predict the next transition given an action. Real agents may only see partial observations. Treating an incomplete observation as a fully informative state can make a method behave poorly.
How reinforcement learning differs from other methods
Supervised learning commonly learns a mapping from inputs to labeled targets such as a document category. Unsupervised methods explore groupings or structure without supplying the target for each example. RL uses the consequences of actions to improve a policy. Although all three can use neural networks, their training objectives and evaluation methods differ.
RL can use a simulator where experiments are inexpensive and reversible. A game agent may try thousands of unsuccessful episodes without harming anyone. By contrast, trying untested actions on real financial accounts, vehicles or equipment can be costly or unsafe. A synthetic or simulated environment should not be mistaken for proof that a policy is ready for real deployment.
The related unsupervised machine learning guide explains clustering and representation learning for comparison. The distinction helps prevent the mistaken claim that all training without manually labeled class examples is identical.
Exploration versus exploitation
An RL agent needs to make use of actions that already appear valuable while also exploring less familiar alternatives. A common strategy is epsilon-greedy action selection: with probability epsilon the agent takes a random action, and otherwise it chooses one that currently has the highest estimated value.
Starting with more exploration can help the agent encounter useful transitions. Reducing exploration later can favor known actions, but no single decay schedule is optimal for every task. Too little exploration may leave the agent committed to an early mistake; too much can prevent stable performance. This trade-off is part of algorithm design and should be measured.
The Gymnasium training guide illustrates tabular Q-learning, epsilon-greedy selection and the update from observed rewards. Its documentation is a useful first-party reference for building a small agent with an explicit environment API.
How Q-learning updates action values
Tabular Q-learning estimates the value of choosing each action in each discrete state. After observing a transition, it moves the current estimate toward a target based on the immediate reward and the best estimated value of the next state. Learning rate alpha controls the update size; discount factor gamma controls the contribution of future rewards.
The usual one-step expression is Q(s,a) ← Q(s,a) + α [r + γ max Q(s′,a′) − Q(s,a)] for a nonterminal transition. If the environment truly terminates, there is no future return to bootstrap from that terminal state. The update may be repeated many times; it does not guarantee convergence when necessary algorithm assumptions are not satisfied.
Some RL implementations treat an externally imposed time limit differently from genuine task termination. Under Gymnasium’s modern API, terminated means an MDP terminal condition was met; truncated describes an interruption such as a time limit. An episode loop ends for either, but a value-learning algorithm may still bootstrap from a truncated observation where appropriate. Confusing the two can bias estimates.
Large or continuous state spaces cannot be represented conveniently by a finite table. Deep Q-networks use neural networks to approximate action values, introducing additional issues such as instability, replay buffers and target networks. The official PyTorch DQN tutorial demonstrates these components on CartPole using Gymnasium.
Run a small tabular Q-learning example
The following Python example uses Gymnasium’s FrozenLake-v1 environment with deterministic transitions. It demonstrates how the training loop, epsilon-greedy exploration and bootstrapped Q-values fit together. Install gymnasium and numpy in an isolated Python environment before running it. The demonstration is not an RL benchmark or an industrial control policy.
import gymnasium as gym
import numpy as np
env = gym.make("FrozenLake-v1", is_slippery=False)
env.action_space.seed(42)
rng = np.random.default_rng(42)
q = np.zeros((env.observation_space.n, env.action_space.n))
alpha = 0.2
gamma = 0.95
for episode in range(3000):
state, info = env.reset(seed=episode)
epsilon = max(0.05, 1.0 - episode / 2500)
for _ in range(100):
if rng.random() < epsilon:
action = env.action_space.sample()
else:
action = int(np.argmax(q[state]))
next_state, reward, terminated, truncated, info = env.step(action)
future = 0.0 if terminated else gamma * np.max(q[next_state])
target = reward + future
q[state, action] += alpha * (target - q[state, action])
state = next_state
if terminated or truncated:
break
print("Start-state estimated values:", q[0].round(3))
env.close()
The script prints the learned action-value estimates for the starting state. Their exact values can depend on implementation versions, exploration and training history, so they should not be treated as fixed expected output. The environment is intentionally simple and deterministic; enabling slippery transitions creates a different, stochastic task that may require other parameters and more training.
For evaluation, use separate episodes without exploratory random actions and record how often the agent reaches the goal. Compare with a simple baseline. The code demonstrates training only; it does not establish reliability in unseen environments or allow the agent to operate a physical system safely.
Compare value-based, policy-based and model-based approaches
Value-based methods estimate how good an action or state is and derive action choices from those estimates. Q-learning and DQN are examples. Policy-based methods directly adjust a parameterized action-selection policy using an optimization objective. Actor-critic systems combine a policy with a learned value estimate to guide updates.
Model-based methods use or learn a representation of environmental transitions so that the agent can plan or simulate future possibilities. Learning a model can improve sample efficiency in some settings but also introduces error when the learned model differs from the real world. A planner that repeatedly reasons through an inaccurate model can amplify its mistakes.
Different approaches suit discrete or continuous actions, partial observability, sparse rewards and computational constraints differently. Algorithm names alone do not establish a benchmark ranking. Record the environment, reward, evaluation method, seeds, computing budget and implementation when making comparisons.
Reward design, evaluation and the risk of shortcuts
An agent can maximize the numerical reward it receives without accomplishing what designers intended. If a simulated robot is rewarded only for speed, it might learn an unsafe route. A recommendation system could optimize immediate interaction while harming long-term user interests. These problems are sometimes described as reward hacking or specification gaming.
Before training, define the task, valid actions, unacceptable states and how independent success will be measured. Evaluate a policy with fresh episodes and meaningful failure cases. Use multiple random seeds and track both average outcomes and tail risks where they matter. A single successful run or best score is not sufficient evidence of robust learning.
Examine distribution shift: the simulator's conditions may differ from real weather, customer behavior, sensor failure or adversarial input. An agent that performs well in one environment can behave unpredictably after a small change. Test recovery and stopping behavior as well as the normal target task.
Where RL can help—and where it is unsuitable
RL research has been applied to games, simulations, robotics, operations research and certain optimization problems. Its suitability depends on having a meaningful reward, feedback and a way to gather experience safely. Not every scheduling or business decision problem benefits from RL; simpler rules, optimization or supervised predictors may be easier to validate and maintain.
For physical systems or consequential services, developers need safety constraints, appropriate human control, monitoring, fallback behavior and domain expertise. Treat a simulated success as a research result until validated under relevant operating conditions. Do not deploy an exploratory agent with unrestricted authority over money, personal data, access controls or hazardous equipment.
Reinforcement learning is best understood as a framework for learning action policies from experience, not an automatic route to autonomous intelligence. Its value depends on explicit assumptions, careful reward and environment design, reproducible experiments and evidence that the resulting behavior is useful and safe for the intended setting.