What the tech

Search

/ opens search from anywhere. Esc closes it.

Tools

Get the newsletter
Technology AI

Reinforcement Learning Explained With a Worked Example

In this story
Reinforcement Learning

Reinforcement learning is a way to train an agent to choose actions that maximize expected cumulative reward. The agent interacts with an environment, receives feedback, and uses experience to improve a policy, its rule for choosing actions. The objective includes future consequences, so the action with the largest immediate reward may be a poor long-term choice.

Imagine a simulated delivery robot learning to reach a destination. Moving closer can require a small cost now, while completing the delivery earns a larger reward later. The training problem is to discover useful behavior across many attempts, including situations where a promising move eventually leads to a dead end.

What makes reinforcement learning different

A supervised learning system might learn to recognize a destination from labeled images. An RL system learns how its actions affect progress toward that destination. Its decisions change the experience it encounters next, making data collection part of the problem.

People still design the task, select observations, and define or learn the reward signal. Trial and error does not mean an agent learns without human choices or automatically discovers the behavior people intended. A high score is evidence about the chosen objective, which needs to represent the actual task well.

How reinforcement learning works

At each step, the policy selects an action from the available information. The environment responds with a new observation and a reward. During training, an algorithm uses these experiences to update action values, the policy, or both. A deployed policy may simply choose actions without continuing to learn.

How reinforcement learning works

Term

Meaning in a delivery example

Agent and environment

The decision-making software and the simulated world it acts in

State and observation

The full relevant situation, versus the information the agent actually receives

Action and policy

A permitted move, and the rule or probability distribution used to select it

Reward and return

Numerical feedback for a step, and accumulated reward over subsequent steps

Episode

One attempt, ending at a defined terminal condition such as delivery or failure

State information and the Markov assumption

A Markov decision process, or MDP, models transitions and rewards using the current state and action. The state must contain the information needed for those predictions. A camera image may omit speed or an obstacle behind the robot, so it can be a partial observation. Memory or a state estimator can help the policy use information from earlier observations. See the Spinning Up guide to states, policies, and returns.

A reinforcement learning example with Q learning

Q-learning estimates the value of taking action a in state s, written Q(s, a). In its tabular form, each state-action pair has a stored number. A higher estimate indicates a better expected discounted return under the optimal continuation. It is an estimate in reward units, not a probability of success.

Consider a small grid with a fully observed position and four possible move directions. An ordinary move costs −1, reaching the goal gives +10, and reaching the goal ends the episode. These are illustrative design choices, not experimental results.

Update one action value

Suppose moving right leads to a nonterminal next state. The current value is 2, the observed reward is −1, and the largest estimated action value in the next state is 6. Choose a learning rate α of 0.5 and discount factor γ of 0.9.

Quantity

Value

Role in this update

Old Q(s, right)

2

Current estimate for the chosen move

Immediate reward r

−1

Feedback observed after moving

Best next Q value

6

Largest estimate among next-state actions

Learning rate α

0.5

Fraction of the update error applied

Discount factor γ

0.9

Weight applied to the next-state estimate

The one-step target combines the immediate reward with the best estimated continuation. The maximum below ranges over actions a′ at the next state s′.

target = r + γ × max Q(s′, a′)

new Q(s, a) = old Q(s, a) + α × [target − old Q(s, a)]

Here, the target is −1 + 0.9 × 6 = 4.4. The updated value is 2 + 0.5 × (4.4 − 2) = 3.2. Despite the negative immediate reward, this estimate rises because the next position offers a promising continuation.

Handle terminal states correctly

If a move instead reaches the terminal goal and receives +10, there is no future continuation to add. Starting from an old value of 2 with the same learning rate gives 2 + 0.5 × (10 − 2) = 6. A training time limit is a separate issue from the task truly ending, so implementations must distinguish those conditions.

Repeated updates can improve a policy, but one calculation does not prove that the estimates are accurate. Classical tabular convergence results require conditions on exploration and learning rates. They do not guarantee that a neural-network implementation will succeed on any task. The original Watkins and Dayan Q-learning paper establishes the tabular result.

Rewards exploration and the main algorithm families

Why delayed rewards matter

A return combines rewards over time. In a discounted formulation, later rewards receive successively smaller weights, such as r₁ + γr₂ + γ²r₃. With γ = 0.9, a reward two steps later has weight 0.81. Discounting changes the objective’s time horizon. It is not a direct estimate of the chance that a reward will arrive.

Sparse rewards arrive infrequently, perhaps only when delivery succeeds. That makes it difficult to identify which earlier decisions helped. Extra feedback for intermediate progress can improve learning, but it also changes the incentives and must be checked for shortcuts.

Exploration versus exploitation

Exploitation chooses actions that currently look valuable. Exploration gathers information about alternatives. A route that has never been tried may be better than the familiar one, but trying it can consume time or cause failure. Exploration belongs inside the task’s constraints, especially when physical equipment or people could be affected.

An epsilon-greedy strategy usually chooses the current highest-value action and occasionally samples an action at random. This is a simple starting point for a small discrete task. It does not provide a complete strategy for exploring a large world or discovering rewards that are rarely encountered.

Approach

What it learns or uses

Useful distinction

Tabular Q-learning

A table of state-action values

Practical for small, discrete state and action spaces

Deep Q-network or DQN

A neural approximation of action values

Handles richer inputs while typically selecting discrete actions

Policy gradient and actor-critic

A policy directly, often with a value estimator

Can handle continuous control; proximal policy optimization (PPO) is one example

Model-based RL

A known or learned model of transitions and rewards

Uses predictions for planning or training, with accuracy limited by the model

Deep reinforcement learning uses neural networks within an RL method. It is not a separate objective. Model-free methods can still learn values and policies, even though they do not explicitly learn an environment model for planning. Offline RL learns from an existing dataset, while online RL collects new interactions during learning. Deployment still needs task-specific validation.

In an actor-critic method, the actor selects actions and the critic estimates values to guide learning. Q-learning is off-policy because its target evaluates the best next action even when the behavior policy explores. PPO typically uses fresh rollouts from the current policy. This on-policy versus off-policy distinction concerns which policy generated the training data, while online versus offline concerns whether new interaction is available.

How this compares with other learning methods

Supervised learning fits input-output examples, such as images paired with object labels. Unsupervised learning discovers structure, such as clusters of similar records. Reinforcement learning evaluates actions through their consequences over time. A real system can combine all three, using perception learned from examples alongside a policy trained with rewards.

Reinforcement learning applications and their limits

Reinforcement learning applications and their limits

Games and simulated decision problems

Games provide repeatable rules, many inexpensive attempts, and measurable outcomes. They make it possible to study planning and exploration without breaking physical equipment. A policy trained for a particular game still needs evaluation when rules, opponents, or starting conditions change. Evaluate performance on each new task separately.

Robotics and embodied AI

Researchers have trained miniature humanoid robots to play a simplified one-versus-one soccer game. A 2024 Science Robotics study reported policies trained in simulation and transferred to real robots, including walking, turning, kicking, and recovering from falls. Training used dynamics randomization and perturbations to support transfer. The research team’s publication summary describes the setup.

Other robots use conventional control, imitation learning, or combinations of methods. Our embodied AI explainer explains the wider relationship between sensing, action, and learning. The humanoid robot design guide covers the physical constraints a learned controller must respect.

Reinforcement learning from human feedback

RLHF uses human feedback to help define what behavior should be rewarded. In the InstructGPT research pipeline, people supplied demonstrations and ranked model outputs. A reward model learned from comparisons, and reinforcement learning then optimized the language model against that signal. This differs from directly rewarding every correct next word. The original InstructGPT paper documents the process.

A reward model approximates the preferences represented in its training data. A high reward score can still accompany a wrong answer. RLHF is one training approach, and a deployed language model need not update its weights during a conversation.

When reinforcement learning is a good fit

RL is worth investigating when actions affect later opportunities, useful feedback exists, and you can collect enough experience at an acceptable cost. Start by comparing it with a simple rule, a conventional controller, or supervised prediction combined with optimization. Use it when better task outcomes justify the data collection and training costs.

Question

What a usable answer should establish

What can the agent control?

Defined actions, valid ranges, and prohibited behavior

What can it observe?

Available measurements, delays, and important hidden information

What counts as success?

A task metric independent of the training reward

Where can it practice?

A simulator, suitable logged data, or a controlled real setting

How will it be evaluated?

Unseen conditions, baseline comparisons, and recorded failures

Check the reward for unintended shortcuts

Imagine rewarding a delivery robot simply for distance traveled. It could move in circles and earn points without delivering anything. That hypothetical failure comes from a mismatch between the score and the goal. Measure completed deliveries separately from distance traveled.

DeepMind’s analysis of specification gaming describes how optimizing a stated objective can exploit gaps in its design. Treat task completion, collisions, energy use, and human intervention as separate checks where relevant. Important constraints may need enforcement outside the reward function.

Evaluate beyond the best training run

Measure performance on conditions held apart from training. Repeat training with several random seeds, report variation, and record the interaction budget. Compare completion rates and failures with a baseline under the same conditions. A single impressive rollout leaves unanswered how often the behavior works.

For physical systems, test changes in friction, lighting, loads, and sensor timing where they affect the task. Our digital twin architecture guide explains how models and live data can support operational representations. A simulator used for RL can be useful without being a live digital twin, and its dynamics still need validation against physical measurements.

Start with a small reproducible task

Use a fully observed grid before adding camera images or a robot body. Specify the actions, rewards, terminal condition, and training budget, then implement a random baseline and tabular Q-learning. Log whether the goal is reached as well as the return. Reproduce the update above by hand before trusting a rising learning curve.

Change one assumption at a time, such as hiding part of the state or adding a misleading intermediate reward. This reveals whether the policy learned behavior that serves the task and whether the measurements would expose a failure.