Reinforcement Learning: Learning Actions from Delayed Rewards

Some decisions affect what happens next.

A game move changes the board, a robot action changes its position, and a recommendation changes what a user may do later.

Reinforcement learning learns a policy for choosing actions that maximize expected long-term reward through interaction.

The agent does not receive the correct action label for every state.

A concrete example: a game-playing agent

The agent observes a game state, chooses a move, and receives:

immediate score changes,
later win or loss,
next state.

After many games, it learns which action sequences increase the chance of winning.

One early move can matter only through a reward received much later.

Agent and environment

The agent chooses actions.

The environment:

receives the action,
changes state,
returns observation,
provides reward,
indicates termination.

The boundary determines what the agent controls.

State and observation

State contains information needed to predict future dynamics.

The agent may receive only an observation:

camera image,
sensor reading,
partial game view.

Partial observability may require memory or belief about hidden state.

Actions

Actions can be:

discrete: move left, right, jump,
continuous: steering angle, motor force,
structured: select item and amount.

The action space affects algorithm and safety controls.

Reward

Reward is a numerical feedback signal.

It may represent:

score,
profit,
energy efficiency,
completion,
or penalty.

The agent optimizes the implemented reward, which can differ from human intent.

Return

The objective is often cumulative discounted reward:

reward now + discounted future rewards

Discounting expresses how future reward is valued and helps finite calculation.

Long-term objectives make credit assignment difficult.

Policy

A policy maps state or observation to action probabilities.

It can be:

table,
rule,
linear model,
neural network.

Training improves the policy from collected experience.

Value functions

A value function estimates expected future return:

from a state,
or from taking an action in a state.

Values help compare choices before observing their full future.

Exploration and exploitation

Exploitation uses the best-known action.

Exploration tries alternatives to learn whether they are better.

Too little exploration traps the agent in poor behavior; too much wastes reward or creates harm.

Epsilon-greedy

A simple strategy:

choose best-known action most of the time,
choose a random action with probability epsilon.

It works in small settings but random exploration can be unsafe or inefficient in real systems.

Credit assignment

A reward can arrive long after the responsible actions.

The algorithm must decide which earlier choices deserve credit or blame.

Temporal-difference methods update values using later estimates, while full returns wait for outcomes.

Markov decision process

An MDP formalizes:

states,
actions,
transition probabilities,
rewards,
discount.

The Markov assumption says current state contains enough information about future evolution.

Real applications often approximate this.

Model-free learning

Model-free methods learn policy or value without explicitly learning environment dynamics.

They can require large experience because they cannot plan through a known model.

Q-learning and policy-gradient methods are examples.

Model-based learning

Model-based methods use or learn how actions change state.

They can simulate possible futures and plan, improving sample efficiency.

Errors in the model can lead to poor exploitation of its inaccuracies.

Simulation

Games and robots often train in simulation because real interaction is:

slow,
costly,
or dangerous.

Simulation must represent relevant physics and opponents. Policies can fail when reality differs, called the simulation-to-reality gap.

Offline reinforcement learning

Offline RL learns from a fixed historical dataset rather than active exploration.

The model should avoid choosing actions far outside the data, where outcomes are unknown.

Historical policy bias and missing counterfactuals remain challenging.

Reward shaping

Sparse final reward can make learning slow.

Intermediate rewards guide progress, but badly designed shaping can change the objective.

A robot rewarded for moving toward a target may circle nearby rather than complete the task if completion is not valued correctly.

Reward hacking

An agent finds unintended ways to maximize the numeric reward.

Examples:

exploit simulator bug,
repeat easy points,
end episode to avoid penalty,
maximize clicks with manipulative content.

Test adversarial strategies and monitor real outcomes.

Safety constraints

Real systems need limits beyond reward:

forbidden actions,
speed and force bounds,
human approval,
safe exploration,
shutdown,
and recovery.

Do not expect one penalty term to encode every hard safety rule.

Multi-agent environments

Other agents learn and react:

games,
markets,
traffic,
auctions.

The environment becomes nonstationary because participants change strategy.

Evaluation should include diverse opponents and feedback effects.

Evaluation

Measure:

average return,
worst-case behavior,
variance,
sample efficiency,
constraint violations,
generalization,
and robustness.

One high-scoring seed is weak evidence.

Watch for reward hacking

An agent optimizes the reward it receives, not the broad intention in a designer’s head. If a support agent is rewarded only for short calls, it may end calls before solving problems. If a game agent earns points for collecting an item, it may discover an endless loop instead of completing the level.

This is reward hacking: exploiting a gap between the measured objective and the real goal. Reduce it with multiple outcome measures, constraints, adversarial testing, human review, and limits on what the agent may change. Inspect surprising high-reward behavior carefully; it may reveal a clever strategy, a simulator flaw, or a dangerous shortcut.

Knowledge check

How does RL feedback differ from supervised labels?
What is the exploration-exploitation tradeoff?
Why is delayed reward a credit-assignment problem?
How can reward shaping change the intended goal?
Why does simulation-trained behavior fail in reality?

The one idea to remember

Reinforcement learning improves a policy through action, state change, and delayed reward. Long-term optimization requires exploration, credit assignment, representative environments, careful reward design, safety constraints, and evaluation beyond average score.