Skip to content
Back to search results

Reinforcement Learning

Introduced by
Sutton, R. S. & Barto, A. G. (formalised); Bellman, R. (foundations)
Year
1950s–1998
Type
Sequential decision-making framework
Family
Deep learning / NLP / CV
Font size

What does it do?

Reinforcement Learning (RL) is a framework in which an agent learns to make sequential decisions by interacting with an environment, receiving scalar reward signals, and updating a policy to maximise cumulative future reward.

Unlike supervised learning, no labeled examples are provided; the agent discovers optimal behavior entirely through experience and delayed feedback.

The intuition behind the method

Imagine training a dog: you do not explain each action; you reward good behavior and let it figure out what to do. Reinforcement Learning works the same way. An agent tries actions in an environment, observes what happens, and receives a reward — positive for desired outcomes, negative for mistakes. Over many interactions it learns which actions in which situations tend to lead to the most reward, building a strategy — called a policy — without needing any pre-labeled dataset of correct answers.

History

Reinforcement Learning has roots in Bellman's dynamic programming (1957) and animal learning psychology. Sutton and Barto formalised the modern framework in their 1998 textbook, unifying temporal-difference learning, Q-learning, and policy-gradient methods. The field transformed in 2013–2015 when DeepMind's DQN combined deep CNNs with Q-learning to achieve human-level Atari performance, launching deep reinforcement learning as a major research direction. AlphaGo (2016) and subsequent RLHF work on large language models further expanded RL's reach.

When to use it

Use reinforcement learning when the task is sequential decision-making with delayed rewards and no labeled training data: game playing, robotic control, resource scheduling, recommendation sequencing, or fine-tuning language models with human feedback (RLHF). RL is appropriate when the environment can be simulated cheaply and the agent can interact many times. Do NOT use RL when a supervised or self-supervised approach can solve the problem — RL is sample-inefficient and unstable; when the reward function is hard to specify; when interactions with the real environment are expensive or dangerous without a simulator; or when the dataset is static and fixed (use offline RL with caution).

Limitations and considerations

  • Extremely sample-inefficient: may require millions of environment interactions to converge.
  • Training is often unstable and sensitive to hyperparameters, reward shaping, and random seeds.
  • Reward function design is difficult; poorly specified rewards cause unintended optimisation (reward hacking).
  • Generalisation to unseen states or environments is not guaranteed.
  • Real-world deployment is risky without extensive simulation; exploration can cause dangerous actions.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Learns directly from interaction without labeled data, making it applicable where annotation is impossible.
  • Capable of discovering superhuman strategies in complex sequential tasks (games, robotics, scheduling).
  • Naturally models long-horizon planning and temporal credit assignment.
  • Scales to high-dimensional state spaces (pixels, text) when combined with deep neural networks.
  • RLHF enables aligning large language models with human preferences.

Applications

  • Game playing and strategy: Atari, Go, chess, StarCraft at superhuman levels.
  • Robotics: locomotion, manipulation, and dexterous hand control.
  • Large language model alignment via Reinforcement Learning from Human Feedback (RLHF).
  • Resource and network scheduling, data center cooling optimisation.
  • Personalised recommendation systems modelling long-term user engagement.
  • Autonomous driving simulation and traffic signal control.

How it works

How it works

1

At each time step the agent observes the current state of the environment and chooses an action according to its current policy. The environment transitions to a new state and emits a scalar reward. This loop of observe-act-reward repeats, generating a trajectory of experience that is used to improve the policy.

2

The agent's objective is to maximise the discounted cumulative reward G_t, which sums all future rewards weighted by a discount factor gamma between 0 and 1. A gamma close to 1 makes the agent far-sighted; a small gamma makes it focus on immediate rewards. Careful reward shaping is critical because the agent optimises whatever signal it receives.

Formula
G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}
3

The policy pi maps states to actions (or action probabilities). It can be represented as a lookup table for small discrete problems, or as a deep neural network — a deep RL agent — for high-dimensional states such as images or text. The policy is what the agent learns and ultimately what is deployed.

Formula
\pi(a \mid s) = P(A_t = a \mid S_t = s)
4

Value functions estimate how good a state (or state-action pair) is under a given policy. The state-value V(s) is the expected cumulative reward from state s onward. Action-value functions Q(s, a) extend this to specific actions. These estimates guide policy improvement in methods such as Q-learning, SARSA, and actor-critic algorithms.

Formula
V^\pi(s) = \mathbb{E}_\pi\left[G_t \mid S_t = s\right]
5

The agent updates its policy using the collected experience. Value-based methods (e.g. DQN) derive the policy implicitly from Q-values; policy-gradient methods (e.g. REINFORCE, PPO) directly adjust policy parameters to increase the probability of high-reward trajectories. Most modern algorithms combine both in actor-critic architectures.

Common pitfalls

  • Specifying a reward function that is easy to optimise but misaligned with the true objective (reward hacking).
  • Skipping a well-tuned simulation environment and going directly to real hardware, leading to costly failures.
  • Choosing RL when a supervised or imitation-learning approach would be far more sample-efficient.
  • Ignoring reproducibility: RL results vary dramatically across random seeds; always report multiple runs.
  • Treating a single run's training curve as evidence of convergence without held-out evaluation episodes.

Frequently asked questions

What distinguishes reinforcement learning from supervised learning?

Supervised learning requires labeled input-output pairs and learns a fixed mapping. RL requires no labels; instead the agent receives reward signals after actions and must discover what to do through trial and error, making it suitable for sequential decision problems where the correct action is not known in advance.

What is the difference between model-free and model-based RL?

Model-free RL (e.g. DQN, PPO) learns policies or value functions directly from experience without an explicit model of environment dynamics. Model-based RL learns or is given a transition model and uses it for planning. Model-based methods tend to be more sample-efficient but require an accurate model, which can be hard to learn.

How do I choose between a value-based and policy-gradient method?

Value-based methods like DQN work well with discrete action spaces and are relatively stable. Policy-gradient methods (REINFORCE, PPO, SAC) handle continuous action spaces naturally and can learn stochastic policies. Actor-critic methods combine both. For most modern applications, PPO or SAC are safe starting points.

How many environment interactions does RL typically need?

RL is notoriously sample-hungry. Simple tabular tasks may converge in thousands of steps; Atari DQN required tens of millions of frames; complex robotics tasks can need hundreds of millions of simulation steps. This makes a fast simulator essential and rules out RL for many real-world problems with expensive interactions.

What is RLHF and why is it important?

Reinforcement Learning from Human Feedback trains a reward model from human preference comparisons and then fine-tunes a language model with RL to maximise that learned reward. It is the dominant technique for aligning large language models with human values and is central to models like InstructGPT and Claude.

Related methods

Sources

  1. Sutton, R. S. & Barto, A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press.

    ISBN: 978-0-262-03924-6
    Copied.
  2. Mnih, V., Kavukcuoglu, K., Silver, D., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518, 529–533.

    Open DOI reference
    Copied.