Skip to content
Back to search results

Deep Reinforcement Learning

Introduced by
Mnih, V. et al. (DQN)
Year
2015
Type
Sequential decision-making (agent–environment interaction)
Font size

What does it do?

Deep Reinforcement Learning combines neural networks with reinforcement learning so an agent learns by interacting with an environment, popularised by Mnih and colleagues' 2015 Nature work on human-level Atari control.

Instead of learning from a fixed labelled dataset, the agent takes actions, observes rewards, and gradually shapes a policy that maximises long-run return.

The intuition behind the method

Imagine learning to play a game with no rulebook: you try moves, see your score change, and slowly figure out which actions pay off later. Deep RL automates exactly this trial-and-error loop. A neural network estimates how good situations and actions are, and the agent keeps the choices that earn more reward over time — even when the payoff comes many steps after the decision.

History

Deep Reinforcement Learning rose to prominence with Mnih and colleagues' 2015 Nature paper, in which a deep Q-network achieved human-level control across many Atari games directly from pixels. Policy-gradient advances followed, including A3C and Schulman and colleagues' 2017 Proximal Policy Optimization (PPO), which became a widely used and more stable default. Together DQN, PPO, and A3C anchor the modern Deep RL toolkit used in games, robotics, and beyond.

When to use it

Use Deep Reinforcement Learning for sequential decision-making and control problems — game playing, robot control, recommendation optimisation, clinical decision support — where an agent can interact with a simulator or live environment and a reward signal can be defined. It handles continuous and categorical variables, makes no normality assumption, but expects a designed reward function and substantial interaction data (at least roughly 1000 episodes). With far fewer interactions the policy will not converge, and a supervised model such as Random Forest or XGBoost is a safer choice.

Limitations and considerations

  • Requires a simulator or interaction environment and a carefully designed reward function.
  • Data-hungry: needs large numbers of interactions (about 1000+) and a GPU is recommended.
  • Below roughly 1000 interactions the policy fails to converge; below ~500 it cannot produce a reliable decision policy.
  • Training can be unstable and sensitive to hyperparameters and reward shaping.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Learns control and sequential decision policies directly from interaction, without labelled examples.
  • Neural function approximation scales to very large state and action spaces.
  • Flexible across domains — games, robotics, recommendation, clinical decision support.
  • No normality assumption; works with continuous and categorical inputs and many data structures.

Applications

  • Game playing and agent control (for example Atari and board games).
  • Robot control and continuous-control tasks.
  • Recommendation and sequential optimisation problems.
  • Clinical decision support framed as sequential decision-making.

How it works

How it works

1

The agent observes the current state of an environment (a game, a simulator, a robot, a recommendation setting), chooses an action, and receives a reward plus the next state. A simulator or live interaction environment is required, and the reward function has to be designed deliberately because it defines what 'good behaviour' means. Each interaction adds to the experience the agent learns from.

2

A neural network approximates the value of states and actions — for example a deep Q-network estimates the expected future return of each action, while policy-gradient methods like PPO and A3C directly parameterise the policy. This is what makes the method 'deep': the network generalises across vast state spaces that a lookup table could never cover.

Formula
Q^*(s,a) = \mathbb{E}\big[r + \gamma \max_{a'} Q^*(s',a')\big]
3

Using the rewards collected through interaction, the network is updated to favour actions that lead to higher cumulative discounted return, steadily improving the policy toward one that maximises long-run reward. Because learning is driven by experience, large amounts of interaction data (the registry recommends at least about 1000 episodes/interactions) and ideally a GPU are needed for the policy to converge.

Formula
\pi^* = \arg\max_{\pi} \mathbb{E}_\pi\Big[\sum_t \gamma^t r_t\Big]

Common pitfalls

  • Applying Deep RL to a small dataset where the policy cannot converge instead of using a supervised model.
  • Designing a poorly specified reward function that the agent exploits in unintended ways.
  • Skipping a simulator/interaction environment and trying to learn from a static table of records.
  • Ignoring convergence diagnostics and trusting an undertrained, high-variance policy.
  • Underestimating compute needs and running without a GPU on too few episodes.

Frequently asked questions

How much data does Deep RL need?

It is data-hungry. The registry recommends at least about 1000 interactions/episodes for the policy to converge; below roughly 500 interactions DQN/PPO cannot produce a reliable decision policy, and a supervised model is preferable.

What is the difference between DQN, PPO, and A3C?

DQN learns a value function (the expected return of each action) and acts greedily with respect to it. PPO and A3C are policy-gradient methods that directly optimise the policy; PPO in particular is valued for its training stability.

Do I need a simulator?

Yes. Deep RL learns from interaction, so a simulator or a live interaction environment is required, along with a deliberately designed reward function that encodes the goal.

Do I need a GPU?

A GPU is recommended. Deep RL trains neural networks over very large numbers of interactions, so without suitable compute training is slow and convergence is hard to reach.

Related methods

Sources

  1. Mnih, V. et al. (2015). Human-Level Control through Deep Reinforcement Learning. Nature, 518, 529–533.

    Open DOI reference
    Copied.
  2. Schulman, J. et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.

    Open publication reference
    Copied.