Skip to content
Back to search results

Policy Gradient Methods

Introduced by
Ronald Williams (REINFORCE); Sutton et al. (policy gradient theorem)
Year
1992
Type
Policy-based reinforcement learning
Family
Reinforcement learning
Font size

What does it do?

Policy gradient methods are reinforcement-learning algorithms that optimize a parameterized policy directly by gradient ascent on the expected return, rather than learning action-values and acting greedily.

Founded on Ronald Williams' 1992 REINFORCE algorithm and the policy gradient theorem of Sutton and colleagues (2000), they naturally handle stochastic and continuous action spaces and underpin modern actor-critic and deep-RL algorithms.

The intuition behind the method

Value-based methods like Q-learning learn how good each action is and then pick the best; policy gradients skip the middleman and adjust the policy itself. The policy is a probability distribution over actions with tunable parameters. After acting and seeing the return, you push the parameters so that actions which led to high reward become more likely and poor ones less likely — gradient ascent on expected reward. Because the policy can be any differentiable distribution, this works for continuous controls (e.g., a steering angle) where taking a 'max over actions' is impractical. Subtracting a baseline (a value estimate) keeps the noisy gradient stable.

History

Ronald Williams introduced REINFORCE, the foundational Monte-Carlo policy gradient, in 1992. Sutton, McAllester, Singh, and Mansour (2000) proved the policy gradient theorem with function approximation, enabling neural-network policies and the actor-critic family. The approach drove a decade of deep-RL advances — A3C, TRPO, PPO, DDPG/SAC — and is central to reinforcement learning from human feedback (RLHF) used to align large language models. It is the policy-based counterpart to value-based methods like Q-learning.

When to use it

Use policy gradient methods when the action space is continuous or high-dimensional, when a stochastic policy is desirable (exploration, partial observability, multi-agent), or when you want to optimize a policy end-to-end with a neural network — robotics control, continuous control benchmarks, dialogue/recommendation policies, and as the foundation of RLHF for language models. They directly optimize the objective and handle continuous actions that value-based methods struggle with. The costs: gradient estimates are high-variance (mitigated by baselines/critics), they are typically on-policy and sample-inefficient, sensitive to step size and reward scaling, and prone to local optima. When the action space is small and discrete, value-based Q-learning/DQN can be simpler and more sample-efficient; trust-region variants (TRPO/PPO) address step-size instability.

Limitations and considerations

  • High-variance gradient estimates; need baselines or critics to be practical.
  • Usually on-policy and sample-inefficient compared with off-policy value methods.
  • Sensitive to learning rate, reward scaling, and can converge to local optima.
  • Stability requires care; naive large steps can collapse the policy.

Source conditions do not replace independent scientific review or validation by an analysis engine.

Strengths

  • Directly optimize the policy; handle continuous and high-dimensional action spaces.
  • Naturally represent stochastic policies, aiding exploration and partial observability.
  • Integrate seamlessly with neural-network function approximation (deep RL).
  • Foundation for modern algorithms (A2C/A3C, TRPO, PPO, DDPG) and RLHF.

Applications

  • Continuous control in robotics and simulated environments.
  • Game playing and complex sequential decision-making.
  • Dialogue, recommendation, and adaptive interaction policies.
  • Reinforcement learning from human feedback (RLHF) for language models.
  • Operations and control problems with continuous actions.

How it works

How it works

1

The goal is to maximize J(θ), the expected discounted return of trajectories generated by the policy π_θ. Unlike value methods, the object being optimized is the policy's parameters directly, and the policy can be stochastic and continuous.

Formula
J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}\!\left[\textstyle\sum_t \gamma^{t} r_t\right]
2

The policy gradient theorem expresses the gradient of expected return as an expectation of the score function (the gradient of the log-policy) weighted by the action's value. Crucially the gradient does not require differentiating the environment dynamics, so it can be estimated from sampled experience.

Formula
\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[\nabla_\theta \log \pi_\theta(a\mid s)\, Q^{\pi}(s,a)\right]
3

REINFORCE is the Monte-Carlo estimator: after an episode, for each step it moves the parameters along the log-policy gradient scaled by the return from that step minus a baseline. The baseline (often a state-value estimate) reduces variance without adding bias, making the noisy gradient usable.

Formula
\theta \leftarrow \theta + \alpha\,\nabla_\theta \log \pi_\theta(a_t\mid s_t)\,(G_t - b(s_t))
4

Actor-critic methods learn two pieces together: an actor (the policy) updated by the policy gradient, and a critic (a value function) that supplies a low-variance estimate of the advantage (how much better an action is than average). The critic's bootstrapped estimates replace high-variance Monte-Carlo returns, yielding more stable, sample-efficient learning and the basis for A2C/A3C, PPO, and DDPG.

Formula
\text{critic estimates } Q^{\pi}/V^{\pi}\ (\text{baseline}),\ \text{actor follows } \nabla_\theta \log\pi_\theta \cdot A

Common pitfalls

  • Omitting a baseline/critic, leaving gradients too noisy to learn.
  • Reusing off-policy data with a plain on-policy estimator, biasing the gradient.
  • Large or unscaled rewards/step sizes causing unstable or collapsing policies.
  • Insufficient exploration when the policy prematurely becomes near-deterministic.

Frequently asked questions

How do policy gradients differ from Q-learning?

Q-learning learns action-values and acts greedily; policy gradients parameterize and optimize the policy directly via gradient ascent on expected return. Policy gradients handle continuous/stochastic actions naturally but are higher-variance and usually on-policy, while Q-learning is off-policy and often more sample-efficient for small discrete actions.

What is an actor-critic method?

It combines a policy (actor), updated by the policy gradient, with a learned value function (critic) that estimates how good states/actions are. The critic provides a low-variance advantage signal, stabilizing and speeding up learning. A2C/A3C, PPO, and DDPG are actor-critic algorithms.

Why do policy gradients use a baseline?

The raw gradient estimate is very noisy. Subtracting a baseline — typically a state-value estimate — from the return reduces the variance of the gradient without introducing bias, which makes learning far more stable and efficient. The advantage function (return minus value) is the common choice.

Related methods

Sources

  1. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4), 229–256.

    Open DOI reference
    Copied.
  2. Sutton, R. S., McAllester, D., Singh, S., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. Advances in Neural Information Processing Systems, 12, 1057–1063.

    Open publication reference
    Copied.