Skip to content

Compare methods

Compare source explanations, strengths, limitations and references for up to four methods.

8 results
Criterion Reinforcement Learning Policy Gradient Recurrent Neural Network
Overview
  • Reinforcement Learning (RL) is a framework in which an agent learns to make sequential decisions by interacting with an environment, receiving scalar reward signals, and updating a policy to maximise cumulative future reward. Unlike supervised learning, no labeled examples are provided; the agent discovers optimal behavior entirely through experience and delayed feedback.
  • Policy gradient methods are reinforcement-learning algorithms that optimize a parameterized policy directly by gradient ascent on the expected return, rather than learning action-values and acting greedily. Founded on Ronald Williams' 1992 REINFORCE algorithm and the policy gradient theorem of Sutton and colleagues (2000), they naturally handle stochastic and continuous action spaces and underpin modern actor-critic and deep-RL algorithms.
  • A Recurrent Neural Network (RNN) is a class of neural network designed to process sequential data by maintaining a hidden state that carries information across time steps. Introduced in its modern form by Rumelhart et al. (1986) and further shaped by Elman (1990), RNNs became the dominant architecture for sequence modelling in NLP, speech, and time-series analysis before the rise of attention-based models.
Use and interpretation
  • Use reinforcement learning when the task is sequential decision-making with delayed rewards and no labeled training data: game playing, robotic control, resource scheduling, recommendation sequencing, or fine-tuning language models with human feedback (RLHF). RL is appropriate when the environment can be simulated cheaply and the agent can interact many times. Do NOT use RL when a supervised or self-supervised approach can solve the problem — RL is sample-inefficient and unstable; when the reward function is hard to specify; when interactions with the real environment are expensive or dangerous without a simulator; or when the dataset is static and fixed (use offline RL with caution).
  • Use policy gradient methods when the action space is continuous or high-dimensional, when a stochastic policy is desirable (exploration, partial observability, multi-agent), or when you want to optimize a policy end-to-end with a neural network — robotics control, continuous control benchmarks, dialogue/recommendation policies, and as the foundation of RLHF for language models. They directly optimize the objective and handle continuous actions that value-based methods struggle with. The costs: gradient estimates are high-variance (mitigated by baselines/critics), they are typically on-policy and sample-inefficient, sensitive to step size and reward scaling, and prone to local optima. When the action space is small and discrete, value-based Q-learning/DQN can be simpler and more sample-efficient; trust-region variants (TRPO/PPO) address step-size instability.
  • Use an RNN when your data is sequential and order carries meaning — text, speech, sensor streams, financial time series, or biological sequences of moderate length (up to a few hundred steps). It is appropriate when you need a compact recurrent architecture or when downstream integration with LSTM/GRU variants is planned. Do not use a vanilla RNN for sequences longer than a few hundred steps: vanishing gradients make it unable to learn long-range dependencies, where LSTM or GRU are far more effective. For natural language understanding tasks in 2024+, pretrained transformers (BERT, RoBERTa) almost always outperform RNNs. Use RNNs mainly as a component in larger architectures or as a baseline for sequence tasks.
Level of measurement
  • State-action-reward trajectories (environment interactions)
  • Sequential / time-series / text data
Strengths
  • Learns directly from interaction without labeled data, making it applicable where annotation is impossible.
  • Capable of discovering superhuman strategies in complex sequential tasks (games, robotics, scheduling).
  • Naturally models long-horizon planning and temporal credit assignment.
  • Scales to high-dimensional state spaces (pixels, text) when combined with deep neural networks.
  • RLHF enables aligning large language models with human preferences.
  • Directly optimize the policy; handle continuous and high-dimensional action spaces.
  • Naturally represent stochastic policies, aiding exploration and partial observability.
  • Integrate seamlessly with neural-network function approximation (deep RL).
  • Foundation for modern algorithms (A2C/A3C, TRPO, PPO, DDPG) and RLHF.
  • Naturally processes variable-length sequences without requiring padding to a fixed window size.
  • Parameter sharing across time steps keeps model size manageable relative to input length.
  • Forms the conceptual and practical foundation for LSTM, GRU, and many modern sequence models.
  • Suitable as a lightweight baseline for sequence classification and regression before more expensive architectures.
  • Can produce outputs at every time step, enabling many-to-many tasks such as sequence tagging and translation.
Limitations
  • Extremely sample-inefficient: may require millions of environment interactions to converge.
  • Training is often unstable and sensitive to hyperparameters, reward shaping, and random seeds.
  • Reward function design is difficult; poorly specified rewards cause unintended optimisation (reward hacking).
  • Generalisation to unseen states or environments is not guaranteed.
  • Real-world deployment is risky without extensive simulation; exploration can cause dangerous actions.
  • High-variance gradient estimates; need baselines or critics to be practical.
  • Usually on-policy and sample-inefficient compared with off-policy value methods.
  • Sensitive to learning rate, reward scaling, and can converge to local optima.
  • Stability requires care; naive large steps can collapse the policy.
  • Severe vanishing/exploding gradient problem for sequences longer than roughly 50–200 steps; LSTM or GRU should be preferred.
  • Sequential computation cannot be parallelised across time steps, making training slow on long sequences compared to convolutional or transformer architectures.
  • Substantially outperformed on most NLP benchmarks by pretrained transformer models such as BERT and RoBERTa.
  • Limited memory capacity: the fixed-dimensional hidden state struggles to compress long-range context.
Common pitfalls
  • Specifying a reward function that is easy to optimise but misaligned with the true objective (reward hacking).
  • Skipping a well-tuned simulation environment and going directly to real hardware, leading to costly failures.
  • Choosing RL when a supervised or imitation-learning approach would be far more sample-efficient.
  • Ignoring reproducibility: RL results vary dramatically across random seeds; always report multiple runs.
  • Treating a single run's training curve as evidence of convergence without held-out evaluation episodes.
  • Omitting a baseline/critic, leaving gradients too noisy to learn.
  • Reusing off-policy data with a plain on-policy estimator, biasing the gradient.
  • Large or unscaled rewards/step sizes causing unstable or collapsing policies.
  • Insufficient exploration when the policy prematurely becomes near-deterministic.
  • Applying a vanilla RNN to long sequences without acknowledging the vanishing gradient limitation and benchmarking against LSTM/GRU.
  • Failing to clip gradients, leading to NaN losses or unstable training on longer inputs.
  • Treating the final hidden state as a rich summary without checking whether early-sequence information is actually retained.
  • Skipping a proper train/validation/test split or cross-validation, especially given RNNs' tendency to overfit small datasets.
  • Not normalising or scaling input features, which amplifies gradient instability.
Introduced by
  • Sutton, R. S. & Barto, A. G. (formalised); Bellman, R. (foundations)
  • Ronald Williams (REINFORCE); Sutton et al. (policy gradient theorem)
  • Rumelhart, D. E.; Elman, J. L.
Year
  • 1950s–1998
  • 1992
  • 1986–1990