Recurrent Neural Network
- Introduced by
- Rumelhart, D. E.; Elman, J. L.
- Year
- 1986–1990
- Type
- Sequential neural network
- Family
- Deep learning / NLP / CV
What does it do?
A Recurrent Neural Network (RNN) is a class of neural network designed to process sequential data by maintaining a hidden state that carries information across time steps.
Introduced in its modern form by Rumelhart et al. (1986) and further shaped by Elman (1990), RNNs became the dominant architecture for sequence modelling in NLP, speech, and time-series analysis before the rise of attention-based models.
The intuition behind the method
Unlike feedforward networks that treat each input independently, an RNN processes a sequence one element at a time and passes a summary of what it has seen so far — its hidden state — to the next step. Think of it like reading a sentence word by word: at each word, you update your understanding in light of all previous words. This recurrent connection allows the network to capture temporal dependencies and context, making it naturally suited to tasks where order matters and past context shapes the meaning of the current element.
History
The foundations of recurrent networks were laid by Rumelhart, Hinton, and Williams in 1986 with the backpropagation through time algorithm, and Elman's 1990 paper popularised the simple recurrent network architecture for cognitive modelling. Throughout the 1990s and 2000s RNNs powered speech recognition and language modelling, though their practical impact was limited by the vanishing gradient problem until Hochreiter and Schmidhuber introduced LSTM in 1997. RNNs remain important as conceptual anchors and lightweight baselines.
When to use it
Use an RNN when your data is sequential and order carries meaning — text, speech, sensor streams, financial time series, or biological sequences of moderate length (up to a few hundred steps). It is appropriate when you need a compact recurrent architecture or when downstream integration with LSTM/GRU variants is planned. Do not use a vanilla RNN for sequences longer than a few hundred steps: vanishing gradients make it unable to learn long-range dependencies, where LSTM or GRU are far more effective. For natural language understanding tasks in 2024+, pretrained transformers (BERT, RoBERTa) almost always outperform RNNs. Use RNNs mainly as a component in larger architectures or as a baseline for sequence tasks.
Limitations and considerations
- Severe vanishing/exploding gradient problem for sequences longer than roughly 50–200 steps; LSTM or GRU should be preferred.
- Sequential computation cannot be parallelised across time steps, making training slow on long sequences compared to convolutional or transformer architectures.
- Substantially outperformed on most NLP benchmarks by pretrained transformer models such as BERT and RoBERTa.
- Limited memory capacity: the fixed-dimensional hidden state struggles to compress long-range context.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Naturally processes variable-length sequences without requiring padding to a fixed window size.
- Parameter sharing across time steps keeps model size manageable relative to input length.
- Forms the conceptual and practical foundation for LSTM, GRU, and many modern sequence models.
- Suitable as a lightweight baseline for sequence classification and regression before more expensive architectures.
- Can produce outputs at every time step, enabling many-to-many tasks such as sequence tagging and translation.
Applications
- Text classification and sentiment analysis on short-to-medium length documents.
- Language modelling and next-word prediction at character or word level.
- Time-series forecasting for sensor, financial, or physiological data of moderate length.
- Sequence tagging tasks such as part-of-speech tagging when used as an encoder.
- Speech-to-text and audio feature extraction pipelines.
- Baseline sequence modelling before upgrading to LSTM, GRU, or transformer architectures.
How it works
How it works
The input is a sequence of vectors (words, time steps, frames) fed to the network one element at a time. Each element x_t is paired with the hidden state h_{t-1} carried from the previous step, allowing context to accumulate across the entire sequence.
At each step t, the hidden state is updated as h_t = tanh(W_h * h_{t-1} + W_x * x_t + b). The weight matrices W_h and W_x are shared across all time steps, so the model learns a single transition rule that is applied repeatedly, regardless of sequence length.
Formula
h_t = \tanh(W_h h_{t-1} + W_x x_t + b)An output y_t is computed from the hidden state at each time step (many-to-many) or only at the final step (many-to-one). For classification tasks the final hidden state is passed to a softmax layer; for sequence-to-sequence tasks outputs are produced at every step.
Formula
y_t = W_y h_t + b_y
Training uses backpropagation through time (BPTT): gradients flow backward through the unrolled sequence to update all shared weights. For long sequences this leads to the vanishing or exploding gradient problem, where gradients become too small or too large to update early time steps reliably.
Shared weights W_h, W_x, and W_y are updated via gradient descent or an adaptive optimizer (Adam, RMSProp). Gradient clipping is commonly applied to prevent exploding gradients. The same weights govern every step, so the model generalises across varying sequence lengths.
Common pitfalls
- Applying a vanilla RNN to long sequences without acknowledging the vanishing gradient limitation and benchmarking against LSTM/GRU.
- Failing to clip gradients, leading to NaN losses or unstable training on longer inputs.
- Treating the final hidden state as a rich summary without checking whether early-sequence information is actually retained.
- Skipping a proper train/validation/test split or cross-validation, especially given RNNs' tendency to overfit small datasets.
- Not normalising or scaling input features, which amplifies gradient instability.
Frequently asked questions
What is the difference between an RNN, an LSTM, and a GRU?
An RNN is the simplest recurrent architecture: it has a single hidden state updated at each step. LSTM adds input, forget, and output gates that protect long-range information from vanishing gradients. GRU is a streamlined version of LSTM with fewer parameters. Both LSTM and GRU almost always outperform a vanilla RNN on tasks requiring memory beyond ~50–100 steps.
Why does training an RNN sometimes produce NaN losses?
NaN losses almost always result from exploding gradients. Apply gradient clipping (clip norm to 1 or 5) and lower the learning rate. Ensure inputs are normalised. If the problem persists, switch to LSTM or GRU, which are more stable due to their gating mechanisms.
Can I still use an RNN for NLP tasks in 2024?
For production NLP, pretrained transformers such as BERT or RoBERTa dominate. Vanilla RNNs are best reserved as lightweight baselines, components inside larger architectures, or for resource-constrained environments where large model sizes are prohibitive.
How do I choose the hidden state dimension?
Common choices range from 64 to 512 units depending on task complexity and dataset size. Start small (e.g., 128), monitor validation loss, and increase capacity only if underfitting is evident. Larger hidden states increase the risk of overfitting on small datasets.
What loss function should I use for sequence classification?
For binary classification use binary cross-entropy; for multi-class use categorical cross-entropy with softmax on the final hidden state. For sequence labelling (one output per step), apply the loss at every time step and average or sum across steps.
Related methods
Sources
Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179–211.
Open DOI referenceCopied.Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533–536.
Open DOI referenceCopied.