LSTM
- Introduced by
- Hochreiter, S. & Schmidhuber, J.
- Year
- 1997
- Type
- Recurrent neural network (gated memory cell)
What does it do?
LSTM (Long Short-Term Memory) is a recurrent neural network architecture, introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997, that can learn long-term dependencies in sequential data and is widely used for time-series and sequence prediction.
It keeps an internal memory that lets information persist across many time steps.
The intuition behind the method
Ordinary recurrent networks struggle to carry information across long sequences because the learning signal fades the further back it has to reach. LSTM solves this by giving each unit a memory cell that can hold a value over time, plus gates that decide what to remember, what to forget, and what to pass on. The network can therefore connect events that are far apart in a sequence, which is exactly what time-series and sequence problems demand.
History
LSTM was introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997 to overcome the vanishing-gradient problem that prevented earlier recurrent networks from learning long-range dependencies. Its gated memory cell became a cornerstone of sequence modelling and a standard choice for time-series and sequence-prediction tasks before attention-based architectures emerged.
When to use it
A good fit for forecasting and prediction on continuous or date-indexed sequences with a time-series or longitudinal structure, when long-range temporal patterns matter and a large dataset (about 500 observations or more) is available. Inputs must be normalised, and GPU acceleration is recommended. On smaller samples the network overfits, so simpler models such as Random Forest (below 500) or XGBoost (below 200) are safer.
Limitations and considerations
- Needs a large dataset (around 500 observations or more) to train reliably.
- Overfits on small samples, where simpler machine-learning models generalise better.
- Inputs must be normalised before training, adding a mandatory preprocessing step.
- Computationally heavy; GPU acceleration is strongly preferred for practical training.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Learns long-term dependencies in sequential data that ordinary recurrent networks lose.
- Well suited to time-series and longitudinal forecasting and sequence prediction.
- Assumption-light on distribution: does not require normally distributed data.
- Handles continuous and date-indexed sequence inputs natively.
Applications
- Time-series forecasting on continuous or date-indexed data.
- Sequence prediction where long-range temporal patterns matter.
- Modelling longitudinal data with dependencies spanning many time steps.
- Predictive tasks on large sequential datasets where temporal memory is essential.
How it works
How it works
Each LSTM unit carries a memory cell whose state runs forward through the sequence. This cell acts as a conveyor belt for information, letting relevant signals persist across many time steps instead of decaying after one or two, which is what enables the network to learn long-term dependencies.
Gates regulate the flow into and out of the memory cell: one gate decides how much of the past state to keep, one decides what new information to write in, and one decides what to read out as the unit's response. By learning these gates from the data, the network controls memory over long sequences rather than being overwhelmed by it.
As the sequence is processed step by step, the gated hidden state at each step feeds an output layer that produces the prediction — for example the next value in a time series. Because inputs must be normalised and the network is large, training needs a sizeable dataset (around 500 or more observations) and ideally GPU acceleration.
Formula
\hat{y}_t = f(h_t)Common pitfalls
- Feeding unnormalised inputs, which destabilises training.
- Training on fewer than about 500 observations and trusting the result instead of falling back to a simpler model.
- Applying a deep recurrent network to a very small sample where classic machine learning would suffice.
- Ignoring overfitting and reporting in-sample fit as if it were predictive performance.
Frequently asked questions
How much data does an LSTM need?
It is intended for large datasets — roughly 500 observations or more. Below that the network tends to overfit and generalises poorly, so a simpler model such as Random Forest is the recommended fallback, and below about 200 observations XGBoost is a better choice.
Do I have to normalise the inputs?
Yes. Normalisation is a mandatory preprocessing step for LSTM; unscaled inputs destabilise training and degrade the result.
What makes LSTM different from a plain recurrent network?
LSTM adds a memory cell and gates that control what to remember, forget, and output. This lets it carry information across long sequences and learn long-term dependencies that a plain recurrent network loses to vanishing gradients.
Is a GPU required?
It is not strictly required, but GPU acceleration is strongly preferred because LSTM training on large sequential datasets is computationally heavy.
Related methods
Sources
Hochreiter, S. & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780.
Open DOI referenceCopied.