Long Short-Term Memory (LSTM)
- Introduced by
- Hochreiter, S. & Schmidhuber, J.
- Year
- 1997
- Type
- Recurrent neural network with gated memory cells
- Family
- Deep learning / NLP / CV
What does it do?
Long Short-Term Memory (LSTM) is a gated recurrent neural network architecture introduced by Hochreiter and Schmidhuber in 1997.
It was designed to learn dependencies across long sequences by using dedicated memory cells and three learned gates — forget, input, and output — that control what information is retained, updated, or passed forward at each time step.
The intuition behind the method
A plain recurrent network tries to carry information through time by reusing a single hidden state, but gradients shrink or explode over long sequences, making it almost impossible to learn dependencies that span many steps. LSTMs solve this with a separate memory cell that acts like a conveyor belt: the forget gate decides what old information to erase, the input gate adds new information, and the output gate determines what the cell reveals to the next layer. Because the cell can carry information largely unchanged for many steps, gradients flow more reliably during training, enabling the network to learn patterns that span sentences, paragraphs, or long time windows.
History
Hochreiter and Schmidhuber introduced the LSTM architecture in 1997 to address the vanishing-gradient problem that prevented plain RNNs from learning long-range dependencies. Through the 2000s it remained a research curiosity; by the early 2010s, driven by work from Graves and Hinton on speech recognition and from Sutskever on sequence-to-sequence learning, LSTMs became the dominant architecture for sequential tasks. They held that position until the Transformer surpassed them around 2018–2019, though LSTMs remain widely used in time-series and resource-constrained settings.
When to use it
LSTM is well-suited for tasks where the order of observations matters and relevant context may span many steps: text classification, sentiment analysis, named-entity recognition, machine translation, time-series forecasting, and speech processing. Choose LSTM when your sequences are moderately long (tens to a few hundred steps) and you have enough labeled data (typically thousands of examples) to learn gate parameters. For very long sequences or when parallelism is critical, the Transformer is usually preferable. For shorter sequences or when simplicity matters, a GRU (fewer parameters) or a fine-tuned pretrained model (BERT, RoBERTa) may be more appropriate.
Limitations and considerations
- Slower to train than Transformers because sequential computation cannot be fully parallelized.
- Requires substantial labeled data; on small datasets it tends to overfit despite dropout.
- Very long sequences (hundreds of steps) still challenge LSTMs due to limited cell capacity.
- Hyperparameter tuning — hidden units, layers, dropout, learning rate — can be time-consuming.
- For many NLP tasks, pretrained Transformer models now outperform LSTMs trained from scratch.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Explicitly designed to capture long-range dependencies without gradient vanishing.
- Flexible architecture: works for sequence classification, token labeling, generation, and forecasting.
- Interpretable gating structure — forget, input, and output gates — relative to black-box alternatives.
- Bidirectional extension (BiLSTM) doubles context by processing sequences in both directions.
- Large body of proven applications across NLP, speech, and time-series domains.
Applications
- Text classification and sentiment analysis on documents or reviews.
- Sequence labeling: named-entity recognition, part-of-speech tagging.
- Time-series forecasting: stock prices, sensor readings, energy demand.
- Speech recognition as an acoustic model component.
- Machine translation via encoder-decoder LSTM architectures.
- Anomaly detection in sequential log or sensor data.
How it works
How it works
At each time step the forget gate receives the previous hidden state and the current input, passes them through a sigmoid, and produces a value between 0 and 1 for each cell-state element. Values near 0 erase that element from memory; values near 1 preserve it. This learned erasing is what allows the LSTM to discard context that is no longer relevant.
Formula
f_t = \sigma(W_f [h_{t-1}, x_t] + b_f)The input gate, also sigmoid-activated, decides which elements of the candidate cell state — computed via tanh — should be written into the cell. Together, the forget and input gates let the network selectively remember new information while discarding stale information, without the gradient vanishing that plagues vanilla RNNs.
Formula
i_t = \sigma(W_i [h_{t-1}, x_t] + b_i),\quad \tilde{c}_t = \tanh(W_c [h_{t-1}, x_t] + b_c)The cell state is updated by element-wise multiplying the old cell state by the forget gate output, then adding the element-wise product of the input gate and the candidate cell state. This additive update is the key reason gradients can flow backward through many time steps without vanishing: the path through the cell state involves addition rather than repeated multiplication.
Formula
c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_tThe output gate controls how much of the (tanh-squashed) cell state is exposed as the hidden state passed to the next time step and to the layer above. This decoupling of the internal memory from the exposed representation gives the LSTM additional flexibility in sequencing tasks.
Formula
o_t = \sigma(W_o [h_{t-1}, x_t] + b_o),\quad h_t = o_t \odot \tanh(c_t)Depending on the task, the LSTM can return the final hidden state (sequence classification), all hidden states (sequence labeling, translation), or both. In practice, stacking multiple LSTM layers and adding dropout between them improves representational power while reducing overfitting.
Common pitfalls
- Using an LSTM without padding and masking when sequences vary in length, causing silent errors from padded zeros.
- Forgetting to set the network to evaluation mode (disabling dropout) at inference time.
- Training without gradient clipping, which can lead to exploding gradients even in LSTMs.
- Reporting accuracy alone on imbalanced sequence datasets instead of per-class F1 or AUC.
- Treating a unidirectional LSTM as sufficient for tasks (such as NER or masked prediction) where future context is available.
Frequently asked questions
What is the difference between LSTM and GRU?
A GRU merges the forget and input gates into a single update gate and has no separate cell state, giving it fewer parameters and faster training. LSTMs tend to slightly outperform GRUs on tasks requiring very precise long-term memory, but the difference is often small and dataset-dependent.
Should I use a bidirectional LSTM?
If your task allows access to the full sequence before making predictions — classification, NER, or offline forecasting — a BiLSTM typically improves accuracy by incorporating both past and future context. If predictions must be made in real time (online forecasting, streaming), use a unidirectional LSTM.
When should I choose a Transformer over an LSTM?
For most modern NLP tasks with sufficient compute, pretrained Transformers (BERT, RoBERTa) outperform LSTMs trained from scratch. Choose an LSTM when compute or latency is constrained, when your sequences are numerical time-series rather than text, or when you have too little data to fine-tune a large pretrained model.
How do I handle variable-length sequences?
Pad shorter sequences to a common length, create a mask to ignore padding tokens in loss computation, and use packed sequences (PyTorch) or masking layers (Keras/TensorFlow) so that padded positions do not affect the hidden state.
How many LSTM layers and hidden units should I use?
For most tasks one to two stacked LSTM layers with 64–512 hidden units is a sensible starting range. More layers and units increase capacity but also training time and the risk of overfitting; use dropout between layers and validate with a held-out set or cross-validation.
Related methods
Sources
Hochreiter, S. & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780.
Open DOI referenceCopied.Graves, A., Mohamed, A.-R. & Hinton, G. (2013). Speech recognition with deep recurrent neural networks. Proceedings of ICASSP 2013, pp. 6645–6649. IEEE.
Open DOI referenceCopied.