Autoencoder
- Introduced by
- Hinton, G.E. & Salakhutdinov, R.R.
- Year
- 2006
- Type
- Neural network (encoder-decoder)
What does it do?
An autoencoder is an encoder-decoder neural network, popularised by Hinton and Salakhutdinov in 2006, that compresses data into a low-dimensional latent code and then reconstructs it, enabling dimensionality reduction and anomaly detection.
By learning to rebuild its own input through a narrow bottleneck, it discovers a compact representation of the data.
The intuition behind the method
Imagine being asked to summarise a long document in a single sentence and then rewrite the full document from that sentence alone. To do this well you must keep only what matters most. An autoencoder learns exactly this discipline: it squeezes each observation through a narrow bottleneck and tries to rebuild the original from the squeezed code. Whatever survives the squeeze is the essence of the data, and inputs the network cannot reconstruct well stand out as anomalies.
History
The autoencoder idea of learning a compact code by reconstructing the input was brought to prominence by Geoffrey Hinton and Ruslan Salakhutdinov in their 2006 Science paper, which showed that deep neural networks could reduce the dimensionality of data more effectively than classical linear techniques. This work helped establish neural networks as a practical tool for representation learning.
When to use it
Use an autoencoder for nonlinear dimensionality reduction or anomaly detection on continuous-valued data when you have a reasonably large sample — roughly 200 observations or more. It does not require normally distributed data, but it needs enough data to learn a meaningful latent space, and choosing the size of the hidden layer is critical. With small samples (below about 200) PCA is the safer choice, and below about 50 neural-network dimensionality reduction fails outright.
Limitations and considerations
- Needs a reasonably large sample — about 200 observations or more — to learn a meaningful latent space.
- Choosing the size of the hidden (latent) layer is critical and strongly affects results.
- On small datasets it cannot learn the latent space and simpler PCA is preferable.
- Below about 50 observations neural-network-based dimensionality reduction fails completely.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Captures nonlinear structure that linear methods such as PCA miss.
- Serves both dimensionality reduction and anomaly detection from the same learned representation.
- Assumption-light: does not require normally distributed data.
- The latent code provides a compact, learned representation of complex continuous data.
Applications
- Nonlinear dimensionality reduction of high-dimensional continuous data.
- Anomaly detection by flagging observations with large reconstruction error.
- Learning compact latent representations as features for downstream tasks.
- Exploratory analysis of complex data where linear reduction is insufficient.
How it works
How it works
The encoder maps the input through one or more hidden layers down to a low-dimensional latent code. The size of this hidden (bottleneck) layer is the critical design choice: too wide and the network simply copies the input, too narrow and it loses meaningful structure.
Formula
z = f(Wx + b)
The decoder takes the latent code and expands it back through hidden layers toward the original dimensionality, producing a reconstruction of the input. Encoder and decoder are trained together as a single network.
Formula
\hat{x} = g(W'z + b')Training minimises the reconstruction error — the difference between the input and its rebuilt version. Once trained, the latent code serves as a compressed representation for dimensionality reduction, while large reconstruction errors flag observations that do not fit the learned patterns, supporting anomaly detection.
Formula
\min \; \lVert x - \hat{x} \rVert^2Common pitfalls
- Applying it to small samples where PCA would be both adequate and more reliable.
- Setting the bottleneck too wide, so the network copies the input instead of compressing it.
- Setting the bottleneck too narrow, discarding meaningful structure.
- Using it on data types it is not suited to instead of continuous-valued features.
Frequently asked questions
How is an autoencoder different from PCA?
PCA finds the best linear projection of the data, while an autoencoder can learn nonlinear representations through its hidden layers. For small samples or when a linear summary suffices, PCA is simpler and more reliable; autoencoders pay off when the data has nonlinear structure and the sample is large enough.
How much data do I need?
Roughly 200 observations or more. Below that, the network cannot learn a meaningful latent space and PCA is the safer choice; below about 50 observations neural-network dimensionality reduction fails outright.
How do I detect anomalies with it?
After training, pass observations through the network and measure their reconstruction error. Points the model rebuilds poorly — those with large error — do not fit the learned patterns and are flagged as anomalies.
What is the most important setting to get right?
The size of the hidden (bottleneck) layer. Too wide and the network just copies the input without learning a useful compression; too narrow and it throws away meaningful structure.
Related methods
Sources
Hinton, G.E. & Salakhutdinov, R.R. (2006). Reducing the Dimensionality of Data with Neural Networks. Science, 313(5786), 504–507.
Open DOI referenceCopied.