Variational Autoencoder
- Introduced by
- Kingma, D. P. & Welling, M.
- Year
- 2014
- Type
- Deep generative latent-variable model (encoder–decoder)
What does it do?
The Variational Autoencoder (VAE) is a deep generative latent-variable model, introduced by Diederik Kingma and Max Welling in 2014, that encodes data as a probability distribution in a latent space and samples from that distribution to generate new examples.
It is used for data generation, anomaly detection, and feature learning.
The intuition behind the method
An ordinary autoencoder squeezes each input into a single point in a compressed space, which is good for reconstruction but leaves no principled way to create new data. A VAE instead maps each input to a small probability cloud — a mean and a spread — and forces those clouds to stay close to a simple reference distribution. Because the latent space is then smooth and continuous, you can draw a fresh point from it and the decoder turns it into a brand-new, plausible example.
History
The Variational Autoencoder was introduced by Diederik Kingma and Max Welling in 2014 in 'Auto-Encoding Variational Bayes', which combined variational inference with neural-network encoders and decoders and introduced the reparameterisation trick that made the model trainable end to end. In 2017 Higgins and colleagues proposed beta-VAE, which reweights the KL term with a coefficient to encourage more disentangled latent factors. The VAE became one of the foundational deep generative models, alongside GANs and, more recently, diffusion models.
When to use it
A good fit when you have at least about 200 continuous-valued observations and want to learn a smooth latent representation for generating new data, detecting anomalies, or compressing high-dimensional inputs, and a GPU is available for training. The model assumes a continuous latent variable and is trained by jointly optimising reconstruction loss and the KL divergence. Below about 200 samples the latent distribution cannot be learned meaningfully and a simpler method such as PCA is preferable; below about 50 it fails entirely.
Limitations and considerations
- Needs a reasonably large sample (about 200+) and typically a GPU to train effectively.
- Samples can look blurry compared with adversarial or diffusion-based generators.
- On small datasets the latent distribution is not learned meaningfully, so PCA is the safer choice.
- Balancing the reconstruction and KL terms is delicate; the KL term can collapse and be ignored.
Source conditions do not replace independent scientific review or validation by an analysis engine.
Strengths
- Learns a smooth, continuous latent space that can be sampled to generate genuinely new examples.
- Provides a principled probabilistic framework, optimising a single objective (the evidence lower bound).
- Useful beyond generation — for anomaly detection, feature learning, and nonlinear dimensionality reduction.
- Does not require normally distributed inputs and works with continuous data of high dimension.
- The reparameterisation trick makes the whole model trainable end to end by gradient descent.
Applications
- Generating new synthetic data samples from a learned latent space.
- Anomaly and outlier detection by flagging inputs the model reconstructs poorly.
- Nonlinear dimensionality reduction and feature learning on high-dimensional data.
- Representation learning as a pre-step for downstream prediction or clustering.
How it works
How it works
An encoder network reads each input and outputs the parameters of a distribution over the latent variable — a mean and a variance — rather than a single fixed code. This is the probabilistic encoding step that represents each observation as a region of latent space instead of one point.
Formula
q_\phi(z \mid x) = \mathcal{N}(z; \mu_\phi(x), \sigma_\phi^2(x))A latent vector is drawn from the encoded distribution using the reparameterisation trick: instead of sampling directly, the model writes the sample as the mean plus the standard deviation times standard Gaussian noise. Because the randomness is pushed into an external noise term, gradients can still flow through the network and the model can be trained end to end.
Formula
z = \mu + \sigma \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)A decoder network maps the sampled latent vector back to data space, and training optimises the evidence lower bound: a reconstruction term that rewards faithfully rebuilding the input, minus a Kullback–Leibler term that pulls each encoded distribution towards the prior. Balancing these two forces is what produces a latent space that both reconstructs well and supports generating new samples.
Formula
\mathcal{L} = \mathbb{E}_{q_\phi}[\log p_\theta(x \mid z)] - D_{KL}(q_\phi(z \mid x) \,\|\, p(z))Common pitfalls
- Training on too few observations (below about 200) and expecting a meaningful generative model.
- Ignoring posterior collapse, where the KL term drives the latent code to be uninformative.
- Judging the model on reconstruction sharpness alone instead of the quality of generated samples.
- Skipping the reparameterisation trick and breaking the gradient flow through the sampling step.
- Reaching for a VAE when simple linear dimensionality reduction such as PCA would suffice.
Frequently asked questions
How is a VAE different from a plain autoencoder?
A plain autoencoder maps each input to a single latent code optimised only for reconstruction. A VAE encodes each input as a probability distribution and adds a KL term that keeps that distribution close to a prior, which yields a smooth latent space you can sample from to generate new data.
What is the reparameterisation trick?
It rewrites a latent sample as the encoded mean plus the standard deviation times standard Gaussian noise. By moving the randomness into an external noise term, gradients can pass through the sampling step so the encoder and decoder can be trained together by backpropagation.
How much data and what hardware do I need?
At least about 200 continuous observations are recommended, and a GPU is advised for training. Below 200 the latent distribution is not learned meaningfully and PCA is preferable; below about 50 the VAE fails entirely.
What is beta-VAE?
beta-VAE, proposed by Higgins and colleagues in 2017, multiplies the KL term by a coefficient (beta). A larger beta pressures the latent dimensions to capture more independent, disentangled factors of variation, at some cost to reconstruction fidelity.
Related methods
Sources
Kingma, D. P. & Welling, M. (2014). Auto-Encoding Variational Bayes. International Conference on Learning Representations (ICLR).
Open publication referenceCopied.Higgins, I. et al. (2017). beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. International Conference on Learning Representations (ICLR).
Open publication referenceCopied.