AI-generated review 5 min read Intermediate 952 words

Variational Autoencoders

Variational Autoencoders learn compressed latent representations with the ELBO objective and reparameterization trick for generative modeling.

vae variational-inference generative latent-variable-models autoencoders
Generative models · 2013

Primary source: Auto-Encoding Variational Bayes (arXiv:1312.6114)

Generative modeling asks a simple question with a hard answer: given a dataset of images, sounds, or sentences, can we learn a model that produces new samples that look like they came from the same distribution? Before 2013, the most principled approaches to this problem relied on graphical models with intractable inference, while neural networks were mostly used for discriminative tasks like classification.

The 2013 paper Auto-Encoding Variational Bayes by Kingma and Welling bridged that gap. It introduced the variational autoencoder (VAE), a neural network that learns a compressed latent representation of the data while remaining a proper probabilistic generative model. The paper's two contributions — a scalable variational objective and the reparameterization trick — made it possible to train deep latent-variable models with ordinary stochastic gradient descent.

The core idea

A VAE assumes each data point x was generated from an unobserved latent variable z: first sample z from a simple prior p(z), usually a standard Gaussian, then sample x from a decoder distribution p(x \mid z) parameterized by a neural network. The catch is that inference — computing the posterior p(z \mid x) — is intractable for any interesting decoder.

The paper's solution is amortized variational inference. Instead of optimizing a separate posterior for every data point, train an encoder network q(z \mid x) that maps any input directly to the parameters of its approximate posterior. One forward pass gives you a distribution over latent codes, and sampling from it and decoding gives you new data. Unlike a plain autoencoder, the latent space is regularized to follow the prior, so random draws from the prior decode into plausible outputs.

Input x Encoder q(z|x) μ, σ Sample z + noise ε Decoder p(x|z) x̂ KL(q(z|x) || p(z)) pulls posterior toward prior loss x̂≈x
The VAE pipeline: the encoder maps input x to posterior parameters, a latent code z is sampled with injected noise, and the decoder reconstructs the input while a KL term regularizes the posterior.

The ELBO objective

Because the true log-likelihood \log p(x) involves an intractable integral over z, the paper optimizes a tractable lower bound on it, the evidence lower bound (ELBO):

\log p(x) \geq \mathbb{E}_{q}[\log p(x \mid z)] - \text{KL}(q(z \mid x) \,\|\, p(z))

The two terms pull in complementary directions. The reconstruction term rewards latent codes from which the decoder can rebuild the input, like a standard autoencoder. The KL divergence term penalizes posteriors that drift far from the prior, which keeps the latent space dense and smooth. With Gaussian posteriors and a Gaussian prior, the KL term has a closed form, so it costs almost nothing to compute.

This framing is what makes the VAE a generative model rather than just a compression scheme.

The reparameterization trick

The subtle problem is gradients. The reconstruction term is an expectation over samples from q(z \mid x), and naive sampling blocks backpropagation because it is a stochastic operation with no gradient. The paper's key technical move is to reparameterize the sample as a deterministic function of the encoder outputs plus independent noise:

z = \mu(x) + \sigma(x) \odot \epsilon, \quad \epsilon \sim \mathcal{N}(0, I)

Now the randomness comes from \epsilon, which has no parameters, and gradients flow freely through \mu and \sigma. These low-variance gradients are why VAEs train stably with ordinary minibatch gradient descent. The same trick has since become standard across variational inference, reinforcement learning, and diffusion models.

Inside the architecture

The paper keeps the networks deliberately simple: multilayer perceptrons for both encoder and decoder, with the encoder emitting a mean and log-variance vector and the decoder emitting Bernoulli logits for binary images or Gaussian means for continuous data. Training uses minibatches of around 100 points with a small number of posterior samples each — often just one — plus the analytic KL term.

Two design details matter in practice. First, the latent dimensionality is a capacity knob: too few dimensions and reconstructions suffer, too many and the KL term can collapse toward zero while the decoder ignores z, a failure known as posterior collapse. Second, the choice of decoder distribution implicitly sets the reconstruction loss — a Gaussian decoder gives mean squared error, which is one reason VAE images look smooth or blurry compared with the crisp samples of GANs.

What the paper showed

The authors demonstrated VAEs on MNIST digits and Frey Faces, showing smooth interpolations through latent space and plausible random samples drawn from the prior. Quantitatively, they estimated held-out log-likelihoods that compared favorably with contemporary deep generative models such as wake-sleep networks and Monte Carlo EM. For the first time, a deep directed generative model with continuous latents could be trained end to end by backpropagation.

Legacy and limitations

The VAE became one of the three pillars of deep generative modeling alongside GANs and, later, diffusion models. Its descendants include VQ-VAE, which discretizes the latent space and underpins modern token-based generators, and the latent diffusion approach, which runs a diffusion process inside a VAE-like latent space.

The main limitation is sample quality: likelihood-based training with simple decoder distributions averages over plausible outputs, producing blur. VAEs also inherit the usual variational gap — the ELBO is only a bound, and a loose one when the true posterior is far from Gaussian.

Source

Want the next review in your inbox? The briefing collects them — at most one email a week. Read signup data and removal options.

Cite this report
@misc{knowyourml-vae-auto-encoding-variational-bayes
  title        = {Variational Autoencoders},
  author       = {{Know Your ML}},
  year         = {2013},
  howpublished = {\url{https://knowyourml.yunusemrevurgun.com/report.php?slug=vae-auto-encoding-variational-bayes}},
  note         = {AI-generated review of arXiv:1312.6114. Primary sources are listed on the page.}
}
Machine Learning Decision Toolkit — decision matrix, scorecards and trade-off analysis
Toolkit 7 decision tools Model-agnostic

Machine Learning Decision Toolkit

Turn this review into a decision: the Machine Learning Decision Toolkit packages the evaluation scorecards, workload templates and deployment checks behind model, provider and architecture choices.

$24 View on Gumroad

References

Frequently asked questions

How is a VAE different from a plain autoencoder?

A standard autoencoder learns a deterministic mapping with no structure on the bottleneck, so random points in it usually decode to garbage. A VAE forces the encoder to output distributions matched to a Gaussian prior, which makes the latent space smooth and samplable.

What is the ELBO objective?

ELBO stands for evidence lower bound: a tractable lower bound on the log-likelihood of the data. Maximizing it simultaneously improves reconstruction quality and keeps the learned posterior close to the prior.

What is the reparameterization trick?

It rewrites sampling from the encoder distribution as a deterministic function of the parameters plus independent noise, typically z = mu + sigma * epsilon. Gradients can then flow through mu and sigma while the randomness comes from epsilon.

Why do VAE samples look blurry?

VAE samples tend to be blurrier than those from GANs or diffusion models because the reconstruction loss averages over plausible outputs. Later work such as VQ-VAE and latent diffusion addressed this while keeping the latent-variable idea.

How to Remain Valuable When Intelligence Becomes Cheap — book cover
eBook 224 pages

How to Remain Valuable When Intelligence Becomes Cheap

This report covered one more way the machine gets smarter. The companion eBook asks the question on the other side: which human advantages stay scarce — and valuable — when intelligence itself is cheap?

Read the eBook