What is "posterior collapse" phenomenon?

I was going through this paper on Towards Text Generation with Adversarially Learned Neural Outlines and it states why the VAEs are hard to train for text generation due to this problem. The paper states

the model ends up relying solely on the auto-regressive properties of the decoder while ignoring the latent variables, which become uninformative.

please simplify and explain the problem in a lucid way.

Topic vae autoencoder deep-learning python

Category Data Science


I think there are some mistakes in the answer by @Esmailian.
The answer said $μ_d$ and $σ_d$ are stable functions, and then later that they are unstable, which is a contradiction. I think noisy signal means that $σ_d$ is too large.
And the referenced paper seems to be wrong. The KL term that vanishes is not KL between learnt posterior and true posterior, it is KL between learnt posterior and the prior that is in Eq. 3 in the paper.


With the help of better explanations provided in Z-Forcing: Training Stochastic Recurrent Networks:

When posterior is not collapsed, $z_d$ (d-th dimension of latent variable $z$) is sampled from $q_{\phi}(z_d|x)=\mathcal{N}(\mu_d, \sigma^2_d)$, where $\mu_d$ and $\sigma_d$ are stable functions of input $x$. In other words, encoder distills useful information from $x$ into $\mu_d$ and $\sigma_d$.

We say a posterior is collapsing, when signal from input $x$ to posterior parameters is either too weak or too noisy, and as a result, decoder starts ignoring $z$ samples drawn from the posterior $q_{\phi}(z|x)$.

The too noisy signal means $\mu_d$ and $\sigma_d$ are unstable and thus sampled $z$'s are also unstable, which forces the decoder to ignore them. By "ignore" I mean: output of decoder $\hat{x}$ becomes almost independent of $z$, which in practice translates to producing some generic outputs $\hat{x}$ that are crude representatives of all seen $x$'s.

The too weak signal translates to $$q_{\phi}(z|x)\simeq q_{\phi}(z)=\mathcal{N}(a,b)$$ which means $\mu$ and $\sigma$ of posterior become almost disconnected from input $x$. In other words, $\mu$ and $\sigma$ collapse to constant values $a$, and $b$ channeling a weak (constant) signal from different inputs to decoder. As a result, decoder tries to reconstruct $x$ by ignoring useless $z$'s which are sampled from $\mathcal{N}(a,b)$.

Here are some explanations from Z-Forcing: Training Stochastic Recurrent Networks:

In these cases, the posterior approximation tends to provide a too weak or noisy signal, due to the variance induced by the stochastic gradient approximation. As a result, the decoder may learn to ignore z and instead to rely solely on the autoregressive properties of x, causing x and z to be independent, i.e. the KL term in Eq. 2 vanishes.

and

In various domains, such as text and images, it has been empirically observed that it is difficult to make use of latent variables when coupled with a strong autoregressive decoder.

where the simplest form of KL term, for the sake of clarity, is $$D_{KL}(q_{\phi}(z|x) \parallel p(z|x)) = D_{KL}(q_{\phi}(z|x) \parallel \mathcal{N}(0,1))$$ The paper uses a more complicated Gaussian prior for $p(z|x)$.

About

Geeks Mental is a community that publishes articles and tutorials about Web, Android, Data Science, new techniques and Linux security.