Resetting a fixed broken ELBO
Robert I. Cukier
Variational autoencoders (VAEs) are one class of generative probabilistic
latent-variable models designed for inference based on known data. They balance
reconstruction and regularizer terms. A variational approximation produces an
evidence lower bound (ELBO). Multiplying the regularizer term by beta provides
a beta-VAE/ELBO, improving disentanglement of the latent space. However, any
beta value different than unity violates the laws of conditional probability.
To provide a similarly-parameterized VAE, we develop a Renyi (versus Shannon)
entropy VAE, and a variational approximation RELBO that introduces a similar
parameter. The Renyi VAE has an additional Renyi regularizer-like term with a
conditional distribution that is not learned. The term is evaluated essentially
analytically using a Singular Value Decomposition method.