Dataset Size Dependence of Rate-Distortion Curve and Threshold of
  Posterior Collapse in Linear VAE
Yuma Ichikawa Koji Hukushima
In the Variational Autoencoder (VAE), the variational posterior often aligns
closely with the prior, which is known as posterior collapse and hinders the
quality of representation learning. To mitigate this problem, an adjustable
hyperparameter beta has been introduced in the VAE. This paper presents a
closed-form expression to assess the relationship between the beta in VAE, the
dataset size, the posterior collapse, and the rate-distortion curve by
analyzing a minimal VAE in a high-dimensional limit. These results clarify that
a long plateau in the generalization error emerges with a relatively larger
beta. As the beta increases, the length of the plateau extends and then becomes
infinite beyond a certain beta threshold. This implies that the choice of beta,
unlike the usual regularization parameters, can induce posterior collapse
regardless of the dataset size. Thus, beta is a risky parameter that requires
careful tuning. Furthermore, considering the dataset-size dependence on the
rate-distortion curve, a relatively large dataset is required to obtain a
rate-distortion curve with high rates. Extensive numerical experiments support
our analysis.