Preventing Model Collapse in Gaussian Process Latent Variable Models
Ying Li Zhidi Lin Feng Yin Michael Minyi Zhang
Gaussian process latent variable models (GPLVMs) are a versatile family of
unsupervised learning models, commonly used for dimensionality reduction.
However, common challenges in modeling data with GPLVMs include inadequate
kernel flexibility and improper selection of the projection noise, which leads
to a type of model collapse characterized primarily by vague latent
representations that do not reflect the underlying structure of the data. This
paper addresses these issues by, first, theoretically examining the impact of
the projection variance on model collapse through the lens of a linear GPLVM.
Second, we address the problem of model collapse due to inadequate kernel
flexibility by integrating the spectral mixture (SM) kernel and a
differentiable random Fourier feature (RFF) kernel approximation, which ensures
computational scalability and efficiency through off-the-shelf automatic
differentiation tools for learning the kernel hyperparameters, projection
variance, and latent representations within the variational inference
framework. The proposed GPLVM, named advisedRFLVM, is evaluated across diverse
datasets and consistently outperforms various salient competing models,
including state-of-the-art variational autoencoders (VAEs) and GPLVM variants,
in terms of informative latent representations and missing data imputation.