Thin and Deep Gaussian Processes
Daniel Augusto de Souza Alexander Nikitin ST John Magnus Ross Mauricio A. Álvarez Marc Peter Deisenroth João P. P. Gomes Diego Mesquita César Lincoln C. Mattos
Gaussian processes (GPs) can provide a principled approach to uncertainty
quantification with easy-to-interpret kernel hyperparameters, such as the
lengthscale, which controls the correlation distance of function values.
However, selecting an appropriate kernel can be challenging. Deep GPs avoid
manual kernel engineering by successively parameterizing kernels with GP
layers, allowing them to learn low-dimensional embeddings of the inputs that
explain the output data. Following the architecture of deep neural networks,
the most common deep GPs warp the input space layer-by-layer but lose all the
interpretability of shallow GPs. An alternative construction is to successively
parameterize the lengthscale of a kernel, improving the interpretability but
ultimately giving away the notion of learning lower-dimensional embeddings.
Unfortunately, both methods are susceptible to particular pathologies which may
hinder fitting and limit their interpretability. This work proposes a novel
synthesis of both previous approaches: Thin and Deep GP (TDGP). Each TDGP layer
defines locally linear transformations of the original input data maintaining
the concept of latent embeddings while also retaining the interpretation of
lengthscales of a kernel. Moreover, unlike the prior solutions, TDGP induces
non-pathological manifolds that admit learning lower-dimensional
representations. We show with theoretical and experimental results that i) TDGP
is, unlike previous models, tailored to specifically discover lower-dimensional
manifolds in the input data, ii) TDGP behaves well when increasing the number
of layers, and iii) TDGP performs well in standard benchmark datasets.