A Unifying Perspective on Non-Stationary Kernels for Deeper Gaussian
  Processes
Marcus M. Noack Hengrui Luo Mark D. Risser
The Gaussian process (GP) is a popular statistical technique for stochastic
function approximation and uncertainty quantification from data. GPs have been
adopted into the realm of machine learning in the last two decades because of
their superior prediction abilities, especially in data-sparse scenarios, and
their inherent ability to provide robust uncertainty estimates. Even so, their
performance highly depends on intricate customizations of the core methodology,
which often leads to dissatisfaction among practitioners when standard setups
and off-the-shelf software tools are being deployed. Arguably the most
important building block of a GP is the kernel function which assumes the role
of a covariance operator. Stationary kernels of the Mat\'ern class are used in
the vast majority of applied studies; poor prediction performance and
unrealistic uncertainty quantification are often the consequences.
Non-stationary kernels show improved performance but are rarely used due to
their more complicated functional form and the associated effort and expertise
needed to define and tune them optimally. In this perspective, we want to help
ML practitioners make sense of some of the most common forms of
non-stationarity for Gaussian processes. We show a variety of kernels in action
using representative datasets, carefully study their properties, and compare
their performances. Based on our findings, we propose a new kernel that
combines some of the identified advantages of existing kernels.