Stationarity without mean reversion: Improper Gaussian process
  regression and improper kernels
Luca Ambrogioni
Gaussian processes (GP) regression has gained substantial popularity in
machine learning applications. The behavior of a GP regression depends on the
choice of covariance function. Stationary covariance functions are favorite in
machine learning applications. However, (non-periodic) stationary covariance
functions are always mean reverting and can therefore exhibit pathological
behavior when applied to data that does not relax to a fixed global mean value.
In this paper, we show that it is possible to use improper GP prior with
infinite variance to define processes that are stationary but not mean
reverting. To this aim, we introduce a large class of improper kernels that can
only be defined in this improper regime. Specifically, we introduce the Smooth
Walk kernel, which produces infinitely smooth samples, and a family of improper
Mat\'ern kernels, which can be defined to be $j$-times differentiable for any
integer $j$. The resulting posterior distributions can be computed analytically
and it involves a simple correction of the usual formulas. By analyzing both
synthetic and real data, we demonstrate that these improper kernels solve some
known pathologies of mean reverting GP regression while retaining most of the
favourable properties of ordinary smooth stationary kernels.