Neural Operator Variational Inference based on Regularized Stein
  Discrepancy for Deep Gaussian Processes
Jian Xu Shian Du Junmei Yang Qianli Ma Delu Zeng
Deep Gaussian Process (DGP) models offer a powerful nonparametric approach
for Bayesian inference, but exact inference is typically intractable,
motivating the use of various approximations. However, existing approaches,
such as mean-field Gaussian assumptions, limit the expressiveness and efficacy
of DGP models, while stochastic approximation can be computationally expensive.
To tackle these challenges, we introduce Neural Operator Variational Inference
(NOVI) for Deep Gaussian Processes. NOVI uses a neural generator to obtain a
sampler and minimizes the Regularized Stein Discrepancy in L2 space between the
generated distribution and true posterior. We solve the minimax problem using
Monte Carlo estimation and subsampling stochastic optimization techniques. We
demonstrate that the bias introduced by our method can be controlled by
multiplying the Fisher divergence with a constant, which leads to robust error
control and ensures the stability and precision of the algorithm. Our
experiments on datasets ranging from hundreds to tens of thousands demonstrate
the effectiveness and the faster convergence rate of the proposed method. We
achieve a classification accuracy of 93.56 on the CIFAR10 dataset,
outperforming SOTA Gaussian process methods. Furthermore, our method guarantees
theoretically controlled prediction error for DGP models and demonstrates
remarkable performance on various datasets. We are optimistic that NOVI has the
potential to enhance the performance of deep Bayesian nonparametric models and
could have significant implications for various practical applications