Gradient-flow adaptive importance sampling for Bayesian leave one out
  cross-validation for sigmoidal classification models
Joshua C Chang Xiangting Li Shixin Xu Hao-Ren Yao Julia Porcino Carson Chow
We introduce a set of gradient-flow-guided adaptive importance sampling (IS)
transformations to stabilize Monte-Carlo approximations of point-wise leave one
out cross-validated (LOO) predictions for Bayesian classification models. One
can leverage this methodology for assessing model generalizability by for
instance computing a LOO analogue to the AIC or computing LOO ROC/PRC curves
and derived metrics like the AUROC and AUPRC. By the calculus of variations and
gradient flow, we derive two simple nonlinear single-step transformations that
utilize gradient information to shift a model's pre-trained full-data posterior
closer to the target LOO posterior predictive distributions. In doing so, the
transformations stabilize importance weights. Because the transformations
involve the gradient of the likelihood function, the resulting Monte Carlo
integral depends on Jacobian determinants with respect to the model Hessian. We
derive closed-form exact formulae for these Jacobian determinants in the cases
of logistic regression and shallow ReLU-activated artificial neural networks,
and provide a simple approximation that sidesteps the need to compute full
Hessian matrices and their spectra. We test the methodology on an $n\ll p$
dataset that is known to produce unstable LOO IS weights.