Approximations to the Fisher Information Metric of Deep Generative
  Models for Out-Of-Distribution Detection
Sam Dauncey Chris Holmes Christopher Williams Fabian Falck
Likelihood-based deep generative models such as score-based diffusion models
and variational autoencoders are state-of-the-art machine learning models
approximating high-dimensional distributions of data such as images, text, or
audio. One of many downstream tasks they can be naturally applied to is
out-of-distribution (OOD) detection. However, seminal work by Nalisnick et al.
which we reproduce showed that deep generative models consistently infer higher
log-likelihoods for OOD data than data they were trained on, marking an open
problem. In this work, we analyse using the gradient of a data point with
respect to the parameters of the deep generative model for OOD detection, based
on the simple intuition that OOD data should have larger gradient norms than
training data. We formalise measuring the size of the gradient as approximating
the Fisher information metric. We show that the Fisher information matrix (FIM)
has large absolute diagonal values, motivating the use of chi-square
distributed, layer-wise gradient norms as features. We combine these features
to make a simple, model-agnostic and hyperparameter-free method for OOD
detection which estimates the joint density of the layer-wise gradient norms
for a given data point. We find that these layer-wise gradient norms are weakly
correlated, rendering their combined usage informative, and prove that the
layer-wise gradient norms satisfy the principle of (data representation)
invariance. Our empirical results indicate that this method outperforms the
Typicality test for most deep generative models and image dataset pairings.