Rethinking Test-time Likelihood: The Likelihood Path Principle and Its
  Application to OOD Detection
Sicong Huang Jiawei He Kry Yik Chau Lui
While likelihood is attractive in theory, its estimates by deep generative
models (DGMs) are often broken in practice, and perform poorly for out of
distribution (OOD) Detection. Various recent works started to consider
alternative scores and achieved better performances. However, such recipes do
not come with provable guarantees, nor is it clear that their choices extract
sufficient information.
  We attempt to change this by conducting a case study on variational
autoencoders (VAEs). First, we introduce the likelihood path (LPath) principle,
generalizing the likelihood principle. This narrows the search for informative
summary statistics down to the minimal sufficient statistics of VAEs'
conditional likelihoods. Second, introducing new theoretic tools such as nearly
essential support, essential distance and co-Lipschitzness, we obtain
non-asymptotic provable OOD detection guarantees for certain distillation of
the minimal sufficient statistics. The corresponding LPath algorithm
demonstrates SOTA performances, even using simple and small VAEs with poor
likelihood estimates. To our best knowledge, this is the first provable
unsupervised OOD method that delivers excellent empirical results, better than
any other VAEs based techniques. We use the same model as
\cite{xiao2020likelihood}, open sourced from:
https://github.com/XavierXiao/Likelihood-Regret