An Interpretable Evaluation of Entropy-based Novelty of Generative
  Models
Jingwei Zhang Cheuk Ting Li Farzan Farnia
The massive developments of generative model frameworks and architectures
require principled methods for the evaluation of a model's novelty compared to
a reference dataset or baseline generative models. While the recent literature
has extensively studied the evaluation of the quality, diversity, and
generalizability of generative models, the assessment of a model's novelty
compared to a baseline model has not been adequately studied in the machine
learning community. In this work, we focus on the novelty assessment under
multi-modal generative models and attempt to answer the following question:
Given the samples of a generative model $\mathcal{G}$ and a reference dataset
$\mathcal{S}$, how can we discover and count the modes expressed by
$\mathcal{G}$ more frequently than in $\mathcal{S}$. We introduce a spectral
approach to the described task and propose the Kernel-based Entropic Novelty
(KEN) score to quantify the mode-based novelty of distribution $P_\mathcal{G}$
with respect to distribution $P_\mathcal{S}$. We analytically interpret the
behavior of the KEN score under mixture distributions with sub-Gaussian
components. Next, we develop a method based on Cholesky decomposition to
compute the KEN score from observed samples. We support the KEN-based
quantification of novelty by presenting several numerical results on synthetic
and real image distributions. Our numerical results indicate the success of the
proposed approach in detecting the novel modes and the comparison of
state-of-the-art generative models.