A Practical Guide to Statistical Distances for Evaluating Generative
  Models in Science
Sebastian Bischoff Alana Darcher Michael Deistler Richard Gao Franziska Gerken Manuel Gloeckler Lisa Haxel Jaivardhan Kapoor Janne K Lappalainen Jakob H Macke Guy Moss Matthijs Pals Felix Pei Rachel Rapp A Erdem Sağtekin Cornelius Schröder Auguste Schulz Zinovia Stefanidi Shoji Toyota Linda Ulmer Julius Vetter
Generative models are invaluable in many fields of science because of their
ability to capture high-dimensional and complicated distributions, such as
photo-realistic images, protein structures, and connectomes. How do we evaluate
the samples these models generate? This work aims to provide an accessible
entry point to understanding popular notions of statistical distances,
requiring only foundational knowledge in mathematics and statistics. We focus
on four commonly used notions of statistical distances representing different
methodologies: Using low-dimensional projections (Sliced-Wasserstein; SW),
obtaining a distance using classifiers (Classifier Two-Sample Tests; C2ST),
using embeddings through kernels (Maximum Mean Discrepancy; MMD), or neural
networks (Fr\'echet Inception Distance; FID). We highlight the intuition behind
each distance and explain their merits, scalability, complexity, and pitfalls.
To demonstrate how these distances are used in practice, we evaluate generative
models from different scientific domains, namely a model of decision making and
a model generating medical images. We showcase that distinct distances can give
different results on similar data. Through this guide, we aim to help
researchers to use, interpret, and evaluate statistical distances for
generative models in science.