For many evaluation metrics commonly used as benchmarks for unconditional image generation, trivially memorizing the training set attains a better score than models which are considered state-of-the-art; we consider this problematic.
 We clarify a necessary condition for an evaluation metric not to behave this way: estimating the function must require a large sample from the model. In search of such a metric, we turn to neural network divergences (NNDs), which are defined in terms of a neural network trained to distinguish between distributions. The resulting benchmarks cannot be ``won'' by training set memorization, while still being perceptually correlated and computable only from samples. We survey past work on using NNDs for evaluation, implement an example black-box metric based on these ideas, and validate experimentally that it can measure a notion of generalization.
 In machine learning, it is often difficult to directly measure progress towards our goals (e.g. "classify images correctly in the real world", "generate valid translations of text"). To enable progress despite this difficulty, it is useful to define a standardized benchmark task which is easy to evaluate and serves as a proxy for some final task. This enables much stronger claims of improvement (albeit towards a somewhat artificial task) by reducing the risk of inadequate baselines or evaluation mistakes, with the hope that progress on the benchmark will yield discoveries and methods which are useful towards the final task. This approach requires that a benchmark task satisfy two properties:1. It should define a straightforward and objective evaluation procedure, such that strong claims of improvement can be made. Any off-limits methods of obtaining a high score (e.g. abusing a test set) should be clearly defined. 2. Improved performance on the benchmark should require insights which are likely to be helpful towards the final task. The benchmark should, by construction, reflect at least some of the kinds of difficulty inherent in the final task.Together these imply that a benchmark should at least be nontrivial: we should not know a priori how to obtain an arbitrarily high score, except perhaps by clearly-off-limits methods. Crucial to a useful benchmark is an evaluation metric which measures what we care about for the final task and which satisfies the requirements outlined above.This paper deals with unconditional generation of natural images, which has been the goal of much recent work in generative modeling (e.g. BID35 BID25 . Our ideas are general, but we hope to improve evaluation practice in models like Generative Adversarial Networks (GANs) BID14 . Generative modeling has many possible final tasks BID47 . Of these, unconditional image generation is perhaps not very useful directly, but the insights and methods discovered in its pursuit have proven useful to other tasks like domain adaptation BID42 , disentangled representation learning , and imitation learning BID21 .Some notion of generalization is crucial to why unconditional generation is difficult (and hence interesting). Otherwise , simply memorizing the training data exactly would yield perfect "generations" Figure 1 : Common GAN benchmarks prefer training set memorization (p train , red) to a model (q, green) which imperfectly fits the true distribution (p, blue) but covers more of p's support. and the task would be meaningless. One might argue that some work-particularly recent GAN research-merely aims to improve convergence properties of the learning algorithm, and so generalization isn't a big concern. However, generalization is an important part of why GANs themselves are interesting. A GAN with better convergence properties but no ability to generalize is arguably not a very interesting GAN; we believe our definition of the task, and our benchmarks, should reflect this.Because our task is to generate samples, and often our models only permit sampling, recent work (e.g. BID23 BID17 BID48 ) has adopted benchmarks based on evaluation metrics BID20 which measure the perceptual quality and diversity of samples. However these particular metrics are, by construction, trivially "won" by a model which memorizes the training set. In other words , they mostly ignore any notion of generalization, which is central to why the task is difficult to begin with. This idea is illustrated in Figure 1 . In this sense they give rise to "trivial" benchmark tasks which can lead to less convincing claims of improvement, and ultimately less progress towards useful methods. While "nontrivial " benchmark tasks based on downstream applications can be used, the goals of these tasks are at best indirectly related to, and at worst opposite from, sample generation (e.g. for semi-supervised learning BID10 ).This paper considers evaluation metrics for generative models which give rise to nontrivial benchmarks, and which are aligned with the final task of generating novel, perceptually realistic and diverse data. We stress the difference between evaluating models and defining benchmarks. The former assumes that models have been chosen ahead of time, independently of the metric. The latter assumes that models will be developed with the metric in mind, and so seeks to avoid falsely high scores resulting from exploiting undesirable solutions to the benchmark (e.g. memorizing the training set). Our goal is to define benchmarks , and many of our decisions follow from this. Our contributions are as follows:• We establish a framework for sample-based evaluation that permits a meaningful notion of generalization. We clarify a necessary condition for the ability to measure generalization: That the evaluation requires a large sample from the model. • We investigate using neural network divergences as evaluation metrics which have attractive properties for this application. We survey past work exploring the use of neural network divergences for evaluation.• We study an example neural network divergence called "CNN divergence" (D CNN ) experimentally.We demonstrate that it can detect and penalize memorization and that it measures diversity relatively more than other evaluation functions. We believe our experiments show that the NNDs are a promising direction for evaluating generative models. They are not trivially solved by memorizing the training set, which satisfies our argument (Section 3) that measuring generalization ability is linked to whether the metric requires a large collection of samples. They also appear to prefer diversity relatively more than the IS and FID metrics. We note that NNDs are almost certainly not the only evaluation metric under which models can meaningfully generalize, and encourage work on alternative metrics.Ultimately, models should be evaluated according to their intended final task BID47 . In our work we assume that our final task is not usefully solved by memorizing the training set, but for many tasks such memorization is a completely valid solution. If so, the evaluation should reflect this: we do not need our model to be closer to the true distribution than the training set, Definition 1 does not apply, and we might be free to consider evaluations which look at only a small sample from the model.
We argue that GAN benchmarks must require a large sample from the model to penalize memorization and investigate whether neural network divergences have this property. Authors propose criterion for evaluating the quality of samples produced by a Generative Adversarial Network.