We consider the question of how to assess generative adversarial networks, in particular with respect to whether or not they generalise beyond memorising the training data. We propose a simple procedure for assessing generative adversarial network performance based on a principled consideration of what the actual goal of generalisation is. Our approach involves using a test set to estimate the Wasserstein distance between the generative distribution produced by our procedure, and the underlying data distribution. We use this procedure to assess the performance of several modern generative adversarial network architectures. We find that this procedure is sensitive to the choice of ground metric on the underlying data space, and suggest a choice of ground metric that substantially improves performance.   We finally suggest that attending to the ground metric used in Wasserstein generative adversarial network training may be fruitful, and outline a concrete pathway towards doing so. Generative adversarial networks (GANs) BID6 have attracted significant interest as a means for generative modelling. However, recently concerns have been raised about their ability to generalise from training data and their capacity to overfit . Moreover, techniques for evaluating the quality of GAN output are either ad hoc, lack theoretical rigor, or are not suitably objective -often times "visual inspection" of samples is the main tool of choice for the practitioner. More fundamentally, it is sometimes unclear exactly what we want a GAN to do: what is the learning task that we are trying to achieve?In this paper, we provide a simple formulation of the GAN training framework, which consists of using a finite dataset to estimate an underlying data distribution. The quality of GAN output is measured precisely in terms of a statistical distance D between the estimated and true distribution. Within this context, we propose an intuitive notion of what it means for a GAN to generalise.We also show how our notion of performance can be measured empirically for any GAN architecture when D is chosen to be a Wasserstein distance, which -unlike other methods such as the inception score BID14 ) -requires no density assumptions about the data-generating distribution. We investigate this choice of D empirically, finding that its performance is heavily dependent on the choice of ground metric underlying the Wasserstein distance. We suggest a novel choice of ground metric that we show performs well, and also discuss how we might otherwise use this observation to improve the design of Wasserstein GANs (WGANs) . We believe our work reveals two promising avenues of future inquiry. First, we suggest that W L p •η is an appealing choice of D, both due to its nice theoretical properties -it metricises weak convergence, and does not require us to make any density assumptions about π -and due to its sound empirical performance demonstrated above. It would be very interesting to use this D to produce a systematic and objective comparison of the performance of all current major GAN implementations, and indeed to use this as a metric for guiding future GAN design. We also view the test (5) as potentially useful for determining whether our algorithms are overfitting. This would be particularly so if applied via a cross-validation procedure: if we consistently observe that (5) holds when training a GAN according to many different X and Y partitions of our total π samples, then it seems reasonable to infer that α(X) has indeed learnt something useful about π.We also believe that the empirical inadequacy of W L 2 that we observed suggests a path towards a better WGAN architecture. At present, WGAN implementations implicitly use W L 2 for their choice of D Γ . We suspect that altering this to our suggested W L 2 •η may yield better quality samples. We briefly give here one possible way to do so that is largely compatible with existing WGAN setups. In particular, following , we take DISPLAYFORM0 for a class F of functions f : X → R that are all (L 2 • η, d R )-Lipschitz for some fixed Lipschitz constant K. Here d R denotes the usual distance on R. To optimise over such an F in practice, we can require our discriminator f : X → R to have the form f (x) := h(η(x)), where h : Y → R is (d Y , d R )-Lipschitz, which entails that f is Lipschitz provided η is (which is almost always the case in practice). In other words, we compute DISPLAYFORM1 where F is a class of (d Y , d R )-Lipschitz functions. Optimising over this objective may now proceed as usual via weight-clipping like , or via a gradient penalty like BID8 . Note that this suggestion may be understood as training a standard WGAN with the initial layers of the discriminator fixed to the embedding η; our analysis here shows that this is equivalent to optimising with respect to W L 2 •η instead of W L 2 . We have begun some experimentation in this area but leave a more detailed empirical inquiry to future work.It is also clearly important to establish better theoretical guarantees for our method. At present, we have no guarantee that the number of samples in A and Y are enough to ensure that DISPLAYFORM2 (perhaps with some fixed bias that is fairly independent of α, so that it is valid to use the value of D(Â,Ŷ ) to compare different choices of α), or that (5) entails (2) with high probability. We do however note that some recent theoretical work on the convergence rate of empirical Wasserstein estimations BID18 does suggest that it is plausible to hope for fast convergence of D(Â,Ŷ ) to D(α(X),Ŷ ). We also believe that the convincing empirical behaviour of W L 2 •η does suggest that it is possible to say something more substantial about our approach, which we leave to future work.
Assess whether or not your GAN is actually doing something other than memorizing the training data. Aims to provide a quality measure/test for GANs and proposes to evaluate the current approximation of a distribution learnt by a GAN by using Wasserstein distance between two distributions made of a sum of Diracs as a baseline performance.  This paper proposed a procedure for assessing the performance of GANs by re-considering the key of observation, using the procedure to test and improve the current GANs