Generative adversarial training can be generally understood as minimizing certain moment matching loss defined by a set of discriminator functions, typically  neural networks. The discriminator set should be large enough to be able to uniquely identify the true distribution (discriminative), and also be small enough to go beyond memorizing samples (generalizable). In this paper, we show that a discriminator set is guaranteed to be discriminative whenever its linear span is dense in the set of bounded continuous functions. This is a very mild condition satisfied even by neural networks with a single neuron. Further, we develop generalization bounds between the learned distribution and true distribution under different evaluation metrics. When evaluated with neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set. When evaluated with KL divergence, our bound provides an explanation on the counter-intuitive behaviors of testing likelihood in GAN training. Our analysis sheds lights on understanding the practical performance of GANs. Generative adversarial networks (GANs) BID14 and their variants can be generally understood as minimizing certain moment matching loss defined by a set of discriminator functions. Mathematically, GANs minimize the integral probability metric (IPM) BID31 , that is, DISPLAYFORM0 whereμ m is the empirical measure of the observed data, and F and G are the sets of discriminators and generators, respectively.1. Wasserstain GAN (W-GAN) BID1 . F = Lip 1 (X) := {f : ||f || Lip ≤ 1}, corresponding to the Wasserstain-1 distance. 2. MMD-GAN BID27 BID13 BID25 . F is taken as the unit ball in a Reproducing Kernel Hilbert Space (RKHS), corresponding to the Maximum Mean Discrepency (MMD). 3. Energy-based GANs BID45 . F is taken as the set of continuous functions bounded between 0 and M for some constant M > 0, corresponding to the total variation distance BID1 .4. When the KL divergence is used as the evaluation metric, our bound (Corollary 3.5) suggests that the generator and discriminator sets have to be compatible in that the log density ratios of the generators and the true distributions should exist and be included inside the linear span of the discriminator set. The strong condition that log-density ratio should exist partially explains the counter-intuitive behavior of testing likelihood in flow GANs (e.g., BID11 BID15 . 5. We extend our analysis to study neural f -divergences that are the learning objective of f -GANs, and establish similar results on the discrimination and generalization properties of neural fdivergences; see Appedix B. Different from neural distance, a neural f -divergence is discriminative if linear span of its discriminators without the output activation function is dense in the bounded continuous function space. We studied the discrimination and generalization properties of GANs with parameterized discriminator class such as neural networks. A neural distance is guaranteed to be discriminative whenever the linear span of its discriminator set is dense in the bounded continuous function space. On the other hand, a neural divergence is discriminative whenever the linear span of features defined by the last linear layer of its discriminators is dense in the bounded continuous function space. We also provided generalization bounds for GANs in different evaluation metrics. In terms of neural distance, our bounds show that generalization is guaranteed as long as the discriminator set is small enough, regardless of the size of the generator or hypothesis set. This raises an interesting discriminationgeneralization balance in GANs. Fortunately, several GAN methods in practice already choose their discriminator set at the sweet point, where both the discrimination and generalization hold. Finally, our generalization bound in KL divergence provides an explanation on the counter-intuitive behaviors of testing likelihood in GAN training.There are several directions that we would like to explore in the future. First of all, in this paper, we do not talk about methods to compute the neural distance/divergence. This is typically a non-concave maximization problem and is extremely difficult to solve. Many methods have been proposed to solve this kind of minimax problems, but both stable training methods and theoretical analysis of these algorithms are still missing. Secondly, our generalization bound depends purely on the discriminator set. It is possible to obtain sharper bounds by incorporating structural information from the generator set. Finally, we would like to extend our analysis to conditional GANs (see, e.g., BID30
This paper studies the discrimination and generalization properties of GANs when the discriminator set is a restricted function class like neural networks. Balances capacities of generator and discriminator classes in GANs by guaranteeing that induced IPMs are metrics and not pseudo metrics This paper provides a mathematical analysis of the role of the size of the adversary/discriminator set in GANs