We study how, in generative adversarial networks, variance in the discriminator's output affects the generator's ability to learn the data distribution. In particular, we contrast the results from various well-known techniques for training GANs when the discriminator is near-optimal and updated multiple times per update to the generator. As an alternative, we propose an additional method to train GANs by explicitly modeling the discriminator's output as a bi-modal Gaussian distribution over the real/fake indicator variables. In order to do this, we train the Gaussian classifier to match the target bi-modal distribution implicitly through meta-adversarial training. We observe that our new method, when trained together with a strong discriminator, provides meaningful, non-vanishing gradients. Generative adversarial networks BID7 are a framework for training a generator of some target (i.e., "real") distribution without explicitly defining a parametric generating distribution or a tractable likelihood function. Training the generator relies on a learning signal from a discriminator or discriminator, which is optimized on a relatively simple objective to distinguish between (i.e., classify) generated (i.e., "fake") and real samples. In order to match the true distribution, the generator parameters are optimized to maximize the loss as defined by the discriminator, which by analogy makes the generator and discriminator adversaries.In recent years, GANs have attained strong recognition as being able to generate high-quality images with sharp edges in comparison to maximum-likelihood estimation-based methods BID4 BID10 BID22 BID2 BID26 . Despite their recent successes, GANs can also be notoriously hard to train, suffering from collapse (i.e., mapping its noise to a very small set of singular outputs), missing modes of the real distribution BID3 , and vanishing and/or unstable gradients . In practice, successful learning is highly reliant on hyperparameter-tuning and model choice, and finding architectures that work with adversarial learning objectives with any / all of the above problems can be challenging BID20 .Many methods have been proposed to address learning difficulties associated with learning instability.• The use of autoencoders or posterior models in the generator or discriminator. These have been shown help to alleviate mode collapse or stabilize learning BID12 BID3 BID5 .• Regularizing the discriminator, such as with input noise , instance noise BID23 , and gradient norm regularization BID21 ).• Alternate difference measures, such as integral probability metrics (IPMs, Sriperumbudur et al., 2009 ) metrics. The most well-known of these use a dual formulation of the Wasserstein distance. These are implemented via either weight clipping or gradient penalty BID8 . These can yeild stable gradients, high-quality samples and work on a variety of architectures.On this last point, it is unclear whether the metric or the associated regularization used to impose Lipschitz is important, as regularization techniques have also been shown to be effective with stabilizing learning in f -divergences BID21 . In this paper, we study the integral role of variance in the discriminator's output in the regime of the generated distribution and how it ultimately affects learning. In the following sections , we describe theoretical motivations, an empirical analysis from multiple variants of GANs, and finally propose a regularization scheme to combat vanishing gradients on part of the discriminator when it is well-trained. In this paper, we have demonstrated the importance of intra-class variance in the discriminator's output. In particular, our results show that methods whose discriminators tend to map inputs of a class to single real values are unable to provide a reliable learning signal for the generator. Furthermore, variance in the discriminator's output is essential to allow the generator to learn in the presence of a well-trained discriminator. We proposed a technique, conceptually in line with LDA, which ensures the discriminator's output distribution follows a specified prior. Taking a broader perspective, we also introduced a new regularization technique called meta-adversarial learning, which can be applied to ensure enforce various desirable properties in GANs.
We introduce meta-adversarial learning, a new technique to regularize GANs, and propose a training method by explicitly controlling the discriminator's output distribution. The paper proposes variance regularizing adversarial learning for training GANs to ensure that the gradient for the generator does not vanish