In most current formulations of adversarial training, the discriminators can be expressed as single-input operators, that is, the mapping they define is separable over observations. In this work, we argue that this property might help explain the infamous mode collapse phenomenon in adversarially-trained generative models. Inspired by discrepancy measures and two-sample tests between probability distributions, we propose distributional adversaries that operate on samples, i.e., on sets of multiple points drawn from a distribution, rather than on single observations. We show how they can be easily implemented on top of existing models. Various experimental results show that generators trained in combination with our distributional adversaries are much more stable and are remarkably less prone to mode collapse than traditional models trained with observation-wise prediction discriminators. In addition, the application of our framework to domain adaptation results in strong improvement over recent state-of-the-art. Adversarial training of neural networks, especially Generative Adversarial Networks (GANs) BID10 , has proven to be a powerful tool for learning rich models, leading to outstanding results in various tasks such as realistic image generation, text to image synthesis, 3D object generation, and video prediction BID25 BID33 BID32 . Despite their success, GANs are known to be difficult to train. The generator and discriminator can oscillate significantly from iteration to iteration, and slight imbalances in their capacities frequently cause the training objective to diverge. Another common problem suffered by GANs is mode collapse, where the distribution learned by the generator concentrates on a few modes of the true data distribution, ignoring the rest of the space. In the case of images, this failure results in generated images that albeit realistic, lack diversity and reduce to a handful of prototypes.A flurry of recent research seeks to understand and address the causes of instability and mode collapse in adversarially-trained models. The first insights come from BID10 , who note that one of the main causes of training instability is saturation of the discriminator. BID1 formalize this idea by showing that if the two distributions have supports that are disjoint or concentrated on low-dimensional manifolds that do not perfectly align, then there exists an optimal discriminator with perfect classification accuracy almost everywhere and the usual divergences (Kullback-Leibler, Jensen-Shannon) max-out for this discriminator. In follow-up work, propose an alternative training scheme (WGAN) based on estimating the Wasserstein distance instead of the Jensen-Shannon divergence between real and generated distributions.In this work, we highlight a further view on mode collapse. The discriminator part of GANs and of variations like WGANs is separable over observations, which, as we will illustrate, can result in serious problems, even when minibatches are used. The underlying issue is that the stochastic gradients are essentially (sums of) functions of single observations (training points). Despite connections to two-sample tests based on Jensen-Shannon divergence, ultimately the updates based on gradients from different single observations are completely independent of each other. We show how this lack of sharing information between observations may explain mode collapses in GANs.Motivated by these insights, we take a different perspective on adversarial training and propose a framework that brings the discriminator closer to a truly distributional adversary, i.e., one that 1 (a set of observations) in its entirety, retaining and sharing global information between gradients. The key insight is that a carefully placed nonlinearity in the form of specific population comparisons can enable information-sharing and thereby stabilize training. We develop and test two such models, and also connect them to other popular ideas in deep learning and statistics.Contributions. The main contributions of this work are as follows:• We introduce a new distributional framework for adversarial training of neural networks that operates on a genuine sample, i.e., a collection of points, rather than an observation. This choice is orthogonal to modifications of the types of loss (e.g. logistic vs. Wasserstein) in the literature.• We show how off-the-shelf discriminator networks can be made distribution-aware via simple modifications to their architecture and how existing models can seamlessly fit into this framework.• Empirically, our distributional adversarial framework leads to more stable training and significantly better mode coverage than common single-observation methods. A direct application of our framework to domain adaptation results in strong improvements over state-of-the-art.
We show that the mode collapse problem in GANs may be explained by a lack of information sharing between observations in a training batch, and propose a distribution-based framework for globally sharing information between gradients that leads to more stable and effective adversarial training. Proposes to replace single-sample discriminators in adversarial training with discriminators that explicitly operate on distributions of examples. Theory on two-sample tests and MMD and how can be beneficially incorporated into GAN framework.