We propose a new Integral Probability Metric (IPM) between distributions: the Sobolev IPM. The Sobolev IPM compares the mean discrepancy of two distributions for functions (critic) restricted to a Sobolev ball defined with respect to a dominant measure mu. We show that the Sobolev IPM compares two distributions in high dimensions based on weighted conditional Cumulative Distribution Functions (CDF) of each coordinate on a leave one out basis. The Dominant measure mu plays a crucial role as it defines the support on which conditional CDFs are compared. Sobolev IPM can be seen as an extension of the one dimensional Von-Mises Cramer statistics to high dimensional distributions. We show how Sobolev IPM can be used to train Generative Adversarial Networks (GANs). We then exploit the intrinsic conditioning implied by Sobolev IPM in text generation. Finally we show that a variant of Sobolev GAN achieves competitive results in semi-supervised learning on CIFAR-10, thanks to the smoothness enforced on the critic by Sobolev GAN which relates to Laplacian regularization. In order to learn Generative Adversarial Networks BID14 , it is now well established that the generator should mimic the distribution of real data, in the sense of a certain discrepancy measure. Discrepancies between distributions that measure the goodness of the fit of the neural generator to the real data distribution has been the subject of many recent studies BID1 BID36 BID20 BID30 BID17 , most of which focus on training stability.In terms of data modalities, most success was booked in plausible natural image generation after the introduction of Deep Convolutional Generative Adversarial Networks (DCGAN) BID39 . This success is not only due to advances in training generative adversarial networks in terms of loss functions and stable algorithms, but also to the representation power of convolutional neural networks in modeling images and in finding sufficient statistics that capture the continuous density function of natural images. When moving to neural generators of discrete sequences generative adversarial networks theory and practice are still not very well understood. Maximum likelihood pre-training or augmentation, in conjunction with the use of reinforcement learning techniques were proposed in many recent works for training GAN for discrete sequences generation BID50 BID40 . Other methods included using the Gumbel Softmax trick BID23 ) and the use of auto-encoders to generate adversarially discrete sequences from a continuous space BID51 . End to end training of GANs for discrete sequence generation is still an open problem BID38 . Empirical successes of end to end training have been reported within the framework of WGAN-GP BID17 , using a proxy for the Wasserstein distance via a pointwise gradient penalty on the critic. Inspired by this success, we propose in this paper a new Integral Probability Metric (IPM) between distributions that we coin Sobolev IPM. Intuitively an IPM BID35 between two probability distributions looks for a witness function f , called critic, that maximally discriminates between samples coming from the two distributions: DISPLAYFORM0 Traditionally, the function f is defined over a function class F that is independent to the distributions at hand BID48 . The Wasserstein-1 distance corresponds for instance to an IPM where the witness functions are defined over the space of Lipschitz functions; The MMD distance corresponds to witness functions defined over a ball in a Reproducing Kernel Hilbert Space (RKHS).We will revisit in this paper Fisher IPM defined in , which extends the IPM definition to function classes defined with norms that depend on the distributions. Fisher IPM can be seen as restricting the critic to a Lebsegue ball defined with respect to a dominant measure µ. The Lebsegue norm is defined as follows: DISPLAYFORM1 where µ is a dominant measure of P and Q.In this paper we extend the IPM framework to critics bounded in the Sobolev norm: DISPLAYFORM2 In contrast to Fisher IPM, which compares joint probability density functions of all coordinates between two distributions, we will show that Sobolev IPM compares weighted (coordinate-wise) conditional Cumulative Distribution Functions for all coordinates on a leave on out basis. Matching conditional dependencies between coordinates is crucial for sequence modeling.Our analysis and empirical verification show that the modeling of the conditional dependencies can be built in to the metric used to learn GANs as in Sobolev IPM. For instance, this gives an advantage to Sobolev IPM in comparing sequences over Fisher IPM. Nevertheless, in sequence modeling when we parametrize the critic and the generator with a neural network, we find an interesting tradeoff between the metric used and the architectures used to parametrize the critic and the generator as well as the conditioning used in the generator. The burden of modeling the conditional long term dependencies can be handled by the IPM loss function as in Sobolev IPM (more accurately the choice of the data dependent function class of the critic) or by a simpler metric such as Fisher IPM together with a powerful architecture for the critic that models conditional long term dependencies such as LSTM or GRUs in conjunction with a curriculum conditioning of the generator as done in BID38 . Highlighting those interesting tradeoffs between metrics, data dependent functions classes for the critic (Fisher or Sobolev) and architectures is crucial to advance sequence modeling and more broadly structured data generation using GANs.3. The intrinsic conditioning and the CDF matching make Sobolev IPM suitable for discrete sequence matching and explain the success of the gradient pernalty in WGAN-GP and Sobolev GAN in discrete sequence generation. 4. We give in Section 5 an ALM (Augmented Lagrangian Multiplier) algorithm for training Sobolev GAN. Similar to Fisher GAN, this algorithm is stable and does not compromise the capacity of the critic. 5. We show in Appendix A that the critic of Sobolev IPM satisfies an elliptic Partial Differential Equation (PDE). We relate this diffusion to the Fokker-Planck equation and show the behavior of the gradient of the optimal Sobolev critic as a transportation plan between distributions. 6. We empirically study Sobolev GAN in character level text generation (Section 6.1). We validate that the conditioning implied by Sobolev GAN is crucial for the success and stability of GAN in text generation. As a take home message from this study, we see that text generation succeeds either by implicit conditioning i.e using Sobolev GAN (or WGAN-GP) together with convolutional critics and generators, or by explicit conditioning i.e using Fisher IPM together with recurrent critic and generator and curriculum learning. 7. We finally show in Section 6.2 that a variant of Sobolev GAN achieves competitive semisupervised learning results on CIFAR-10, thanks to the smoothness implied by the Sobolev regularizer. We introduced the Sobolev IPM and showed that it amounts to a comparison between weighted (coordinate-wise) CDFs. We presented an ALM algorithm for training Sobolev GAN. The intrinsic conditioning implied by the Sobolev IPM explains the success of gradient regularization in Sobolev GAN and WGAN-GP on discrete sequence data, and particularly in text generation. We highlighted the important tradeoffs between the implicit conditioning introduced by the gradient regularizer in Sobolev IPM, and the explicit conditioning of Fisher IPM via recurrent critics and generators in conjunction with the curriculum conditioning. Both approaches succeed in text generation. We showed that Sobolev GAN achieves competitive semi-supervised learning results without the need of any normalization, thanks to the smoothness induced by the gradient regularizer. We think the Sobolev IPM point of view will open the door for designing new regularizers that induce different types of conditioning for general structured/discrete/graph data beyond sequences.
We define a new Integral Probability Metric (Sobolev IPM) and show how it can be used for training GANs for text generation and semi-supervised learning. Suggests a novel regularization scheme for GANs based on a Sobolev norm, measuring deviations between L2 norms of derivatives. The authors provide another type of GAN using the typical setup of a GAN but with a different function class, and produce a recipe for training GANs with that sort of function class. The paper proposes a different gradient penalty for GAN critics that forces the expected squared norm of the gradient to be equal to 1