Since their invention, generative adversarial networks (GANs) have become a popular approach for learning to model a distribution of real (unlabeled) data. Convergence problems during training are overcome by Wasserstein GANs which minimize the distance between the model and the empirical distribution in terms of a different metric, but thereby introduce a Lipschitz constraint into the optimization problem. A simple way to enforce the Lipschitz constraint on the class of functions, which can be modeled by the neural network, is weight clipping. Augmenting the loss by a regularization term that penalizes the deviation of the gradient norm of the critic (as a function of the network's input) from one, was proposed as an alternative that improves training. We present theoretical arguments why using a weaker regularization term enforcing the Lipschitz constraint is preferable. These arguments are supported by experimental results on several data sets. General adversarial networks (GANs) BID8 are a class of generative models that have recently gained a lot of attention. They are based on the idea of defining a game between two competing neural networks (NNs): a generator and a classifier (or discriminator). While the classifier aims at distinguishing generated from real data, the generator tries to generate samples which the classifier can not distinguish from the ones from the empirical distribution. Realizing the potential behind this new approach to generative models, more recent contributions focused on the stabilization of training, including ensemble methods BID18 , improved network structure BID14 BID15 and theoretical improvements BID13 BID15 that helped to successfully model complex distributions using GANs.It was proposed by to train generator and discriminator networks by minimizing the Wasserstein-1 distance, a distance with properties superior to the Jensen-Shannon distance (used in the original GAN) in terms of convergence. Accordingly, this version of GAN was called Wasserstein GAN (WGAN) . The change of metric introduces a new minimization problem, which requires the discriminator function to lie in the space of 1-Lipschitz functions. In the same paper, the Lipschitz constraint was guaranteed by performing weight clipping, i.e., by constraining the parameters of the discriminator NN to be smaller than a given value in magnitude. An improved training strategy was proposed by BID9 based on results from optimal transport theory (see BID19 . Here, instead of clipping weights, the loss gets augmented by a regularization term that penalizes any deviation of the norm of the gradient of the critic function (with respect to its input) from one.We review these results and present both theoretical considerations and empirical results, leading to the proposal of a less restrictive regularization term for WGANs.1 More precisely, our contributions are as follows:• We review the arguments that the regularization technique proposed by BID9 is based on and make the following two observations: (i) The regularization strategy requires training samples and generated samples to be drawn from a certain joint distribution. In practice, however, samples are drawn independently from their marginals.(ii ) The arguments further assume the discriminator to be differentiable. We explain why both can be harmful for training.• We propose a less restrictive regularization term and present empirical results strongly supporting our theoretical considerations. For stable training of Wasserstein GANs, we propose to use the following penalty term to enforce the Lipschitz constraint that appears in the objective function: DISPLAYFORM0 We presented theoretical and empirical evidence that this gradient penalty performs better than the previously considered approaches of clipping weights and of applying the stronger gradient penalty given by Ex ∼τ [(||∇f (x)|| 2 − 1) 2 ]. In addition to more stable learning behavior, the proposed regularization term leads to lower sensitivity to the value of the penalty weight λ (demonstrating smooth convergence and well-behaved critic scores throughout the whole training process for different values of λ).
A new regularization term can improve your training of wasserstein gans The paper proposes a regularization scheme for Wasserstein GAN based on relaxation of the constraints on the Lipschitz constant of 1. The article deals with regularization/penalization in the fitting of GANs, when based on a L_1 Wasserstein metric.