We propose in this paper a new approach to train the Generative Adversarial Nets (GANs) with a mixture of generators to overcome the mode collapsing problem. The main intuition is to employ multiple generators, instead of using a single one as in the original GAN. The idea is simple, yet proven to be extremely effective at covering diverse data modes, easily overcoming the mode collapsing problem and delivering state-of-the-art results. A minimax formulation was able to establish among a classifier, a discriminator, and a set of generators in a similar spirit with GAN. Generators create samples that are intended to come from the same distribution as the training data, whilst the discriminator determines whether samples are true data or generated by generators, and the classifier specifies which generator a sample comes from. The distinguishing feature is that internal samples are created from multiple generators, and then one of them will be randomly selected as final output similar to the mechanism of a probabilistic mixture model. We term our method Mixture Generative Adversarial Nets (MGAN). We develop theoretical analysis to prove that, at the equilibrium, the Jensen-Shannon divergence (JSD) between the mixture of generators’ distributions and the empirical data distribution is minimal, whilst the JSD among generators’ distributions is maximal, hence effectively avoiding the mode collapsing problem. By utilizing parameter sharing, our proposed model adds minimal computational cost to the standard GAN, and thus can also efficiently scale to large-scale datasets. We conduct extensive experiments on synthetic 2D data and natural image databases (CIFAR-10, STL-10 and ImageNet) to demonstrate the superior performance of our MGAN in achieving state-of-the-art Inception scores over latest baselines, generating diverse and appealing recognizable objects at different resolutions, and specializing in capturing different types of objects by the generators. Generative Adversarial Nets (GANs) BID10 are a recent novel class of deep generative models that are successfully applied to a large variety of applications such as image, video generation, image inpainting, semantic segmentation, image-to-image translation, and text-to-image synthesis, to name a few BID9 . From the game theory metaphor, the model consists of a discriminator and a generator playing a two-player minimax game, wherein the generator aims to generate samples that resemble those in the training data whilst the discriminator tries to distinguish between the two as narrated in BID10 . Training GAN, however, is challenging as it can be easily trapped into the mode collapsing problem where the generator only concentrates on producing samples lying on a few modes instead of the whole data space BID9 .Many GAN variants have been recently proposed to address this problem. They can be grouped into two main categories: training either a single generator or many generators. Methods in the former include modifying the discriminator's objective (Salimans et al., 2016; BID22 , modifying the generator's objective (Warde-Farley & Bengio, 2016) , or employing additional discriminators to yield more useful gradient signals for the generators (Nguyen et al., 2017; BID7 . The common theme in these variants is that generators are shown, at equilibrium, to be able to recover the data distribution, but convergence remains elusive in practice. Most experiments are conducted on toy datasets or on narrow-domain datasets such as LSUN (Yu et al., 2015) or CelebA BID20 . To our knowledge , only Warde-Farley & Bengio (2016) and Nguyen et al. (2017) perform quantitative evaluation of models trained on much more diverse datasets such as STL-10 (Coates et al., 2011) and ImageNet (Russakovsky et al., 2015) .Given current limitations in the training of single-generator GANs, some very recent attempts have been made following the multi-generator approach. Tolstikhin et al. (2017) apply boosting techniques to train a mixture of generators by sequentially training and adding new generators to the mixture. However, sequentially training many generators is computational expensive. Moreover, this approach is built on the implicit assumption that a single-generator GAN can generate very good images of some modes, so reweighing the training data and incrementally training new generators will result in a mixture that covers the whole data space. This assumption is not true in practice since current single-generator GANs trained on diverse datasets such as ImageNet tend to generate images of unrecognizable objects. BID2 train a mixture of generators and discriminators, and optimize the minimax game with the reward function being the weighted average reward function between any pair of generator and discriminator. This model is computationally expensive and lacks a mechanism to enforce the divergence among generators. BID8 train many generators by using a multi-class discriminator that, in addition to detecting whether a data sample is fake, predicts which generator produces the sample. The objective function in this model punishes generators for generating samples that are detected as fake but does not directly encourage generators to specialize in generating different types of data.We propose in this paper a novel approach to train a mixture of generators. Unlike aforementioned multi-generator GANs, our proposed model simultaneously trains a set of generators with the objective that the mixture of their induced distributions would approximate the data distribution, whilst encouraging them to specialize in different data modes. The result is a novel adversarial architecture formulated as a minimax game among three parties: a classifier, a discriminator, and a set of generators. Generators create samples that are intended to come from the same distribution as the training data, whilst the discriminator determines whether samples are true data or generated by generators, and the classifier specifies which generator a sample comes from. We term our proposed model as Mixture Generative Adversarial Nets (MGAN). We provide analysis that our model is optimized towards minimizing the Jensen-Shannon Divergence (JSD) between the mixture of distributions induced by the generators and the data distribution while maximizing the JSD among generators.Empirically, our proposed model can be trained efficiently by utilizing parameter sharing among generators, and between the classifier and the discriminator. In addition, simultaneously training many generators while enforcing JSD among generators helps each of them focus on some modes of the data space and learn better. Trained on CIFAR-10, each generator learned to specialize in generating samples from a different class such as horse, car, ship, dog, bird or airplane. Overall, the models trained on the CIFAR-10, STL-10 and ImageNet datasets successfully generated diverse, recognizable objects and achieved state-of-the-art Inception scores (Salimans et al., 2016) . The model trained on the CIFAR-10 even outperformed GANs trained in a semi-supervised fashion (Salimans et al., 2016; Odena et al., 2016) .In short, our main contributions are: (i) a novel adversarial model to efficiently train a mixture of generators while enforcing the JSD among the generators; (ii) a theoretical analysis that our objective function is optimized towards minimizing the JSD between the mixture of all generators' distributions and the real data distribution, while maximizing the JSD among generators; and (iii) a comprehensive evaluation on the performance of our method on both synthetic and real-world large-scale datasets of diverse natural scenes. We have presented a novel adversarial model to address the mode collapse in GANs. Our idea is to approximate data distribution using a mixture of multiple distributions wherein each distribution captures a subset of data modes separately from those of others. To achieve this goal, we propose a minimax game of one discriminator, one classifier and many generators to formulate an optimization problem that minimizes the JSD between P data and P model , i.e., a mixture of distributions induced by the generators, whilst maximizes JSD among such generator distributions. This helps our model generate diverse images to better cover data modes, thus effectively avoids mode collapse. We term our proposed model Mixture Generative Adversarial Network (MGAN).The MGAN can be efficiently trained by sharing parameters between its discriminator and classifier, and among its generators, thus our model is scalable to be evaluated on real-world largescale datasets. Comprehensive experiments on synthetic 2D data, CIFAR-10, STL-10 and ImageNet databases demonstrate the following capabilities of our model: (i) achieving state-of-the-art Inception scores; (ii) generating diverse and appealing recognizable objects at different resolutions; and (iv) specializing in capturing different types of objects by the generators. In our proposed method, generators G 1 , G 2 , ... G K are deep convolutional neural networks parameterized by θ G . These networks share parameters in all layers except for the input layers. The input layer for generator G k is parameterized by the mapping f θ G ,k (z) that maps the sampled noise z to the first hidden layer activation h. The shared layers are parameterized by the mapping g θ G (h) that maps the first hidden layer to the generated data. The pseudo-code of sampling from the mixture is described in Alg. 1. Classifier C and classifier D are also deep convolutional neural networks that are both parameterized by θ CD . They share parameters in all layers except for the last layer . The pseudo-code of alternatively learning θ G and θ CD using stochastic gradient descend is described in Alg. 2.Algorithm 1 Sampling from MGAN's mixture of generators. 1: Sample noise z from the prior P z . 2: Sample a generator index u from Mult (π 1 , π 2 , ..., π K ) with predefined mixing probability π = (π 1 , π 2 , ..., π K ). DISPLAYFORM0 Return generated data x and the index u.Algorithm 2 Alternative training of MGAN using stochastic gradient descent.1: for number of training iterations do 2:Sample a minibatch of M data points DISPLAYFORM1 Sample a minibatch of N generated data points ) and N indices (u 1 , u 2 , ..., u N ) from the current mixture. DISPLAYFORM2 4: DISPLAYFORM3 5: DISPLAYFORM4 Update classifier C and discriminator D by descending along their gradient: DISPLAYFORM5 Sample a minibatch of N generated data points ) and N indices (u 1 , u 2 , ..., u N ) from the current mixture. DISPLAYFORM6 8: DISPLAYFORM7 Update the mixture of generators G by ascending along its gradient: ∇ θ G L G . 10: end for B APPENDIX: PROOFS FOR SECTION 3.1 Proposition 1 FIG0 . For fixed generators G 1 , G 2 , ..., G K and mixture weights π 1 , π 2 , ..., π K , the optimal classifier C * = C * 1:K and discriminator D * for J (G, C, D) are: DISPLAYFORM8 Proof. The optimal D * was proved in Prop. 1 in BID9 . This section shows a similar proof for the optimal C * . Assuming that C * can be optimized in the functional space, we can calculate the functional derivatives of J (G, C, D)with respect to each C k (x) for k ∈ {2, ..., K} and set them equal to zero: DISPLAYFORM9 Setting DISPLAYFORM10 to 0 for k ∈ {2, ..., K}, we get: DISPLAYFORM11 DISPLAYFORM12 results from Eq. (6) due to the fact that DISPLAYFORM13 Reformulation of L (G 1:K ). Replacing the optimal C * and D * into Eq. (2), we can reformulate the objective function for the generator as follows: DISPLAYFORM14 The sum of the first two terms in Eq. FORMULA29 was shown in BID10 to be 2 · JSD (P data P model ) − log 4. The last term β{ * } of Eq. FORMULA29 is related to the JSD for the K distributions: DISPLAYFORM15 where H (P ) is the Shannon entropy for distribution P . Thus, L (G 1:K ) can be rewritten as: DISPLAYFORM16 Theorem 3 (Thm. 3 restated). If the data distribution has the form: DISPLAYFORM17 where the mixture components q k (x)(s) are well-separated, the minimax problem in Eq. (2) or the optimization problem in Eq. (3) has the following solution: DISPLAYFORM18 , and the corresponding objective value of the optimization problem in Eq. (3) DISPLAYFORM19 Proof. We first recap the optimization problem for finding the optimal G * : DISPLAYFORM20 The JSD in Eq. FORMULA30 is given by: DISPLAYFORM21 The i-th expectation in Eq. (9) can be derived as follows: DISPLAYFORM22 and the equality occurs if DISPLAYFORM23 = 1 almost everywhere or equivalently for almost every x except for those in a zero measure set, we have: DISPLAYFORM24 Therefore, we obtain the following inequality: DISPLAYFORM25 and the equality occurs if for almost every x except for those in a zero measure set, we have: DISPLAYFORM26 and we peak the minimum if p G k = q k , ∀k since this solution satisfies both DISPLAYFORM27 and the conditions depicted in Eq. (10). That concludes our proof.C APPENDIX: ADDITIONAL EXPERIMENTS DISPLAYFORM28 The true data is sampled from a 2D mixture of 8 Gaussian distributions with a covariance matrix 0.02I and means arranged in a circle of zero centroid and radius 2.0. We use a simple architecture of 8 generators with two fully connected hidden layers and a classifier and a discriminator with one shared hidden layer. All hidden layers contain the same number of 128 ReLU units. The input layer of generators contains 256 noise units sampled from isotropic multivariate Gaussian distribution N (0, I). We do not use batch normalization in any layer. We refer to Tab. 3 for more specifications of the network and hyperparameters. "Shared" is short for parameter sharing among generators or between the classifier and the discriminator . Feature maps of 8/1 in the last layer for C and D means that two separate fully connected layers are applied to the penultimate layer, one for C that outputs 8 logits and another for D that outputs 1 logit. Optimizer Adam(β 1 = 0.5, β 2 = 0.999) Weight, bias initialization N (µ = 0, σ = 0.02I), 0The effect of the number of generators on generated samples. Fig. 6 shows samples produced by MGANs with different numbers of generators trained on synthetic data for 25,000 epochs . The model with 1 generator behaves similarly to the standard GAN as expected. The models with 2, 3 and 4 generators all successfully cover 8 modes, but the ones with more generators draw fewer points scattered between adjacent modes. Finally, the model with 10 generators also covers 8 modes wherein 2 generators share one mode and one generator hovering around another mode. Figure 6: Samples generated by MGAN models trained on synthetic data with 2, 3, 4 and 10 generators. Data samples from the 8 Gaussians are in red, and generated data by each generator are in a different color.The effect of β on generated samples. To examine the behavior of the diversity coefficient β, Fig. 7 compares samples produced by our MGAN with 4 generators after 25,000 epochs of training with different values of β. Without the JSD force (β = 0), generated samples cluster around one mode. When β = 0.25, generated data clusters near 4 different modes. When β = 0.75 or 1.0, the JSD force is too strong and causes the generators to collapse, generating 4 increasingly tight clusters. When β = 0.5, generators successfully cover all of the 8 modes. DISPLAYFORM29 Figure 7: Samples generated by MGAN models trained on synthetic data with different values of diversity coefficient β. Generated data are in blue and data samples from the 8 Gaussians are in red.
We propose a new approach to train GANs with a mixture of generators to overcome the mode collapsing problem. Address the problem of mode collapse in GANs using a constrained mixture distribution for the generator and an auxiliary classifier which predicts the source mixture component. The paper proposes a mixture of generators to train GANs without extra computational cost The authors present that using MGAN, which aims to overcome model collapsing problem by mixture generators, achieves state-of-art results