Most existing GANs architectures that generate images use transposed convolution or resize-convolution as their upsampling algorithm from lower to higher resolution feature maps in the generator. We argue that this kind of fixed operation is problematic for GANs to model objects that have very different visual appearances. We propose a novel adaptive convolution method that learns the upsampling algorithm based on the local context at each location to address this problem. We modify a baseline GANs architecture by replacing normal convolutions with adaptive convolutions in the generator. Experiments on CIFAR-10 dataset show that our modified models improve the baseline model by a large margin. Furthermore, our models achieve state-of-the-art performance on CIFAR-10 and STL-10 datasets in the unsupervised setting. Generative Adversarial Networks BID6 (GANs) are an unsupervised learning method that is able to generate realistic looking images from noise. GANs employs a minimax game where a generator network learns to generate synthesized data from random noise and in conjunction, a discriminator network learns to distinguish between real and generated data. Theoretically, the training processes of the two networks intertwine and iterate until both networks reach a Nash equilibrium where real and synthesized data are indistinguishable.However, in practice, GANs are notoriously hard to train. For the learning of the generator to happen effectively, hyper-parameters, as well as the architectures of the generator and discriminator, must be chosen carefully. Nevertheless, on datasets with visually similar images, such as bedroom scenes and faces , GANs have produced good results BID15 . This success, however, does not translate to datasets that have diverse visual categories. GANs models trained on ImageNet BID16 were only able to learn basic image statistics and some shapes, but they did not learn any object BID17 . Recent works address this problem by incorporating additional high-level information to guide the generator, such as training the discriminator in a semi-supervised manner BID17 , adding a second training objective to direct the generator toward similar samples from the training set BID19 or using artificial class labels by clustering in the representation space BID7 .We take a different approach to tackle this problem. We hypothesize that the rigidity of the normal convolution operator is partially responsible for the difficulty of GANs to learn on diverse visual datasets. Most GANs generators upsample low resolution feature maps toward higher resolution using fixed convolutions (note that a transposed convolution is equivalent to a convolution) or resize-convolution BID13 . Such operations are unintuitive, because, pixel locations have different local contexts and belong to diverse object categories. Consequently , different pixel locations should have different upsampling schemes. This shortcoming of normal convolution is especially problematic in the early upsampling layers where higher level information usually associates with the object shapes and the context of images.We propose the use of a novel adaptive convolution BID12 architecture, called Adaptive Convolution Block, that replaces normal convolutions to address the aforementioned shortcoming of GANs generators. Instead of learning a fixed convolution for the upsampling of all pixels from the lower to the higher resolution feature map, an AdaConvBlock learns to generate the convolution weights and biases of the upsampling operation adaptively based on the local feature map at each pixel location. AdaConvBlock helps the generator to learn to generate upsampling algorithms that take into account the local context. We believe that this kind of adaptive convolution is more intuitive and more akin to the process when a human draws something: the same thought process is used in the whole drawing but the style of each stroke should vary and depend on the local context around each pixel position.We conduct experiments to compare our adaptive convolution method to normal convolution in the unsupervised setting. We progressively replace all convolutions of a GANs generator with AdaConvBlocks from the lowest resolution to the highest resolution. Experiments on CIFAR-10 dataset show that the modified adaptive convolution models have superior qualitative and quantitative performance over the baseline architecture and just replacing convolution of the upsampling from the lowest resolution feature map with adaptive convolution can have significant impacts on the baseline model. Furthermore, our best models achieve state-of-the-art unsupervised performance on both CIFAR-10 and STL-10 datasets. Our code and models will be released. We have demonstrated that using adaptive convolutions to replace normal convolutions in a GANs generator can improve the performance of a weak baseline model significantly on visually diverse datasets. Our AdaGAN models were able to beat other state-of-the-art methods without using any augmented training objectives. The samples generated by our models show that they seem to be able to learn the global context pretty well and be able to learn the rough shapes of the objects in most cases and the sample quality is quite reasonable on CIFAR-10 dataset. Furthermore, there are not much visible convolution artifacts in the generated images. The success of our models suggests that non-trivial performance improvement can be gained from modifying architectures for GANs.The approach we take is different from other methods that try to inject high level information into the discriminator. These existing methods and AdaGAN can complement each other. More experiments need to be done, but we believe that our architectures can benefit from the augmented training objectives from existing methods.Our method is not without a downside. Even though we used depthwise separable convolution to reduce the cost, the amount of memory and computation is still extremely high. More tricks could be applied to alleviate this issue. For example, in a similar manner to BID12 work, both the local convolutions and the convolution to regress the adaptive weights for the local convolutions in our AdaConvBlock can be approximated by separate 1-D convolutions. This can reduce the cost by more than 50%. Another idea is to exploit locality. We expect the adaptive convolution weights and biases of a pixel location to be quite similar to its neighbors and can be interpolated in a certain way. We will address this issue in our future work.A SAMPLES GENERATED BY OUR MODELS ON CIFAR-10 AND STL-10
We replace normal convolutions with adaptive convolutions to improve GANs generator. Proposes to replace convolutions in the generator with an Adaptive Convolution Block that learns to generate convolution weigths adn biases of upsampling operations adaptively per pixel location Uses Adaptive Convolution in the context of GANs with a block called AdaConvBlock that replaces regular Convolution, this gives more local context per kernel weight so that it can generate locally flexible objects.