Deconvolutional layers have been widely used in a variety of deep
 models for up-sampling, including encoder-decoder networks for
 semantic segmentation and deep generative models for unsupervised
 learning. One of the key limitations of deconvolutional operations
 is that they result in the so-called checkerboard problem. This is
 caused by the fact that no direct relationship exists among adjacent
 pixels on the output feature map. To address this problem, we
 propose the pixel deconvolutional layer (PixelDCL) to establish
 direct relationships among adjacent pixels on the up-sampled feature
 map. Our method is based on a fresh interpretation of the regular
 deconvolution operation. The resulting PixelDCL can be used to
 replace any deconvolutional layer in a plug-and-play manner without
 compromising the fully trainable capabilities of original models.
 The proposed PixelDCL may result in slight decrease in efficiency,
 but this can be overcome by an implementation trick. Experimental
 results on semantic segmentation demonstrate that PixelDCL can
 consider spatial features such as edges and shapes and yields more
 accurate segmentation outputs than deconvolutional layers. When used
 in image generation tasks, our PixelDCL can largely overcome the
 checkerboard problem suffered by regular deconvolution operations. Deep learning methods have shown great promise in a variety of artificial intelligence tasks such as image classification BID9 BID26 , semantic segmentation BID16 BID24 BID23 , and natural image generation BID3 BID8 . Some key network layers, such as convolutional layers BID11 , pooling layers, fully connected layers and deconvolutional layers, have been frequently used to create deep models for different tasks. Deconvolutional layers, also known as transposed convolutional layers BID28 , are initially proposed in BID29 . They have been primarily used in deep models that require up-sampling of feature maps, such as generative models BID19 BID15 BID22 and encoder-decoder architectures BID23 BID16 . Although deconvolutional layers are capable of producing larger feature maps from smaller ones, they suffer from the problem of checkerboard artifacts BID17 . This greatly limits deep model's capabilities in generating photo-realistic images and producing smooth outputs on semantic segmentation. To date, very little efforts have been devoted to improving the deconvolution operation.In this work, we propose a simple, efficient, yet effective method, known as the pixel deconvolutional layer (PixelDCL), to address the checkerboard problem suffered by deconvolution operations. Our method is motivated from a fresh interpretation of deconvolution operations, which clearly pinpoints the root of checkerboard artifacts. That is, the up-sampled feature map generated by deconvolution can be considered as the result of periodical shuffling of multiple intermediate feature maps computed from the input feature map by independent convolutions. As a result, adjacent pixels on the output feature map are not directly related, leading to the checkerboard artifacts. To overcome this problem, we propose the pixel deconvolutional operation to be used in PixelDCL. In this new layer, the intermediate feature maps are generated sequentially so that feature maps generated in a later stage are required to depend on previously generated ones. In this way, direct relationships among adjacent pixels on the output feature map have been established. Sequential generation of intermediate feature maps in PixelDCL may result in slight decrease in computational efficiency, but we show Figure 1 : Comparison of semantic segmentation results. The first and second rows are images and ground true labels, respectively. The third and fourth rows are the results of using regular deconvolution and our proposed pixel deconvolution PixelDCL, respectively. that this can be largely overcome by an implementation trick. Experimental results on semantic segmentation (samples in Figure 1) and image generation tasks demonstrate that the proposed PixelDCL can effectively overcome the checkerboard problem and improve predictive and generative performance.Our work is related to the pixel recurrent neural networks (PixelRNNs) and PixelCNNs BID21 , which are generative models that consider the relationship among units on the same feature map. They belong to a more general class of autoregressive methods for probability density estimation BID2 BID10 . By using masked convolutions in training, the training time of PixelRNNs and PixelCNNs is comparable to that of other generative models such as generative adversarial networks (GANs) BID3 BID20 and variational autoencoders (VAEs) BID8 BID6 . However, the prediction time of PixelRNNs or PixelCNNs is very slow since it has to generate images pixel by pixel. In contrast, our PixelDCL can be used to replace any deconvolutional layer in a plug-and-play manner, and the slight decrease in efficiency can be largely overcome by an implementation trick. In this work, we propose pixel deconvolutional layers that can solve the checkerboard problem in deconvolutional layers. The checkerboard problem is caused by the fact that there is no direct relationship among intermediate feature maps generated in deconvolutional layers. PixelDCL proposed here try to add direct dependencies among these generated intermediate feature maps. PixelDCL generates intermediate feature maps sequentially so that the intermediate feature maps generated in a later stage are required to depend on previously generated ones. The establishment of dependencies in PixelDCL can ensure adjacent pixels on output feature maps are directly related. Experimental results on semantic segmentation and image generation tasks show that PixelDCL is effective in overcoming the checkerboard artifacts. Results on semantic segmentation also show that PixelDCL is able to consider local spatial features such as edges and shapes, leading to better segmentation results. In the future, we plan to employ our PixelDCL in a broader class of models, such as the generative adversarial networks (GANs).
Solve checkerboard problem in Deconvolutional layer by building dependencies between pixels This work proposes pixel deconvolutional layers for convolutional neural networks as a way to alleviate the checkerboard effect. A novel technique to generalize deconvolution operations used in standard CNN architectures, which proposes doing sequential prediction of adjacent pixel features, resulting in more spatially smooth outputs for deconvolution layers.