For the challenging semantic image segmentation task the best performing models
 have traditionally combined the structured modelling capabilities of Conditional
 Random Fields (CRFs) with the feature extraction power of CNNs. In more recent
 works however, CRF post-processing has fallen out of favour. We argue that this
 is mainly due to the slow training and inference speeds of CRFs, as well as the
 difficulty of learning the internal CRF parameters. To overcome both issues we
 propose to add the assumption of conditional independence to the framework of
 fully-connected CRFs. This allows us to reformulate the inference in terms of
 convolutions, which can be implemented highly efficiently on GPUs.Doing so
 speeds up inference and training by two orders of magnitude. All parameters of
 the convolutional CRFs can easily be optimized using backpropagation. Towards
 the goal of facilitating further CRF research we have made our implementations
 publicly available. Semantic image segmentation, which aims to produce a categorical label for each pixel in an image, is a very import task for visual perception. Convolutional Neural Networks have been proven to be very strong in tackling semantic segmentation tasks BID23 BID5 BID40 . While simple feed-forward CNNs are extremely powerful in extracting local features and performing good predictions utilizing a small field of view, they lack the capability to utilize context information and cannot model interactions between predictions directly. Thus it has been suggested that such deep neural networks may not be the perfect model for structured predictions tasks such as semantic segmentation BID40 BID20 BID41 . Several authors have successfully combined the effectiveness of CNNs to extract powerful features, with the modelling power of CRFs in order to address the discussed issues BID20 Chandra & Kokkinos, 2016; BID41 . Despite their indisputable success, structured models have fallen out of favour in more recent approaches BID38 BID4 BID40 .We believe that the main reasons for this development are that CRFs are notoriously slow and hard to optimize. Learning the features for the structured component of the CRF is an open research problem BID37 BID20 and many approaches rely on entirely hand-crafted Gaussian features BID41 BID31 BID5 . In addition , CRF inference is typically two orders of magnitude slower than CNN inference. This makes CRF based approaches too slow for many practical applications. The long training times of the current generation of CRFs also make more in-depth research and experiments with such structured models impractical.To solve both of these issues we propose to add the strong and valid assumption of conditional independence to the existing framework of fully-connected CRFs (FullCRFs) introduced by . This allows us to reformulate a large proportion of the inference as convolutions, which can be implemented highly efficiently on GPUs. We call our method convolutional CRFs (ConvCRFs). Backpropagation BID30 can be used to train all parameters of the ConvCRF. Inference in ConvCRFs can be performed in less then 10ms. This is a speed increase of two-orders of magnitude compared to FullCRFs. We believe that those fast train and inference speeds will greatly benefit future research and hope that our results help to revive CRFs as a popular method to solve structured tasks. In this work we proposed Convolutional CRFs, a novel CRF design. Adding the strong and valid assumption of conditional independence enables us to remove the permutohedral lattice approximation. This allows us to implement the message passing highly efficiently on GPUs as convolution operations. This increases training and inference speed by two orders of magnitude. In addition we observe a modest accuracy improvement when computing the message passing exactly. Our method also enables us to easily train the Gaussian features of the CRF using backpropagation.In future work we will investigate the potential of learning Gaussian features further. We are also going to examine more sophisticated CRF architectures, towards the goal of capturing context information even better. Lastly we are particularly interested in exploring the potential of ConvCRFs in other structured applications such as instance segmentation, landmark recognition and weakly supervised learning.
We propose Convolutional CRFs a fast, powerful and trainable alternative to Fully Connected CRFs. The authors replace the large filtering step in the permutohedral lattice with a spatially varying convolutional kernel and show that inference is more efficient and training is easier.  Proposes to perform message passing on a truncated Gaussian kernel CRF using a defined kernel and parallelized message passing on GPU.