In this paper, we propose a capsule-based neural network model to solve the semantic segmentation problem. By taking advantage of the extractable part-whole dependencies available in capsule layers, we derive the probabilities of the class labels for individual capsules through a recursive, layer-by-layer procedure. We model this procedure as a traceback pipeline and take it as a central piece to build an end-to-end segmentation network. Under the proposed framework, image-level class labels and object boundaries are jointly sought in an explicit manner, which poses a significant advantage over the state-of-the-art fully convolutional network (FCN) solutions. Experiments conducted on modified MNIST and neuroimages demonstrate that our model considerably enhance the segmentation performance compared to the leading FCN variant.
 An effective segmentation solution should have a well-equipped mechanism to capture both semantic (i.e., what) and location (i.e., where) information. The fully convolutional network (FCN) BID19 and its variants BID24 BID21 BID1 constitute a popular class of solutions for this task, producing state-of-the-art results in a variety of applications. FCN and its variants (FCNs) are commonly constructed with an encoder-decoder architecture. In the encoding path, input images are processed through a number of "convolution + pooling" layers to generate high-level latent features, which are then progressively upsampled in the decoder to reconstruct the target pixel labels. The feature maps produced in higher (coarser) layers and those in lower (finer) layers contain complementary information: the former is richer in semantics, while the latter carries more spatial details that define class boundaries.Originated from and constructed upon convolutional neural networks (CNNs) BID13 BID26 , FCNs' encoders inherit some common drawbacks of CNNs, one of which is the lack of an internal mechanism in achieving viewpoint-invariant recognition. Traditional CNNs, as well as FCNs, rely on convolution operations to capture various visual patterns, and utilize poolings to enable multi-scale processing of the input images. Rotation invariance, however, is not readily available in both models. As a result, more data samples or additional network setups BID6 BID7 would be required for objects from different viewpoints to be correctly recognized. The absence of explicit part-whole relationships among objects imposes another limitation for FCNs -without such a mechanism, the rich semantic information residing in the higher layers and the precise boundary information in the lower layers can only be integrated in an implicit manner .Capsule nets BID25 BID10 , operating on a different paradigm, can provide a remedy. Capsule nets are built on capsules, each of which is a group of neurons representing one instance of a visual entity, i.e., an object or one of its parts BID9 . Capsules output both activation probabilities of their presence and the instantiation parameters that describe their properties, such as pose, deformation and texture, relative to a viewer BID9 . During inference propagation, the principle of coincidence filtering is employed to activate higher-level capsules and set up part-whole relationships among capsule entities. Such part-whole hierarchy equips capsule nets with a solid foundation for viewpoint-invariant recognition, which can be implemented through dynamic routing BID25 or EM routing BID10 . The same hierarchy , if properly embedded into a segmentation network, would provide a well-grounded platform to specify contextual constraints and enforce label consistency.With this thought, we develop a capsule-based semantic segmentation solution in this paper. Our approach treats capsule nets as probabilistic graphical models capable of inferring probabilistic dependences among visual entities, through which part-whole relationships can be explicitly constructed. As a concrete implementation , we propose a new operation sequence, which we call traceback pipeline, to capture such part-whole information through a recursive procedure to derive the class memberships for individual pixels. We term our model Tr-CapsNet .The contributions of our Tr-CapsNet can be summarized as:1. In Tr-CapsNet, the class labels for individual spatial coordinates within each capsule layer are analytically derived. The traceback pipeline in our model , taking advantage of the graphical properties of capsule nets, is mathematically rigorous. To the best of our knowledge, this is the first work to explore a capsule traceback approach for image segmentation. In addition, probability maps at each capsule layer are readily available, which makes it convenient to conduct feature visualization and layer interpretation.2. In parallel with segmentation, Tr-CapsNet carries out explicit class recognition at the same time. Such explicitness poses a powerful practical advantage over FCNs.3. The traceback pipeline is designed under a general context, making it applicable to many other potential tasks, including object localization and detection, action localization and network interpretation.
A capsule-based semantic segmentation, in which the probabilities of the class labels are traced back through capsule pipeline.  The authors present a trace-back mechanism to associate lowest level of Capsules with their respective classes Proposes a traceback layer for capsule networks to do semantic segmentation and makes explicit use of part-whole relationship in the capsule layers Proposes a trace-back method based on the CapsNet concept of Sabour to perform a semantic segmentation in parallel to classification.