Channel and Spatial Relation-Propagation Network for RGB-Thermal
  Semantic Segmentation
Zikun Zhou Shukun Wu Guoqing Zhu Hongpeng Wang Zhenyu He
RGB-Thermal (RGB-T) semantic segmentation has shown great potential in
handling low-light conditions where RGB-based segmentation is hindered by poor
RGB imaging quality. The key to RGB-T semantic segmentation is to effectively
leverage the complementarity nature of RGB and thermal images. Most existing
algorithms fuse RGB and thermal information in feature space via concatenation,
element-wise summation, or attention operations in either unidirectional
enhancement or bidirectional aggregation manners. However, they usually
overlook the modality gap between RGB and thermal images during feature fusion,
resulting in modality-specific information from one modality contaminating the
other. In this paper, we propose a Channel and Spatial Relation-Propagation
Network (CSRPNet) for RGB-T semantic segmentation, which propagates only
modality-shared information across different modalities and alleviates the
modality-specific information contamination issue. Our CSRPNet first performs
relation-propagation in channel and spatial dimensions to capture the
modality-shared features from the RGB and thermal features. CSRPNet then
aggregates the modality-shared features captured from one modality with the
input feature from the other modality to enhance the input feature without the
contamination issue. While being fused together, the enhanced RGB and thermal
features will be also fed into the subsequent RGB or thermal feature extraction
layers for interactive feature fusion, respectively. We also introduce a
dual-path cascaded feature refinement module that aggregates multi-layer
features to produce two refined features for semantic and boundary prediction.
Extensive experimental results demonstrate that CSRPNet performs favorably
against state-of-the-art algorithms.