Residual Spatial Fusion Network for RGB-Thermal Semantic Segmentation
Ping Li Junjie Chen Binbin Lin Xianghua Xu
Semantic segmentation plays an important role in widespread applications such
as autonomous driving and robotic sensing. Traditional methods mostly use RGB
images which are heavily affected by lighting conditions, \eg, darkness. Recent
studies show thermal images are robust to the night scenario as a compensating
modality for segmentation. However, existing works either simply fuse
RGB-Thermal (RGB-T) images or adopt the encoder with the same structure for
both the RGB stream and the thermal stream, which neglects the modality
difference in segmentation under varying lighting conditions. Therefore, this
work proposes a Residual Spatial Fusion Network (RSFNet) for RGB-T semantic
segmentation. Specifically, we employ an asymmetric encoder to learn the
compensating features of the RGB and the thermal images. To effectively fuse
the dual-modality features, we generate the pseudo-labels by saliency detection
to supervise the feature learning, and develop the Residual Spatial Fusion
(RSF) module with structural re-parameterization to learn more promising
features by spatially fusing the cross-modality features. RSF employs a
hierarchical feature fusion to aggregate multi-level features, and applies the
spatial weights with the residual connection to adaptively control the
multi-spectral feature fusion by the confidence gate. Extensive experiments
were carried out on two benchmarks, \ie, MFNet database and PST900 database.
The results have shown the state-of-the-art segmentation performance of our
method, which achieves a good balance between accuracy and speed.