Variational Probabilistic Fusion Network for RGB-T Semantic Segmentation
Baihong Lin Zengrong Lin Yulan Guo Yulan Zhang Jianxiao Zou Shicai Fan
RGB-T semantic segmentation has been widely adopted to handle hard scenes
with poor lighting conditions by fusing different modality features of RGB and
thermal images. Existing methods try to find an optimal fusion feature for
segmentation, resulting in sensitivity to modality noise, class-imbalance, and
modality bias. To overcome the problems, this paper proposes a novel
Variational Probabilistic Fusion Network (VPFNet), which regards fusion
features as random variables and obtains robust segmentation by averaging
segmentation results under multiple samples of fusion features. The random
samples generation of fusion features in VPFNet is realized by a novel
Variational Feature Fusion Module (VFFM) designed based on variation attention.
To further avoid class-imbalance and modality bias, we employ the weighted
cross-entropy loss and introduce prior information of illumination and category
to control the proposed VFFM. Experimental results on MFNet and PST900 datasets
demonstrate that the proposed VPFNet can achieve state-of-the-art segmentation
performance.