Multi-Modal Hybrid Learning and Sequential Training for RGB-T Saliency
  Detection
Guangyu Ren Jitesh Joshi Youngjun Cho
RGB-T saliency detection has emerged as an important computer vision task,
identifying conspicuous objects in challenging scenes such as dark
environments. However, existing methods neglect the characteristics of
cross-modal features and rely solely on network structures to fuse RGB and
thermal features. To address this, we first propose a Multi-Modal Hybrid loss
(MMHL) that comprises supervised and self-supervised loss functions. The
supervised loss component of MMHL distinctly utilizes semantic features from
different modalities, while the self-supervised loss component reduces the
distance between RGB and thermal features. We further consider both spatial and
channel information during feature fusion and propose the Hybrid Fusion Module
to effectively fuse RGB and thermal features. Lastly, instead of jointly
training the network with cross-modal features, we implement a sequential
training strategy which performs training only on RGB images in the first stage
and then learns cross-modal features in the second stage. This training
strategy improves saliency detection performance without computational
overhead. Results from performance evaluation and ablation studies demonstrate
the superior performance achieved by the proposed method compared with the
existing state-of-the-art methods.