Dual-modal Prior Semantic Guided Infrared and Visible Image Fusion for
  Intelligent Transportation System
Jing Li Lu Bai Bin Yang Chang Li Lingfei Ma Lixin Cui Edwin R. Hancock
Infrared and visible image fusion (IVF) plays an important role in
intelligent transportation system (ITS). The early works predominantly focus on
boosting the visual appeal of the fused result, and only several recent
approaches have tried to combine the high-level vision task with IVF. However,
they prioritize the design of cascaded structure to seek unified suitable
features and fit different tasks. Thus, they tend to typically bias toward to
reconstructing raw pixels without considering the significance of semantic
features. Therefore, we propose a novel prior semantic guided image fusion
method based on the dual-modality strategy, improving the performance of IVF in
ITS. Specifically, to explore the independent significant semantic of each
modality, we first design two parallel semantic segmentation branches with a
refined feature adaptive-modulation (RFaM) mechanism. RFaM can perceive the
features that are semantically distinct enough in each semantic segmentation
branch. Then, two pilot experiments based on the two branches are conducted to
capture the significant prior semantic of two images, which then is applied to
guide the fusion task in the integration of semantic segmentation branches and
fusion branches. In addition, to aggregate both high-level semantics and
impressive visual effects, we further investigate the frequency response of the
prior semantics, and propose a multi-level representation-adaptive fusion
(MRaF) module to explicitly integrate the low-frequent prior semantic with the
high-frequent details. Extensive experiments on two public datasets demonstrate
the superiority of our method over the state-of-the-art image fusion
approaches, in terms of either the visual appeal or the high-level semantics.