From Text to Pixels: A Context-Aware Semantic Synergy Solution for
  Infrared and Visible Image Fusion
Xingyuan Li Yang Zou Jinyuan Liu Zhiying Jiang Long Ma Xin Fan Risheng Liu
With the rapid progression of deep learning technologies, multi-modality
image fusion has become increasingly prevalent in object detection tasks.
Despite its popularity, the inherent disparities in how different sources
depict scene content make fusion a challenging problem. Current fusion
methodologies identify shared characteristics between the two modalities and
integrate them within this shared domain using either iterative optimization or
deep learning architectures, which often neglect the intricate semantic
relationships between modalities, resulting in a superficial understanding of
inter-modal connections and, consequently, suboptimal fusion outcomes. To
address this, we introduce a text-guided multi-modality image fusion method
that leverages the high-level semantics from textual descriptions to integrate
semantics from infrared and visible images. This method capitalizes on the
complementary characteristics of diverse modalities, bolstering both the
accuracy and robustness of object detection. The codebook is utilized to
enhance a streamlined and concise depiction of the fused intra- and
inter-domain dynamics, fine-tuned for optimal performance in detection tasks.
We present a bilevel optimization strategy that establishes a nexus between the
joint problem of fusion and detection, optimizing both processes concurrently.
Furthermore, we introduce the first dataset of paired infrared and visible
images accompanied by text prompts, paving the way for future research.
Extensive experiments on several datasets demonstrate that our method not only
produces visually superior fusion results but also achieves a higher detection
mAP over existing methods, achieving state-of-the-art results.