Graph Representation Learning for Infrared and Visible Image Fusion
Jing Li Lu Bai Bin Yang Chang Li Lingfei Ma Edwin R. Hancock
Infrared and visible image fusion aims to extract complementary features to
synthesize a single fused image. Many methods employ convolutional neural
networks (CNNs) to extract local features due to its translation invariance and
locality. However, CNNs fail to consider the image's non-local self-similarity
(NLss), though it can expand the receptive field by pooling operations, it
still inevitably leads to information loss. In addition, the transformer
structure extracts long-range dependence by considering the correlativity among
all image patches, leading to information redundancy of such transformer-based
methods. However, graph representation is more flexible than grid (CNN) or
sequence (transformer structure) representation to address irregular objects,
and graph can also construct the relationships among the spatially repeatable
details or texture with far-space distance. Therefore, to address the above
issues, it is significant to convert images into the graph space and thus adopt
graph convolutional networks (GCNs) to extract NLss. This is because the graph
can provide a fine structure to aggregate features and propagate information
across the nearest vertices without introducing redundant information.
Concretely, we implement a cascaded NLss extraction pattern to extract NLss of
intra- and inter-modal by exploring interactions of different image pixels in
intra- and inter-image positional distance. We commence by preforming GCNs on
each intra-modal to aggregate features and propagate information to extract
independent intra-modal NLss. Then, GCNs are performed on the concatenate
intra-modal NLss features of infrared and visible images, which can explore the
cross-domain NLss of inter-modal to reconstruct the fused image. Ablation
studies and extensive experiments illustrates the effectiveness and superiority
of the proposed method on three datasets.