Interpretation on Multi-modal Visual Fusion
Hao Chen Haoran Zhou Yongjian Deng
In this paper, we present an analytical framework and a novel metric to shed
light on the interpretation of the multimodal vision community. Our approach
involves measuring the proposed semantic variance and feature similarity across
modalities and levels, and conducting semantic and quantitative analyses
through comprehensive experiments. Specifically, we investigate the consistency
and speciality of representations across modalities, evolution rules within
each modality, and the collaboration logic used when optimizing a
multi-modality model. Our studies reveal several important findings, such as
the discrepancy in cross-modal features and the hybrid multi-modal cooperation
rule, which highlights consistency and speciality simultaneously for
complementary inference. Through our dissection and findings on multi-modal
fusion, we facilitate a rethinking of the reasonability and necessity of
popular multi-modal vision fusion strategies. Furthermore, our work lays the
foundation for designing a trustworthy and universal multi-modal fusion model
for a variety of tasks in the future.