Unlocking the Potential of Multimodal Unified Discrete Representation
  through Training-Free Codebook Optimization and Hierarchical Alignment
Hai Huang Yan Xia Shengpeng Ji Shulei Wang Hanting Wang Jieming Zhu Zhenhua Dong Zhou Zhao
Recent advances in representation learning have demonstrated the significance
of multimodal alignment. The Dual Cross-modal Information Disentanglement
(DCID) model, utilizing a unified codebook, shows promising results in
achieving fine-grained representation and cross-modal generalization. However,
it is still hindered by equal treatment of all channels and neglect of minor
event information, resulting in interference from irrelevant channels and
limited performance in fine-grained tasks. Thus, in this work, We propose a
Training-free Optimization of Codebook (TOC) method to enhance model
performance by selecting important channels in the unified space without
retraining. Additionally, we introduce the Hierarchical Dual Cross-modal
Information Disentanglement (H-DCID) approach to extend information separation
and alignment to two levels, capturing more cross-modal details. The experiment
results demonstrate significant improvements across various downstream tasks,
with TOC contributing to an average improvement of 1.70% for DCID on four
tasks, and H-DCID surpassing DCID by an average of 3.64%. The combination of
TOC and H-DCID further enhances performance, exceeding DCID by 4.43%. These
findings highlight the effectiveness of our methods in facilitating robust and
nuanced cross-modal learning, opening avenues for future enhancements. The
source code and pre-trained models can be accessed at
https://github.com/haihuangcode/TOC_H-DCID.