Sparse multimodal fusion with modal channel attention
Josiah Bjorgaard
The ability of masked multimodal transformer architectures to learn a robust
embedding space when modality samples are sparsely aligned is studied by
measuring the quality of generated embedding spaces as a function of modal
sparsity. An extension to the masked multimodal transformer model is proposed
which incorporates modal-incomplete channels in the multihead attention
mechanism called modal channel attention (MCA). Two datasets with 4 modalities
are used, CMU-MOSEI for multimodal sentiment recognition and TCGA for
multiomics. Models are shown to learn uniform and aligned embedding spaces with
only two out of four modalities in most samples. It was found that, even with
no modal sparsity, the proposed MCA mechanism improves the quality of generated
embedding spaces, recall metrics, and subsequent performance on downstream
tasks.