Multimodal Transformer With a Low-Computational-Cost Guarantee
Sungjin Park Edward Choi
Transformer-based models have significantly improved performance across a
range of multimodal understanding tasks, such as visual question answering and
action recognition. However, multimodal Transformers significantly suffer from
a quadratic complexity of the multi-head attention with the input sequence
length, especially as the number of modalities increases. To address this, we
introduce Low-Cost Multimodal Transformer (LoCoMT), a novel multimodal
attention mechanism that aims to reduce computational cost during training and
inference with minimal performance loss. Specifically, by assigning different
multimodal attention patterns to each attention head, LoCoMT can flexibly
control multimodal signals and theoretically ensures a reduced computational
cost compared to existing multimodal Transformer variants. Experimental results
on two multimodal datasets, namely Audioset and MedVidCL demonstrate that
LoCoMT not only reduces GFLOPs but also matches or even outperforms established
models.