Improving Unimodal Inference with Multimodal Transformers
Kateryna Chumachenko Alexandros Iosifidis Moncef Gabbouj
This paper proposes an approach for improving performance of unimodal models
with multimodal training. Our approach involves a multi-branch architecture
that incorporates unimodal models with a multimodal transformer-based branch.
By co-training these branches, the stronger multimodal branch can transfer its
knowledge to the weaker unimodal branches through a multi-task objective,
thereby improving the performance of the resulting unimodal models. We evaluate
our approach on tasks of dynamic hand gesture recognition based on RGB and
Depth, audiovisual emotion recognition based on speech and facial video, and
audio-video-text based sentiment analysis. Our approach outperforms the
conventionally trained unimodal counterparts. Interestingly, we also observe
that optimization of the unimodal branches improves the multimodal branch,
compared to a similar multimodal model trained from scratch.