A Theory of Unimodal Bias in Multimodal Learning
Yedi Zhang Peter E. Latham Andrew Saxe
Using multiple input streams simultaneously in training multimodal neural
networks is intuitively advantageous, but practically challenging. A key
challenge is unimodal bias, where a network overly relies on one modality and
ignores others during joint training. While unimodal bias is well-documented
empirically, our theoretical understanding of how architecture and data
statistics influence this bias remains incomplete. Here we develop a theory of
unimodal bias with deep multimodal linear networks. We calculate the duration
of the unimodal phase in learning as a function of the depth at which
modalities are fused within the network, dataset statistics, and
initialization. We find that the deeper the layer at which fusion occurs, the
longer the unimodal phase. A long unimodal phase can lead to a generalization
deficit and permanent unimodal bias in the overparametrized regime. In
addition, our theory reveals the modality learned first is not necessarily the
modality that contributes more to the output. Our results, derived for
multimodal linear networks, extend to ReLU networks in certain settings. Taken
together, this work illuminates pathologies of multimodal learning under joint
training, showing that late and intermediate fusion architectures can give rise
to long unimodal phases and permanent unimodal bias.