Distilling Privileged Multimodal Information for Expression Recognition
  using Optimal Transport
Muhammad Haseeb Aslam Muhammad Osama Zeeshan Soufiane Belharbi Marco Pedersoli Alessandro Koerich Simon Bacon Eric Granger
Multimodal affect recognition models have reached remarkable performance in
the lab environment due to their ability to model complementary and redundant
semantic information. However, these models struggle in the wild, mainly
because of the unavailability or quality of modalities used for training. In
practice, only a subset of the training-time modalities may be available at
test time. Learning with privileged information (PI) enables deep learning
models (DL) to exploit data from additional modalities only available during
training. State-of-the-art knowledge distillation (KD) methods have been
proposed to distill multiple teacher models (each trained on a modality) to a
common student model. These privileged KD methods typically utilize
point-to-point matching and have no explicit mechanism to capture the
structural information in the teacher representation space formed by
introducing the privileged modality. We argue that encoding this same structure
in the student space may lead to enhanced student performance. This paper
introduces a new structural KD mechanism based on optimal transport (OT), where
entropy-regularized OT distills the structural dark knowledge. Privileged KD
with OT (PKDOT) method captures the local structures in the multimodal teacher
representation by calculating a cosine similarity matrix and selects the top-k
anchors to allow for sparse OT solutions, resulting in a more stable
distillation process. Experiments were performed on two different problems:
pain estimation on the Biovid dataset (ordinal classification) and
arousal-valance prediction on the Affwild2 dataset (regression). Results show
that the proposed method can outperform state-of-the-art privileged KD methods
on these problems. The diversity of different modalities and fusion
architectures indicates that the proposed PKDOT method is modality and
model-agnostic.