From the Lab to the Wild: Affect Modeling via Privileged Information
Konstantinos Makantasis Kosmas Pinitas Antonios Liapis Georgios N. Yannakakis
How can we reliably transfer affect models trained in controlled laboratory
conditions (in-vitro) to uncontrolled real-world settings (in-vivo)? The
information gap between in-vitro and in-vivo applications defines a core
challenge of affective computing. This gap is caused by limitations related to
affect sensing including intrusiveness, hardware malfunctions and availability
of sensors. As a response to these limitations, we introduce the concept of
privileged information for operating affect models in real-world scenarios (in
the wild). Privileged information enables affect models to be trained across
multiple modalities available in a lab, and ignore, without significant
performance drops, those modalities that are not available when they operate in
the wild. Our approach is tested in two multimodal affect databases one of
which is designed for testing models of affect in the wild. By training our
affect models using all modalities and then using solely raw footage frames for
testing the models, we reach the performance of models that fuse all available
modalities for both training and testing. The results are robust across both
classification and regression affect modeling tasks which are dominant
paradigms in affective computing. Our findings make a decisive step towards
realizing affect interaction in the wild.