HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with
  Cross-person Memory Transformer
Yubin Kim Dong Won Lee Paul Pu Liang Sharifa Algohwinem Cynthia Breazeal Hae Won Park
Accurately modeling affect dynamics, which refers to the changes and
fluctuations in emotions and affective displays during human conversations, is
crucial for understanding human interactions. By analyzing affect dynamics, we
can gain insights into how people communicate, respond to different situations,
and form relationships. However, modeling affect dynamics is challenging due to
contextual factors, such as the complex and nuanced nature of interpersonal
relationships, the situation, and other factors that influence affective
displays. To address this challenge, we propose a Cross-person Memory
Transformer (CPM-T) framework which is able to explicitly model affective
dynamics (intrapersonal and interpersonal influences) by identifying verbal and
non-verbal cues, and with a large language model to utilize the pre-trained
knowledge and perform verbal reasoning. The CPM-T framework maintains memory
modules to store and update the contexts within the conversation window,
enabling the model to capture dependencies between earlier and later parts of a
conversation. Additionally, our framework employs cross-modal attention to
effectively align information from multi-modalities and leverage cross-person
attention to align behaviors in multi-party interactions. We evaluate the
effectiveness and generalizability of our approach on three publicly available
datasets for joint engagement, rapport, and human beliefs prediction tasks.
Remarkably, the CPM-T framework outperforms baseline models in average
F1-scores by up to 7.3%, 9.3%, and 2.0% respectively. Finally, we demonstrate
the importance of each component in the framework via ablation studies with
respect to multimodal temporal behavior.