Sim-to-Real Causal Transfer: A Metric Learning Approach to
  Causally-Aware Interaction Representations
Yuejiang Liu Ahmad Rahimi Po-Chien Luan Frano Rajič Alexandre Alahi
Modeling spatial-temporal interactions among neighboring agents is at the
heart of multi-agent problems such as motion forecasting and crowd navigation.
Despite notable progress, it remains unclear to which extent modern
representations can capture the causal relationships behind agent interactions.
In this work, we take an in-depth look at the causal awareness of these
representations, from computational formalism to real-world practice. First, we
cast doubt on the notion of non-causal robustness studied in the recent
CausalAgents benchmark. We show that recent representations are already
partially resilient to perturbations of non-causal agents, and yet modeling
indirect causal effects involving mediator agents remains challenging. To
address this challenge, we introduce a metric learning approach that
regularizes latent representations with causal annotations. Our controlled
experiments show that this approach not only leads to higher degrees of causal
awareness but also yields stronger out-of-distribution robustness. To further
operationalize it in practice, we propose a sim-to-real causal transfer method
via cross-domain multi-task learning. Experiments on pedestrian datasets show
that our method can substantially boost generalization, even in the absence of
real-world causal annotations. We hope our work provides a new perspective on
the challenges and potential pathways towards causally-aware representations of
multi-agent interactions. Our code is available at
https://github.com/socialcausality.