Towards Inferring Users' Impressions of Robot Performance in Navigation
  Scenarios
Qiping Zhang Nathan Tsoi Booyeon Choi Jie Tan Hao-Tien Lewis Chiang Marynel Vázquez
Human impressions of robot performance are often measured through surveys. As
a more scalable and cost-effective alternative, we study the possibility of
predicting people's impressions of robot behavior using non-verbal behavioral
cues and machine learning techniques. To this end, we first contribute the SEAN
TOGETHER Dataset consisting of observations of an interaction between a person
and a mobile robot in a Virtual Reality simulation, together with impressions
of robot performance provided by users on a 5-point scale. Second, we
contribute analyses of how well humans and supervised learning techniques can
predict perceived robot performance based on different combinations of
observation types (e.g., facial, spatial, and map features). Our results show
that facial expressions alone provide useful information about human
impressions of robot performance; but in the navigation scenarios we tested,
spatial features are the most critical piece of information for this inference
task. Also, when evaluating results as binary classification (rather than
multiclass classification), the F1-Score of human predictions and machine
learning models more than doubles, showing that both are better at telling the
directionality of robot performance than predicting exact performance ratings.
Based on our findings, we provide guidelines for implementing these predictions
models in real-world navigation scenarios.