Offline Risk-sensitive RL with Partial Observability to Enhance
  Performance in Human-Robot Teaming
Giorgio Angelotti Caroline P. C. Chanel Adam H. M. Pinto Christophe Lounis Corentin Chauffaut Nicolas Drougard
The integration of physiological computing into mixed-initiative human-robot
interaction systems offers valuable advantages in autonomous task allocation by
incorporating real-time features as human state observations into the
decision-making system. This approach may alleviate the cognitive load on human
operators by intelligently allocating mission tasks between agents.
Nevertheless, accommodating a diverse pool of human participants with varying
physiological and behavioral measurements presents a substantial challenge. To
address this, resorting to a probabilistic framework becomes necessary, given
the inherent uncertainty and partial observability on the human's state. Recent
research suggests to learn a Partially Observable Markov Decision Process
(POMDP) model from a data set of previously collected experiences that can be
solved using Offline Reinforcement Learning (ORL) methods. In the present work,
we not only highlight the potential of partially observable representations and
physiological measurements to improve human operator state estimation and
performance, but also enhance the overall mission effectiveness of a
human-robot team. Importantly, as the fixed data set may not contain enough
information to fully represent complex stochastic processes, we propose a
method to incorporate model uncertainty, thus enabling risk-sensitive
sequential decision-making. Experiments were conducted with a group of
twenty-six human participants within a simulated robot teleoperation
environment, yielding empirical evidence of the method's efficacy. The obtained
adaptive task allocation policy led to statistically significant higher scores
than the one that was used to collect the data set, allowing for generalization
across diverse participants also taking into account risk-sensitive metrics.