Bootstrapping Adaptive Human-Machine Interfaces with Offline
  Reinforcement Learning
Jensen Gao Siddharth Reddy Glen Berseth Anca D. Dragan Sergey Levine
Adaptive interfaces can help users perform sequential decision-making tasks
like robotic teleoperation given noisy, high-dimensional command signals (e.g.,
from a brain-computer interface). Recent advances in human-in-the-loop machine
learning enable such systems to improve by interacting with users, but tend to
be limited by the amount of data that they can collect from individual users in
practice. In this paper, we propose a reinforcement learning algorithm to
address this by training an interface to map raw command signals to actions
using a combination of offline pre-training and online fine-tuning. To address
the challenges posed by noisy command signals and sparse rewards, we develop a
novel method for representing and inferring the user's long-term intent for a
given trajectory. We primarily evaluate our method's ability to assist users
who can only communicate through noisy, high-dimensional input channels through
a user study in which 12 participants performed a simulated navigation task by
using their eye gaze to modulate a 128-dimensional command signal from their
webcam. The results show that our method enables successful goal navigation
more often than a baseline directional interface, by learning to denoise user
commands signals and provide shared autonomy assistance. We further evaluate on
a simulated Sawyer pushing task with eye gaze control, and the Lunar Lander
game with simulated user commands, and find that our method improves over
baseline interfaces in these domains as well. Extensive ablation experiments
with simulated user commands empirically motivate each component of our method.