Reinforcement Learning Interventions on Boundedly Rational Human Agents
  in Frictionful Tasks
Eura Nofshin Siddharth Swaroop Weiwei Pan Susan Murphy Finale Doshi-Velez
Many important behavior changes are frictionful; they require individuals to
expend effort over a long period with little immediate gratification. Here, an
artificial intelligence (AI) agent can provide personalized interventions to
help individuals stick to their goals. In these settings, the AI agent must
personalize rapidly (before the individual disengages) and interpretably, to
help us understand the behavioral interventions. In this paper, we introduce
Behavior Model Reinforcement Learning (BMRL), a framework in which an AI agent
intervenes on the parameters of a Markov Decision Process (MDP) belonging to a
boundedly rational human agent. Our formulation of the human decision-maker as
a planning agent allows us to attribute undesirable human policies (ones that
do not lead to the goal) to their maladapted MDP parameters, such as an
extremely low discount factor. Furthermore, we propose a class of tractable
human models that captures fundamental behaviors in frictionful tasks.
Introducing a notion of MDP equivalence specific to BMRL, we theoretically and
empirically show that AI planning with our human models can lead to helpful
policies on a wide range of more complex, ground-truth humans.