Interactively Teaching an Inverse Reinforcement Learner with Limited
  Feedback
Rustam Zayanov Francisco S. Melo Manuel Lopes
We study the problem of teaching via demonstrations in sequential
decision-making tasks. In particular, we focus on the situation when the
teacher has no access to the learner's model and policy, and the feedback from
the learner is limited to trajectories that start from states selected by the
teacher. The necessity to select the starting states and infer the learner's
policy creates an opportunity for using the methods of inverse reinforcement
learning and active learning by the teacher. In this work, we formalize the
teaching process with limited feedback and propose an algorithm that solves
this teaching problem. The algorithm uses a modified version of the active
value-at-risk method to select the starting states, a modified maximum causal
entropy algorithm to infer the policy, and the difficulty score ratio method to
choose the teaching demonstrations. We test the algorithm in a synthetic car
driving environment and conclude that the proposed algorithm is an effective
solution when the learner's feedback is limited.