Introspective Action Advising for Interpretable Transfer Learning
Joseph Campbell Yue Guo Fiona Xie Simon Stepputtis Katia Sycara
Transfer learning can be applied in deep reinforcement learning to accelerate
the training of a policy in a target task by transferring knowledge from a
policy learned in a related source task. This is commonly achieved by copying
pretrained weights from the source policy to the target policy prior to
training, under the constraint that they use the same model architecture.
However, not only does this require a robust representation learned over a wide
distribution of states -- often failing to transfer between specialist models
trained over single tasks -- but it is largely uninterpretable and provides
little indication of what knowledge is transferred. In this work, we propose an
alternative approach to transfer learning between tasks based on action
advising, in which a teacher trained in a source task actively guides a
student's exploration in a target task. Through introspection, the teacher is
capable of identifying when advice is beneficial to the student and should be
given, and when it is not. Our approach allows knowledge transfer between
policies agnostic of the underlying representations, and we empirically show
that this leads to improved convergence rates in Gridworld and Atari
environments while providing insight into what knowledge is transferred.