Sequential Decision Making with Expert Demonstrations under Unobserved
  Heterogeneity
Vahid Balazadeh Keertana Chidambaram Viet Nguyen Rahul G. Krishnan Vasilis Syrgkanis
We study the problem of online sequential decision-making given auxiliary
demonstrations from experts who made their decisions based on unobserved
contextual information. These demonstrations can be viewed as solving related
but slightly different tasks than what the learner faces. This setting arises
in many application domains, such as self-driving cars, healthcare, and
finance, where expert demonstrations are made using contextual information,
which is not recorded in the data available to the learning agent. We model the
problem as a zero-shot meta-reinforcement learning setting with an unknown task
distribution and a Bayesian regret minimization objective, where the unobserved
tasks are encoded as parameters with an unknown prior. We propose the
Experts-as-Priors algorithm (ExPerior), a non-parametric empirical Bayes
approach that utilizes the principle of maximum entropy to establish an
informative prior over the learner's decision-making problem. This prior
enables the application of any Bayesian approach for online decision-making,
such as posterior sampling. We demonstrate that our strategy surpasses existing
behaviour cloning and online algorithms for multi-armed bandits and
reinforcement learning, showcasing the utility of our approach in leveraging
expert demonstrations across different decision-making setups.