IBCB: Efficient Inverse Batched Contextual Bandit for Behavioral
  Evolution History
Yi Xu Weiran Shen Xiao Zhang Jun Xu
Traditional imitation learning focuses on modeling the behavioral mechanisms
of experts, which requires a large amount of interaction history generated by
some fixed expert. However, in many streaming applications, such as streaming
recommender systems, online decision-makers typically engage in online learning
during the decision-making process, meaning that the interaction history
generated by online decision-makers includes their behavioral evolution from
novice expert to experienced expert. This poses a new challenge for existing
imitation learning approaches that can only utilize data from experienced
experts. To address this issue, this paper proposes an inverse batched
contextual bandit (IBCB) framework that can efficiently perform estimations of
environment reward parameters and learned policy based on the expert's
behavioral evolution history. Specifically, IBCB formulates the inverse problem
into a simple quadratic programming problem by utilizing the behavioral
evolution history of the batched contextual bandit with inaccessible rewards.
We demonstrate that IBCB is a unified framework for both deterministic and
randomized bandit policies. The experimental results indicate that IBCB
outperforms several existing imitation learning algorithms on synthetic and
real-world data and significantly reduces running time. Additionally, empirical
analyses reveal that IBCB exhibits better out-of-distribution generalization
and is highly effective in learning the bandit policy from the interaction
history of novice experts.