Adversarial Batch Inverse Reinforcement Learning: Learn to Reward from
  Imperfect Demonstration for Interactive Recommendation
Jialin Liu Xinyan Su Zeyu He Xiangyu Zhao Jun Li
Rewards serve as a measure of user satisfaction and act as a limiting factor
in interactive recommender systems. In this research, we focus on the problem
of learning to reward (LTR), which is fundamental to reinforcement learning.
Previous approaches either introduce additional procedures for learning to
reward, thereby increasing the complexity of optimization, or assume that
user-agent interactions provide perfect demonstrations, which is not feasible
in practice. Ideally, we aim to employ a unified approach that optimizes both
the reward and policy using compositional demonstrations. However, this
requirement presents a challenge since rewards inherently quantify user
feedback on-policy, while recommender agents approximate off-policy future
cumulative valuation. To tackle this challenge, we propose a novel batch
inverse reinforcement learning paradigm that achieves the desired properties.
Our method utilizes discounted stationary distribution correction to combine
LTR and recommender agent evaluation. To fulfill the compositional requirement,
we incorporate the concept of pessimism through conservation. Specifically, we
modify the vanilla correction using Bellman transformation and enforce KL
regularization to constrain consecutive policy updates. We use two real-world
datasets which represent two compositional coverage to conduct empirical
studies, the results also show that the proposed method relatively improves
both effectiveness (2.3\%) and efficiency (11.53\%)