Offline Inverse RL: New Solution Concepts and Provably Efficient
  Algorithms
Filippo Lazzati Mirco Mutti Alberto Maria Metelli
Inverse reinforcement learning (IRL) aims to recover the reward function of
an expert agent from demonstrations of behavior. It is well known that the IRL
problem is fundamentally ill-posed, i.e., many reward functions can explain the
demonstrations. For this reason, IRL has been recently reframed in terms of
estimating the feasible reward set, thus, postponing the selection of a single
reward. However, so far, the available formulations and algorithmic solutions
have been proposed and analyzed mainly for the online setting, where the
learner can interact with the environment and query the expert at will. This is
clearly unrealistic in most practical applications, where the availability of
an offline dataset is a much more common scenario. In this paper, we introduce
a novel notion of feasible reward set capturing the opportunities and
limitations of the offline setting and we analyze the complexity of its
estimation. This requires the introduction an original learning framework that
copes with the intrinsic difficulty of the setting, for which the data coverage
is not under control. Then, we propose two computationally and statistically
efficient algorithms, IRLO and PIRLO, for addressing the problem. In
particular, the latter adopts a specific form of pessimism to enforce the novel
desirable property of inclusion monotonicity of the delivered feasible set.
With this work, we aim to provide a panorama of the challenges of the offline
IRL problem and how they can be fruitfully addressed.