Bi-Level Offline Policy Optimization with Limited Exploration
Wenzhuo Zhou
We study offline reinforcement learning (RL) which seeks to learn a good
policy based on a fixed, pre-collected dataset. A fundamental challenge behind
this task is the distributional shift due to the dataset lacking sufficient
exploration, especially under function approximation. To tackle this issue, we
propose a bi-level structured policy optimization algorithm that models a
hierarchical interaction between the policy (upper-level) and the value
function (lower-level). The lower level focuses on constructing a confidence
set of value estimates that maintain sufficiently small weighted average
Bellman errors, while controlling uncertainty arising from distribution
mismatch. Subsequently, at the upper level, the policy aims to maximize a
conservative value estimate from the confidence set formed at the lower level.
This novel formulation preserves the maximum flexibility of the implicitly
induced exploratory data distribution, enabling the power of model
extrapolation. In practice, it can be solved through a computationally
efficient, penalized adversarial estimation procedure. Our theoretical regret
guarantees do not rely on any data-coverage and completeness-type assumptions,
only requiring realizability. These guarantees also demonstrate that the
learned policy represents the "best effort" among all policies, as no other
policies can outperform it. We evaluate our model using a blend of synthetic,
benchmark, and real-world datasets for offline RL, showing that it performs
competitively with state-of-the-art methods.