Doubly-Robust Off-Policy Evaluation with Estimated Logging Policy
Kyungbok Lee Myunghee Cho Paik
We introduce a novel doubly-robust (DR) off-policy evaluation (OPE) estimator
for Markov decision processes, DRUnknown, designed for situations where both
the logging policy and the value function are unknown. The proposed estimator
initially estimates the logging policy and then estimates the value function
model by minimizing the asymptotic variance of the estimator while considering
the estimating effect of the logging policy. When the logging policy model is
correctly specified, DRUnknown achieves the smallest asymptotic variance within
the class containing existing OPE estimators. When the value function model is
also correctly specified, DRUnknown is optimal as its asymptotic variance
reaches the semiparametric lower bound. We present experimental results
conducted in contextual bandits and reinforcement learning to compare the
performance of DRUnknown with that of existing methods.