Doubly Calibrated Estimator for Recommendation on Data Missing Not At
  Random
Wonbin Kweon Hwanjo Yu
Recommender systems often suffer from selection bias as users tend to rate
their preferred items. The datasets collected under such conditions exhibit
entries missing not at random and thus are not randomized-controlled trials
representing the target population. To address this challenge, a doubly robust
estimator and its enhanced variants have been proposed as they ensure
unbiasedness when accurate imputed errors or predicted propensities are
provided. However, we argue that existing estimators rely on miscalibrated
imputed errors and propensity scores as they depend on rudimentary models for
estimation. We provide theoretical insights into how miscalibrated imputation
and propensity models may limit the effectiveness of doubly robust estimators
and validate our theorems using real-world datasets. On this basis, we propose
a Doubly Calibrated Estimator that involves the calibration of both the
imputation and propensity models. To achieve this, we introduce calibration
experts that consider different logit distributions across users. Moreover, we
devise a tri-level joint learning framework, allowing the simultaneous
optimization of calibration experts alongside prediction and imputation models.
Through extensive experiments on real-world datasets, we demonstrate the
superiority of the Doubly Calibrated Estimator in the context of debiased
recommendation tasks.