High Precision Causal Model Evaluation with Conditional Randomization
Chao Ma Cheng Zhang
The gold standard for causal model evaluation involves comparing model
predictions with true effects estimated from randomized controlled trials
(RCT). However, RCTs are not always feasible or ethical to perform. In
contrast, conditionally randomized experiments based on inverse probability
weighting (IPW) offer a more realistic approach but may suffer from high
estimation variance. To tackle this challenge and enhance causal model
evaluation in real-world conditional randomization settings, we introduce a
novel low-variance estimator for causal error, dubbed as the pairs estimator.
By applying the same IPW estimator to both the model and true experimental
effects, our estimator effectively cancels out the variance due to IPW and
achieves a smaller asymptotic variance. Empirical studies demonstrate the
improved of our estimator, highlighting its potential on achieving near-RCT
performance. Our method offers a simple yet powerful solution to evaluate
causal inference models in conditional randomization settings without
complicated modification of the IPW estimator itself, paving the way for more
robust and reliable model assessments.