The Cram Method for Efficient Simultaneous Learning and Evaluation
Zeyang Jia Kosuke Imai Michael Lingzhi Li
We introduce the "cram" method, a general and efficient approach to
simultaneous learning and evaluation using a generic machine learning (ML)
algorithm. In a single pass of batched data, the proposed method repeatedly
trains an ML algorithm and tests its empirical performance. Because it utilizes
the entire sample for both learning and evaluation, cramming is significantly
more data-efficient than sample-splitting. The cram method also naturally
accommodates online learning algorithms, making its implementation
computationally efficient. To demonstrate the power of the cram method, we
consider the standard policy learning setting where cramming is applied to the
same data to both develop an individualized treatment rule (ITR) and estimate
the average outcome that would result if the learned ITR were to be deployed.
We show that under a minimal set of assumptions, the resulting crammed
evaluation estimator is consistent and asymptotically normal. While our
asymptotic results require a relatively weak stabilization condition of ML
algorithm, we develop a simple, generic method that can be used with any policy
learning algorithm to satisfy this condition. Our extensive simulation studies
show that, when compared to sample-splitting, cramming reduces the evaluation
standard error by more than 40% while improving the performance of learned
policy. We also apply the cram method to a randomized clinical trial to
demonstrate its applicability to real-world problems. Finally, we briefly
discuss future extensions of the cram method to other learning and evaluation
settings.