Calibration-then-Calculation: A Variance Reduced Metric Framework in
  Deep Click-Through Rate Prediction Models
Yewen Fan Nian Si Xiangchen Song Kun Zhang
Deep learning has been widely adopted across various fields, but there has
been little focus on evaluating the performance of deep learning pipelines.
With the increased use of large datasets and complex models, it has become
common to run the training process only once and compare the result to previous
benchmarks. However, this procedure can lead to imprecise comparisons due to
the variance in neural network evaluation metrics. The metric variance comes
from the randomness inherent in the training process of deep learning
pipelines. Traditional solutions such as running the training process multiple
times are usually not feasible in deep learning due to computational
limitations. In this paper, we propose a new metric framework, Calibrated Loss
Metric, that addresses this issue by reducing the variance in its vanilla
counterpart. As a result, the new metric has a higher accuracy to detect
effective modeling improvement. Our approach is supported by theoretical
justifications and extensive experimental validations in the context of Deep
Click-Through Rate Prediction Models.