Helen: Optimizing CTR Prediction Models with Frequency-wise Hessian
  Eigenvalue Regularization
Zirui Zhu Yong Liu Zangwei Zheng Huifeng Guo Yang You
Click-Through Rate (CTR) prediction holds paramount significance in online
advertising and recommendation scenarios. Despite the proliferation of recent
CTR prediction models, the improvements in performance have remained limited,
as evidenced by open-source benchmark assessments. Current researchers tend to
focus on developing new models for various datasets and settings, often
neglecting a crucial question: What is the key challenge that truly makes CTR
prediction so demanding?
  In this paper, we approach the problem of CTR prediction from an optimization
perspective. We explore the typical data characteristics and optimization
statistics of CTR prediction, revealing a strong positive correlation between
the top hessian eigenvalue and feature frequency. This correlation implies that
frequently occurring features tend to converge towards sharp local minima,
ultimately leading to suboptimal performance. Motivated by the recent
advancements in sharpness-aware minimization (SAM), which considers the
geometric aspects of the loss landscape during optimization, we present a
dedicated optimizer crafted for CTR prediction, named Helen. Helen incorporates
frequency-wise Hessian eigenvalue regularization, achieved through adaptive
perturbations based on normalized feature frequencies.
  Empirical results under the open-source benchmark framework underscore
Helen's effectiveness. It successfully constrains the top eigenvalue of the
Hessian matrix and demonstrates a clear advantage over widely used optimization
algorithms when applied to seven popular models across three public benchmark
datasets on BARS. Our code locates at github.com/NUS-HPC-AI-Lab/Helen.