Multi-Epoch Learning for Deep Click-Through Rate Prediction Models
Zhaocheng Liu Zhongxiang Fan Jian Liang Dongying Kong Han Li
The one-epoch overfitting phenomenon has been widely observed in industrial
Click-Through Rate (CTR) applications, where the model performance experiences
a significant degradation at the beginning of the second epoch. Recent advances
try to understand the underlying factors behind this phenomenon through
extensive experiments. However, it is still unknown whether a multi-epoch
training paradigm could achieve better results, as the best performance is
usually achieved by one-epoch training. In this paper, we hypothesize that the
emergence of this phenomenon may be attributed to the susceptibility of the
embedding layer to overfitting, which can stem from the high-dimensional
sparsity of data. To maintain feature sparsity while simultaneously avoiding
overfitting of embeddings, we propose a novel Multi-Epoch learning with Data
Augmentation (MEDA), which can be directly applied to most deep CTR models.
MEDA achieves data augmentation by reinitializing the embedding layer in each
epoch, thereby avoiding embedding overfitting and simultaneously improving
convergence. To our best knowledge, MEDA is the first multi-epoch training
paradigm designed for deep CTR prediction models. We conduct extensive
experiments on several public datasets, and the effectiveness of our proposed
MEDA is fully verified. Notably, the results show that MEDA can significantly
outperform the conventional one-epoch training. Besides, MEDA has exhibited
significant benefits in a real-world scene on Kuaishou.