Dynamic Sparse Learning: A Novel Paradigm for Efficient Recommendation
Shuyao Wang Yongduo Sui Jiancan Wu Zhi Zheng Hui Xiong
In the realm of deep learning-based recommendation systems, the increasing
computational demands, driven by the growing number of users and items, pose a
significant challenge to practical deployment. This challenge is primarily
twofold: reducing the model size while effectively learning user and item
representations for efficient recommendations. Despite considerable
advancements in model compression and architecture search, prevalent approaches
face notable constraints. These include substantial additional computational
costs from pre-training/re-training in model compression and an extensive
search space in architecture design. Additionally, managing complexity and
adhering to memory constraints is problematic, especially in scenarios with
strict time or space limitations. Addressing these issues, this paper
introduces a novel learning paradigm, Dynamic Sparse Learning (DSL), tailored
for recommendation models. DSL innovatively trains a lightweight sparse model
from scratch, periodically evaluating and dynamically adjusting each weight's
significance and the model's sparsity distribution during the training. This
approach ensures a consistent and minimal parameter budget throughout the full
learning lifecycle, paving the way for "end-to-end" efficiency from training to
inference. Our extensive experimental results underline DSL's effectiveness,
significantly reducing training and inference costs while delivering comparable
recommendation performance.