On the Embedding Collapse when Scaling up Recommendation Models
Xingzhuo Guo Junwei Pan Ximei Wang Baixu Chen Jie Jiang Mingsheng Long
Recent advances in deep foundation models have led to a promising trend of
developing large recommendation models to leverage vast amounts of available
data. However, we experiment to scale up existing recommendation models and
observe that the enlarged models do not improve satisfactorily. In this
context, we investigate the embedding layers of enlarged models and identify a
phenomenon of embedding collapse, which ultimately hinders scalability, wherein
the embedding matrix tends to reside in a low-dimensional subspace. Through
empirical and theoretical analysis, we demonstrate that the feature interaction
module specific to recommendation models has a two-sided effect. On the one
hand, the interaction restricts embedding learning when interacting with
collapsed embeddings, exacerbating the collapse issue. On the other hand,
feature interaction is crucial in mitigating the fitting of spurious features,
thereby improving scalability. Based on this analysis, we propose a simple yet
effective multi-embedding design incorporating embedding-set-specific
interaction modules to capture diverse patterns and reduce collapse. Extensive
experiments demonstrate that this proposed design provides consistent
scalability for various recommendation models.