USE: Dynamic User Modeling with Stateful Sequence Models
Zhihan Zhou Qixiang Fang Leonardo Neves Francesco Barbieri Yozen Liu Han Liu Maarten W. Bos Ron Dotsch
User embeddings play a crucial role in user engagement forecasting and
personalized services. Recent advances in sequence modeling have sparked
interest in learning user embeddings from behavioral data. Yet behavior-based
user embedding learning faces the unique challenge of dynamic user modeling. As
users continuously interact with the apps, user embeddings should be
periodically updated to account for users' recent and long-term behavior
patterns. Existing methods highly rely on stateless sequence models that lack
memory of historical behavior. They have to either discard historical data and
use only the most recent data or reprocess the old and new data jointly. Both
cases incur substantial computational overhead. To address this limitation, we
introduce User Stateful Embedding (USE). USE generates user embeddings and
reflects users' evolving behaviors without the need for exhaustive reprocessing
by storing previous model states and revisiting them in the future.
Furthermore, we introduce a novel training objective named future W-behavior
prediction to transcend the limitations of next-token prediction by forecasting
a broader horizon of upcoming user behaviors. By combining it with the Same
User Prediction, a contrastive learning-based objective that predicts whether
different segments of behavior sequences belong to the same user, we further
improve the embeddings' distinctiveness and representativeness. We conducted
experiments on 8 downstream tasks using Snapchat users' behavioral logs in both
static (i.e., fixed user behavior sequences) and dynamic (i.e., periodically
updated user behavior sequences) settings. We demonstrate USE's superior
performance over established baselines. The results underscore USE's
effectiveness and efficiency in integrating historical and recent user behavior
sequences into user embeddings in dynamic user modeling.