PMG : Personalized Multimodal Generation with Large Language Models
Xiaoteng Shen Rui Zhang Xiaoyan Zhao Jieming Zhu Xi Xiao
The emergence of large language models (LLMs) has revolutionized the
capabilities of text comprehension and generation. Multi-modal generation
attracts great attention from both the industry and academia, but there is
little work on personalized generation, which has important applications such
as recommender systems. This paper proposes the first method for personalized
multimodal generation using LLMs, showcases its applications and validates its
performance via an extensive experimental study on two datasets. The proposed
method, Personalized Multimodal Generation (PMG for short) first converts user
behaviors (e.g., clicks in recommender systems or conversations with a virtual
assistant) into natural language to facilitate LLM understanding and extract
user preference descriptions. Such user preferences are then fed into a
generator, such as a multimodal LLM or diffusion model, to produce personalized
content. To capture user preferences comprehensively and accurately, we propose
to let the LLM output a combination of explicit keywords and implicit
embeddings to represent user preferences. Then the combination of keywords and
embeddings are used as prompts to condition the generator. We optimize a
weighted sum of the accuracy and preference scores so that the generated
content has a good balance between them. Compared to a baseline method without
personalization, PMG has a significant improvement on personalization for up to
8% in terms of LPIPS while retaining the accuracy of generation.