Regularized Conditional Diffusion Model for Multi-Task Preference
  Alignment
Xudong Yu Chenjia Bai Haoran He Changhong Wang Xuelong Li
Sequential decision-making is desired to align with human intents and exhibit
versatility across various tasks. Previous methods formulate it as a
conditional generation process, utilizing return-conditioned diffusion models
to directly model trajectory distributions. Nevertheless, the
return-conditioned paradigm relies on pre-defined reward functions, facing
challenges when applied in multi-task settings characterized by varying reward
functions (versatility) and showing limited controllability concerning human
preferences (alignment). In this work, we adopt multi-task preferences as a
unified condition for both single- and multi-task decision-making, and propose
preference representations aligned with preference labels. The learned
representations are used to guide the conditional generation process of
diffusion models, and we introduce an auxiliary objective to maximize the
mutual information between representations and corresponding generated
trajectories, improving alignment between trajectories and preferences.
Extensive experiments in D4RL and Meta-World demonstrate that our method
presents favorable performance in single- and multi-task scenarios, and
exhibits superior alignment with preferences.