InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interaction
Sirui Xu Ziyin Wang Yu-Xiong Wang Liang-Yan Gui
Text-conditioned human motion generation has experienced significant
advancements with diffusion models trained on extensive motion capture data and
corresponding textual annotations. However, extending such success to 3D
dynamic human-object interaction (HOI) generation faces notable challenges,
primarily due to the lack of large-scale interaction data and comprehensive
descriptions that align with these interactions. This paper takes the
initiative and showcases the potential of generating human-object interactions
without direct training on text-interaction pair data. Our key insight in
achieving this is that interaction semantics and dynamics can be decoupled.
Being unable to learn interaction semantics through supervised training, we
instead leverage pre-trained large models, synergizing knowledge from a large
language model and a text-to-motion model. While such knowledge offers
high-level control over interaction semantics, it cannot grasp the intricacies
of low-level interaction dynamics. To overcome this issue, we further introduce
a world model designed to comprehend simple physics, modeling how human actions
influence object motion. By integrating these components, our novel framework,
InterDreamer, is able to generate text-aligned 3D HOI sequences in a zero-shot
manner. We apply InterDreamer to the BEHAVE and CHAIRS datasets, and our
comprehensive experimental analysis demonstrates its capability to generate
realistic and coherent interaction sequences that seamlessly align with the
text directives.