Idea-2-3D: Collaborative LMM Agents Enable 3D Model Generation from
  Interleaved Multimodal Inputs
Junhao Chen Xiang Li Xiaojun Ye Chao Li Zhaoxin Fan Hao Zhao
In this paper, we pursue a novel 3D AIGC setting: generating 3D content from
IDEAs. The definition of an IDEA is the composition of multimodal inputs
including text, image, and 3D models. To our knowledge, this challenging and
appealing 3D AIGC setting has not been studied before. We propose the novel
framework called Idea-2-3D to achieve this goal, which consists of three agents
based upon large multimodel models (LMMs) and several existing algorithmic
tools for them to invoke. Specifically, these three LMM-based agents are
prompted to do the jobs of prompt generation, model selection and feedback
reflection. They work in a cycle that involves both mutual collaboration and
criticism. Note that this cycle is done in a fully automatic manner, without
any human intervention. The framework then outputs a text prompt to generate 3D
models that well align with input IDEAs. We show impressive 3D AIGC results
that are beyond any previous methods can achieve. For quantitative comparisons,
we construct caption-based baselines using a whole bunch of state-of-the-art 3D
AIGC models and demonstrate Idea-2-3D out-performs significantly. In 94.2% of
cases, Idea-2-3D meets users' requirements, marking a degree of match between
IDEA and 3D models that is 2.3 times higher than baselines. Moreover, in 93.5%
of the cases, users agreed that Idea-2-3D was better than baselines. Codes,
data and models will made publicly available.