LucidDreaming: Controllable Object-Centric 3D Generation
Zhaoning Wang Ming Li Chen Chen
With the recent development of generative models, Text-to-3D generations have
also seen significant growth. Nonetheless, achieving precise control over 3D
generation continues to be an arduous task, as using text to control often
leads to missing objects and imprecise locations. Contemporary strategies for
enhancing controllability in 3D generation often entail the introduction of
additional parameters, such as customized diffusion models. This often induces
hardness in adapting to different diffusion models or creating distinct
objects.
  In this paper, we present LucidDreaming as an effective pipeline capable of
fine-grained control over 3D generation. It requires only minimal input of 3D
bounding boxes, which can be deduced from a simple text prompt using a Large
Language Model. Specifically, we propose clipped ray sampling to separately
render and optimize objects with user specifications. We also introduce
object-centric density blob bias, fostering the separation of generated
objects. With individual rendering and optimizing of objects, our method excels
not only in controlled content generation from scratch but also within the
pre-trained NeRF scenes. In such scenarios, existing generative approaches
often disrupt the integrity of the original scene, and current editing methods
struggle to synthesize new content in empty spaces. We show that our method
exhibits remarkable adaptability across a spectrum of mainstream Score
Distillation Sampling-based 3D generation frameworks, and achieves superior
alignment of 3D content when compared to baseline approaches. We also provide a
dataset of prompts with 3D bounding boxes, benchmarking 3D spatial
controllability.