3D Copy-Paste: Physically Plausible Object Insertion for Monocular 3D
  Detection
Yunhao Ge Hong-Xing Yu Cheng Zhao Yuliang Guo Xinyu Huang Liu Ren Laurent Itti Jiajun Wu
A major challenge in monocular 3D object detection is the limited diversity
and quantity of objects in real datasets. While augmenting real scenes with
virtual objects holds promise to improve both the diversity and quantity of the
objects, it remains elusive due to the lack of an effective 3D object insertion
method in complex real captured scenes. In this work, we study augmenting
complex real indoor scenes with virtual objects for monocular 3D object
detection. The main challenge is to automatically identify plausible physical
properties for virtual assets (e.g., locations, appearances, sizes, etc.) in
cluttered real scenes. To address this challenge, we propose a physically
plausible indoor 3D object insertion approach to automatically copy virtual
objects and paste them into real scenes. The resulting objects in scenes have
3D bounding boxes with plausible physical locations and appearances. In
particular, our method first identifies physically feasible locations and poses
for the inserted objects to prevent collisions with the existing room layout.
Subsequently, it estimates spatially-varying illumination for the insertion
location, enabling the immersive blending of the virtual objects into the
original scene with plausible appearances and cast shadows. We show that our
augmentation method significantly improves existing monocular 3D object models
and achieves state-of-the-art performance. For the first time, we demonstrate
that a physically plausible 3D object insertion, serving as a generative data
augmentation technique, can lead to significant improvements for discriminative
downstream tasks such as monocular 3D object detection. Project website:
https://gyhandy.github.io/3D-Copy-Paste/