CADSim: Robust and Scalable in-the-wild 3D Reconstruction for
  Controllable Sensor Simulation
Jingkang Wang Sivabalan Manivasagam Yun Chen Ze Yang Ioan Andrei Bârsan Anqi Joyce Yang Wei-Chiu Ma Raquel Urtasun
Realistic simulation is key to enabling safe and scalable development of %
self-driving vehicles. A core component is simulating the sensors so that the
entire autonomy system can be tested in simulation. Sensor simulation involves
modeling traffic participants, such as vehicles, with high quality appearance
and articulated geometry, and rendering them in real time. The self-driving
industry has typically employed artists to build these assets. However, this is
expensive, slow, and may not reflect reality. Instead, reconstructing assets
automatically from sensor data collected in the wild would provide a better
path to generating a diverse and large set with good real-world coverage.
Nevertheless, current reconstruction approaches struggle on in-the-wild sensor
data, due to its sparsity and noise. To tackle these issues, we present CADSim,
which combines part-aware object-class priors via a small set of CAD models
with differentiable rendering to automatically reconstruct vehicle geometry,
including articulated wheels, with high-quality appearance. Our experiments
show our method recovers more accurate shapes from sparse data compared to
existing approaches. Importantly, it also trains and renders efficiently. We
demonstrate our reconstructed vehicles in several applications, including
accurate testing of autonomy perception systems.