A Real-time Method for Inserting Virtual Objects into Neural Radiance
  Fields
Keyang Ye Hongzhi Wu Xin Tong Kun Zhou
We present the first real-time method for inserting a rigid virtual object
into a neural radiance field, which produces realistic lighting and shadowing
effects, as well as allows interactive manipulation of the object. By
exploiting the rich information about lighting and geometry in a NeRF, our
method overcomes several challenges of object insertion in augmented reality.
For lighting estimation, we produce accurate, robust and 3D spatially-varying
incident lighting that combines the near-field lighting from NeRF and an
environment lighting to account for sources not covered by the NeRF. For
occlusion, we blend the rendered virtual object with the background scene using
an opacity map integrated from the NeRF. For shadows, with a precomputed field
of spherical signed distance field, we query the visibility term for any point
around the virtual object, and cast soft, detailed shadows onto 3D surfaces.
Compared with state-of-the-art techniques, our approach can insert virtual
object into scenes with superior fidelity, and has a great potential to be
further applied to augmented reality systems.