Virtual Occlusions Through Implicit Depth
Jamie Watson Mohamed Sayed Zawar Qureshi Gabriel J. Brostow Sara Vicente Oisin Mac Aodha Michael Firman
For augmented reality (AR), it is important that virtual assets appear to
`sit among' real world objects. The virtual element should variously occlude
and be occluded by real matter, based on a plausible depth ordering. This
occlusion should be consistent over time as the viewer's camera moves.
Unfortunately, small mistakes in the estimated scene depth can ruin the
downstream occlusion mask, and thereby the AR illusion. Especially in real-time
settings, depths inferred near boundaries or across time can be inconsistent.
In this paper, we challenge the need for depth-regression as an intermediate
step.
  We instead propose an implicit model for depth and use that to predict the
occlusion mask directly. The inputs to our network are one or more color
images, plus the known depths of any virtual geometry. We show how our
occlusion predictions are more accurate and more temporally stable than
predictions derived from traditional depth-estimation models. We obtain
state-of-the-art occlusion results on the challenging ScanNetv2 dataset and
superior qualitative results on real scenes.