The Sky's the Limit: Re-lightable Outdoor Scenes via a Sky-pixel
  Constrained Illumination Prior and Outside-In Visibility
James A. D. Gardner Evgenii Kashin Bernhard Egger William A. P. Smith
Inverse rendering of outdoor scenes from unconstrained image collections is a
challenging task, particularly illumination/albedo ambiguities and occlusion of
the illumination environment (shadowing) caused by geometry. However, there are
many cues in an image that can aid in the disentanglement of geometry, albedo
and shadows. We exploit the fact that any sky pixel provides a direct
measurement of distant lighting in the corresponding direction and, via a
neural illumination prior, a statistical cue as to the remaining illumination
environment. We also introduce a novel `outside-in' method for computing
differentiable sky visibility based on a neural directional distance function.
This is efficient and can be trained in parallel with the neural scene
representation, allowing gradients from appearance loss to flow from shadows to
influence estimation of illumination and geometry. Our method estimates
high-quality albedo, geometry, illumination and sky visibility, achieving
state-of-the-art results on the NeRF-OSR relighting benchmark. Our code and
models can be found https://github.com/JADGardner/neusky