DeepShaRM: Multi-View Shape and Reflectance Map Recovery Under Unknown
  Lighting
Kohei Yamashita Shohei Nobuhara Ko Nishino
Geometry reconstruction of textureless, non-Lambertian objects under unknown
natural illumination (i.e., in the wild) remains challenging as correspondences
cannot be established and the reflectance cannot be expressed in simple
analytical forms. We derive a novel multi-view method, DeepShaRM, that achieves
state-of-the-art accuracy on this challenging task. Unlike past methods that
formulate this as inverse-rendering, i.e., estimation of reflectance,
illumination, and geometry from images, our key idea is to realize that
reflectance and illumination need not be disentangled and instead estimated as
a compound reflectance map. We introduce a novel deep reflectance map
estimation network that recovers the camera-view reflectance maps from the
surface normals of the current geometry estimate and the input multi-view
images. The network also explicitly estimates per-pixel confidence scores to
handle global light transport effects. A deep shape-from-shading network then
updates the geometry estimate expressed with a signed distance function using
the recovered reflectance maps. By alternating between these two, and, most
important, by bypassing the ill-posed problem of reflectance and illumination
decomposition, the method accurately recovers object geometry in these
challenging settings. Extensive experiments on both synthetic and real-world
data clearly demonstrate its state-of-the-art accuracy.