Correspondences of the Third Kind: Camera Pose Estimation from Object
  Reflection
Kohei Yamashita Vincent Lepetit Ko Nishino
Computer vision has long relied on two kinds of correspondences: pixel
correspondences in images and 3D correspondences on object surfaces. Is there
another kind, and if there is, what can they do for us? In this paper, we
introduce correspondences of the third kind we call reflection correspondences
and show that they can help estimate camera pose by just looking at objects
without relying on the background. Reflection correspondences are point
correspondences in the reflected world, i.e., the scene reflected by the object
surface. The object geometry and reflectance alters the scene geometrically and
radiometrically, respectively, causing incorrect pixel correspondences.
Geometry recovered from each image is also hampered by distortions, namely
generalized bas-relief ambiguity, leading to erroneous 3D correspondences. We
show that reflection correspondences can resolve the ambiguities arising from
these distortions. We introduce a neural correspondence estimator and a RANSAC
algorithm that fully leverages all three kinds of correspondences for robust
and accurate joint camera pose and object shape estimation just from the object
appearance. The method expands the horizon of numerous downstream tasks,
including camera pose estimation for appearance modeling (e.g., NeRF) and
motion estimation of reflective objects (e.g., cars on the road), to name a
few, as it relieves the requirement of overlapping background.