LoCUS: Learning Multiscale 3D-consistent Features from Posed Images
Dominik A. Kloepfer Dylan Campbell João F. Henriques
An important challenge for autonomous agents such as robots is to maintain a
spatially and temporally consistent model of the world. It must be maintained
through occlusions, previously-unseen views, and long time horizons (e.g., loop
closure and re-identification). It is still an open question how to train such
a versatile neural representation without supervision. We start from the idea
that the training objective can be framed as a patch retrieval problem: given
an image patch in one view of a scene, we would like to retrieve (with high
precision and recall) all patches in other views that map to the same
real-world location. One drawback is that this objective does not promote
reusability of features: by being unique to a scene (achieving perfect
precision/recall), a representation will not be useful in the context of other
scenes. We find that it is possible to balance retrieval and reusability by
constructing the retrieval set carefully, leaving out patches that map to
far-away locations. Similarly, we can easily regulate the scale of the learned
features (e.g., points, objects, or rooms) by adjusting the spatial tolerance
for considering a retrieval to be positive. We optimize for (smooth) Average
Precision (AP), in a single unified ranking-based objective. This objective
also doubles as a criterion for choosing landmarks or keypoints, as patches
with high AP. We show results creating sparse, multi-scale, semantic spatial
maps composed of highly identifiable landmarks, with applications in landmark
retrieval, localization, semantic segmentation and instance segmentation.