Improving Semantic Correspondence with Viewpoint-Guided Spherical Maps
Octave Mariotti Oisin Mac Aodha Hakan Bilen
Recent progress in self-supervised representation learning has resulted in
models that are capable of extracting image features that are not only
effective at encoding image level, but also pixel-level, semantics. These
features have been shown to be effective for dense visual semantic
correspondence estimation, even outperforming fully-supervised methods.
Nevertheless, current self-supervised approaches still fail in the presence of
challenging image characteristics such as symmetries and repeated parts. To
address these limitations, we propose a new approach for semantic
correspondence estimation that supplements discriminative self-supervised
features with 3D understanding via a weak geometric spherical prior. Compared
to more involved 3D pipelines, our model only requires weak viewpoint
information, and the simplicity of our spherical representation enables us to
inject informative geometric priors into the model during training. We propose
a new evaluation metric that better accounts for repeated part and
symmetry-induced mistakes. We present results on the challenging SPair-71k
dataset, where we show that our approach demonstrates is capable of
distinguishing between symmetric views and repeated parts across many object
categories, and also demonstrate that we can generalize to unseen classes on
the AwA dataset.