GIM: Learning Generalizable Image Matcher From Internet Videos
Xuelun Shen Zhipeng Cai Wei Yin Matthias Müller Zijun Li Kaixuan Wang Xiaozhi Chen Cheng Wang
Image matching is a fundamental computer vision problem. While learning-based
methods achieve state-of-the-art performance on existing benchmarks, they
generalize poorly to in-the-wild images. Such methods typically need to train
separate models for different scene types and are impractical when the scene
type is unknown in advance. One of the underlying problems is the limited
scalability of existing data construction pipelines, which limits the diversity
of standard image matching datasets. To address this problem, we propose GIM, a
self-training framework for learning a single generalizable model based on any
image matching architecture using internet videos, an abundant and diverse data
source. Given an architecture, GIM first trains it on standard domain-specific
datasets and then combines it with complementary matching methods to create
dense labels on nearby frames of novel videos. These labels are filtered by
robust fitting, and then enhanced by propagating them to distant frames. The
final model is trained on propagated data with strong augmentations. We also
propose ZEB, the first zero-shot evaluation benchmark for image matching. By
mixing data from diverse domains, ZEB can thoroughly assess the cross-domain
generalization performance of different methods. Applying GIM consistently
improves the zero-shot performance of 3 state-of-the-art image matching
architectures; with 50 hours of YouTube videos, the relative zero-shot
performance improves by 8.4%-18.1%. GIM also enables generalization to extreme
cross-domain data such as Bird Eye View (BEV) images of projected 3D point
clouds (Fig. 1(c)). More importantly, our single zero-shot model consistently
outperforms domain-specific baselines when evaluated on downstream tasks
inherent to their respective domains. The video presentation is available at
https://www.youtube.com/watch?v=FU_MJLD8LeY.