Optical Flow boosts Unsupervised Localization and Segmentation
Xinyu Zhang Abdeslam Boularias
Unsupervised localization and segmentation are long-standing robot vision
challenges that describe the critical ability for an autonomous robot to learn
to decompose images into individual objects without labeled data. These tasks
are important because of the limited availability of dense image manual
annotation and the promising vision of adapting to an evolving set of object
categories in lifelong learning. Most recent methods focus on using visual
appearance continuity as object cues by spatially clustering features obtained
from self-supervised vision transformers (ViT). In this work, we leverage
motion cues, inspired by the common fate principle that pixels that share
similar movements tend to belong to the same object. We propose a new loss term
formulation that uses optical flow in unlabeled videos to encourage
self-supervised ViT features to become closer to each other if their
corresponding spatial locations share similar movements, and vice versa. We use
the proposed loss function to finetune vision transformers that were originally
trained on static images. Our fine-tuning procedure outperforms
state-of-the-art techniques for unsupervised semantic segmentation through
linear probing, without the use of any labeled data. This procedure also
demonstrates increased performance over original ViT networks across
unsupervised object localization and semantic segmentation benchmarks.