Unsupervised Open-Vocabulary Object Localization in Videos
Ke Fan Zechen Bai Tianjun Xiao Dominik Zietlow Max Horn Zixu Zhao Carl-Johann Simon-Gabriel Mike Zheng Shou Francesco Locatello Bernt Schiele Thomas Brox Zheng Zhang Yanwei Fu Tong He
In this paper, we show that recent advances in video representation learning
and pre-trained vision-language models allow for substantial improvements in
self-supervised video object localization. We propose a method that first
localizes objects in videos via a slot attention approach and then assigns text
to the obtained slots. The latter is achieved by an unsupervised way to read
localized semantic information from the pre-trained CLIP model. The resulting
video object localization is entirely unsupervised apart from the implicit
annotation contained in CLIP, and it is effectively the first unsupervised
approach that yields good results on regular video benchmarks.