Region-Based Representations Revisited
Michal Shlapentokh-Rothman Ansel Blume Yao Xiao Yuqun Wu Sethuraman T V Heyi Tao Jae Yong Lee Wilfredo Torres Yu-Xiong Wang Derek Hoiem
We investigate whether region-based representations are effective for
recognition. Regions were once a mainstay in recognition approaches, but pixel
and patch-based features are now used almost exclusively. We show that recent
class-agnostic segmenters like SAM can be effectively combined with strong
unsupervised representations like DINOv2 and used for a wide variety of tasks,
including semantic segmentation, object-based image retrieval, and multi-image
analysis. Once the masks and features are extracted, these representations,
even with linear decoders, enable competitive performance, making them well
suited to applications that require custom queries. The compactness of the
representation also makes it well-suited to video analysis and other problems
requiring inference across many images.