Renovating Names in Open-Vocabulary Segmentation Benchmarks
Haiwen Huang Songyou Peng Dan Zhang Andreas Geiger
Names are essential to both human cognition and vision-language models.
Open-vocabulary models utilize class names as text prompts to generalize to
categories unseen during training. However, name qualities are often overlooked
and lack sufficient precision in existing datasets. In this paper, we address
this underexplored problem by presenting a framework for "renovating" names in
open-vocabulary segmentation benchmarks (RENOVATE). Through human study, we
demonstrate that the names generated by our model are more precise descriptions
of the visual segments and hence enhance the quality of existing datasets by
means of simple renaming. We further demonstrate that using our renovated names
enables training of stronger open-vocabulary segmentation models. Using
open-vocabulary segmentation for name quality evaluation, we show that our
renovated names lead to up to 16% relative improvement from the original names
on various benchmarks across various state-of-the-art models. We provide our
code and relabelings for several popular segmentation datasets (ADE20K,
Cityscapes, PASCAL Context) to the research community.