Rethinking Evaluation Metrics of Open-Vocabulary Segmentaion
Hao Zhou Tiancheng Shen Xu Yang Hai Huang Xiangtai Li Lu Qi Ming-Hsuan Yang
In this paper, we highlight a problem of evaluation metrics adopted in the
open-vocabulary segmentation. That is, the evaluation process still heavily
relies on closed-set metrics on zero-shot or cross-dataset pipelines without
considering the similarity between predicted and ground truth categories. To
tackle this issue, we first survey eleven similarity measurements between two
categorical words using WordNet linguistics statistics, text embedding, and
language models by comprehensive quantitative analysis and user study. Built
upon those explored measurements, we designed novel evaluation metrics, namely
Open mIoU, Open AP, and Open PQ, tailored for three open-vocabulary
segmentation tasks. We benchmarked the proposed evaluation metrics on 12
open-vocabulary methods of three segmentation tasks. Even though the relative
subjectivity of similarity distance, we demonstrate that our metrics can still
well evaluate the open ability of the existing open-vocabulary segmentation
methods. We hope that our work can bring with the community new thinking about
how to evaluate the open ability of models. The evaluation code is released in
github.