Training-Free Semantic Segmentation via LLM-Supervision
Wenfang Sun Yingjun Du Gaowen Liu Ramana Kompella Cees G. M. Snoek
Recent advancements in open vocabulary models, like CLIP, have notably
advanced zero-shot classification and segmentation by utilizing natural
language for class-specific embeddings. However, most research has focused on
improving model accuracy through prompt engineering, prompt learning, or
fine-tuning with limited labeled data, thereby overlooking the importance of
refining the class descriptors. This paper introduces a new approach to
text-supervised semantic segmentation using supervision by a large language
model (LLM) that does not require extra training. Our method starts from an
LLM, like GPT-3, to generate a detailed set of subclasses for more accurate
class representation. We then employ an advanced text-supervised semantic
segmentation model to apply the generated subclasses as target labels,
resulting in diverse segmentation results tailored to each subclass's unique
characteristics. Additionally, we propose an assembly that merges the
segmentation maps from the various subclass descriptors to ensure a more
comprehensive representation of the different aspects in the test images.
Through comprehensive experiments on three standard benchmarks, our method
outperforms traditional text-supervised semantic segmentation methods by a
marked margin.