Textual Knowledge Matters: Cross-Modality Co-Teaching for Generalized
  Visual Class Discovery
Haiyang Zheng Nan Pu Wenjing Li Nicu Sebe Zhun Zhong
In this paper, we study the problem of Generalized Category Discovery (GCD),
which aims to cluster unlabeled data from both known and unknown categories
using the knowledge of labeled data from known categories. Current GCD methods
rely on only visual cues, which however neglect the multi-modality perceptive
nature of human cognitive processes in discovering novel visual categories. To
address this, we propose a two-phase TextGCD framework to accomplish
multi-modality GCD by exploiting powerful Visual-Language Models. TextGCD
mainly includes a retrieval-based text generation (RTG) phase and a
cross-modality co-teaching (CCT) phase. First, RTG constructs a visual lexicon
using category tags from diverse datasets and attributes from Large Language
Models, generating descriptive texts for images in a retrieval manner. Second,
CCT leverages disparities between textual and visual modalities to foster
mutual learning, thereby enhancing visual GCD. In addition, we design an
adaptive class aligning strategy to ensure the alignment of category
perceptions between modalities as well as a soft-voting mechanism to integrate
multi-modality cues. Experiments on eight datasets show the large superiority
of our approach over state-of-the-art methods. Notably, our approach
outperforms the best competitor, by 7.7% and 10.8% in All accuracy on
ImageNet-1k and CUB, respectively.