Supervised Knowledge May Hurt Novel Class Discovery Performance
Ziyun Li Jona Otholt Ben Dai Di Hu Christoph Meinel Haojin Yang
Novel class discovery (NCD) aims to infer novel categories in an unlabeled
dataset by leveraging prior knowledge of a labeled set comprising disjoint but
related classes. Given that most existing literature focuses primarily on
utilizing supervised knowledge from a labeled set at the methodology level,
this paper considers the question: Is supervised knowledge always helpful at
different levels of semantic relevance? To proceed, we first establish a novel
metric, so-called transfer flow, to measure the semantic similarity between
labeled/unlabeled datasets. To show the validity of the proposed metric, we
build up a large-scale benchmark with various degrees of semantic similarities
between labeled/unlabeled datasets on ImageNet by leveraging its hierarchical
class structure. The results based on the proposed benchmark show that the
proposed transfer flow is in line with the hierarchical class structure; and
that NCD performance is consistent with the semantic similarities (measured by
the proposed metric). Next, by using the proposed transfer flow, we conduct
various empirical experiments with different levels of semantic similarity,
yielding that supervised knowledge may hurt NCD performance. Specifically,
using supervised information from a low-similarity labeled set may lead to a
suboptimal result as compared to using pure self-supervised knowledge. These
results reveal the inadequacy of the existing NCD literature which usually
assumes that supervised knowledge is beneficial. Finally, we develop a
pseudo-version of the transfer flow as a practical reference to decide if
supervised knowledge should be used in NCD. Its effectiveness is supported by
our empirical studies, which show that the pseudo transfer flow (with or
without supervised knowledge) is consistent with the corresponding accuracy
based on various datasets. Code is released at
https://github.com/J-L-O/SK-Hurt-NCD