No Representation Rules Them All in Category Discovery
Sagar Vaze Andrea Vedaldi Andrew Zisserman
In this paper we tackle the problem of Generalized Category Discovery (GCD).
Specifically, given a dataset with labelled and unlabelled images, the task is
to cluster all images in the unlabelled subset, whether or not they belong to
the labelled categories. Our first contribution is to recognize that most
existing GCD benchmarks only contain labels for a single clustering of the
data, making it difficult to ascertain whether models are using the available
labels to solve the GCD task, or simply solving an unsupervised clustering
problem. As such, we present a synthetic dataset, named 'Clevr-4', for category
discovery. Clevr-4 contains four equally valid partitions of the data, i.e
based on object shape, texture, color or count. To solve the task, models are
required to extrapolate the taxonomy specified by the labelled set, rather than
simply latching onto a single natural grouping of the data. We use this dataset
to demonstrate the limitations of unsupervised clustering in the GCD setting,
showing that even very strong unsupervised models fail on Clevr-4. We further
use Clevr-4 to examine the weaknesses of existing GCD algorithms, and propose a
new method which addresses these shortcomings, leveraging consistent findings
from the representation learning literature to do so. Our simple solution,
which is based on 'mean teachers' and termed $\mu$GCD, substantially
outperforms implemented baselines on Clevr-4. Finally, when we transfer these
findings to real data on the challenging Semantic Shift Benchmark (SSB), we
find that $\mu$GCD outperforms all prior work, setting a new state-of-the-art.
For the project webpage, see https://www.robots.ox.ac.uk/~vgg/data/clevr4/