The Pursuit of Human Labeling: A New Perspective on Unsupervised
  Learning
Artyom Gadetsky Maria Brbic
We present HUME, a simple model-agnostic framework for inferring human
labeling of a given dataset without any external supervision. The key insight
behind our approach is that classes defined by many human labelings are
linearly separable regardless of the representation space used to represent a
dataset. HUME utilizes this insight to guide the search over all possible
labelings of a dataset to discover an underlying human labeling. We show that
the proposed optimization objective is strikingly well-correlated with the
ground truth labeling of the dataset. In effect, we only train linear
classifiers on top of pretrained representations that remain fixed during
training, making our framework compatible with any large pretrained and
self-supervised model. Despite its simplicity, HUME outperforms a supervised
linear classifier on top of self-supervised representations on the STL-10
dataset by a large margin and achieves comparable performance on the CIFAR-10
dataset. Compared to the existing unsupervised baselines, HUME achieves
state-of-the-art performance on four benchmark image classification datasets
including the large-scale ImageNet-1000 dataset. Altogether, our work provides
a fundamentally new view to tackle unsupervised learning by searching for
consistent labelings between different representation spaces.