Neural Clustering based Visual Representation Learning
Guikun Chen Xia Li Yi Yang Wenguan Wang
We investigate a fundamental aspect of machine vision: the measurement of
features, by revisiting clustering, one of the most classic approaches in
machine learning and data analysis. Existing visual feature extractors,
including ConvNets, ViTs, and MLPs, represent an image as rectangular regions.
Though prevalent, such a grid-style paradigm is built upon engineering practice
and lacks explicit modeling of data distribution. In this work, we propose
feature extraction with clustering (FEC), a conceptually elegant yet
surprisingly ad-hoc interpretable neural clustering framework, which views
feature extraction as a process of selecting representatives from data and thus
automatically captures the underlying data distribution. Given an image, FEC
alternates between grouping pixels into individual clusters to abstract
representatives and updating the deep features of pixels with current
representatives. Such an iterative working mechanism is implemented in the form
of several neural layers and the final representatives can be used for
downstream tasks. The cluster assignments across layers, which can be viewed
and inspected by humans, make the forward process of FEC fully transparent and
empower it with promising ad-hoc interpretability. Extensive experiments on
various visual recognition models and tasks verify the effectiveness,
generality, and interpretability of FEC. We expect this work will provoke a
rethink of the current de facto grid-style paradigm.