CEIR: Concept-based Explainable Image Representation Learning
Yan Cui Shuhong Liu Liuzhuozheng Li Zhiyuan Yuan
In modern machine learning, the trend of harnessing self-supervised learning
to derive high-quality representations without label dependency has garnered
significant attention. However, the absence of label information, coupled with
the inherently high-dimensional nature, improves the difficulty for the
interpretation of learned representations. Consequently, indirect evaluations
become the popular metric for evaluating the quality of these features, leading
to a biased validation of the learned representation rationale. To address
these challenges, we introduce a novel approach termed Concept-based
Explainable Image Representation (CEIR). Initially, using the Concept-based
Model (CBM) incorporated with pretrained CLIP and concepts generated by GPT-4,
we project input images into a concept vector space. Subsequently, a
Variational Autoencoder (VAE) learns the latent representation from these
projected concepts, which serves as the final image representation. Due to the
capability of the representation to encapsulate high-level, semantically
relevant concepts, the model allows for attributions to a human-comprehensible
concept space. This not only enhances interpretability but also preserves the
robustness essential for downstream tasks. For instance, our method exhibits
state-of-the-art unsupervised clustering performance on benchmarks such as
CIFAR10, CIFAR100, and STL10. Furthermore, capitalizing on the universality of
human conceptual understanding, CEIR can seamlessly extract the related concept
from open-world images without fine-tuning. This offers a fresh approach to
automatic label generation and label manipulation.