Representing visual classification as a linear combination of words
Shobhit Agarwal Yevgeniy R. Semenov William Lotter
Explainability is a longstanding challenge in deep learning, especially in
high-stakes domains like healthcare. Common explainability methods highlight
image regions that drive an AI model's decision. Humans, however, heavily rely
on language to convey explanations of not only "where" but "what".
Additionally, most explainability approaches focus on explaining individual AI
predictions, rather than describing the features used by an AI model in
general. The latter would be especially useful for model and dataset auditing,
and potentially even knowledge generation as AI is increasingly being used in
novel tasks. Here, we present an explainability strategy that uses a
vision-language model to identify language-based descriptors of a visual
classification task. By leveraging a pre-trained joint embedding space between
images and text, our approach estimates a new classification task as a linear
combination of words, resulting in a weight for each word that indicates its
alignment with the vision-based classifier. We assess our approach using two
medical imaging classification tasks, where we find that the resulting
descriptors largely align with clinical knowledge despite a lack of
domain-specific language training. However, our approach also identifies the
potential for 'shortcut connections' in the public datasets used. Towards a
functional measure of explainability, we perform a pilot reader study where we
find that the AI-identified words can enable non-expert humans to perform a
specialized medical task at a non-trivial level. Altogether, our results
emphasize the potential of using multimodal foundational models to deliver
intuitive, language-based explanations of visual tasks.