Do Concept Bottleneck Models Obey Locality?
Naveen Raman Mateo Espinosa Zarlenga Juyeon Heo Mateja Jamnik
Concept-based learning improves a deep learning model's interpretability by
explaining its predictions via human-understandable concepts. Deep learning
models trained under this paradigm heavily rely on the assumption that neural
networks can learn to predict the presence or absence of a given concept
independently of other concepts. Recent work, however, strongly suggests that
this assumption may fail to hold in Concept Bottleneck Models (CBMs), a
quintessential family of concept-based interpretable architectures. In this
paper, we investigate whether CBMs correctly capture the degree of conditional
independence across concepts when such concepts are localised both spatially,
by having their values entirely defined by a fixed subset of features, and
semantically, by having their values correlated with only a fixed subset of
predefined concepts. To understand locality, we analyse how changes to features
outside of a concept's spatial or semantic locality impact concept predictions.
Our results suggest that even in well-defined scenarios where the presence of a
concept is localised to a fixed feature subspace, or whose semantics are
correlated to a small subset of other concepts, CBMs fail to learn this
locality. These results cast doubt upon the quality of concept representations
learnt by CBMs and strongly suggest that concept-based explanations may be
fragile to changes outside their localities.