Knowing Your Nonlinearities: Shapley Interactions Reveal the Underlying
  Structure of Data
Divyansh Singhvi Andrej Erkelens Raghav Jain Diganta Misra Naomi Saphra
Measuring nonlinear feature interaction is an established approach to
understanding complex patterns of attribution in many models. In this paper, we
use Shapley Taylor interaction indices (STII) to analyze the impact of
underlying data structure on model representations in a variety of modalities,
tasks, and architectures. Considering linguistic structure in masked and
auto-regressive language models (MLMs and ALMs), we find that STII increases
within idiomatic expressions and that MLMs scale STII with syntactic distance,
relying more on syntax in their nonlinear structure than ALMs do. Our speech
model findings reflect the phonetic principal that the openness of the oral
cavity determines how much a phoneme varies based on its context. Finally, we
study image classifiers and illustrate that feature interactions intuitively
reflect object boundaries. Our wide range of results illustrates the benefits
of interdisciplinary work and domain expertise in interpretability research.