Observable Propagation: A Data-Efficient Approach to Uncover Feature
  Vectors in Transformers
Jacob Dunefsky Arman Cohan
A key goal of current mechanistic interpretability research in NLP is to find
linear features (also called "feature vectors") for transformers: directions in
activation space corresponding to concepts that are used by a given model in
its computation. Present state-of-the-art methods for finding linear features
require large amounts of labelled data -- both laborious to acquire and
computationally expensive to utilize. In this work, we introduce a novel
method, called "observable propagation" (in short: ObsProp), for finding linear
features used by transformer language models in computing a given task -- using
almost no data. Our paradigm centers on the concept of observables, linear
functionals corresponding to given tasks. We then introduce a mathematical
theory for the analysis of feature vectors: we provide theoretical motivation
for why LayerNorm nonlinearities do not affect the direction of feature
vectors; we also introduce a similarity metric between feature vectors called
the coupling coefficient which estimates the degree to which one feature's
output correlates with another's. We use ObsProp to perform extensive
qualitative investigations into several tasks, including gendered occupational
bias, political party prediction, and programming language detection. Our
results suggest that ObsProp surpasses traditional approaches for finding
feature vectors in the low-data regime, and that ObsProp can be used to better
understand the mechanisms responsible for bias in large language models. Code
for experiments can be found at github.com/jacobdunefsky/ObservablePropagation.