Feature Accentuation: Revealing 'What' Features Respond to in Natural
  Images
Chris Hamblin Thomas Fel Srijani Saha Talia Konkle George Alvarez
Efforts to decode neural network vision models necessitate a comprehensive
grasp of both the spatial and semantic facets governing feature responses
within images. Most research has primarily centered around attribution methods,
which provide explanations in the form of heatmaps, showing where the model
directs its attention for a given feature. However, grasping 'where' alone
falls short, as numerous studies have highlighted the limitations of those
methods and the necessity to understand 'what' the model has recognized at the
focal point of its attention. In parallel, 'Feature visualization' offers
another avenue for interpreting neural network features. This approach
synthesizes an optimal image through gradient ascent, providing clearer
insights into 'what' features respond to. However, feature visualizations only
provide one global explanation per feature; they do not explain why features
activate for particular images. In this work, we introduce a new method to the
interpretability tool-kit, 'feature accentuation', which is capable of
conveying both where and what in arbitrary input images induces a feature's
response. At its core, feature accentuation is image-seeded (rather than
noise-seeded) feature visualization. We find a particular combination of
parameterization, augmentation, and regularization yields naturalistic
visualizations that resemble the seed image and target feature simultaneously.
Furthermore, we validate these accentuations are processed along a natural
circuit by the model. We make our precise implementation of feature
accentuation available to the community as the Faccent library, an extension of
Lucent.