Measuring Feature Dependency of Neural Networks by Collapsing Feature
  Dimensions in the Data Manifold
Yinzhu Jin Matthew B. Dwyer P. Thomas Fletcher
This paper introduces a new technique to measure the feature dependency of
neural network models. The motivation is to better understand a model by
querying whether it is using information from human-understandable features,
e.g., anatomical shape, volume, or image texture. Our method is based on the
principle that if a model is dependent on a feature, then removal of that
feature should significantly harm its performance. A targeted feature is
"removed" by collapsing the dimension in the data distribution that corresponds
to that feature. We perform this by moving data points along the feature
dimension to a baseline feature value while staying on the data manifold, as
estimated by a deep generative model. Then we observe how the model's
performance changes on the modified test data set, with the target feature
dimension removed. We test our method on deep neural network models trained on
synthetic image data with known ground truth, an Alzheimer's disease prediction
task using MRI and hippocampus segmentations from the OASIS-3 dataset, and a
cell nuclei classification task using the Lizard dataset.