Deep Neural Networks (DNNs) excel on many complex perceptual tasks but it has proven notoriously difficult to understand how they reach their decisions. We here introduce a high-performance DNN architecture on ImageNet whose decisions are considerably easier to explain. Our model, a simple variant of the ResNet-50 architecture called BagNet, classifies an image based on the occurrences of small local image features without taking into account their spatial ordering. This strategy is closely related to the bag-of-feature (BoF) models popular before the onset of deep learning and reaches a surprisingly high accuracy on ImageNet (87.6% top-5 for 32 x 32 px features and Alexnet performance for 16 x16 px features). The constraint on local features makes it straight-forward to analyse how exactly each part of the image influences the classification. Furthermore, the BagNets behave similar to state-of-the art deep neural networks such as VGG-16, ResNet-152 or DenseNet-169 in terms of feature sensitivity, error distribution and interactions between image parts. This suggests that the improvements of DNNs over previous bag-of-feature classifiers in the last few years is mostly achieved by better fine-tuning rather than by qualitatively different decision strategies. A big obstacle in understanding the decision-making of DNNs is due to the complex dependencies between input and hidden activations: for one, the effect of any part of the input on a hidden activation depends on the state of many other parts of the input. Likewise, the role of a hidden unit on downstream representations depends on the activity of many other units. This dependency makes it extremely difficult to understand how DNNs reach their decisions.To circumvent this problem we here formulate a new DNN architecture that is easier to interpret by design. Our architecture is inspired by bag-of-feature (BoF) models which -alongside extensions such as VLAD encoding or Fisher Vectors -have been the most successful approaches to large-scale object recognition before the advent of deep learning (up to 75% top-5 on ImageNet) and classify images based on the counts, but not the spatial relationships, of a set of local image features. This structure makes the decisions of BoF models particularly easy to explain.To be concise, throughout this manuscript the concept of interpretability refers to the way in which evidence from small image patches is integrated to reach an image-level decision. While basic BoF models perform just a simple and transparent spatial aggregate of the patch-wise evidences, DNNs non-linearly integrate information across the whole image.In this paper we show that it is possible to combine the performance and flexibility of DNNs with the interpretability of BoF models, and that the resulting model family (called BagNets) is able to reach high accuracy on ImageNet even if limited to fairly small image patches. Given the simplicity of BoF models we imagine many use cases for which it can be desirable to trade a bit of accuracy for better interpretability, just as this is common e.g. for linear function approximation. This includes diagnosing failure cases (e.g. adversarial examples) or non-iid. settings (e.g. domain transfer), benchmarking diagnostic tools (e.g. attribution methods) or serving as interpretable parts of a computer vision pipeline (e.g. with a relational network on top of the local features).In addition, we demonstrate similarities between the decision-making behaviour of BagNets and popular DNNs in computer vision. These similarities suggest that current network architectures base their decisions on a large number of relatively weak and local statistical regularities and are not sufficiently encouraged -either through their architecture, training procedure or task specification -to learn more holistic features that can better appreciate causal relationships between different parts of the image. In this paper we introduced and analysed a novel interpretable DNN architecture -coined BagNets -that classifies images based on linear bag-of-local-features representations. The results demonstrate that even complex perceptual tasks like ImageNet can be solved just based on small image features and without any notion of spatial relationships. In addition we showed that the key properties of BagNets, in particlar invariance to spatial relationships as well as weak interactions between image features, are also present to varying degrees in many common computer vision models like ResNet-50 Table 1 : Average probability of leading class after masking the 100 patches (8 × 8 pixels) with the highest attribution according to different heatmaps (columns).or VGG-16, suggesting that the decision-making of many DNNs trained on ImageNet follows at least in part a similar bag-of-feature strategy. In contrast to the perceived "leap" in performance from bag-of-feature models to deep neural networks, the representations learnt by DNNs may in the end still be similar to the pre-deep learning era.VGG-16 is particularly close to bag-of-feature models, as demonstrated by the weak interactions (Figure 6 ) and the sensitivity to the same small image patches as BagNets (Figure 8 ). Deeper networks, on the other hand, exhibit stronger nonlinear interactions between image parts and are less sensitive to local maskings. This might explain why texturisation (Figure 5 ) works well in VGG-16 but fails for ResNet-and DenseNet-architectures.Clearly, ImageNet alone is not sufficient to force DNNs to learn more physical and causal representation of the world -simply because such a representation is not necessary to solve the task (local image features are enough). This might explain why DNNs generalise poorly to distribution shifts: a DNN trained on natural images has learnt to recognize the textures and local image features associated with different objects (like the fur and eyes of a cat or the keys of a typewriter) and will inevitably fail if presented with cartoon-like images as they lack the key local image features upon which it bases its decisions.One way forward is to define novel tasks that cannot be solved using local statistical regularities. Here the BagNets can serve as a way to evaluate a lower-bound on the task performance as a function of the observable length-scales. Furthermore, BagNets can be an interesting tool in any application in which it is desirable to trade some accuracy for better interpretability. For example, BagNets can make it much easier to spot the relevant spatial locations and image features that are predictive of certain diseases in medical imaging. Likewise, they can serve as diagnostic tools to benchmark feature attribution techniques since ground-truth attributions are directly available. BagNets can also serve as interpretable parts of a larger computer vision pipeline (e.g. in autonomous cars) as they make it easier to understand edge and failure cases. We released the pretrained BagNets (BagNet-9, BagNet-17 and BagNet-33) for PyTorch and Keras at https://github.com/wielandbrendel/ bag-of-local-features-models.Taken together, DNNs might be more powerful than previous hand-tuned bag-of-feature algorithms in discovering weak statistical regularities, but that does not necessarily mean that they learn substantially different representations. We hope that this work will encourage and inspire future work to adapt tasks, architectures and training algorithms to encourage models to learn more causal models of the world.A APPENDIXThe architecture of the BagNets is detailed in Figure A. 1. Training of the models was performed in PyTorch using the default ImageNet training script of Torchvision (https://github.com/ pytorch/vision, commit 8a4786a) with default parameters. In brief, we used SGD with momentum (0.9), a batchsize of 256 and an initial learning rate of 0.01 which we decreased by a factor of 10 every 30 epochs. Images were resized to 256 pixels (shortest side) after which we extracted a random crop of size 224 × 224 pixels.
Aggregating class evidence from many small image patches suffices to solve ImageNet, yields more interpretable models and can explain aspects of the decision-making of popular DNNs. This paper suggests a novel and compact neural network architecture which uses the information within bag-of-words features. The proposed algorithm only uses the patch information independently and performs majority voting using independently classified patches.