Deep neural networks are able to solve tasks across a variety of domains and modalities of data. Despite many empirical successes, we lack the ability to clearly understand and interpret the learned mechanisms that contribute to such effective behaviors and more critically, failure modes. In this work, we present a general method for visualizing an arbitrary neural network's inner mechanisms and their power and limitations. Our dataset-centric method produces visualizations of how a trained network attends to components of its inputs. The computed "attention masks" support improved interpretability by highlighting which input attributes are critical in determining output. We demonstrate the effectiveness of our framework on a variety of deep neural network architectures in domains from computer vision and natural language processing. The primary contribution of our approach is an interpretable visualization of attention that provides unique insights into the network's underlying decision-making process irrespective of the data modality. Machine-learning systems are ubiquitous, even in safety-critical areas. Trained models used in self-driving cars, healthcare, and environmental science must not only strive to be error free but, in the face of failures, must be amenable to rapid diagnosis and recovery. This trend toward realworld applications is largely being driven by recent advances in the area of deep learning. Deep neural networks have achieved state-of-the-art performance on fundamental domains such as image classification BID15 , language modeling BID2 BID20 , and reinforcement learning from raw pixels BID22 . Unlike traditional linear models, deep neural networks offer the significant advantage of being able to learn their own feature representation for the completion of a given task. While learning such a representation removes the need for manual feature engineering and generally boosts performance, the resulting models are often hard to interpret, making it significantly more difficult to assign credit (or blame) to the model's behaviors. The use of deep learning models in increasingly important application areas underscores the need for techniques to gain insight into their failure modes, limitations, and decision-making mechanisms.Substantial prior work investigates methods for increasing interpretability of these systems. One body of work focuses on visualizing various aspects of networks or their relationship to each datum they take as input BID33 ; BID35 . Other work investigates algorithms for eliciting an explanation from trained machine-learning systems for each decision they make BID26 ; BID0 ; BID27 . A third line of work, of which our method is most aligned, seeks to capture and understand what networks focus on and what they ignore through attention mechanisms.Attention-based approaches focus on network architectures that specifically attend to regions of their input space. These "explicit" attention mechanisms were developed primarily to improve network behavior, but additionally offer increased interpretability of network decision making through highlighting key attributes of the input data BID30 BID12 BID25 BID16 . Crucially, these explicit attention mechanisms act as filters on the input. As such, the filtered components of the input could be replaced with reasonably generated noise without dramatically affecting the final network output. The ability to selectively replace irrelevant components of the input space is a direct consequence of the explicit attention mechanism. The insight at the heart of the present work is that it is possible to evaluate the property of "selective replaceability" to better understand a network that lacks any explicit attention mechanism. An architecture without explicit attention may still depend more on specific facets of its input data when constructing its learned, internal representation, resulting in a "latent" attention mechanism.In this work, we propose a novel approach for indirectly measuring latent attention mechanisms in arbitrary neural networks using the notion of selective replaceability. Concretely, we learn an auxiliary, "Latent Attention Network" (LAN) , that consumes an input data sample and generates a corresponding mask (of the same shape) indicating the degree to which each of the input's components are replaceable with noise. We train this LAN by corrupting the inputs to a pre-trained network according to generated LAN masks and observing the resulting corrupted outputs. We define a loss function that trades off maximizing the corruption of the input while minimizing the deviation between the outputs generated by the pre-trained network using the true and corrupted inputs, independently. The resultant LAN masks must learn to identify the components of the input data that are most critical to producing the existing network's output (i.e. those regions that are given the most attention by the existing network.)We empirically demonstrate that the LAN framework can provide unique insights into the inner workings of various pre-trained networks. Specifically, we show that classifiers trained on a Translated MNIST domain learn a two-stage process of first localizing a digit within the image before determining its class. We use this interpretation to predict regions on the screen where digits are less likely to be properly classified. Additionally, we use our framework to visualize the latent attention mechanisms of classifiers on both image classification (to learn the visual features most important to the network's prediction), and natural language document classification domains (to identify the words most relevant to certain output classes). Finally, we examine techniques for generating attention masks for specific samples, illustrating the capability of our approach to highlight salient features in individual members of a dataset. As deep neural networks continue to find application to a growing collection of tasks, understanding their decision-making processes becomes increasingly important. Furthermore, as this space of tasks grows to include areas where there is a small margin for error, the ability to explore and diagnose problems within erroneous models becomes crucial.In this work, we proposed Latent Attention Networks as a framework for capturing the latent attention mechanisms of arbitrary neural networks that draws parallels between noise-based input corruption and attention. We have shown that the analysis of these attention measurements can effectively diagnose failure modes in pre-trained networks and provide unique perspectives on the mechanism by which arbitrary networks perform their designated tasks.We believe there are several interesting research directions that arise from our framework. First, there are interesting parallels between this work and the popular Generative Adversarial Networks BID9 . It may be possible to simultaneously train F and A as adversaries. Since both F and A are differentiable, one could potentially exploit this property and use A to encourage a specific attention mechanism on F , speeding up learning in challenging domains and otherwise allowing for novel interactions between deep networks. Furthermore, we explored two types of noise for input corruption: η const and η boot . It may be possible to make the process of generating noise a part of the network itself by learning a nonlinear transformation and applying it to some standard variety of noise (such as Normal or Uniform). Since our method depends on being able to sample noise that is similar to the "background noise" of the domain, better mechanisms for capturing noise could potentially enhance the LAN's ability to pick out regions of attention and eliminate the need for choosing a specific type of noise at design time. Doing so would allow the LAN to pick up more specific features of the input space that are relevant to the decision-making process of arbitrary classifier networks.In the following experiment subsections we describe network architectures by sequentially listing their layers using an abbreviated notation: DISPLAYFORM0 for convolutional, convolutional-transpose and fully connected layers respectively. In all network architectures, -ReLU denotes the leaky-ReLU BID18 .We now describe each experiment in greater detail.
We develop a technique to visualize attention mechanisms in arbitrary neural networks.  Proposes to learn a Latent Attention Network that can help to visualize the inner structure of a deep neural network. The authors of this paper propose a data-driven black-box visualization scheme. 