Recently popularized graph neural networks achieve the state-of-the-art accuracy on a number of standard benchmark datasets for graph-based semi-supervised learning, improving significantly over existing approaches. These architectures alternate between a propagation layer that aggregates the hidden states of the local neighborhood and a fully-connected layer. Perhaps surprisingly, we show that a linear model, that removes all the intermediate fully-connected layers, is still able to achieve a performance comparable to the state-of-the-art models. This significantly reduces the number of parameters, which is critical for semi-supervised learning where number of labeled examples are small. This in turn allows a room for designing more innovative propagation layers. Based on this insight, we propose a novel graph neural network that removes all the intermediate fully-connected layers, and replaces the propagation layers with attention mechanisms that respect the structure of the graph. The attention mechanism allows us to learn a dynamic and adaptive local summary of the neighborhood to achieve more accurate predictions. In a number of experiments on benchmark citation networks datasets, we demonstrate that our approach outperforms competing methods. By examining the attention weights among neighbors, we show that our model provides some interesting insights on how neighbors influence each other. One of the major bottlenecks in applying machine learning in practice is collecting sizable and reliable labeled data, essential for accurate predictions. One way to overcome the problem of limited labeled data is semi-supervised learning, using additional unlabeled data that might be freely available. In this paper, we are interested in a scenario when this additional unlabeled data is available in a form of a graph. The graph provides underlying pairwise relations among the data points, both labeled and unlabeled.Of particular interest are those applications where the presence or absence of an edge between two data points is determined by nature, for instance as a result of human activities or natural relations. As a concrete example, consider a citation network. Each node in the graph is a published research paper, associated with a bag-of-words feature vector. An (directed) edge indicates a citation link. Presence of an edge indicates that the authors of a paper have consciously determined to refer to the other paper, and hence captures some underlying relation that might not be inferred from the bag-of-words feature vectors alone. Such external graph data are available in several applications of interest, such as classifying users connected via a social network, items and customers connected by purchase history, users and movies connected by viewing history, and entities in a knowledge graph connected by relationships. In this paper, we are interested in the setting where the graph is explicitly given and represents additional information not present in the feature vectors.The goal of such graph-based semi-supervised learning problems is to classify the nodes in a graph using a small subset of labeled nodes and all the node features. There is a long line of literature on this topic since BID6 which seeks graph cuts that preserve the known labels and BID50 which uses graph Laplacian to regularize the nearby nodes to have similar labels. However, BID27 recently demonstrated that the existing approaches can be significantly improved upon on a number of standard benchmark datasets, using an innovative neural network architecture on graph-based data known collectively as graph neural networks.Inspired by this success, we seek to understand the reason behind the power of graph neural networks, to guide our design of a novel architecture for semi-supervised learning on graphs. To this end, we first found that a linear classifier of multinomial logistic regression achieves the accuracy comparable to the best known graph neural network. This linear classifier removes all intermediate non-linear activation layers, and only keeps the linear propagation function from neighbors in graph neural networks. This suggests the importance of aggregation information form the neighbors in the graph. This further motivates us to design a new way of aggregating neighborhood information through attention mechanism since, intuitively, neighbors might not be equally important. This proposed attention-based graph neural network captures this intuition and (a) greatly reduces the model complexity, with only a single scalar parameter at each intermediate layer; (b) discovers dynamically and adaptively which nodes are relevant to the target node for classification; and (c) improves upon state-of-the-art methods in terms of accuracy on standard benchmark datasets. Further, the learned attention strengths provide some form of interpretability. They provide insights on why a particular prediction is made on a target node and which neighbors are more relevant in making that decision. In this paper, we present an attention-based graph neural network model for semi-supervised classification on a graph. We demonstrate that our method consistently outperforms competing methods on the standard benchmark citation network datasets. We also show that the learned attention also provides interesting insights on how neighbors influence each other. In training, we have tried more
We propose a novel attention-based interpretable Graph Neural Network architecture which outperforms the current state-of-the-art Graph Neural Networks in standard benchmark datasets The authors propose two extensions of GCNs, by removing intermediate non-linearities from the GCN computation and adding an attention mechanism in the aggregation layer. The paper proposes a semi supervised learning algorithm for graph node classification with is inspired from Graph Neural Networks.