Different kinds of representation learning techniques on graph have shown significant effect in downstream machine learning tasks. Recently, in order to inductively learn representations for graph structures that is unobservable during training, a general framework with sampling and aggregating (GraphSAGE) was proposed by Hamilton and Ying and had been proved more efficient than transductive methods on fileds like transfer learning or evolving dataset. However, GraphSAGE is uncapable of selective neighbor sampling and lack of memory of known nodes that've been trained. To address these problems, we present an unsupervised method that samples neighborhood information attended by co-occurring structures and optimizes a trainable global bias as a representation expectation for each node in the given graph. Experiments show that our approach outperforms the state-of-the-art inductive and unsupervised methods for representation learning on graphs. Graphs and networks, e.g., social network analysis BID7 , molecule screening BID4 , knowledge base reasoning BID19 , and biological proteinprotein networks BID24 ), emerge in many real-world applications. Learning low-dimensional vector embeddings of nodes in large graphs has been proved effective for a wide variety of prediction and graph analysis tasks BID5 ; BID18 ). The high-level idea of node embedding is to explore high-dimensional information about the neighborhood of a node with a dense vector embedding, which can be fed to off-the-shelf machine learning approaches to tasks such as node classification and link prediction BID14 ).Whereas previous approaches BID14 ; BID5 ; BID18 ) can transductively learn embeddings on graphs, without re-training they cannot generalize to new nodes that are newly added to graphs. It is ubiquitous in real-world evolving networks, e.g., new users joining in a social friendship circle such as facebook. To address the problem , BID8 propose an approach, namely GraphSAGE, to leverage node feature information (e.g., text attributes) to efficiently generate node embeddings for previously unseen nodes. Despite the success of GraphSAGE, it randomly and uniformly samples neighbors of nodes, which suggests it is difficult to explore the most useful neighbor nodes. It could be helpful if we can take advantage of the most relevant neighbors and ignore irrelevant neighbors of the target node. Besides, GraphSAGE only focuses on training parameters of the hierarchical aggregator functions, but lose sight of preserving the memory of the training nodes, which means when training is finished, those nodes that have been trained over and over again would still be treated like unseen nodes, which causes a huge waste.To address the first issue, inspired by GAT BID20 ), a supervised approach that assigns different weights to all neighbors of each node in each aggregating layer, we introduce a bi-attention architecture BID16 ) to perform selective neighbor sampling in unsupervised learning scenarios. In unsupervised representation learning, when encoding embeddings of a positive 1 node pair before calculating their proximity loss BID7 ), we assume that neighbor nodes positive to both of the pair should have larger chance to be selected, since they are statistically more relevant to the current positive pair than other neighbors. For example, when embedding words like "mouse", in FIG0 , it's more reasonable to choose "keyboard" rather than "cat" as sampled neighbor while maximizing co-occrrence probability between "mouse" and "PC", because "keyboard" also tends to co-occurr with "PC", which means its imformation should be more relevant. We thus stack a bi-attention architecture BID16 ) on representations aggregated from both side in a positive node pair. In this way, we learn the most relevant representations for each positive node pair corresponding to their most relevant neighbors, and simply use a fixed-size uniform sampling which allows us to efficiently generate node embeddings in batches.To address the second issue, we combine the idea behind transductive approaches and inductive approaches, by intuitively applying an additive global embedding bias to each node's aggregated embedding. The global embedding biases are trainable as well as parameters of aggregator functions and can be considered as a memorable global identification of each node in training sets. When the training is completed, we generate the embedding for each node by calculating an average of multiple embedding outputs corresponding to different sampled neighbors with respect to different positive nodes. In this way, nodes that tend to co-occur in short random-walks will have more similar embeddings based on our bi-attention mechanism.Based on the above-mentioned two techniques, we propose a novel approach, called BIGSAGE (which stands for the BI-attention architeture, global BIas and the original framework GraphSAGE,) to explore most relevant neighbors and preserve previously learnt knowledge of nodes by utilizing bi-attention architecture and introducing global bias, respectively. In this paper, we proposed BIGSAGE, an unsupervised and inductive network embedding approach which is able to preserve local proximity wisely as well as learn and memorize global identities for seen nodes while generalizing to unseen nodes or networks. We apply a bi-attention architeture upon hierarchical aggregating layers to directly capture the most relevant representations of co-occurring nodes. We also present an efficient way of combining inductive and transductive approaches by allowing trainable global embedding bias to be retrieved in all layers within the hierarchical aggregating framework. Experiments demenstrate the superiority of BIGSAGE over the state-of-art baselines on unsupervised and inductive tasks.
For unsupervised and inductive network embedding, we propose a novel approach to explore most relevant neighbors and preserve previously learnt knowledge of nodes by utilizing bi-attention architecture and introducing global bias, respectively This proposes an extension to GraphSAGE using a global embedding bias matrix in the local aggregating functions and a method to sample interesting nodes.