Recent approaches have successfully demonstrated the benefits of learning the parameters of shallow networks in hyperbolic space. We extend this line of work by imposing hyperbolic geometry on the embeddings used to compute the ubiquitous attention mechanisms for different neural networks architectures. By only changing the geometry of embedding of object representations, we can use the embedding space more efficiently without increasing the number of parameters of the model. Mainly as the number of objects grows exponentially for any semantic distance from the query, hyperbolic geometry  --as opposed to Euclidean geometry-- can encode those objects without having any interference. Our method shows improvements in generalization on neural machine translation on WMT'14 (English to German), learning on graphs (both on synthetic and real-world graph tasks) and visual question answering (CLEVR) tasks while keeping the neural representations compact. The focus of this work is to endow neural network representations with suitable geometry to capture fundamental properties of data, including hierarchy and clustering behaviour. These properties emerge in many real-world scenarios that approximately follow power-law distributions BID28 BID9 ). This includes a wide range of natural phenomena in physics BID23 , biology BID26 , and even human-made structures such as metabolic-mass relationships BID4 , social networks , and frequencies of words BID33 BID32 BID38 ).Complex networks , which connect distinguishable heterogeneous sets of elements represented as nodes, provide us an intuitive way of understanding these structures. They will also serve as our starting point for introducing hyperbolic geometry, which is by itself difficult to visualize. Nodes in complex networks are referred to as heterogeneous, in the sense that they can be divided into sub-nodes which are themselves distinguishable from each other. The scale-free structure of natural data manifests itself as a power law distribution on the node degrees of the complex network that describes it.Complex networks can be approximated with tree-like structures, such as taxonomies and dendrograms, and as lucidly presented by , hyperbolic spaces can be thought of as smooth trees abstracting the hierarchical organization of complex networks. Let us begin by recalling a simple property of n-ary trees that will help us understand hyperbolic space and why hyperbolic geometry is well suited to model relational data.In an n-ary tree, the number of nodes at distance r from the root and the number of nodes at distance no more than r from the root both grow as n r . Similarly, in a two-dimensional hyperbolic space with curvature −ζ 2 ,ζ > 0, the circumference and area of a disc of radius r grows as 2πsinh(ζr) and 2π(cosh(ζr)−1), respectively, both of are exponential in r BID20 . The growth of volume in hyperbolic space should be contrasted with Euclidean space where the corresponding quantities expand polynomially, circumference as 2πr and area as πr 2 .In the two-dimensional example of Figure 1 , the expanding rings show examples at a fixed semantic distance from the central object ("pug"). The number of concepts grows quickly with semantic distance forcing each successive ring to be more crowded in order to maintain a fixed distance to the center. In contrast, the extra volume of hyperbolic spheres (depicted by reducing the size of the examples) allows all of the examples to remain well separated from their semantic neighbours. Figure 1: An intuitive depiction of how images might be embedded in 2D. The location of the embeddings reflects the similarity between each image and that of a pug. Since the number of instances within a given semantic distance from the central object grows exponentially, the Euclidean space is not able to compactly represent such structure (left). In hyperbolic space (right) the volume grows exponentially, allowing for sufficient room to embed the images. For visualization, we have shrunk the images in this Euclidean diagram, a trick also used by Escher.Mechanically, the computed embeddings by a random network for objects at a given semantic distance might still seem epsilon distance away from each other (or crowded) as the ones obtained by using Euclidean geometry. However, enforcing hyperbolic geometry intuitively means that all operations with these embeddings take into account, the density in that particular region of the space. For example, any noise introduced in the system (e.g., in gradients) will also be corrected by the density. In contrast to working in Euclidean space , this means that the embeddings will be equally distinguishable regardless of the density.The intimate connection between hyperbolic space and scale free networks (where node degree follows a power law) is made more precise in . In particular, there it is shown that the heterogeneous topology implies hyperbolic geometry, and conversely hyperbolic geometry yields heterogeneous topology. Moreover, Sarkar (2011) describes a construction that embeds trees in two-dimensional hyperbolic space with arbitrarily low distortion, which is not possible in Euclidean space of any dimension BID24 . Following this exciting line of research, recently the machine learning community has gained interest in learning non-Euclidean embeddings directly from data BID29 BID7 BID34 BID30 BID39 BID5 .Fuelled by the desire of increasing the capacity of neural networks without increasing the number of trainable parameters so as to match the complexity of data, we propose hyperbolic attention networks. As opposed to previous approaches, which impose hyperbolic geometry on the parameters of shallow networks BID29 BID7 , we impose hyperbolic geometry on the activations of deep networks. This allows us to exploit hyperbolic geometry to reason about embeddings produced by deep networks. We introduce efficient hyperbolic operations to express the popular , ubiquitous mechanism of attention BID1 BID12 BID42 BID47 . Our method shows improvements in terms of generalization on neural machine translation BID42 , learning on graphs and visual question answering BID0 BID25 BID16 tasks while keeping the representations compact. Simultaneously to our work, BID8 proposed a method to learn SVMs in the hyperboloid model of hyperbolic space, and Nickel and Kiela (2018) proposed a method to learn shallow embeddings of graphs in hyperbolic space by using the hyperboloid model. We have presented a novel way to impose the inductive biases from hyperbolic geometry on the activations of deep neural networks. Our proposed hyperbolic attention operation makes use of hyperbolic geometry in both the computation of the attention weights, and in the aggregation operation over values. We implemented our proposed hyperbolic attention mechanism in both Relation Networks and the Transformer and showed that we achieve improved performance on a diverse set of tasks. We have shown improved performance on link prediction and shortest path length prediction in scale free graphs, on two visual question answering datasets, real-world graph transduction tasks and finally on English to German machine translation. The gains are particularly prominent in relatively small models, which confirms our hypothesis that hyperbolic geometry induces more compact representations.Yang and Rush (2016) have proposed to imposed the activations of the neural network to lie on a Lie-group manifold in the memory. Similarly as a future work, an interesting potential future direction is to use hyperbolic geometry as an inductive bias for the activation of neural networks in the memory.
We propose to incorporate inductive biases and operations coming from hyperbolic geometry to improve the attention mechanism of the neural networks. This paper replaces the dot-product similarity used in attention mechanisms with the negative hyperbolic distance, and applies it to the existing Transformer model, graph attention networks, and Relation Networks The authors propose a novel approach to improve relational-attention by changing the matching and aggregation functions to use hyperbolic geometric. 