Riemannian Residual Neural Networks
Isay Katsman Eric Ming Chen Sidhanth Holalkere Anna Asch Aaron Lou Ser-Nam Lim Christopher De Sa
Recent methods in geometric deep learning have introduced various neural
networks to operate over data that lie on Riemannian manifolds. Such networks
are often necessary to learn well over graphs with a hierarchical structure or
to learn over manifold-valued data encountered in the natural sciences. These
networks are often inspired by and directly generalize standard Euclidean
neural networks. However, extending Euclidean networks is difficult and has
only been done for a select few manifolds. In this work, we examine the
residual neural network (ResNet) and show how to extend this construction to
general Riemannian manifolds in a geometrically principled manner. Originally
introduced to help solve the vanishing gradient problem, ResNets have become
ubiquitous in machine learning due to their beneficial learning properties,
excellent empirical results, and easy-to-incorporate nature when building
varied neural networks. We find that our Riemannian ResNets mirror these
desirable properties: when compared to existing manifold neural networks
designed to learn over hyperbolic space and the manifold of symmetric positive
definite matrices, we outperform both kinds of networks in terms of relevant
testing metrics and training dynamics.