Convolutional Neural Networks (CNNs) have become the method of choice for learning problems involving 2D planar images. However, a number of problems of recent interest have created a demand for models that can analyze spherical images. Examples include omnidirectional vision for drones, robots, and autonomous cars, molecular regression problems, and global weather and climate modelling. A naive application of convolutional networks to a planar projection of the spherical signal is destined to fail, because the space-varying distortions introduced by such a projection will make translational weight sharing ineffective.

 In this paper we introduce the building blocks for constructing spherical CNNs. We propose a definition for the spherical cross-correlation that is both expressive and rotation-equivariant. The spherical correlation satisfies a generalized Fourier theorem, which allows us to compute it efficiently using a generalized (non-commutative) Fast Fourier Transform (FFT) algorithm. We demonstrate the computational efficiency, numerical accuracy, and effectiveness of spherical CNNs applied to 3D model recognition and atomization energy regression. Figure 1: Any planar projection of a spherical signal will result in distortions. Rotation of a spherical signal cannot be emulated by translation of its planar projection.Convolutional networks are able to detect local patterns regardless of their position in the image. Like patterns in a planar image, patterns on the sphere can move around, but in this case the "move" is a 3D rotation instead of a translation. In analogy to the planar CNN, we would like to build a network that can detect patterns regardless of how they are rotated over the sphere.As shown in Figure 1 , there is no good way to use translational convolution or cross-correlation 1 to analyze spherical signals. The most obvious approach, then, is to change the definition of crosscorrelation by replacing filter translations by rotations. Doing so, we run into a subtle but important difference between the plane and the sphere: whereas the space of moves for the plane (2D translations) is itself isomorphic to the plane, the space of moves for the sphere (3D rotations) is a different, three-dimensional manifold called SO(3) 2 . It follows that the result of a spherical correlation (the output feature map) is to be considered a signal on SO(3), not a signal on the sphere, S 2 . For this reason, we deploy SO(3) group correlation in the higher layers of a spherical CNN BID4 .The implementation of a spherical CNN (S 2 -CNN) involves two major challenges. Whereas a square grid of pixels has discrete translation symmetries, no perfectly symmetrical grids for the sphere exist. This means that there is no simple way to define the rotation of a spherical filter by one pixel. Instead, in order to rotate a filter we would need to perform some kind of interpolation. The other challenge is computational efficiency; SO(3) is a three-dimensional manifold, so a naive implementation of SO(3) correlation is O(n 6 ).We address both of these problems using techniques from non-commutative harmonic analysis BID3 BID11 . This field presents us with a far-reaching generalization of the Fourier transform, which is applicable to signals on the sphere as well as the rotation group. It is known that the SO(3) correlation satisfies a Fourier theorem with respect to the SO(3) Fourier transform, and the same is true for our definition of S 2 correlation. Hence, the S 2 and SO(3) correlation can be implemented efficiently using generalized FFT algorithms.Because we are the first to use cross-correlation on a continuous group inside a multi-layer neural network, we rigorously evaluate the degree to which the mathematical properties predicted by the continuous theory hold in practice for our discretized implementation.Furthermore, we demonstrate the utility of spherical CNNs for rotation invariant classification and regression problems by experiments on three datasets. First, we show that spherical CNNs are much better at rotation invariant classification of Spherical MNIST images than planar CNNs. Second, we use the CNN for classifying 3D shapes. In a third experiment we use the model for molecular energy regression, an important problem in computational chemistry. In this paper we have presented the theory of Spherical CNNs and evaluated them on two important learning problems. We have defined S 2 and SO(3) cross-correlations, analyzed their properties, and implemented a Generalized FFT-based correlation algorithm. Our numerical results confirm the stability and accuracy of this algorithm, even for deep networks. Furthermore, we have shown that Spherical CNNs can effectively generalize across rotations, and achieve near state-of-the-art results on competitive 3D Model Recognition and Molecular Energy Regression challenges, without excessive feature engineering and task-tuning.For intrinsically volumetric tasks like 3D model recognition, we believe that further improvements can be attained by generalizing further beyond SO(3) to the roto-translation group SE(3). The development of Spherical CNNs is an important first step in this direction. Another interesting generalization is the development of a Steerable CNN for the sphere , which would make it possible to analyze vector fields such as global wind directions, as well as other sections of vector bundles over the sphere.Perhaps the most exciting future application of the Spherical CNN is in omnidirectional vision. Although very little omnidirectional image data is currently available in public repositories, the increasing prevalence of omnidirectional sensors in drones, robots, and autonomous cars makes this a very compelling application of our work. We use the ZYZ Euler parameterization for SO(3). An element R ∈ SO(3) is written as DISPLAYFORM0 where α ∈ [0, 2π], β ∈ [0, π] and γ ∈ [0, 2π], and Z resp. Y are rotations around the Z and Y axes.Using this parameterization, the normalized Haar measure is DISPLAYFORM1 We have SO(3) dR = 1. The Haar measure BID23 BID3 ) is sometimes called the invariant measure because it has the property that SO(3) f (R R)dR = SO FORMULA2 f (R)dR (this is analogous to the more familiar property DISPLAYFORM2 f (x)dx for functions on the line). This invariance property allows us to do many useful substitutions.We have a related parameterization for the sphere. An element x ∈ S 2 is written DISPLAYFORM3 where n is the north pole.This parameterization makes explicit the fact that the sphere is a quotient S 2 = SO(3)/ SO(2), where H = SO(2) is the subgroup of rotations around the Z axis. Elements of this subgroup H leave the north pole invariant, and have the form Z(γ). The point x(α, β) ∈ S 2 is associated with the coset representativex = R(α, β, 0) ∈ SO(3). This element represents the cosetxH = {R(α, β, γ)|γ DISPLAYFORM4 The normalized Haar measure for the sphere is DISPLAYFORM5 The normalized Haar measure for SO(2) is DISPLAYFORM6 So we have dR = dx dh, again reflecting the quotient structure.We can think of a function on S 2 as a γ-invariant function on SO(3). Given a function f : S 2 → C we associate the functionf (α, β, γ) = f (α, β). When using normalized Haar measures, we have: DISPLAYFORM7 This will allow us to define the Fourier transform on S 2 from the Fourier transform on SO(3), by viewing a function on S 2 as a γ-invariant function on SO(3) and taking its SO(3)-Fourier transform.
We introduce Spherical CNNs, a convolutional network for spherical signals, and apply it to 3D model recognition and molecular energy regression. The paper proposes a framework for constructing spherical convolutional networks based on a novel synthesis of several existing concepts This paper focuses on how to extend convolutional neural networks to have built-in spherical invariance, and adapts tools from non-Abelian harmonic analysis to achieve this goal. The authors develop a novel scheme for representing spherical data from the ground up