Infinite-width neural networks have been extensively used to study the theoretical properties underlying the extraordinary empirical success of standard, finite-width neural networks. Nevertheless, until now, infinite-width networks have been limited to at most two hidden layers. To address this shortcoming, we study the initialisation requirements of these networks and show that the main challenge for constructing them is defining the appropriate sampling distributions for the weights. Based on these observations, we propose a principled approach to weight initialisation that correctly accounts for the functional nature of the hidden layer activations and facilitates the construction of arbitrarily many infinite-width layers, thus enabling the construction of arbitrarily deep infinite-width networks. The main idea of our approach is to iteratively reparametrise the hidden-layer activations into appropriately defined reproducing kernel Hilbert spaces and use the canonical way of constructing probability distributions over these spaces for specifying the required weight distributions in a principled way. Furthermore, we examine the practical implications of this construction for standard, finite-width networks. In particular, we derive a novel weight initialisation scheme for standard, finite-width networks that takes into account the structure of the data and information about the task at hand. We demonstrate the effectiveness of this weight initialisation approach on the MNIST, CIFAR-10 and Year Prediction MSD datasets. While deep neural networks have achieved impressive empirical success on many tasks across a wide range of domains in recent years BID20 BID35 BID33 BID34 BID29 , they remain hard to interpret black boxes whose performance crucially depends on many heuristics. In an attempt to better understand them, significant research effort has been directed towards examining the theoretical properties of these models with one important line of research focusing on the profound connections between infinite-width networks and kernel methods. In particular, BID30 established a correspondence between single-layer infinite-width networks and Gaussian processes (GP) BID32 showing the equivalence of the prior over functions that is induced by the network and a GP with a particular covariance kernel. The appropriate covariance kernel has been analytically derived for a few particular activation functions and weight priors BID38 .Although there is a large body of research on infinite-width networks with surging recent interest in the topic BID13 BID24 BID27 , until now the construction of these networks has been limited to at most two hidden layers. In order to overcome this shortcoming and enable deep infinite-width networks, we propose a novel approach to the construction of networks with infinitely wide hidden layers. To the best of our knowledge, this is the first method that enables the construction of infinite-width networks with more than two hidden layers. The main challenge in constructing this type of networks lies in ensuring that the inner products between the hidden layer representations and the corresponding weights, which are both functions, are well-defined. In particular, this amounts to ensuring that the weights lie in the same function space as the hidden layer representations with which the inner product is taken. In other words, the weights connecting layers l and l + 1 need to be in the same function space as the activations of layer l. To construct the infinite-width layer l + 1, we need to construct infinitely many weights connecting layers l and l + 1 that fulfill this requirement, i.e. we need to define a probability distribution over the function space of activations of layer l. As the number of layers grows, the function spaces of activations grow increasingly more complex, thus making it increasingly more difficult to satisfy the requirements on the weights.In order to tackle this challenge, we propose a principled approach to weight initialisation that automatically ensures that the weights are in the appropriate function space. The main idea of our approach is to make use of the canonical way of defining probability distributions over reproducing kernel Hilbert spaces (RKHS) and iteratively define the appropriate weight distributions facilitating the composition of arbitrarily many layers of infinite width. To this end, we first construct a kernel corresponding to each hidden layer and examine the associated RKHS of functions that is induced by this kernel. Next, for every layer, we establish a correspondence between the space of activations at a layer and the corresponding RKHS by reparametrising the hidden layer activations of a datapoint with the RKHS function corresponding to that point. Establishing this correspondence allows us to use a principled approach to defining probability distributions over RKHSs for constructing the appropriate sampling distribution of the weights in the infinite-width network.We also examine some practical implications of this construction for the case of standard, finitewidth neural networks in terms of weight initialisation. Using Monte Carlo approximations, we derive a novel data-and task-dependent weight initialisation scheme for finite-width networks that incorporates the structure of the data and information about the task at hand into the network.The main contributions of this paper are• a novel approach to the construction of infinite-width networks that enables the construction of networks with arbitrarily many hidden layers of infinite-width,• a hierarchy of increasingly complex kernels that capture the geometry and inductive biases of individual layers in the network,• a novel weight initialisation scheme for deep neural networks with theoretical foundations established in the infinite-width case.The rest of the paper is organised as follows. Section 2 discusses related work and Section 3 introduces our proposed approach to the construction of deep infinite-width networks. In Section 4, we showcase the practical implications of our theoretical contribution for the case of standard, finite-width deep networks. Section 5 discusses the experimental results followed by a conclusion in Section 6. In this paper, we have studied the initialisation requirements of infinite-width networks and have shown that the main challenge in constructing these networks lies in defining the appropriate sampling distributions of the weights. To address this problem, we have presented a novel method for the construction of infinite-width networks that, unlike previous approaches, enables the construction of deep infinite-width networks with arbitrarily many hidden layers. In particular, we have proposed a principled approach to weight initialisation using the theory of reproducing kernel Hilbert spaces. In order to appropriately account for the functional form of the hidden layer activations and to facilitate the construction of arbitrarily many infinite-width layers, we proposed to construct the sampling distributions of the weights at every hidden layer as Gaussian processes with specific covariance kernels that take into account the geometry of the underlying space of activations. To achieve this, we have constructed a hierarchy of kernels that capture the geometry and inductive biases of individual layers in the neural network. Furthermore, using Monte Carlo approximations, we have examined the practical implications of this construction for standard, finite-width networks. In particular, we have derived a novel data-and task-dependent weight initialisation method for this type of network and showcased its competitive performance on three diverse datasets.7 SUPPLEMENTARY MATERIAL 7.1 PROOFS For completeness, we restate the proposition and lemma and provide the corresponding proofs.7.1.1 PROPOSITION 1 Proposition 1. Let x, x ∈ R d be inputs to an network with l infinite-width layers, k l be the kernel corresponding to the l-th layer with k l (·, x) the corresponding canonical feature maps and H k l the induced RKHS. We can extend the network with an (l + 1)-th layer of infinite width by sampling the weights w l connecting the l-th and (l + 1)-th layers from a Gaussian process with zero mean function and covariance function DISPLAYFORM0
We propose a method for the construction of arbitrarily deep infinite-width networks, based on which we derive a novel weight initialisation scheme for finite-width networks and demonstrate its competitive performance. Proposes a weight initialization approach to enable infinitely deep and infinite-width networks with experimental results on small datasets. Proposes deep neural networks of infinite width.