We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large. This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature. Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs “approximate inference” as posed by Scellier et al. (2018), and its connection to reversibility considered by several theoretical studies. Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts. Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization. Our analysis is not specific to any depths or any Lipschitz activations, and our analytical techniques may have broader applicability. The autoencoder is a cornerstone in machine learning, first as a response to the unsupervised learning problem (Rumelhart & Zipser (1985) ), then with applications to dimensionality reduction (Hinton & Salakhutdinov (2006) ), unsupervised pre-training (Erhan et al. (2010) ), and also as a precursor to many modern generative models (Goodfellow et al. (2016) ). Its reconstruction power is well utilized in applications such as anomaly detection (Chandola et al. (2009) ) and image recovery (Mousavi et al. (2015) ). With the surge of deep learning, thousands of papers have studied multilayer variants of this architecture, but theoretical understanding has been limited, since analyzing the learning dynamics of a highly nonlinear structure is typically a difficult problem even for the shallow autoencoder. To get around this, we tackle the task with a critical assumption: the weights are random and the autoencoder is weight-tied. One enjoys much analytical tractability from the randomness assumption, whereas weight tying enforces the random autoencoder to perform "autoencoding". We also study this in the high-dimensional setting, where all dimensions are comparably large and ideally jointly approaching infinity. We consider the simplest setting: vanilla autoencoders (i.e., ones with fully connected layers only) and their reconstruction capability. This is done for the sake of understanding the effect of depth, while we note our techniques may have broader applicability.The aforementioned assumptions are not without justifications. There is a growing literature on deep neural networks with random weights, (Li & Saad (2018) ; Giryes et al. (2016) ; Poole et al. (2016) ; Schoenholz et al. (2016) ; Gabrié et al. (2018) ; Amari et al. (2018) ) to name a few, revealing certain properties of deep feedforward networks 1 . Several recent works have also studied random multilayer feedforward networks through the lens of statistical inference (Manoel et al. (2017) ; Reeves (2017); Fletcher et al. (2018) ). The idea of weight tying is considered in the important paper Vincent et al. (2010) with an empirical finding that autoencoders with and without weight tying perform comparably, and has become standard in autoencoders. Similar features of random connection and symmetry also appear in other neural models (Lillicrap et al. (2016) ; Scellier et al. (2018) ). Finally the high-dimensional setting is common in recent statistical learning advances (Bühlmann & Van De Geer (2011) ), and not too far from the actual practice where many large datasets have dimensions of at least a few hundreds and are harnessed by large-scaled models.We seek quantitative answers to three specific questions that are motivated by previous works:• In exactly what way does the (vanilla) random weight-tied autoencoder perform "approximate inference"? This term is coined in Scellier et al. (2018) in connection with the theoretical results in Arora et al. (2015) , which implicitly studies the said model. In particular, Arora et al. (2015) proves an upper bound on x − x 2 , where x andx are the input and the output of the network, but is limited in the number of layers and specific to the ReLU activation. This direction has been recently extended by Gilbert et al. (2017) . In our work, we establish precisely what this approximate inference is by obtaining a general and asymptotically exact characterization 2 ofx, for any number of layers and any Lipschitz continuous activations (Theorem 1 and Section 3.3). Theorem 1 is the key theoretical result of our work and lays the foundation for all analyses that follow.• In what way is the deep autoencoder different from the shallow counterpart? Li & Saad (2018) ; Poole et al. (2016) reveal this in terms of the candidate function space and expressivity for feedforward networks. It is unclear how these notions are applicable to weighttied autoencoders, which seek replication of the input rather than a generic mapping. In this work, we show that the deep autoencoder exhibits a higher order of sensitivity to perturbations of the parameters (Section 3.4). Burkholz & Dubatovka (2018) demonstrate a connection between the study of random networks, or ones at initialization, and their trainability. Note that these works either do not study weight-tied structures, or assume the analysis of the untying case for weight-tied structures. In our work , we derive and experimentally verify insights on how (not) to initialize deep weight-tied autoencoders, demonstrating that it is possible to train them without resorting to techniques such as greedy layer-wise pretraining, drop-out and batch normalization (Section 3.5). Specifically we experiment with 200-layer autoencoders.No prior works have attempted all three tasks. The quantitative difference between weight-tied and weight-untied networks is in fact not negligible, yet the analysis is non-trivial due to the weight tying constraint (Arora et al. (2015) ; Chen et al. (2018) ). To address this issue and obtain Theorem 1, we apply the Gaussian conditioning technique, which first appears in the studies of TAP equations in spin glass theory (Bolthausen (2014) ) and is extensively used in the approximate message passing algorithm literature (Bayati & Montanari (2011); Javanmard & Montanari (2013) ; Berthier et al. (2017) ). This should be contrasted with untied random networks, whose analysis is typically more straightforward. More importantly , the difference is not only analytical: the overall picture of deep random weight-tied autoencoders is rich and drastically different from that of feedforward networks. An analysis in the limit of infinite depth reveals three fundamental equations governing the picture (Section 3.1), which displays multiple phase transition phenomena (Section 3.2) . Consider the following 2L-layers autoencoder with weight tying: DISPLAYFORM0 Here x ∈ R n0 is the input, W ∈ R n ×n −1 is the weight, b ∈ R n is the encoder bias, and v ∈ R n −1 is the decoder bias, for = 1, ..., L. Also ϕ : R → R and σ : R → R are the activations (where for a vector u ∈ R n and a function ϕ : R → R, we write ϕ (u) to denote the vector (ϕ (u 1 ) , ..., ϕ (u n )) ). It is usually the case in practice that σ 0 (u) = u the identity function. We introduce some convenient quantities inductively: FIG6 of Appendix A.1 for a schematic diagram. We assume weights are random . Specifically we generate the weights and biases according to DISPLAYFORM1 DISPLAYFORM2 independently of each other. The scaling of the variances accords with the literature and actual practice (Glorot & Bengio (2010); Vincent et al. (2010) ). We also consider the asymptotic highdimensional regime, indexed by n: DISPLAYFORM3 Here σ W, , σ b, , σ v, and α are finite constants independent of n. We enforce σ W, > 0, but allow σ b, and σ v, to be zero. We assume that all activations are Lipschitz continuous, and the encoder activations σ 's are non-trivial in the sense that for any τ > 0, E z σ (τ z) This paper has shown quantitative answers to the three questions posed in Section 1. This feat is enabled by an exact analysis via Theorem 1. The theorem is stated in a general setting, allowing varying activations, weight variances, etc, but our analyses in Section 3 have made several simplifications. This leaves a question of whether these simplifications can be relaxed, and how the picture changes accordingly, for instance, when the parameters vary across layers, similar to Yang & Schoenholz (2018) . Many other questions also remain. For example, what would be the covariance structure between the outputs of two distinct inputs? How does the network's Jacobian matrix look like? These questions have been answered in the feedforward case (Poole et al. (2016) ; Pennington et al. FORMULA12 ), but we believe answering them is more technically involved in our case. We have also seen that an autoencoder that shows initial progress may not necessarily produce meaningful reconstruction eventually after training, and hence much more work is needed to understand the training dynamics far beyond initialization. Recent works Mei et al. FORMULA12 In the following, we give an outline of the proof of Theorem 1, and the complete proof. First, we start with a few notations and definitions. DISPLAYFORM0
We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena. A theoretical analysis of autoencoders with weights tied between encoder and decoder (weight-tied) via mean field analysis Analyses the performances of weighted tied auto-encoders by building on recent progress in analysis of high-dimensional statistics problems and specifically, the message passing algorithm This paper studies auto-encoders under several assumptions, and points out that this model of random autoencoder can be elegantly and rigorously analysed with one-dimensional equations.