High-dimensional Asymptotics of Denoising Autoencoders
Hugo Cui Lenka Zdeborová
We address the problem of denoising data from a Gaussian mixture using a
two-layer non-linear autoencoder with tied weights and a skip connection. We
consider the high-dimensional limit where the number of training samples and
the input dimension jointly tend to infinity while the number of hidden units
remains bounded. We provide closed-form expressions for the denoising
mean-squared test error. Building on this result, we quantitatively
characterize the advantage of the considered architecture over the autoencoder
without the skip connection that relates closely to principal component
analysis. We further show that our results accurately capture the learning
curves on a range of real data sets.