Variational auto-encoders
 (VAEs) offer a tractable approach when performing approximate inference in otherwise intractable generative models. However, standard VAEs often produce latent codes that are disperse and lack interpretability, thus making the resulting representations unsuitable for auxiliary tasks (e.g. classiﬁcation) and human interpretation. We address these issues by merging ideas from variational auto-encoders and sparse coding, and propose to explicitly model sparsity in the latent space of a VAE with a Spike and Slab prior distribution. We derive the evidence lower bound using a discrete mixture recognition function thereby making approximate posterior inference as computational efﬁcient as in the standard VAE case. With the new approach, we are able to infer truly sparse representations with generally intractable non-linear probabilistic models. We show that these sparse representations are advantageous over standard VAE representations on two benchmark classiﬁcation tasks (MNIST and Fashion-MNIST) by demonstrating improved classiﬁcation accuracy and signiﬁcantly increased robustness to the number of latent dimensions. Furthermore, we demonstrate qualitatively that the sparse elements capture subjectively understandable sources of variation. Variational auto-encoders (VAEs) offer an efficient way of performing approximate posterior inference with otherwise intractable generative models and yield probabilistic encoding functions that can map complicated high-dimensional data to lower dimensional representations BID11 BID26 BID31 BID25 . Making such representations meaningful and efficient, however, is a particularly difficult task and currently a major challenge in representation learning BID8 BID2 BID9 BID33 . Large latent spaces often give rise to many latent dimensions that do not carry any information, and obtaining codes that properly capture the complexity of the observed data is generally problematic BID33 BID6 BID2 .In the case of linear mappings, sparse coding offers an elegant solution to the aforementioned problem; the representation space is induced to be sparse. In such a way, the encoding function is encouraged to use the minimum number of non-zero elements necessary to describe each observation and condense information in few active variables, different for each sample BID22 BID7 . In fact, due to their efficiency of representation, sparse codes have been used in many learning and recognition systems, as they provide easier interpretation BID14 BID1 BID18 BID0 and increased efficiency in, for example, classification, clustering, and transmission tasks when used as learning inputs BID38 BID35 BID12 .In this work, we aim to extent the aforementioned capability of linear sparse coding to non-linear probabilistic generative models thus allowing efficient, informative and interpretable representations in the general case. To this end we formulate a new variation of the classical VAE in which we employ a sparsity inducing prior in the latent space based on the Spike and Slab distribution. We match this by a discrete mixture recognition function that can map observations to sparse latent vectors. Efficient inference, comparable in complexity to that of standard VAEs, is achieved by deriving an evidence lower bound (ELBO) for the new model which is optimized using standard gradient methods to recover the encoding and decoding functions. In our experiments , we consider two benchmark dataset (MNIST and Fashion-MNIST) and show how the resulting ELBO is able to recover sparse, informative and interpretable representations regardless of the predefined number of latent dimensions. The ability to adjust to data complexity allows to automatically discover the sources of variation in given observations, without the need to carefully adjust the architecture of a model to the given representation task. We demonstrate these properties by first performing classification experiments using latent vectors as inputs, where we demonstrate that VSC representations marginally outperform VAE ones and display greatly improved robustness over large variations in latent space dimensionality. Secondly we show that many sparse elements in retrieved codes control subjectively recognisable features in the generated observations. 2 BACKGROUND AND RELATED WORK 2.1 SPARSE CODING Sparse coding aims to approximately represent input vectors x i with a weighted linear combination of few unknown basis vectors b j BID14 BID1 BID15 . The problem of determining the optimal basis and weights is generally formulated as the minimisation of an objective function of the following form arg min DISPLAYFORM0 where X ∈ R M ×N is the matrix of data, having as columns the input vectors x i ∈ R M ×1 , B ∈ R M ×J is the matrix having as columns the basis vectors b j ∈ R M ×1 , Z ∈ R J×N is the sparse codes matrix, having as columns the sparse codes z i ∈ R J×1 corresponding to the inputs x i , λ is a real positive parameter and φ(z i ) is a sparsity inducing function.Sparse coding can be probabilistically interpreted as a generative model, where the observed vectors x i are generated from the unobserved latent variables z i through the linear process x i = Bz i + , where is the observation noise and is drawn from an isotropic normal distribution with zero mean BID14 BID1 . The model can then be described with the following prior and likelihood distributions DISPLAYFORM1 where β is a real positive parameter, σ is the standard deviation of the observation noise and I is the identity matrix. Performing maximum a posteriori (MAP) estimation with this model results in the minimisation shown in equation 1 with λ = σ 2 β.In contrast to the MAP formulation, we are interested in maximising the marginal likelihood p(x) = p(x i ) and being able to perform such optimisation for arbitrarily complicated likelihood functions p(x|z).Previous work has demonstrated variational EM inference for such maximisation in the linear generative model case, with a particular choice of sparsity inducing prior BID32 BID4 . However, EM inference becomes intractable for more complicated non-linear posteriors and a large number of input vectors BID11 , making such an approach unsuitable to scale to our desired model. Conversely, some work has been done in generalising sparse coding to non-linear transformations, by defining sparsity on Riemannian manifolds BID7 BID3 . These generalisations, however, perform MAP inference as they define a non-linear equivalent of the objective function in equation 1 and are limited to simple manifolds due to the need to compute the manifold's logarithmic map. In this paper, we lay the general framework to induce sparsity in the latent space of VAEs, allowing approximate variational inference with arbitrarily complicated and probabilistic sparse coding models. We derived a lower bound which is of clear interpretation and efficient to estimate and optimise, as the ELBO of a standard VAE. With the resulting encoders, we recovered efficient sparse codes, which proved to be optimal learning inputs in standard classification benchmarks and exhibit good interpretation in many of their non-zero components. We conclude that inducing sparsity in the latent space of generative models appears to be a promising route to obtaining useful codes, interpretable representations and controlled data synthesis, which are all outstanding challenges in VAEs and representation learning in general. In future work, we aim to further study the properties of a sparse latent space with respect to its interpretation and features disentanglement capability. We expect VSC to be able to model huge ensembles of varied data by sparsely populating large latent spaces, hence isolating the features that govern variability among similar objects in widely diverse aggregates of data.
We explore the intersection of VAEs and sparse coding. This paper proposes an extension of VAEs with sparse priors and posteriors to learn sparse interpretable representations.