Representations learnt through deep neural networks tend to be highly informative, but opaque in terms of what information they learn to encode. We introduce an approach to probabilistic modelling that learns to represent data with two separate deep representations: an invariant representation that encodes the information of the class from which the data belongs, and an equivariant representation that encodes the symmetry transformation defining the particular data point within the class manifold (equivariant in the sense that the representation varies naturally with symmetry transformations). This approach to representation learning is conceptually transparent, easy to implement, and in-principle generally applicable to any data comprised of discrete classes of continuous distributions (e.g. objects in images, topics in language, individuals in behavioural data). We demonstrate qualitatively compelling representation learning and competitive quantitative performance, in both supervised and semi-supervised settings, versus comparable modelling approaches in the literature with little fine tuning. Representation learning BID0 is part of the foundation of deep learning; powerful deep neural network models appear to derive their performance from sequentially representing data in more-and-more refined structures, tailored to the training task.However, representation learning has a broader impact than just model performance. Transferable representations are leveraged efficiently for new tasks BID22 , representations are used for human interpretation of machine learning models BID18 , and meaningfully structured (disentangled) representations can be used for model control (e.g. semisupervised learning as in , topic modelling as in BID1 ).Consequently , it is often preferable to have interpretable data representations within a model, in the sense that the information contained in the representation is easily understood and the representation can be used to control the output of the model (e.g. to generate data of a given class or with a particular characteristic). Unfortunately , there is often a tension between optimal model performance and cleanly disentangled or controllable representations.To overcome this, some practitioners have proposed modifying their model's objective functions by inserting parameters in front of particular terms BID2 BID11 , while others have sought to modify the associated generative models BID20 . Further still , attempts have been made to build the symmetries of the data directly into the neural network architecture in order to force the learning of latent variables that transform meaningfully under those symmetries BID27 . The diversity and marginal success of these approaches point to the importance and difficulty of learning meaningful representations in deep generative modelling.In this work we present an approach to probabilistic modelling of data comprised of a finite number of distinct classes, each described by a smooth manifold of instantiations of that class. For convenience , we call our approach EQUIVAE for Equivariant Variational Autoencoder. EQUIVAE is a probabilistic model with 2 latent variables: an invariant latent that represents the global class information, and an equivariant latent that smoothly interpolates between all of the members of that class. The EQUIVAE approach is general in that the symmetry group of the manifold must not be specified (as in for example BID6 ; BID8 ), and it can be used for any number of classes and any dimensionality of both underlying representations. The price that must be paid for this level of model control and flexibility is that some labelled data is needed in order to provide the concept of class invariance versus equivariance to the model. The endeavor to model the content and the style of data separately is certainly not new to this work BID29 . BID25 and BID24 go further, disentangling the continuous sources of variation in their representations using a clamping technique that exposes specific latent components to a single source of variation in the data during training. In the same vein, other approaches have used penalty terms in the objective function that encourage the learning of disentangled representations BID5 BID4 . EQUIVAE does not require any modification to the training algorithm, nor additional penalty terms in the objective function in order to bifurcate the information stored in the two latent variables. This is due to the way in which multiple data points are used to reconstruct a single data point from the same-class manifold, which we consider the primary novel aspect of our approach. In particular, our invariant representation takes as input multiple data points that come from the same class, but are different from the data point to be reconstructed. This invariant representation thus directly learns to encode the information common to the overall class, but not the individual data point, simply due to the information flowing through it.Of further note, we deliberately use a deterministic latent for the invariant representation, and a stochastic latent for the smooth equivariant representation (an idea also employed by BID30 ). This choice is why we do not need to explicitly force the equivariant latent to not contain any class-level information: it is available and easier to access from the deterministic latent.EQUIVAE is also comparable to BID28 , where the authors leverage labelled data explicitly in their generative model in order to force the VAE latent to learn the non-class information BID19 do similarly using adversarial training). The primary difference between those works and ours is that EQUIVAE provides a non-trivial representation of the global information instead of simply using the integer-valued label. Furthermore, this invariant representation can be deterministically evaluated directly on unlabelled data. Practitioners can reuse this embedding on unlabelled data in downstream tasks, along with the equivariant encoder if needed. The invariant representation provides more information than a simple prediction of the class-label distribution.The encoding procedure for the invariant representation in EQUIVAE is partially inspired by BID7 , who use images from various, known coordinates in a scene in order to reconstruct a new image of that scene at new, known coordinates. In contrast, we do not have access to the exact coordinates of the class instance , which in our case corresponds to the unknown, non-trivial manifold structure of the class; we must infer these manifold coordinates in an unsupervised way. BID9 similarly explore the simultaneous usage of multiple data points in generative modelling in order to better capture modelling uncertainty. We have introduced a technique for jointly learning invariant and equivariant representations of data comprised of discrete classes of continuous values. The invariant representation encodes global information about the given class manifold which is ensured by the procedure of reconstructing a data point through complementary samples from the same class. The equivariant representation is a stochastic VAE latent that learns the smooth set of transformations that cover the instances of data on that class manifold. We showed that the invariant latents are so widely separated that a 99.18% accuracy can be achieved on MNIST (87.70% on SVHN) with a simple 0-parameter distance metric based on the invariant embedding. The equivariant latent learns to cover the manifold for each class of data with qualitatively excellent samples and interpolations for each class. Finally, we showed that semi-supervised learning based on such latent variable models is competitive with similar approaches in the literature with essentially no hyperparameter tuning.
This paper presents a novel latent-variable generative modelling technique that enables the representation of global information into one latent variable and local information into another latent variable. The paper presents a VAE that uses labels to separate the learned representation into an invariant and a covariant part.