One of the most successful techniques in generative models has been decomposing a complicated generation task into a series of simpler generation tasks.   For example, generating an image at a low resolution and then learning to refine that into a high resolution image often improves results substantially.   Here we explore a novel strategy for decomposing generation for complicated objects in which we first generate latent variables which describe a subset of the observed variables, and then map from these latent variables to the observed space.   We show that this allows us to achieve decoupled training of complicated generative models and present both theoretical and experimental results supporting the benefit of such an approach.   Learning useful intermediate representations in a hierarchical manner has been a driving factor in the recent success of deep learning BID11 . When ample amounts of labelled data are available, supervised learning methods are successful in learning useful intermediate representations BID20 BID14 ). However, the task is significantly more challenging in the context of unsupervised learning. One such approach to unsupervised learning is to learn a generative model of high-dimensional observed variables with low-dimensional latent variables, such that the latent variables capture the salient features of the data, which could then be used for other upstream tasks.Recent work by BID22 argues why hierarchical latent variables models are often not able to take advantage of the hierarchy, and only the lowest-level latent variables learn any useful representations. We posit that this is possibly because the vanilla hierarchical latent variable structure by itself only adds a very weak prior (that of devoting more processing to higher-level latent variables). When parameterizing the conditional distributions with powerful deep neural networks, this could admit a local optima in which all factors of variation are sub-optimally explained by the lowest-level latent variable. Notably, this phenomenon was also common in supervised training of deep neural networks, before BID6 introduced batch normalization, which successfully disentangles the learning dynamics at each layer, as if each layer has an independent objective function.For example, the resolution-based hierarchy is well suited to images because lower resolution images capture some factors of variation (such as objects) but discards other factors of variation (such as texture and details), giving the low and high level models distinct responsibilities. However, this decomposition is a strong prior and may not work well or apply to other types of data (for example, it is not clear how it would apply to language or video). This motivates the need for an unsupervised method for learning hierarchical latent variables with a requisite but general prior to facilitate disentangled learning dynamics in each level.Our proposed approach, which we call Locally Disentangled Factors (LDF), has the following desired features:• Decoupled level-wise training objectives which significantly accelerate training.• A graphical model based on spatial locality which aids in credit assignment, and can be thought of as a generalization to resolution-based hierarchies.• Vastly reduced memory consumption which allows training generative models on large objects, such as videos, where this is known to be a prohibitive limitation.• Applicable to variable-length objects, such as videos and text.2 PROPOSED APPROACH We have proposed Locally Disentangled Factors (LDF), a powerful new approach to decomposing the training of generative models. We have shown that LDF is able to successfully generate joint distributions over complicated objects, even though the discriminators and gradient flow are entirely local within the hierarchy. We have also shown that this allows for decoupled training and improved ability to learn from small amounts of data from the joint distribution. While our method assumes a more general prior than resolution-hierarchy style approaches, it still leaves the decision of what the local factors would be on the practitioner. Finding methods that enjoy the same computational and sample-complexity benefits with fewer assumptions about the data is an interesting research direction.
Decompose the task of learning a generative model into learning disentangled latent factors for subsets of the data and then learning the joint over those latent factors.   Locally Disentangled Factors for hierarchical latent variable generative model, which can be seen as a hierarchical variant of Adversarially Learned Inference The paper investigates the potential of hierarchical latent variable models for generating images and image sequences and proposes to train several ALI models stacked on top of each other to create a hierarchical representation of the data. The paper aims to learn the hierarchies for training GAN in a hierarchical optimization schedule directly instead of being designed by a human