Generative models are important tools to capture and investigate the properties of complex empirical data. Recent developments such as Generative Adversarial Networks (GANs) and Variational Auto-Encoders (VAEs) use two very similar, but \textit{reverse}, deep convolutional architectures, one to generate and one to extract information from data. Does learning the parameters of both architectures obey the same rules? We exploit the causality principle of independence of mechanisms to quantify how the weights of successive layers adapt to each other. Using the recently introduced Spectral Independence Criterion, we quantify the dependencies between the kernels of successive convolutional layers and show that those are more independent for the generative process than for information extraction, in line with results from the field of causal inference. In addition, our experiments on generation of human faces suggest that more independence between successive layers of generators results in improved performance of these architectures.
 Deep generative models have proven powerful in learning to design realistic images in a variety of complex domains (handwritten digits, human faces, interior scenes). In particular, two approaches have recently emerged: Generative Adversarial Networks (GANs) BID8 , which train an image generator by having it fool a discriminator that should tell apart real from artificially generated images; and Variational Autoencoders (VAEs) BID15 BID21 ) that learn both a mapping from latent variables to the data (the decoder) and the converse mapping from the data to the latent variables (the encoder), such that correspondences between latent variables and data features can be easily investigated. Although these architecture have been lately the subject of extensive investigations, understanding why and how they work, and how they can be improved, remains elusive.An interesting feature of GANs and VAEs is that they both involve the learning of two deep subnetworks. These sub-networks have a "mirrored" architecture, as they both consist in a hierarchy of convolutional layers, but information flows in opposite ways: generators and decoders map latent variables to the data space, while discriminators and encoders extract information from the same space. Interestingly, this difference could be framed in a causal perspective, with information flowing in the causal direction in the case of generators (from the putative causes of variations in the observed data), while extracting high level properties from the observations (with encoders or discriminators) would operate in the anti-causal direction.Generative models in machine learning are usually not required to be causal, as modeling the data distribution is considered to be the goal to achieve. However, the idea that a generative model able to capture the causal structure of the data by disentangling the contribution of independent factors should perform better has been suggested in the literature BID0 BID17 and evidence supports that this can help the learning procedure BID1 . Although many approaches have been implemented on specific examples, principles and automated ways of learning disentangled representations from data remains largely an open problem both for learning representations (targeting supervised learning applications) and for fitting good generative models. GANs have for example recently been a subject of intensive work in this direction, leading to algorithms disentangling high level properties of the data such as InfoGans BID2 or conditional GANs BID18 . However such models require supervision (e.g. feeding digit labels as additional inputs) to disentangle factors of interest. Unsupervised learning of disentangled representations has been addressed in various frameworks including Restricted Boltzmann Machines BID6 , tensor analyzers BID26 and Lie groups BID4 . A recent attempt to address unsupervised learning in VAEs is β-VAE BID10 which introduces and adjustable parameter β in the VAE objective to strengthen the data compression constraint with respect to reconstruction error.While the above approaches envision the disentangling of representations as finding subsets of latent variables that relate to different properties of the generated data, parameters of the network can be also considered as factors affecting the generated data. Ideally, to ensure modularity of deep generative models, different layers should encode different aspects of the data in their parameters. Intuitively for deep convolutional networks, different layers should encode image features at different scales. The idea that successive layers can be used as modules encoding different levels of details has for example been exploited to build high-resolution generative models by training iteratively a GAN with an increasing number of layers BID14 . Enforcing modularity of trained neural architecture may not only allow to adapt them to the task at hand with minimum additional training, but also to better understand the structure and function of these highly complex black-box systems. However, to the best of our knowledge, assessing how independent (or disentangled) are the properties encoded by the weights distributed across the structure of deep networks has not been addressed quantitatively in the literature.We propose that the coupling between high dimensional parameters can be quantified and exploited in a causal framework to infer whether the layered architecture disentangles different aspects of the data. This hypothesis relies on recent work exploiting the postulate of Independence of Cause and Mechanism stating that Nature chooses independently the properties of a cause and those of the mechanism that generate effects from the cause BID13 . Several methods relying on this principle have been proposed in the literature in association to different model classes BID28 BID5 BID24 BID23 . Among these methods, the Spectral Independence Criterion (SIC) BID24 can be used in the context of linear dynamical systems, which involve a convolution mechanism.In this paper, we show how SIC can be adapted to investigate the coupling between the parameters of successive convolutional layers. Empirical investigation shows that SIC is approximately valid between successive layers of generative models, and suggests SIC violations indicate deficiencies in the learning algorithm or the architecture of the network. Interestingly, and in line with theoretical predictions BID24 , SIC tends to be more satisfied for generative sub-networks (mapping latent variables to data), than for the part of the system that map the anti-causal direction (data to latent variables). In addition, comparison of different generative models indicates that more independence between layers is associated to better performance of the model. Overall, our study suggests that quantifying Independence of Mechanisms in deep architecture can help analyze and design better generative models. In this work, we derived a measure of independence between the weights learned by convolutional layers of deep networks. The results suggest that generative models that map latent variables to data tend to have more independence between successive layers than discriminative or encoding networks. This is in line with theoretical predictions about independence of mechanisms for causal and anti-causal systems. In addition, our results suggest the dependency between successive layers relates to the bad performance of the trained generative models. Moreover, the SDR analysis also indicates which layers should be modified to improve the performance. Enforcing independence during training may thus help diagnose and improve generative models. Finally, we speculate that independence between successive layers, by favoring modularity of the network, may help build architectures that can be easily adapted to new purposes. In particular, separation of spatial scales in such models may help build networks in which one can intervene on one scale without affecting others, with applications such as style transfer BID7 . One specific feature of our approach is that this quantitative measure of the network performance in not statistical and as such requires neither extensive sampling form the fitted generative distribution nor from real datasets to be computed: only the parameters of the model are used. This is in strong contrast with state-ofthe-art approaches such as the Fréchet Inception Distance (FID) BID9 , and makes the approach easy to apply to any neural network equipped with convolutional layers.
We use causal inference to characterise the architecture of generative models This paper examines the nature of convolutional filters in the encoder and a decoder of a VAE, and a generator and a discriminator of a GAN. This work exploits the causality principle to quantify how the weights of successive layers adapt to each other.