The application of multi-modal generative models by means of a Variational Auto Encoder (VAE) is an upcoming research topic for sensor fusion and bi-directional modality exchange.
 This contribution gives insights into the learned joint latent representation and shows that expressiveness and coherence are decisive properties for multi-modal datasets.
 Furthermore, we propose a multi-modal VAE derived from the full joint marginal log-likelihood that is able to learn the most meaningful representation for ambiguous observations.
 Since the properties of multi-modal sensor setups are essential for our approach but hardly available, we also propose a technique to generate correlated datasets from uni-modal ones.
 Auto Encoder (AE), Variational Auto Encoder (VAE), and more recently Disentangled Variational Auto Encoder (β-VAE) have a considerable impact on the field of data-driven leaning of generative models. Furthermore, recent investigations have shown the fruitful applicability to deep reinforcement learning (DRL) as well as bi-directionally exchange of multi-modal data. VAEs tend to encode the data into latent space features that are (ideally) linearly separable as shown by BID4 . They also allow the discovery of generative joint models (e.g. BID17 ), as well as zero-shot domain transfer in DRL as shown by BID5 .However , a good generative model should not just generate good data and achieve a good quantitative score, but also gives a coherent and expressive latent space representation. This property is decisive for multi-modal approaches if the data shows correlation, as it is the case for every sensor setup designed for sensor fusion. With this contribution , we investigate the characteristic of the latent space as well as the quantitative features for existing multi-modal VAEs. Furthermore, we propose a novel approach to build and train a novel multi-modal VAE (M 2 VAE) which comprises the complete marginal joint log-likelihood without simplifying assumptions. As our objective is the consideration of raw multi-modal sensor data, we also propose an approach to generate correlated multi-modal datasets from available uni-modal ones. Lastly, we draw connections to in-place sensor fusion and epistemic (ambiguity-resolving) active-sensing.Section 2 comprises the related work on multi-modal VAEs. Our comprehensive approach (i.e. M 2 VAE) is given in Sec. 3. Furthermore, we describe multi-modal datasets as well as the generation of correlated sets in Sec. 4 which are evaluated in Sec. 5. Finally, we conclude our work in Sec. 6. This work presents a novel multi-modal Variational Auto Encoder which is derived from the complete marginal joint log-likelihood. We showed that this expression can jointly be trained on an Mixture-of-Gaussian dataset with ambiguous observations, as well as on a complex dataset derived from MNIST and fashion-MNIST. Furthermore, we formulated requirements and characteristics for multi-modal data for sensor fusion and derived a technique to learn new datasets, namely the proposed entangled-MNIST, which suffice these requirements. Lastly, we developed the idea of in-place sensor fusion in distributed, active sensing scenarios and formulated the requirements, by means of auto re-encoding, to VAEs. This revealed the properties of VAEs, that they tend to denoise the observable data which leads to an attractor behavior in latent space. However, we performed all qualitative evaluations of the latent space with the premise in mind, that a good generative model should not just generate good data but also gives a good latent representation. This does also correlate with the quantitative behaviors, as our proposed model achieved the highest ELBO values. Future work will concentrate on the integration of the ambiguous resolving characteristics to an epistemic-exploration scenario.
Deriving a general formulation of a multi-modal VAE from the joint marginal log-likelihood. Proposes a multi-modal VAE with a variational bound derived from the chain rule. This paper proposes an objective, M^2VAE, for multi-modal VAEs, which is supposed to learn a more meaningful latent space representation.