Multi-modal Gaussian Process Variational Autoencoders for Neural and
  Behavioral Data
Rabia Gondur Usama Bin Sikandar Evan Schaffer Mikio Christian Aoi Stephen L Keeley
Characterizing the relationship between neural population activity and
behavioral data is a central goal of neuroscience. While latent variable models
(LVMs) are successful in describing high-dimensional time-series data, they are
typically only designed for a single type of data, making it difficult to
identify structure shared across different experimental data modalities. Here,
we address this shortcoming by proposing an unsupervised LVM which extracts
temporally evolving shared and independent latents for distinct, simultaneously
recorded experimental modalities. We do this by combining Gaussian Process
Factor Analysis (GPFA), an interpretable LVM for neural spiking data with
temporally smooth latent space, with Gaussian Process Variational Autoencoders
(GP-VAEs), which similarly use a GP prior to characterize correlations in a
latent space, but admit rich expressivity due to a deep neural network mapping
to observations. We achieve interpretability in our model by partitioning
latent variability into components that are either shared between or
independent to each modality. We parameterize the latents of our model in the
Fourier domain, and show improved latent identification using this approach
over standard GP-VAE methods. We validate our model on simulated multi-modal
data consisting of Poisson spike counts and MNIST images that scale and rotate
smoothly over time. We show that the multi-modal GP-VAE (MM-GPVAE) is able to
not only identify the shared and independent latent structure across modalities
accurately, but provides good reconstructions of both images and neural rates
on held-out trials. Finally, we demonstrate our framework on two real world
multi-modal experimental settings: Drosophila whole-brain calcium imaging
alongside tracked limb positions, and Manduca sexta spike train measurements
from ten wing muscles as the animal tracks a visual stimulus.