Learning Latent Dynamics via Invariant Decomposition and
  (Spatio-)Temporal Transformers
Kai Lagemann Christian Lagemann Sach Mukherjee
We propose a method for learning dynamical systems from high-dimensional
empirical data that combines variational autoencoders and (spatio-)temporal
attention within a framework designed to enforce certain
scientifically-motivated invariances. We focus on the setting in which data are
available from multiple different instances of a system whose underlying
dynamical model is entirely unknown at the outset. The approach rests on a
separation into an instance-specific encoding (capturing initial conditions,
constants etc.) and a latent dynamics model that is itself universal across all
instances/realizations of the system. The separation is achieved in an
automated, data-driven manner and only empirical data are required as inputs to
the model. The approach allows effective inference of system behaviour at any
continuous time but does not require an explicit neural ODE formulation, which
makes it efficient and highly scalable. We study behaviour through simple
theoretical analyses and extensive experiments on synthetic and real-world
datasets. The latter investigate learning the dynamics of complex systems based
on finite data and show that the proposed approach can outperform
state-of-the-art neural-dynamical models. We study also more general inductive
bias in the context of transfer to data obtained under entirely novel system
interventions. Overall, our results provide a promising new framework for
efficiently learning dynamical models from heterogeneous data with potential
applications in a wide range of fields including physics, medicine, biology and
engineering.