Modeling Spatio-temporal Dynamical Systems with Neural Discrete Learning
  and Levels-of-Experts
Kun Wang Hao Wu Guibin Zhang Junfeng Fang Yuxuan Liang Yuankai Wu Roger Zimmermann Yang Wang
In this paper, we address the issue of modeling and estimating changes in the
state of the spatio-temporal dynamical systems based on a sequence of
observations like video frames. Traditional numerical simulation systems depend
largely on the initial settings and correctness of the constructed partial
differential equations (PDEs). Despite recent efforts yielding significant
success in discovering data-driven PDEs with neural networks, the limitations
posed by singular scenarios and the absence of local insights prevent them from
performing effectively in a broader real-world context. To this end, this paper
propose the universal expert module -- that is, optical flow estimation
component, to capture the evolution laws of general physical processes in a
data-driven fashion. To enhance local insight, we painstakingly design a
finer-grained physical pipeline, since local characteristics may be influenced
by various internal contextual information, which may contradict the
macroscopic properties of the whole system. Further, we harness currently
popular neural discrete learning to unveil the underlying important features in
its latent space, this process better injects interpretability, which can help
us obtain a powerful prior over these discrete random variables. We conduct
extensive experiments and ablations to demonstrate that the proposed framework
achieves large performance margins, compared with the existing SOTA baselines.