Masked Autoencoders are PDE Learners
Anthony Zhou Amir Barati Farimani
Neural solvers for partial differential equations (PDEs) have great
potential, yet their practicality is currently limited by their
generalizability. PDEs evolve over broad scales and exhibit diverse behaviors;
predicting these phenomena will require learning representations across a wide
variety of inputs, which may encompass different coefficients, geometries, or
equations. As a step towards generalizable PDE modeling, we adapt masked
pretraining for PDEs. Through self-supervised learning across PDEs, masked
autoencoders can learn useful latent representations for downstream tasks. In
particular, masked pretraining can improve coefficient regression and
timestepping performance of neural solvers on unseen equations. We hope that
masked pretraining can emerge as a unifying method across large, unlabeled, and
heterogeneous datasets to learn latent physics at scale.