Operator-learning-inspired Modeling of Neural Ordinary Differential
  Equations
Woojin Cho Seunghyeon Cho Hyundong Jin Jinsung Jeon Kookjin Lee Sanghyun Hong Dongeun Lee Jonghyun Choi Noseong Park
Neural ordinary differential equations (NODEs), one of the most influential
works of the differential equation-based deep learning, are to continuously
generalize residual networks and opened a new field. They are currently
utilized for various downstream tasks, e.g., image classification, time series
classification, image generation, etc. Its key part is how to model the
time-derivative of the hidden state, denoted dh(t)/dt. People have habitually
used conventional neural network architectures, e.g., fully-connected layers
followed by non-linear activations. In this paper, however, we present a neural
operator-based method to define the time-derivative term. Neural operators were
initially proposed to model the differential operator of partial differential
equations (PDEs). Since the time-derivative of NODEs can be understood as a
special type of the differential operator, our proposed method, called branched
Fourier neural operator (BFNO), makes sense. In our experiments with general
downstream tasks, our method significantly outperforms existing methods.