Exploiting Inductive Biases in Video Modeling through Neural CDEs
Johnathan Chiu Samuel Duffield Max Hunter-Gordon Kaelan Donatella Max Aifer Andi Gu
We introduce a novel approach to video modeling that leverages controlled
differential equations (CDEs) to address key challenges in video tasks, notably
video interpolation and mask propagation. We apply CDEs at varying resolutions
leading to a continuous-time U-Net architecture. Unlike traditional methods,
our approach does not require explicit optical flow learning, and instead makes
use of the inherent continuous-time features of CDEs to produce a highly
expressive video model. We demonstrate competitive performance against
state-of-the-art models for video interpolation and mask propagation tasks.