Video Dynamics Prior: An Internal Learning Approach for Robust Video
  Enhancements
Gaurav Shrivastava Ser-Nam Lim Abhinav Shrivastava
In this paper, we present a novel robust framework for low-level vision
tasks, including denoising, object removal, frame interpolation, and
super-resolution, that does not require any external training data corpus. Our
proposed approach directly learns the weights of neural modules by optimizing
over the corrupted test sequence, leveraging the spatio-temporal coherence and
internal statistics of videos. Furthermore, we introduce a novel spatial
pyramid loss that leverages the property of spatio-temporal patch recurrence in
a video across the different scales of the video. This loss enhances robustness
to unstructured noise in both the spatial and temporal domains. This further
results in our framework being highly robust to degradation in input frames and
yields state-of-the-art results on downstream tasks such as denoising, object
removal, and frame interpolation. To validate the effectiveness of our
approach, we conduct qualitative and quantitative evaluations on standard video
datasets such as DAVIS, UCF-101, and VIMEO90K-T.