PNeRV: Enhancing Spatial Consistency via Pyramidal Neural Representation
  for Videos
Qi Zhao M. Salman Asif Zhan Ma
The primary focus of Neural Representation for Videos (NeRV) is to
effectively model its spatiotemporal consistency. However, current NeRV systems
often face a significant issue of spatial inconsistency, leading to decreased
perceptual quality. To address this issue, we introduce the Pyramidal Neural
Representation for Videos (PNeRV), which is built on a multi-scale information
connection and comprises a lightweight rescaling operator, Kronecker
Fully-connected layer (KFc), and a Benign Selective Memory (BSM) mechanism. The
KFc, inspired by the tensor decomposition of the vanilla Fully-connected layer,
facilitates low-cost rescaling and global correlation modeling. BSM merges
high-level features with granular ones adaptively. Furthermore, we provide an
analysis based on the Universal Approximation Theory of the NeRV system and
validate the effectiveness of the proposed PNeRV.We conducted comprehensive
experiments to demonstrate that PNeRV surpasses the performance of contemporary
NeRV models, achieving the best results in video regression on UVG and DAVIS
under various metrics (PSNR, SSIM, LPIPS, and FVD). Compared to vanilla NeRV,
PNeRV achieves a +4.49 dB gain in PSNR and a 231% increase in FVD on UVG, along
with a +3.28 dB PSNR and 634% FVD increase on DAVIS.