Real-World Video for Zoom Enhancement based on Spatio-Temporal Coupling
Zhiling Guo Yinqiang Zheng Haoran Zhang Xiaodan Shi Zekun Cai Ryosuke Shibasaki Jinyue Yan
In recent years, single-frame image super-resolution (SR) has become more
realistic by considering the zooming effect and using real-world short- and
long-focus image pairs. In this paper, we further investigate the feasibility
of applying realistic multi-frame clips to enhance zoom quality via
spatio-temporal information coupling. Specifically, we first built a real-world
video benchmark, VideoRAW, by a synchronized co-axis optical system. The
dataset contains paired short-focus raw and long-focus sRGB videos of different
dynamic scenes. Based on VideoRAW, we then presented a Spatio-Temporal Coupling
Loss, termed as STCL. The proposed STCL is intended for better utilization of
information from paired and adjacent frames to align and fuse features both
temporally and spatially at the feature level. The outperformed experimental
results obtained in different zoom scenarios demonstrate the superiority of
integrating real-world video dataset and STCL into existing SR models for zoom
quality enhancement, and reveal that the proposed method can serve as an
advanced and viable tool for video zoom.