Deep Video Restoration for Under-Display Camera
Xuanxi Chen Tao Wang Ziqian Shao Kaihao Zhang Wenhan Luo Tong Lu Zikun Liu Tae-Kyun Kim Hongdong Li
Images or videos captured by the Under-Display Camera (UDC) suffer from
severe degradation, such as saturation degeneration and color shift. While
restoration for UDC has been a critical task, existing works of UDC restoration
focus only on images. UDC video restoration (UDC-VR) has not been explored in
the community. In this work, we first propose a GAN-based generation pipeline
to simulate the realistic UDC degradation process. With the pipeline, we build
the first large-scale UDC video restoration dataset called PexelsUDC, which
includes two subsets named PexelsUDC-T and PexelsUDC-P corresponding to
different displays for UDC. Using the proposed dataset, we conduct extensive
benchmark studies on existing video restoration methods and observe their
limitations on the UDC-VR task. To this end, we propose a novel
transformer-based baseline method that adaptively enhances degraded videos. The
key components of the method are a spatial branch with local-aware
transformers, a temporal branch embedded temporal transformers, and a
spatial-temporal fusion module. These components drive the model to fully
exploit spatial and temporal information for UDC-VR. Extensive experiments show
that our method achieves state-of-the-art performance on PexelsUDC. The
benchmark and the baseline method are expected to promote the progress of
UDC-VR in the community, which will be made public.