Rolling Shutter Correction with Intermediate Distortion Flow Estimation
Mingdeng Cao Sidi Yang Yujiu Yang Yinqiang Zheng
This paper proposes to correct the rolling shutter (RS) distorted images by
estimating the distortion flow from the global shutter (GS) to RS directly.
Existing methods usually perform correction using the undistortion flow from
the RS to GS. They initially predict the flow from consecutive RS frames,
subsequently rescaling it as the displacement fields from the RS frame to the
underlying GS image using time-dependent scaling factors. Following this,
RS-aware forward warping is employed to convert the RS image into its GS
counterpart. Nevertheless, this strategy is prone to two shortcomings. First,
the undistortion flow estimation is rendered inaccurate by merely linear
scaling the flow, due to the complex non-linear motion nature. Second, RS-aware
forward warping often results in unavoidable artifacts. To address these
limitations, we introduce a new framework that directly estimates the
distortion flow and rectifies the RS image with the backward warping operation.
More specifically, we first propose a global correlation-based flow attention
mechanism to estimate the initial distortion flow and GS feature jointly, which
are then refined by the following coarse-to-fine decoder layers. Additionally,
a multi-distortion flow prediction strategy is integrated to mitigate the issue
of inaccurate flow estimation further. Experimental results validate the
effectiveness of the proposed method, which outperforms state-of-the-art
approaches on various benchmarks while maintaining high efficiency. The project
is available at \url{https://github.com/ljzycmd/DFRSC}.