LatentWarp: Consistent Diffusion Latents for Zero-Shot Video-to-Video
  Translation
Yuxiang Bao Di Qiu Guoliang Kang Baochang Zhang Bo Jin Kaiye Wang Pengfei Yan
Leveraging the generative ability of image diffusion models offers great
potential for zero-shot video-to-video translation. The key lies in how to
maintain temporal consistency across generated video frames by image diffusion
models. Previous methods typically adopt cross-frame attention, \emph{i.e.,}
sharing the \textit{key} and \textit{value} tokens across attentions of
different frames, to encourage the temporal consistency. However, in those
works, temporal inconsistency issue may not be thoroughly solved, rendering the
fidelity of generated videos limited.%The current state of the art cross-frame
attention method aims at maintaining fine-grained visual details across frames,
but it is still challenged by the temporal coherence problem. In this paper, we
find the bottleneck lies in the unconstrained query tokens and propose a new
zero-shot video-to-video translation framework, named \textit{LatentWarp}. Our
approach is simple: to constrain the query tokens to be temporally consistent,
we further incorporate a warping operation in the latent space to constrain the
query tokens. Specifically, based on the optical flow obtained from the
original video, we warp the generated latent features of last frame to align
with the current frame during the denoising process. As a result, the
corresponding regions across the adjacent frames can share closely-related
query tokens and attention outputs, which can further improve latent-level
consistency to enhance visual temporal coherence of generated videos. Extensive
experiment results demonstrate the superiority of \textit{LatentWarp} in
achieving video-to-video translation with temporal coherence.