Dynamic Texture Transfer using PatchMatch and Transformers
Guo Pu Shiyao Xu Xixin Cao Zhouhui Lian
How to automatically transfer the dynamic texture of a given video to the
target still image is a challenging and ongoing problem. In this paper, we
propose to handle this task via a simple yet effective model that utilizes both
PatchMatch and Transformers. The key idea is to decompose the task of dynamic
texture transfer into two stages, where the start frame of the target video
with the desired dynamic texture is synthesized in the first stage via a
distance map guided texture transfer module based on the PatchMatch algorithm.
Then, in the second stage, the synthesized image is decomposed into
structure-agnostic patches, according to which their corresponding subsequent
patches can be predicted by exploiting the powerful capability of Transformers
equipped with VQ-VAE for processing long discrete sequences. After getting all
those patches, we apply a Gaussian weighted average merging strategy to
smoothly assemble them into each frame of the target stylized video.
Experimental results demonstrate the effectiveness and superiority of the
proposed method in dynamic texture transfer compared to the state of the art.