Continuous Piecewise-Affine Based Motion Model for Image Animation
Hexiang Wang Fengqi Liu Qianyu Zhou Ran Yi Xin Tan Lizhuang Ma
Image animation aims to bring static images to life according to driving
videos and create engaging visual content that can be used for various purposes
such as animation, entertainment, and education. Recent unsupervised methods
utilize affine and thin-plate spline transformations based on keypoints to
transfer the motion in driving frames to the source image. However, limited by
the expressive power of the transformations used, these methods always produce
poor results when the gap between the motion in the driving frame and the
source image is large. To address this issue, we propose to model motion from
the source image to the driving frame in highly-expressive diffeomorphism
spaces. Firstly, we introduce Continuous Piecewise-Affine based (CPAB)
transformation to model the motion and present a well-designed inference
algorithm to generate CPAB transformation from control keypoints. Secondly, we
propose a SAM-guided keypoint semantic loss to further constrain the keypoint
extraction process and improve the semantic consistency between the
corresponding keypoints on the source and driving images. Finally, we design a
structure alignment loss to align the structure-related features extracted from
driving and generated images, thus helping the generator generate results that
are more consistent with the driving action. Extensive experiments on four
datasets demonstrate the effectiveness of our method against state-of-the-art
competitors quantitatively and qualitatively. Code will be publicly available
at: https://github.com/DevilPG/AAAI2024-CPABMM.