Text2AC-Zero: Consistent Synthesis of Animated Characters using 2D
  Diffusion
Abdelrahman Eldesokey Peter Wonka
We propose a zero-shot approach for consistent Text-to-Animated-Characters
synthesis based on pre-trained Text-to-Image (T2I) diffusion models. Existing
Text-to-Video (T2V) methods are expensive to train and require large-scale
video datasets to produce diverse characters and motions. At the same time,
their zero-shot alternatives fail to produce temporally consistent videos. We
strive to bridge this gap, and we introduce a zero-shot approach that produces
temporally consistent videos of animated characters and requires no training or
fine-tuning. We leverage existing text-based motion diffusion models to
generate diverse motions that we utilize to guide a T2I model. To achieve
temporal consistency, we introduce the Spatial Latent Alignment module that
exploits cross-frame dense correspondences that we compute to align the latents
of the video frames. Furthermore, we propose Pixel-Wise Guidance to steer the
diffusion process in a direction that minimizes visual discrepancies. Our
proposed approach generates temporally consistent videos with diverse motions
and styles, outperforming existing zero-shot T2V approaches in terms of
pixel-wise consistency and user preference.