OneActor: Consistent Character Generation via Cluster-Conditioned
  Guidance
Jiahao Wang Caixia Yan Haonan Lin Weizhan Zhang
Text-to-image diffusion models benefit artists with high-quality image
generation. Yet its stochastic nature prevent artists from creating consistent
images of the same character. Existing methods try to tackle this challenge and
generate consistent content in various ways. However, they either depend on
external data or require expensive tuning of the diffusion model. For this
issue, we argue that a lightweight but intricate guidance is enough to
function. Aiming at this, we lead the way to formalize the objective of
consistent generation, derive a clustering-based score function and propose a
novel paradigm, OneActor. We design a cluster-conditioned model which
incorporates posterior samples to guide the denoising trajectories towards the
target cluster. To overcome the overfitting challenge shared by one-shot tuning
pipelines, we devise auxiliary components to simultaneously augment the tuning
and regulate the inference. This technique is later verified to significantly
enhance the content diversity of generated images. Comprehensive experiments
show that our method outperforms a variety of baselines with satisfactory
character consistency, superior prompt conformity as well as high image
quality. And our method is at least 4 times faster than tuning-based baselines.
Furthermore, to our best knowledge, we first prove that the semantic space has
the same interpolation property as the latent space dose. This property can
serve as another promising tool for fine generation control.