ECNet: Effective Controllable Text-to-Image Diffusion Models
Sicheng Li Keqiang Sun Zhixin Lai Xiaoshi Wu Feng Qiu Haoran Xie Kazunori Miyata Hongsheng Li
The conditional text-to-image diffusion models have garnered significant
attention in recent years. However, the precision of these models is often
compromised mainly for two reasons, ambiguous condition input and inadequate
condition guidance over single denoising loss. To address the challenges, we
introduce two innovative solutions. Firstly, we propose a Spatial Guidance
Injector (SGI) which enhances conditional detail by encoding text inputs with
precise annotation information. This method directly tackles the issue of
ambiguous control inputs by providing clear, annotated guidance to the model.
Secondly, to overcome the issue of limited conditional supervision, we
introduce Diffusion Consistency Loss (DCL), which applies supervision on the
denoised latent code at any given time step. This encourages consistency
between the latent code at each time step and the input signal, thereby
enhancing the robustness and accuracy of the output. The combination of SGI and
DCL results in our Effective Controllable Network (ECNet), which offers a more
accurate controllable end-to-end text-to-image generation framework with a more
precise conditioning input and stronger controllable supervision. We validate
our approach through extensive experiments on generation under various
conditions, such as human body skeletons, facial landmarks, and sketches of
general objects. The results consistently demonstrate that our method
significantly enhances the controllability and robustness of the generated
images, outperforming existing state-of-the-art controllable text-to-image
models.