ControlNet-XS: Designing an Efficient and Effective Architecture for
  Controlling Text-to-Image Diffusion Models
Denis Zavadski Johann-Friedrich Feiden Carsten Rother
The field of image synthesis has made tremendous strides forward in the last
years. Besides defining the desired output image with text-prompts, an
intuitive approach is to additionally use spatial guidance in form of an image,
such as a depth map. For this, a recent and highly popular approach is to use a
controlling network, such as ControlNet, in combination with a pre-trained
image generation model, such as Stable Diffusion. When evaluating the design of
existing controlling networks, we observe that they all suffer from the same
problem of a delay in information flowing between the generation and
controlling process. This, in turn, means that the controlling network must
have generative capabilities. In this work we propose a new controlling
architecture, called ControlNet-XS, which does not suffer from this problem,
and hence can focus on the given task of learning to control. In contrast to
ControlNet, our model needs only a fraction of parameters, and hence is about
twice as fast during inference and training time. Furthermore, the generated
images are of higher quality and the control is of higher fidelity. All code
and pre-trained models will be made publicly available.