Bridge Diffusion Model: bridge non-English language-native text-to-image
  diffusion model with English communities
Shanyuan Liu Dawei Leng Yuhui Yin
Text-to-Image generation (TTI) technologies are advancing rapidly, especially
in the English language communities. However, English-native TTI models
inherently carry biases from English world centric training data, which creates
a dilemma for development of other language-native TTI models. One common
choice is fine-tuning the English-native TTI model with translated samples from
non-English communities. It falls short of fully addressing the model bias
problem. Alternatively, training non-English language native models from
scratch can effectively resolve the English world bias, but diverges from the
English TTI communities, thus not able to utilize the strides continuously
gaining in the English TTI communities any more. To build non-English language
native TTI model meanwhile keep compatability with the English TTI communities,
we propose a novel model structure referred as "Bridge Diffusion Model" (BDM).
The proposed BDM employs a backbone-branch network structure to learn the
non-English language semantics while keep the latent space compatible with the
English-native TTI backbone, in an end-to-end manner. The unique advantages of
the proposed BDM are that it's not only adept at generating images that
precisely depict non-English language semantics, but also compatible with
various English-native TTI plugins, such as different checkpoints, LoRA,
ControlNet, Dreambooth, and Textual Inversion, etc. Moreover, BDM can
concurrently generate content seamlessly combining both non-English native and
English-native semantics within a single image, fostering cultural interaction.
We verify our method by applying BDM to build a Chinese-native TTI model,
whereas the method is generic and applicable to any other language.