Attack Deterministic Conditional Image Generative Models for Diverse and
  Controllable Generation
Tianyi Chu Wei Xing Jiafu Chen Zhizhong Wang Jiakai Sun Lei Zhao Haibo Chen Huaizhong Lin
Existing generative adversarial network (GAN) based conditional image
generative models typically produce fixed output for the same conditional
input, which is unreasonable for highly subjective tasks, such as large-mask
image inpainting or style transfer. On the other hand, GAN-based diverse image
generative methods require retraining/fine-tuning the network or designing
complex noise injection functions, which is computationally expensive,
task-specific, or struggle to generate high-quality results. Given that many
deterministic conditional image generative models have been able to produce
high-quality yet fixed results, we raise an intriguing question: is it possible
for pre-trained deterministic conditional image generative models to generate
diverse results without changing network structures or parameters? To answer
this question, we re-examine the conditional image generation tasks from the
perspective of adversarial attack and propose a simple and efficient plug-in
projected gradient descent (PGD) like method for diverse and controllable image
generation. The key idea is attacking the pre-trained deterministic generative
models by adding a micro perturbation to the input condition. In this way,
diverse results can be generated without any adjustment of network structures
or fine-tuning of the pre-trained models. In addition, we can also control the
diverse results to be generated by specifying the attack direction according to
a reference text or image. Our work opens the door to applying adversarial
attack to low-level vision tasks, and experiments on various conditional image
generation tasks demonstrate the effectiveness and superiority of the proposed
method.