Prompt Evolution for Generative AI: A Classifier-Guided Approach
Melvin Wong Yew-Soon Ong Abhishek Gupta Kavitesh K. Bali Caishun Chen
Synthesis of digital artifacts conditioned on user prompts has become an
important paradigm facilitating an explosion of use cases with generative AI.
However, such models often fail to connect the generated outputs and desired
target concepts/preferences implied by the prompts. Current research addressing
this limitation has largely focused on enhancing the prompts before output
generation or improving the model's performance up front. In contrast, this
paper conceptualizes prompt evolution, imparting evolutionary selection
pressure and variation during the generative process to produce multiple
outputs that satisfy the target concepts/preferences better. We propose a
multi-objective instantiation of this broader idea that uses a multi-label
image classifier-guided approach. The predicted labels from the classifiers
serve as multiple objectives to optimize, with the aim of producing diversified
images that meet user preferences. A novelty of our evolutionary algorithm is
that the pre-trained generative model gives us implicit mutation operations,
leveraging the model's stochastic generative capability to automate the
creation of Pareto-optimized images more faithful to user preferences.