Capability-aware Prompt Reformulation Learning for Text-to-Image
  Generation
Jingtao Zhan Qingyao Ai Yiqun Liu Jia Chen Shaoping Ma
Text-to-image generation systems have emerged as revolutionary tools in the
realm of artistic creation, offering unprecedented ease in transforming textual
prompts into visual art. However, the efficacy of these systems is intricately
linked to the quality of user-provided prompts, which often poses a challenge
to users unfamiliar with prompt crafting. This paper addresses this challenge
by leveraging user reformulation data from interaction logs to develop an
automatic prompt reformulation model. Our in-depth analysis of these logs
reveals that user prompt reformulation is heavily dependent on the individual
user's capability, resulting in significant variance in the quality of
reformulation pairs. To effectively use this data for training, we introduce
the Capability-aware Prompt Reformulation (CAPR) framework. CAPR innovatively
integrates user capability into the reformulation process through two key
components: the Conditional Reformulation Model (CRM) and Configurable
Capability Features (CCF). CRM reformulates prompts according to a specified
user capability, as represented by CCF. The CCF, in turn, offers the
flexibility to tune and guide the CRM's behavior. This enables CAPR to
effectively learn diverse reformulation strategies across various user
capacities and to simulate high-capability user reformulation during inference.
Extensive experiments on standard text-to-image generation benchmarks showcase
CAPR's superior performance over existing baselines and its remarkable
robustness on unseen systems. Furthermore, comprehensive analyses validate the
effectiveness of different components. CAPR can facilitate user-friendly
interaction with text-to-image systems and make advanced artistic creation more
achievable for a broader range of users.