Leveraging GPT-4 for Food Effect Summarization to Enhance
  Product-Specific Guidance Development via Iterative Prompting
Yiwen Shi Ping Ren Jing Wang Biao Han Taha ValizadehAslani Felix Agbavor Yi Zhang Meng Hu Liang Zhao Hualou Liang
Food effect summarization from New Drug Application (NDA) is an essential
component of product-specific guidance (PSG) development and assessment.
However, manual summarization of food effect from extensive drug application
review documents is time-consuming, which arouses a need to develop automated
methods. Recent advances in large language models (LLMs) such as ChatGPT and
GPT-4, have demonstrated great potential in improving the effectiveness of
automated text summarization, but its ability regarding the accuracy in
summarizing food effect for PSG assessment remains unclear. In this study, we
introduce a simple yet effective approach, iterative prompting, which allows
one to interact with ChatGPT or GPT-4 more effectively and efficiently through
multi-turn interaction. Specifically, we propose a three-turn iterative
prompting approach to food effect summarization in which the keyword-focused
and length-controlled prompts are respectively provided in consecutive turns to
refine the quality of the generated summary. We conduct a series of extensive
evaluations, ranging from automated metrics to FDA professionals and even
evaluation by GPT-4, on 100 NDA review documents selected over the past five
years. We observe that the summary quality is progressively improved throughout
the process. Moreover, we find that GPT-4 performs better than ChatGPT, as
evaluated by FDA professionals (43% vs. 12%) and GPT-4 (64% vs. 35%).
Importantly, all the FDA professionals unanimously rated that 85% of the
summaries generated by GPT-4 are factually consistent with the golden reference
summary, a finding further supported by GPT-4 rating of 72% consistency. These
results strongly suggest a great potential for GPT-4 to draft food effect
summaries that could be reviewed by FDA professionals, thereby improving the
efficiency of PSG assessment cycle and promoting the generic drug product
development.