ChatCite: LLM Agent with Human Workflow Guidance for Comparative
  Literature Summary
Yutong Li Lu Chen Aiwei Liu Kai Yu Lijie Wen
The literature review is an indispensable step in the research process. It
provides the benefit of comprehending the research problem and understanding
the current research situation while conducting a comparative analysis of prior
works. However, literature summary is challenging and time consuming. The
previous LLM-based studies on literature review mainly focused on the complete
process, including literature retrieval, screening, and summarization. However,
for the summarization step, simple CoT method often lacks the ability to
provide extensive comparative summary. In this work, we firstly focus on the
independent literature summarization step and introduce ChatCite, an LLM agent
with human workflow guidance for comparative literature summary. This agent, by
mimicking the human workflow, first extracts key elements from relevant
literature and then generates summaries using a Reflective Incremental
Mechanism. In order to better evaluate the quality of the generated summaries,
we devised a LLM-based automatic evaluation metric, G-Score, in refer to the
human evaluation criteria. The ChatCite agent outperformed other models in
various dimensions in the experiments. The literature summaries generated by
ChatCite can also be directly used for drafting literature reviews.