Assisting humans in complex comparisons: automated information
  comparison at scale
Truman Yuen Graham A. Watt Yuri Lawryshyn
Generative Large Language Models enable efficient analytics across knowledge
domains, rivalling human experts in information comparisons. However, the
applications of LLMs for information comparisons face scalability challenges
due to the difficulties in maintaining information across large contexts and
overcoming model token limitations. To address these challenges, we developed
the novel Abstractive Summarization \& Criteria-driven Comparison Endpoint
(ASC$^2$End) system to automate information comparison at scale. Our system
employs Semantic Text Similarity comparisons for generating evidence-supported
analyses. We utilize proven data-handling strategies such as abstractive
summarization and retrieval augmented generation to overcome token limitations
and retain relevant information during model inference. Prompts were designed
using zero-shot strategies to contextualize information for improved model
reasoning. We evaluated abstractive summarization using ROUGE scoring and
assessed the generated comparison quality using survey responses. Models
evaluated on the ASC$^2$End system show desirable results providing insights on
the expected performance of the system. ASC$^2$End is a novel system and tool
that enables accurate, automated information comparison at scale across
knowledge domains, overcoming limitations in context length and retrieval.