Characterizing Multimodal Long-form Summarization: A Case Study on
  Financial Reports
Tianyu Cao Natraj Raman Danial Dervovic Chenhao Tan
As large language models (LLMs) expand the power of natural language
processing to handle long inputs, rigorous and systematic analyses are
necessary to understand their abilities and behavior. A salient application is
summarization, due to its ubiquity and controversy (e.g., researchers have
declared the death of summarization). In this paper, we use financial report
summarization as a case study because financial reports not only are long but
also use numbers and tables extensively. We propose a computational framework
for characterizing multimodal long-form summarization and investigate the
behavior of Claude 2.0/2.1, GPT-4/3.5, and Command. We find that GPT-3.5 and
Command fail to perform this summarization task meaningfully. For Claude 2 and
GPT-4, we analyze the extractiveness of the summary and identify a position
bias in LLMs. This position bias disappears after shuffling the input for
Claude, which suggests that Claude has the ability to recognize important
information. We also conduct a comprehensive investigation on the use of
numeric data in LLM-generated summaries and offer a taxonomy of numeric
hallucination. We employ prompt engineering to improve GPT-4's use of numbers
with limited success. Overall, our analyses highlight the strong capability of
Claude 2 in handling long multimodal inputs compared to GPT-4.