NovelQA: A Benchmark for Long-Range Novel Question Answering
Cunxiang Wang Ruoxi Ning Boqi Pan Tonghui Wu Qipeng Guo Cheng Deng Guangsheng Bao Qian Wang Yue Zhang
The rapid advancement of Large Language Models (LLMs) has introduced a new
frontier in natural language processing, particularly in understanding and
processing long-context information. However, the evaluation of these models'
long-context abilities remains a challenge due to the limitations of current
benchmarks. To address this gap, we introduce NovelQA, a benchmark specifically
designed to test the capabilities of LLMs with extended texts. Constructed from
English novels, NovelQA offers a unique blend of complexity, length, and
narrative coherence, making it an ideal tool for assessing deep textual
understanding in LLMs. This paper presents the design and construction of
NovelQA, highlighting its manual annotation, and diverse question types. Our
evaluation of Long-context LLMs on NovelQA reveals significant insights into
the models' performance, particularly emphasizing the challenges they face with
multi-hop reasoning, detail-oriented questions, and extremely long input with
more than 100,000 tokens. The results underscore the necessity for further
advancements in LLMs to improve their long-context comprehension and
computational literary studies.