Disordered-DABS: A Benchmark for Dynamic Aspect-Based Summarization in
  Disordered Texts
Xiaobo Guo Soroush Vosoughi
Aspect-based summarization has seen significant advancements, especially in
structured text. Yet, summarizing disordered, large-scale texts, like those
found in social media and customer feedback, remains a significant challenge.
Current research largely targets predefined aspects within structured texts,
neglecting the complexities of dynamic and disordered environments. Addressing
this gap, we introduce Disordered-DABS, a novel benchmark for dynamic
aspect-based summarization tailored to unstructured text. Developed by adapting
existing datasets for cost-efficiency and scalability, our comprehensive
experiments and detailed human evaluations reveal that Disordered-DABS poses
unique challenges to contemporary summarization models, including
state-of-the-art language models such as GPT-3.5.