MegaWika: Millions of reports and their sources across 50 diverse
  languages
Samuel Barham Orion Weller Michelle Yuan Kenton Murray Mahsa Yarmohammadi Zhengping Jiang Siddharth Vashishtha Alexander Martin Anqi Liu Aaron Steven White Jordan Boyd-Graber Benjamin Van Durme
To foster the development of new models for collaborative AI-assisted report
generation, we introduce MegaWika, consisting of 13 million Wikipedia articles
in 50 diverse languages, along with their 71 million referenced source
materials. We process this dataset for a myriad of applications, going beyond
the initial Wikipedia citation extraction and web scraping of content,
including translating non-English articles for cross-lingual applications and
providing FrameNet parses for automated semantic analysis. MegaWika is the
largest resource for sentence-level report generation and the only report
generation dataset that is multilingual. We manually analyze the quality of
this resource through a semantically stratified sample. Finally, we provide
baseline results and trained models for crucial steps in automated report
generation: cross-lingual question answering and citation retrieval.