American Stories: A Large-Scale Structured Text Dataset of Historical
  U.S. Newspapers
Melissa Dell Jacob Carlson Tom Bryan Emily Silcock Abhishek Arora Zejiang Shen Luca D'Amico-Wong Quan Le Pablo Querubin Leander Heldring
Existing full text datasets of U.S. public domain newspapers do not recognize
the often complex layouts of newspaper scans, and as a result the digitized
content scrambles texts from articles, headlines, captions, advertisements, and
other layout regions. OCR quality can also be low. This study develops a novel,
deep learning pipeline for extracting full article texts from newspaper images
and applies it to the nearly 20 million scans in Library of Congress's public
domain Chronicling America collection. The pipeline includes layout detection,
legibility classification, custom OCR, and association of article texts
spanning multiple bounding boxes. To achieve high scalability, it is built with
efficient architectures designed for mobile phones. The resulting American
Stories dataset provides high quality data that could be used for pre-training
a large language model to achieve better understanding of historical English
and historical world knowledge. The dataset could also be added to the external
database of a retrieval-augmented language model to make historical information
- ranging from interpretations of political events to minutiae about the lives
of people's ancestors - more widely accessible. Furthermore, structured article
texts facilitate using transformer-based methods for popular social science
applications like topic classification, detection of reproduced content, and
news story clustering. Finally, American Stories provides a massive silver
quality dataset for innovating multimodal layout analysis models and other
multimodal applications.