Quality assurance in high volume document digitization: a survey
---
paper_title: Automated Data Entry System: Performance Issues

paper_content:

This paper discusses the performance of a system for extracting bibliographic fields from scanned pages in biomedical journals to populate MEDLINE, the flagship database of the national Library of Medicine (NLM), and heavily used worldwide. This system consists of automated processes to extract the article title, author names, affiliations and abstract, and manual workstations for the entry of other required fields such as pagination, grant support information, databank accession numbers and others needed for a completed bibliographic record in MEDLINE. Labor and time data are given for (1) a wholly manual keyboarding process to create the records, (2) an OCR-based system that requires all fields except the abstract to be manually input, and (3) a more automated system that relies on document image analysis and understanding techniques for the extraction of several fields. It is shown that this last, most automated, approach requires less than 25% of the labor effort in the first, manual, process.
---
paper_title: Deployment of Quality Assurance Procedures for Digital Library Programms

paper_content:

Many digital library programmes have a development philosophy based on use of open standards. In practice, however, projects may not have procedures in place to ensure that project deliverables make use of appropriate open standards. In addition there will be occasions when open standards are not sufficiently mature for deployment in a service environment or use of open standards will require expertise or resources which are not readily available.
---
paper_title: Automated quality assurance for document understanding systems

paper_content:

To process high-volume input data, such as the scanned images of publishers' book and journal collections, content understanding systems should run automatically, continuously, and without human attendance. Ensuring the output quality of such systems is a challenging task, however, and automated quality assurance techniques are thus essential to its success. The author discusses three automated QA techniques that were developed for Hewlett-Packard's Digital Content ReMastering system.
---
paper_title: Difficult and urgent open problems in document image analysis for libraries

paper_content:

Open problems in document image analysis research and development relevant to digital libraries (DLs) are briefly described and their difficulty and urgency estimated. They are grouped according to the stage of processing, in the construction of DLs, where they first apply: image capture, early processing, content extraction and recognition, structure analysis, retrieval, & presentation; and in personal and interactive DLs.
---
paper_title: Digital mountain: from granite archive to global access

paper_content:

Large-scale, multiterabyte digital libraries are becoming feasible due to decreasing costs of storage, CPU, and bandwidth. However, costs associated with preparing content for input into the library remain high due to the amount of human labor required. We describe the digital microfilm pipeline -sequence of image processing operations used to populate a large-scale digital library from a "mountain" of microfilm and reduce the human labor involved. Essential parts of the pipeline include algorithms for document zoning and labeling, consensus-based template creation, reversal of geometric transformations and just-in-time browsing, an interactive technique for progressive access of image content over a low-bandwidth medium. We also suggest more automated approaches to cropping, enhancement and data extraction.
---
paper_title: Generic approach for OCR performance evaluation

paper_content:

This paper presents the limits of the character recognition engines (commercial OCRs) and how to exceed these limits to achieve the industrial goals in terms of document capture and coding performances. The recent integration of these OCRs in several industrial capture chains leads to think that a solution is possible to reach electronically the same performances obtained by human typists. After a global description of the problems and the exposure of the OCR limits, the paper will focus on the methodology used and details the different steps proposed for the individual performance improvement. The first step consists in the individual evaluation of the OCRs. This is made by comparing the OCR result with a ground truth, which allows to highlight its defects and catalogue its main errors on the document processed. The second step allows to increase these individual performances by combination the OCR with some others. Our choice has been fixed on the combination of only two OCRs deemed very efficient and complementary on the same class of documents. The residual errors are treated in the last step which be able to propose a list of heuristics resolving punctually the OCR defects on the limit cases. In order to validate our approach, we present in the second part of the paper a practical case of experimentation to reach industrial performances. This approach has been tested in the framework of an industrial application for automatic document capture, by attempting the lowest score, imposed on one specific document class, of 1 error for 10000 characters.© (2001) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: Creating digital libraries: content generation and re-mastering

paper_content:

This article has two main goals: to describe the automatic creation of a digital library and to provide an overview of the metaalgorithmic patterns that can be applied to increase the accuracy of its creation. Automating the creation of useful digital libraries - that is, digital libraries affording searchable text and reusable ("repurposable") output - is a complicated process, whether the original library is paper-based or already available in electronic form. We outline the steps involved in the creation of a deployable digital library (>1.2 /spl times/ 10/sup 6/ pages) for MIT Press, as well as its implications to other aspects of digital library creation, management, use and repurposing. Input, transformation, information extraction, and output processes are considered in light of their utility in creating layers of content. Interestingly, in some aspects, scanning directly from paper offers extra opportunities for error-checking through feedback-feedforward combination. Strategies for quality assurance (QA) at the document, chapter and book level are also discussed. We emphasize the use of metaalgorithmic design patterns for application towards improving the content generation, extraction and remastering. This approach also increases the ease with which modules and algorithms are added to and deprecated from the system.
---
paper_title: Estimating scanning characteristics from corners in bilevel images

paper_content:

Degradations that occur during scanning can cause errors in Optical Character Recognition (OCR). Scans made in bilevel mode (no gray scale) from high contrast source patterns are the input to the estimation processes. Two scanner system parameters are estimated from bilevel scans using models of the scanning process and bilevel source patterns. The scanner's point spread function (PSF) width and the binarization threshold are estimated by using corner features in the scanned images. These estimation algorithms were tested in simulation and with scanned test patterns. The resulting estimates are close in value to what is expected based on gray-level analysis. The results of estimation are used to produce synthetically scanned characters that in most cases bear a strong resemblance to the characters scanned on the scanner at the same settings as the test pattern used for estimation.
---
paper_title: Difficult and urgent open problems in document image analysis for libraries

paper_content:

Open problems in document image analysis research and development relevant to digital libraries (DLs) are briefly described and their difficulty and urgency estimated. They are grouped according to the stage of processing, in the construction of DLs, where they first apply: image capture, early processing, content extraction and recognition, structure analysis, retrieval, & presentation; and in personal and interactive DLs.
---
paper_title: Measuring document image skew and orientation

paper_content:

Several approaches have previously been taken for identifying document image skew. At issue are efficiency, accuracy, and robustness. We work dire ctly with the image, maximizing a function of the number of ON pixels in a scanline. Image rotation is simulated by either vertical shear or accumulation of pixel counts along sloped lines. Pixel sum differences on adjacent scanlines reduce isotropic background noise from non-text regions. To find the skew angle, a succession of values of this function are found. Angles are chosen hierarchically, typically with both a coarse sweep and a fine angular bifurcation. To inc rease efficiency, measurements are made on subsampled images that have been pre-filtered to m aximize sensitivity to image skew. Results are given for a large set of images, including multiple and unaligned text columns, graphics and large area halftones. The measured intrinsic angular error is inversely proportional to the number of sampling points on a scanline. This method does not indicate when text is upside-down, and it also requires sampling the function at 90 degrees of rotation to measure text skew in landscape mode. However, such text orientation can be determined (as one of four directions) by noting that roman characters in all languages have many more ascenders than descenders, and using morphological operations to identify such pixels. Only a small amount of text is required for accurate statistical determination of orientation, and images without text are identified as such.
---
paper_title: A comprehensive image processing suite for book re-mastering

paper_content:

Converting paper books into electronic form provides benefits for archiving, distribution and content reuse. However, directly scanned images are usually undesirable for electronic books, and automated content re-mastering is required. In this paper, we describe a comprehensive image processing suite consisting of three major components: 1) image enhancement with deskew, cropping, color correction, contrast enhancement and text sharpening, 2) compound document image compression, and 3) extraction of TOC (table of content) and linking. We built a processing pipeline that automatically converts a set of scanned page images into a high quality and highly compressed e-book in the popular PDF format.
---
paper_title: A multiscale approach to restoring scanned color document images with show-through effects

paper_content:

This paper describes a new approach to restoring scanned color document images where the backside image shows through the paper sheet. A new framework is presented for correcting show-through components using digital image processing techniques. First, the foreground components on the front side are separated from the background and backside components through locally adaptive binarization for each color component and edge magnitude thresholding. Background colors are estimated locally through color thresholding to generate a restored image, and then corrected adaptively through multiscale analysis along with comparison of edge distributions between the original and the restored image. The proposed method does not require specific input devices or the backside to be input; it is able to correct unneeded image components through analysis of the front side image alone. Experimental results are given to verify effectiveness of the proposed method.
---
paper_title: Creating digital libraries: content generation and re-mastering

paper_content:

This article has two main goals: to describe the automatic creation of a digital library and to provide an overview of the metaalgorithmic patterns that can be applied to increase the accuracy of its creation. Automating the creation of useful digital libraries - that is, digital libraries affording searchable text and reusable ("repurposable") output - is a complicated process, whether the original library is paper-based or already available in electronic form. We outline the steps involved in the creation of a deployable digital library (>1.2 /spl times/ 10/sup 6/ pages) for MIT Press, as well as its implications to other aspects of digital library creation, management, use and repurposing. Input, transformation, information extraction, and output processes are considered in light of their utility in creating layers of content. Interestingly, in some aspects, scanning directly from paper offers extra opportunities for error-checking through feedback-feedforward combination. Strategies for quality assurance (QA) at the document, chapter and book level are also discussed. We emphasize the use of metaalgorithmic design patterns for application towards improving the content generation, extraction and remastering. This approach also increases the ease with which modules and algorithms are added to and deprecated from the system.
---
paper_title: Rectifying the bound document image captured by the camera: a model based approach

paper_content:

A model based approach for rectifying the camera image of the bound document has been developed, i.e., the surface of the document is represented by a general cylindrical surface. The principle of using the model to unwrap the image is discussed. Practically, the skeleton of each horizontal text line is extracted to help estimate the parameter of the model, and rectify the images. To use the model, only a few priori is required, and no more auxiliary device is necessary. Experiment results are given to demonstrate the feasibility and the stability of the method.
---
paper_title: Structuring documents according to their table of contents

paper_content:

In this paper, we present a method for structuring a document according to the information present in its Table of Contents. The detection of the ToC as well as the determination of the parts it refers to in the document body rely on a series of generic properties characterizing any ToC, while its hierarchization is achieved using clustering techniques. We also report on the robustness and performance of the method before discussing it, in light of related work.
---
paper_title: On foreground-background separation in low quality color document images

paper_content:

This paper proposes an adaptive method for separation of foreground and background in low quality color document images. A connected component labelling is initially implemented to capture the spatially connected similar color pixels. Next, dominant background components are determined to divide the entire image into number of grids each representing local uniformity in illumination, background, etc. Finally foreground parts are located using local information around them. Several color images of old historical documents including manuscripts of high importance are used in the experiment. Apart from a qualitative evaluation, results are quantitatively compared with one popular foreground/background separation technique.
---
paper_title: Color documents on the Web with DjVu

paper_content:

We present a new image compression technique called "DjVu" that is specifically geared towards the compression of scanned documents in color at high resolution. With DjVu, a magazine page in color at 300 dpi typically occupies between 40 KB and 80 KB, approximately 5 to 10 times better than JPEG for a similar level of readability. Using a combination of hidden Markov model techniques and MDL-driven heuristics, DjVu first classifies each pixel in the image as either foreground (text, drawings) or background (pictures, photos, paper texture). The pixel categories form a bitonal image which is compressed using a pattern matching technique that takes advantage of the similarities between character shapes. A progressive, wavelet-based compression technique, combined with a masking algorithm, is then used to compress the foreground and background images at lower resolutions while minimizing the number of bits spent on the pixels that are not visible in the foreground and background planes. Encoders, decoders, and real-time, memory efficient plug-ins for various web browsers are available for all the major platforms.
---
paper_title: Generic approach for OCR performance evaluation

paper_content:

This paper presents the limits of the character recognition engines (commercial OCRs) and how to exceed these limits to achieve the industrial goals in terms of document capture and coding performances. The recent integration of these OCRs in several industrial capture chains leads to think that a solution is possible to reach electronically the same performances obtained by human typists. After a global description of the problems and the exposure of the OCR limits, the paper will focus on the methodology used and details the different steps proposed for the individual performance improvement. The first step consists in the individual evaluation of the OCRs. This is made by comparing the OCR result with a ground truth, which allows to highlight its defects and catalogue its main errors on the document processed. The second step allows to increase these individual performances by combination the OCR with some others. Our choice has been fixed on the combination of only two OCRs deemed very efficient and complementary on the same class of documents. The residual errors are treated in the last step which be able to propose a list of heuristics resolving punctually the OCR defects on the limit cases. In order to validate our approach, we present in the second part of the paper a practical case of experimentation to reach industrial performances. This approach has been tested in the framework of an industrial application for automatic document capture, by attempting the lowest score, imposed on one specific document class, of 1 error for 10000 characters.© (2001) COPYRIGHT SPIE--The International Society for Optical Engineering. Downloading of the abstract is permitted for personal use only.
---
paper_title: A comprehensive image processing suite for book re-mastering

paper_content:

Converting paper books into electronic form provides benefits for archiving, distribution and content reuse. However, directly scanned images are usually undesirable for electronic books, and automated content re-mastering is required. In this paper, we describe a comprehensive image processing suite consisting of three major components: 1) image enhancement with deskew, cropping, color correction, contrast enhancement and text sharpening, 2) compound document image compression, and 3) extraction of TOC (table of content) and linking. We built a processing pipeline that automatically converts a set of scanned page images into a high quality and highly compressed e-book in the popular PDF format.
---
paper_title: Evaluation of model-based retrieval effectiveness with OCR text

paper_content:

We give a comprehensive report on our experiments with retrieval from OCR-generated text using systems based on standard models of retrieval. More specifically, we show that average precision and recall is not affected by OCR errors across systems for several collections. The collections used in these experiments include both actual OCR-generated text and standard information retrieval collections corrupted through the simulation of OCR errors. Both the actual and simulation experiments include full-text and abstract-length documents. We also demonstrate that the ranking and feedback methods associated with these models are generally not robust enough to deal with OCR errors. It is further shown that the OCR errors and garbage strings generated from the mistranslation of graphic objects increase the size of the index by a wide margin. We not only point out problems that can arise from applying OCR text within an information retrieval environment, we also suggest solutions to overcome some of these problems.
---
paper_title: PerfectDoc: a ground truthing environment for complex documents

paper_content:

In this paper, we present PerfectDoc; a ground truthing and document correction tool. The tool provides post processing correction capabilities that are required after complex document analysis and understanding tasks. The tool has the advantage of being comprehensive (integration of most common correction tasks), easy to use (minimal clicks for corrections), configurable (can be used for different types of documents), and provides separate correction views. We used the tool to correct the output from a document understanding system used to extract articles from 80-years archive of Time weekly magazine.
---
paper_title: Multiple classifier decision combination strategies for character recognition: A review

paper_content:

Abstract.Two research strands, each identifying an area of markedly increasing importance in the current development of pattern analysis technology, underlie the review covered by this paper, and are drawn together to offer both a task-oriented and a fundamentally generic perspective on the discipline of pattern recognition. The first of these is the concept of decision fusion for high-performance pattern recognition, where (often very diverse) classification technologies, each providing complementary sources of information about class membership, can be integrated to provide more accurate, robust and reliable classification decisions. The second is the rapid expansion in technology for the automated analysis of (especially) handwritten data for OCR applications including document and form processing, pen-based computing, forensic analysis, biometrics and security, and many other areas, especially those which seek to provide online or offline processing of data which is available in a human-oriented medium. Classifier combination/multiple expert processing has a long history, but the sheer volume and diversity of possible strategies now available suggest that it is timely to consider a structured review of the field. Handwritten character processing provides an ideal context for such a review, both allowing engagement with a problem area which lends itself ideally to the performance enhancements offered by multi-classifier configurations, but also allowing a clearer focus to what otherwise, because of the unlimited application horizons, would be a task of unmanageable proportions. Hence, this paper explicitly reviews the field of multiple classifier decision combination strategies for character recognition, from some of its early roots to the present day. In order to give structure and a sense of direction to the review, a new taxonomy for categorising approaches is defined and explored, and this both imposes a discipline on the presentation of the material available and helps to clarify the mechanisms by which multi-classifier configurations deliver performance enhancements. The review incorporates a discussion both of processing structures themselves and a range of important related topics which are essential to maximise an understanding of the potential of such structures. Most importantly, the paper illustrates explicitly how the principles underlying the application of multi-classifier approaches to character recognition can easily generalise to a wide variety of different task domains.
---
paper_title: A comprehensive image processing suite for book re-mastering

paper_content:

Converting paper books into electronic form provides benefits for archiving, distribution and content reuse. However, directly scanned images are usually undesirable for electronic books, and automated content re-mastering is required. In this paper, we describe a comprehensive image processing suite consisting of three major components: 1) image enhancement with deskew, cropping, color correction, contrast enhancement and text sharpening, 2) compound document image compression, and 3) extraction of TOC (table of content) and linking. We built a processing pipeline that automatically converts a set of scanned page images into a high quality and highly compressed e-book in the popular PDF format.
---
paper_title: Evaluation and Application of Recognition Confidence in OCR

paper_content:

Recognition confidence plays an important role in the selection of rejection threshold and the combination of multiple classifiers. In this paper, we first present a systematic theory on classifier's confidence, which includes the definition, the concept of generalized confidence, optimal rejection theorem and the relationship between confidence value and recognition rate. Then we propose a method for the evaluation of recognition confidence. The theory and method are strongly supported by the practice in handwritten numeral recognition and off-line handwritten Chinese character recognition.
---
paper_title: Correcting the document layout: a machine learning approach

paper_content:

In this paper, a machine learning approach to support the user during the correction of the layout analysis is proposed. Layout analysis is the process of extracting a hierarchical structure describing the layout of a page. In our approach, the layout analysis is performed in two steps: firstly, the global analysis determines possible areas containing paragraphs, sections, columns, figures and tables, and secondly, the local analysis groups together blocks that possibly fall within the same area. The result of the local analysis process strongly depends on the quality of the results of the first step. We investigate the possibility of supporting the user during the correction of the results of the global analysis. This is done by allowing the user to correct the results of the global analysis and then by learning rules for layout correction from the sequence of user actions. Experimental results on a set of multi-page documents are reported and commented.
---
paper_title: Bootstrapping structured page segmentation

paper_content:

In this paper, we present an approach to the bootstrap learning of a page segmentation model. The idea evolves from attempts to segment dictionaries that often have a consistent page structure, and is extended to the segmentation of more general structured documents. In cases of highly regular structure, the layout can be learned from examples of only a few pages. The system is first trained using a small number of samples, and a larger test set is processed based on the training result. After making corrections to a selected subset of the test set, these corrected samples are combined with the original training samples to generate bootstrap samples. The newly created samples are used to retrain the system, refine the learned features and resegment the test samples. This procedure is applied iteratively until the learned parameters are stable. Using this approach, we do not need to initially provide a large set of training samples. We have applied this segmentation to many structured documents such as dictionaries, phone books, spoken language transcripts, and obtained satisfying segmentation performance.
---
paper_title: Creating digital libraries: content generation and re-mastering

paper_content:

This article has two main goals: to describe the automatic creation of a digital library and to provide an overview of the metaalgorithmic patterns that can be applied to increase the accuracy of its creation. Automating the creation of useful digital libraries - that is, digital libraries affording searchable text and reusable ("repurposable") output - is a complicated process, whether the original library is paper-based or already available in electronic form. We outline the steps involved in the creation of a deployable digital library (>1.2 /spl times/ 10/sup 6/ pages) for MIT Press, as well as its implications to other aspects of digital library creation, management, use and repurposing. Input, transformation, information extraction, and output processes are considered in light of their utility in creating layers of content. Interestingly, in some aspects, scanning directly from paper offers extra opportunities for error-checking through feedback-feedforward combination. Strategies for quality assurance (QA) at the document, chapter and book level are also discussed. We emphasize the use of metaalgorithmic design patterns for application towards improving the content generation, extraction and remastering. This approach also increases the ease with which modules and algorithms are added to and deprecated from the system.
---
paper_title: Digital mountain: from granite archive to global access

paper_content:

Large-scale, multiterabyte digital libraries are becoming feasible due to decreasing costs of storage, CPU, and bandwidth. However, costs associated with preparing content for input into the library remain high due to the amount of human labor required. We describe the digital microfilm pipeline -sequence of image processing operations used to populate a large-scale digital library from a "mountain" of microfilm and reduce the human labor involved. Essential parts of the pipeline include algorithms for document zoning and labeling, consensus-based template creation, reversal of geometric transformations and just-in-time browsing, an interactive technique for progressive access of image content over a low-bandwidth medium. We also suggest more automated approaches to cropping, enhancement and data extraction.
---
paper_title: A comprehensive image processing suite for book re-mastering

paper_content:

Converting paper books into electronic form provides benefits for archiving, distribution and content reuse. However, directly scanned images are usually undesirable for electronic books, and automated content re-mastering is required. In this paper, we describe a comprehensive image processing suite consisting of three major components: 1) image enhancement with deskew, cropping, color correction, contrast enhancement and text sharpening, 2) compound document image compression, and 3) extraction of TOC (table of content) and linking. We built a processing pipeline that automatically converts a set of scanned page images into a high quality and highly compressed e-book in the popular PDF format.
---
paper_title: PerfectDoc: a ground truthing environment for complex documents

paper_content:

In this paper, we present PerfectDoc; a ground truthing and document correction tool. The tool provides post processing correction capabilities that are required after complex document analysis and understanding tasks. The tool has the advantage of being comprehensive (integration of most common correction tasks), easy to use (minimal clicks for corrections), configurable (can be used for different types of documents), and provides separate correction views. We used the tool to correct the output from a document understanding system used to extract articles from 80-years archive of Time weekly magazine.
---
paper_title: A generic recognition system for making archives documents accessible to public

paper_content:

This paper presents annotations needed for handwritten archive document retrieval by content. We propose two complementary ways of producing those annotations: automatically by using optical document recognition and collectively by using the Internet and manual input by users. A platform for managing those annotations is presented as well as examples of automatic annotations on civil status registers, military forms (tested on 60,000 pages) and naturalization decrees, using a generic document recognition method. Examples of collective annotations built on automatic annotations are also given.
---
paper_title: Digital mountain: from granite archive to global access

paper_content:

Large-scale, multiterabyte digital libraries are becoming feasible due to decreasing costs of storage, CPU, and bandwidth. However, costs associated with preparing content for input into the library remain high due to the amount of human labor required. We describe the digital microfilm pipeline -sequence of image processing operations used to populate a large-scale digital library from a "mountain" of microfilm and reduce the human labor involved. Essential parts of the pipeline include algorithms for document zoning and labeling, consensus-based template creation, reversal of geometric transformations and just-in-time browsing, an interactive technique for progressive access of image content over a low-bandwidth medium. We also suggest more automated approaches to cropping, enhancement and data extraction.
---
paper_title: Document image analysis for World War II personal records

paper_content:

Complete collections of invaluable documents of unique historical and political significance are decaying and at the same time they are virtually inaccessible, necessitating the invention of robust and efficient methods for their conversion into a searchable electronic form. We present the issues encountered and problems addressed in the MEMORIAL project, whose goal is the establishment of a digital document workbench enabling the creation of distributed virtual archives based on documents existing in libraries, archives, museums, memorials, and public record offices. Successful approaches are described in the context of the chosen data class: a variety of typewritten documents containing personal information relating to the presence of individuals in World War II Nazi concentration camps.
---

Title: Quality assurance in high volume document digitization: a survey

Section 1: Introduction
Description 1: Provide an overview of the importance and challenges of quality assurance (QA) in high volume document digitization projects.

Section 2: Role of QA in high volume document digitization
Description 2: Discuss the definition and significance of QA specifically in the context of document digitization, highlighting unique characteristics and challenges.

Section 3: QA throughout the digitization pipeline
Description 3: Explain the different steps in the document digitization process and the associated QA problems and solutions.

Section 4: Cataloging
Description 4: Detail the QA issues in the cataloging step and solutions to ensure accurate and complete metadata.

Section 5: Image capture
Description 5: Discuss QA challenges and solutions related to converting paper documents to digital images.

Section 6: Image analysis and recognition
Description 6: Examine various QA issues in text recognition, document structure analysis, and image segmentation, along with the respective solutions.

Section 7: Error cascading
Description 7: Analyze how errors in upstream processing steps can affect downstream processes and suggest measures to control error cascading.

Section 8: General QA strategies
Description 8: Generalize QA strategies that are applicable across different stages in the digitization pipeline, including the combination of automatic and manual QA, batch QA, and usability of QA systems.

Section 9: Combination of automatic QA and manual QA
Description 9: Elaborate on the strategy of combining automatic and manual QA to balance quality with cost and time efficiency.

Section 10: Batch QA
Description 10: Discuss the benefits and challenges of batch QA in high volume digitization projects.

Section 11: Usability of QA system
Description 11: Explore the importance of designing user-friendly and efficient QA systems, especially for manual corrections.

Section 12: Open source QA
Description 12: Describe the "Open source QA" model and its application in large digitization projects, along with its advantages and drawbacks.

Section 13: Conclusions
Description 13: Summarize the key observations and findings regarding QA in high volume document digitization, and highlight the areas not covered in the survey.

Section 14: Acknowledgements
Description 14: Acknowledge the contributions of individuals and collaborators that supported the research and writing of the survey.