Handbook for Developing
Quality Assurance Project Plans

This document is a review draft. This information is distributed solely for the purpose of pre-dissemination review under applicable information quality guidelines. It has not been formally disseminated by EPA. It does not represent and should not be construed to represent any Agency determination or policy. It is being circulated for review of its technical accuracy.
                                   FOREWORD
The U.S. Environmental Protection Agency (EPA) developed the Quality Assurance Project Plan (QAPP) as a tool for project managers and planners to document the type and quality of data needed to make environmental decisions and to describe the methods for collecting and assessing those data. The development, review, approval, and implementation of the QAPP are part of EPA's mandatory Quality Program. The EPA Quality Program requires all organizations to develop and operate management structures and processes to ensure that data used in Agency decisions are of the type and quality needed for their intended use. The QAPP is an integral part of the fundamental principles and practices that form the foundation of the EPA Quality Program.
This Handbook is designed to assist in the creation of QAPPs that address the specifications listed in Annex B of Quality Standard For Environmental Data Collection, Production, and Use By EPA Organizations (EPA CIO 2106-S-01) and Quality Standard For Environmental Data Collection, Production, and Use By Non-EPA (External) Organizations (EPA CIO 2106-S-02.0) (current versions). It is intended both for EPA organizations and for organizations conducting environmental data operations under external agreements with EPA.
This document:
   * provides technical and policy advice;
   * provides implementation information;
   * does not substitute for statutes EPA administers or their implementing regulations;
   * is not a regulation;
   * does not impose legally-binding requirements on EPA, states, or the regulated community;
   * is not mandatory.
EPA may:
   * apply this handbook to any particular program or project to the extent appropriate in light of site-specific facts;
   * not apply this handbook to a particular situation based upon the specific circumstances; and
   * update, improve, revise, replace or withdraw this handbook at any time.

This document is one of several guidance and handbook documents from the U.S. Environmental Protection Agency, Office of Environmental Information (OEI) Quality Staff. EPA Quality Program documents may be downloaded from the Quality Home Page:
http://www.epa.gov/quality/qa_docs.html

In many instances, regional or program quality assurance personnel and project management personnel will be engaged for specific implementation information. These personnel are listed here:
http://www.epa.gov/quality/contacts.html
Questions regarding this document or other documents can be directed to the OEI Quality Staff:
U.S. EPA
OEI/Quality Staff (2811R)
Washington, DC 20460
Phone:<PHONE_NUMBER>
FAX:<PHONE_NUMBER>
e-mail<EMAIL_ADDRESS>

                                   PROLOGUE
The Environmental Protection Agency's (EPA) mission is to protect human health and to safeguard the natural environment  --  air, water, and land  --  upon which life depends. EPA provides leadership in the nation's environmental science, research, education, and assessment efforts. In order to accomplish this mission, EPA must ensure that our efforts meet quality standards that inspire confidence in our conclusions. The EPA Quality Program provides a framework for ensuring that our products and services meet quality standards that are appropriate for their intended use.
Quality Assurance Project Plans (QAPPs) are one component of the EPA Quality Program. Each project subject to the Internal or External Standards (CIO 2106-S-01.0 and CIO 2106-S-02.0, respectively) is required to develop and maintain a QAPP. QAPPs are intended to help a project team document how they are going to conduct their project. Each project team has innate ideas about the type and quantity of data they will need, how they'll know if the data are "good enough" or not, and how they'll "double-check" to be sure that their results make sense and are meaningful. This document offers some guidance on how to document those ideas. It is meant to cover the broad spectrum of activities conducted by the Agency such as setting national standards, pollution prevention, permitting, water and air quality, or human health and ecological risk assessment. Depending on the organization, the title of the person having quality assurance responsibilities can differ from those used in this guidance.  For example, a Quality Assurance Manager in one organization may be a Quality Assurance Director in another. From the contextual use in the guidance, the appropriate person should be recognized.
There are some areas of environmental data that are minimally addressed in this version of the Handbook including use of questionnaire surveys and method development studies; later versions of the Handbook will address these.
There are three principal chapters: Chapter 2 is for projects requiring the collection of new data to answer the study question; Chapter 3 is for projects making use of existing data; Chapter 4 is for projects involving modeling. Each chapter is self contained and should be used in conjunction with the Internal and External Standards (specifically, Clauses 7.5  -  7.10, and Annex B).  Chapters and appendices of greatest utility when creating a QAPP vary with the intent and purpose of the data collection:
   * New data for a new project:  Chapters 1 and 2, Appendices A, B, C, and D
   * New data for an established program: Chapters 1 and 2, Appendices A, B, and E
   * Using existing data: Chapters 1 and 3, Appendices A, B, and D
   * Use in modeling: Chapters 1 and 4, Appendix A
For those familiar with EPA's previous guidance on QAPPs Guidance for Quality Assurance Project Plans, EPA QA/G-5, a crosswalk between that obsolete document and this QAPP handbook is provided in Appendix E.
Questions on implementation of EPA's Quality Management System, including development of QAPPs, may be directed to the Quality Staff, Office of Environmental Information (quality@epa.gov).

                               TABLE OF CONTENTS
CHAPTER 1  INTRODUCTION	1
1.1	QAPPs, EPA QUALITY MANAGEMENT SYSTEM, EPA POLICY 2106 and ANSI/ASQ E4-2004	1
1.2	WHAT IS A QUALITY ASSURANCE PROJECT PLAN?	3
1.3	THE GRADED APPROACH	7
1.4	GENERIC QAPPs	7
1.5	PHASED QAPPs	9
1.6	WHEN SHOULD A QMP BE COMBINED WITH A QAPP?	9
1.7	DEVELOPING, REVIEWING, AND APPROVING A QAPP	11
1.7.1	Developing a QAPP	11
1.7.2	Reviewing a QAPP	12
1.7.3	Approving a QAPP	12
1.8	DISTRIBUTING, IMPLEMENTING, AND MODIFYING A QAPP	13
1.8.1	Distributing a QAPP	13
1.8.2	Implementing a QAPP	13
1.8.3	Modifying a QAPP	13
1.9	PERIOD OF APPLICABILITY	14
1.10	SUPERSESSION	14
CHAPTER 2  QAPP ELEMENTS FOR THE COLLECTION OF DATA BY DIRECT MEASUREMENT	15
2.1	OVERVIEW OF QAPP ELEMENTS FOR THE COLLECTION OF DATA BY DIRECT MEASUREMENT	15
2.2	PROJECT MANAGEMENT (Plan)	16
2.2.1	Title, Version, and Approval/Sign-Off	16
2.2.2	Document Format and Table of Contents	17
2.2.3	Distribution List	17
2.2.4	Project Organization and Schedule	18
2.2.5	Project Background, Overview, and Intended Use of Data	18
2.2.6	Data/Project Quality Objectives and Measurement Performance Criteria	19
2.2.7	Special Training Requirements and Certification	21
2.2.8	Documentation and Records Requirements	21
2.3	DATA ACQUISITION (Do)	22
2.3.1	Data Collection Procedure, Experimental Design, and Sampling Tasks	23
2.3.2	Sampling Procedures and Requirements	24
2.3.3	Sample Handling, Custody Procedures, and Documentation	25
2.3.4	Analytical Methods Requirements and Task Description	28
2.3.5	Quality Control Requirements	29
2.3.6	Instrument/Equipment Testing, Calibration and Maintenance Requirements, QA/QC for Supplies and Consumables	30
2.3.7	Data Management Requirements	32
2.4	ASSESSMENTS (Check)	32
2.4.1	Technical Systems Assessments	33
2.4.2	Performance Audits of Measurement and Analytical Systems	34
2.4.3	Surveillance of Operations	35
2.4.4	Audits of Data Quality	36
2.4.5	Qualitative and Quantitative Comparisons to Acceptance Criteria	37
2.4.6	Interim Assessments of Data Quality	37
2.4.7	Evaluation of Unconventional Measurements	38
2.4.8	Evaluation of Unconventional Monitoring Projects	39
2.5	REVIEW, EVALUATION OF USABILITY, and REPORTING REQUIREMENTS (Act)	39
2.5.1	Data Verification and Validation Targets and Methods	40
2.5.2	Quantitative and Qualitative Evaluations of Usability	41
2.5.3	Potential Limitations on Data Interpretation	42
2.5.4	Reconciliation with Project Requirements	42
2.5.5	Reports to Management	44
CHAPTER 3  QAPP ELEMENTS FOR EVALUATING EXISTING DATA	46
3.1	OVERVIEW OF QAPP ELEMENTS FOR EVALUATING EXISTING DATA	46
3.2	PROJECT MANAGEMENT (Plan)	47
3.2.1	Title, Version, and Approval/Sign-Off	47
3.2.2	Document Format and Table of Contents	48
3.2.3	Distribution List	48
3.2.4	Project Organization and Schedule	49
3.2.5	Project Background, Overview, and Intended Use of Data	49
3.2.6	Data/Project Quality Objectives and Measurement Performance Criteria	50
3.2.7	Special Training Requirements and Certification	51
3.2.8	Documentation and Records Requirements	51
3.3	DATA ACQUISITION (Do)	52
3.3.1	Proposed Data Source Originator and Publication Information	52
3.3.2	Data Format and Accessibility	52
3.3.3	Establishment of Acceptance Criteria	53
3.3.4	Sample Data Collection Methodology	54
3.3.5	Quality Program and Quality Assurance Procedures Used by Data Originator	54
3.3.6	Documentation of Sample Quality Assurance Procedures	55
3.3.7	Data Management Requirements	55
3.4	ASSESSMENTS (Check)	56
3.4.1	Qualitative Comparisons to Acceptance Criteria	56
3.4.2	Quantitative Comparisons to Acceptance Criteria	62
3.4.3	Assessments to Other Criteria	63
3.4.4	Interim Assessments of Data Quality	63
3.4.5	Evaluation of Unconventional Measurements	63
3.4.6	Evaluation of Unconventional Monitoring Projects	64
3.5	REVIEW, EVALUATION OF USABILITY, AND REPORTING REQUIREMENTS (Act)	64
3.5.1	Data Verification and Validation Targets and Methods	64
3.5.2	Quantitative and Qualitative Evaluations of Usability	64
3.5.3	Potential Limitations on Data Interpretation	65
3.5.4	Reconciliation with Project Requirements	65
3.5.5	Reports to Management	67
CHAPTER 4  QAPP ELEMENTS FOR DEVELOPMENT, MODIFICATION, AND USE OF MODELS	69
4.1	OVERVIEW OF QAPP ELEMENTS FOR MODELS	69
4.2	PROJECT MANAGEMENT (Plan)	70
4.2.1	Title, Version, and Approval/Sign-Off	72
4.2.2	Document Format and Table of Contents	72
4.2.3	Distribution List	73
4.2.4	Project Organization and Schedule	73
4.2.5	Project Background, Overview, and Intended Use of the Model	74
4.2.6	Data/Project Quality Objectives and Acceptance Criteria	78
4.2.7	Special Training Requirements and Certification	79
4.2.8	Documentation and Records Requirements	80
4.3	DATA ACQUISITION: MODEL DEVELOPMENT, MODIFICATION, AND USE (Do)	80
4.3.1	Data Requirements for Model Input	81
4.3.2	Data Management Tasks	83
4.4	ASSESSMENTS: MODEL EVALUATION ACTIONS (Check)	84
4.4.1 	Model Evaluation Methods and Activities	84
4.4.2 Evaluation of the Model	85
4.4.3	Model Output Sensitivity and Uncertainty Analysis	87
4.5	REVIEW, EVALUATION OF USABILITY: MODELING RESULTS USABILITY AND REPORTING REQUIREMENTS (Act)	87
4.5.1	Peer Review	88
4.5.2	Description of Model Documentation	89
4.5.3	Specifications for Model Maintenance and User Support	90
4.5.4	Reports to Management	90
GLOSSARY	91
REFERENCES	97
APPENDIX A CHECKLIST OF QAPP ELEMENTS	A1
APPENDIX B  DATA QUALITY INDICATORS	B1
B.1	Sources of Measurement Uncertainty	B3
B.2	Establishing MQOs in the Context of DQOs	B3
B.3	Precision	B5
B.4	Bias	B5
B.5	Accuracy	B6
B.6	Representativeness	B7
B.7	Comparability	B9
B.8	Completeness	B10
B.9	Sensitivity	B11
B.10	The Role of DQIs in the Project Lifecycle	B14
APPENDIX C  QUALITY CONTROL SAMPLES	C1
APPENDIX D  DATA REVIEW, VERIFICATION, AND VALIDATION FOR PROJECT PURPOSES	D1
D.1	Verification	D4
D.2	Validation	D5
D.2.1	Validation Activities  -  Step A: Compliance	D6
D.2.2	Validation Activities  -  Step B: Comparison	D6
APPENDIX E  CROSSWALKS TO OTHER DOCUMENTS	E1
E.1 	Crosswalk: Standards CLauses 7.5 to 7.10 to Handbook QAPP and HANDBOOK QMP	E1
E.2 	Crosswalk: Standards ANNEX B to QAPP Handbook	E2
E.3	Crosswalk: EPA QA/G-5 to QAPP Handbook	E4
E.4	Crosswalk: QAPP Handbook to EPA QA/G-5, QA/G-5M, and QA/G-8	E6
E.5	Crosswalk: UFP-QAPP Workbook to QAPP HANDBOOK	E9

                                 ACRONYM LIST

ANSI	American National Standards Institute
ASTM	American Society for Testing and Materials
ASQ	American Society for Quality
CFR	Code of Federal Regulations
CIO	Chief Information Officer
DL	Detection Limit
DQA	Data Quality Assessment
DQI	Data Quality Indicator
DQO	Data Quality Objective
EPA	Environmental Protection Agency
FIFRA	Federal Insecticide, Fungicide, and Rodenticide Act
FORMS	Field Operations and Records Management System
GPS	Global Positioning System
ICP	Inductively Coupled Plasma
IQG	Information Quality Guidelines
ISO	International Organization for Standardization (not an acronym)
IT	Information Technology
LOD	Limit of Detection
LOQ	Limit of Quantitation
MCL	Maximum Contaminant Level
MDL	Method Detection Limit
MPC	Measurement Performance Criteria
MQO	Measurement Quality Objective
OEI	Office of Environmental Information
OMB	Office of Management and Budget
ORD	Office of Research and Development
OSWER	Office of Solid Waste and Emergency Response
PCB	Polychlorinated biphenyls
PE	Performance Evaluation
PQL	Practical Quantitation Limit
PQO	Project Quality Objectives
PRP	Potentially Responsible Parties
PT	Proficiency Test
QA	Quality Assurance
QAPP	Quality Assurance Project Plan
QC	Quality Control
QL	Quantitation Limit
QMP	Quality Management Plan
QMS	Quality Management System
RCRA	Resource Conservation and Recovery Act
SOP	Standard Operating Procedure
TSA	Technical Systems Assessment
TSCA	Toxic Substances Control Act
UFP	Uniform Federal Policy
XRF	X-ray Fluorescence

CHAPTER 1

INTRODUCTION
1.1	QAPPs, EPA QUALITY MANAGEMENT SYSTEM, EPA POLICY 2106 and ANSI/ASQ E4-2004
This Chapter lays a general foundation for a QAPP and provides a more detailed explanation of what is required in the Internal and External Standards. The Chapter discusses key aspects of what a QAPP contains, what it applies to, its connection to a QMP, and who is responsible for developing one. It lays out the general alignment of required content in a Plan-Do-Check-Act paradigm.
This document presents non-mandatory guidance intended to help its users prepare Quality Assurance Project Plans (QAPPs) that meet the requirements of the following two Environmental Protection Agency (EPA) Standards:
Quality Standard for Environmental Data Collection, Production, and Use by EPA Organizations, (CIO 2106-S-01), "Internal Standard" (EPA 2013a); and
Quality Standard for Environmental Data Collection, Production, and Use by Non-EPA (External) Organizations, (CIO 2106-S-02), "External Standard" (EPA 2013b).
These Standards provide the foundation for the Agency-wide Quality Program for environmental data-related products and services that are disseminated outside the Agency. They conform to EPA Quality Policy, CIO 2106.0, "Quality Policy" (EPA 2008a) and to EPA Procedure for Quality Policy, CIO 2106-P-01.0, "Quality Procedure" (EPA 2008b) as issued by the Chief Information Officer (CIO), EPA. The use of this Handbook is discretionary but should be used in conjunction with the two EPA Quality Standards. Throughout Chapters 2, 3, and 4 of this document, the corresponding sections of the Internal and External Standards are noted in italics.
This handbook follows the lead of the Standards in reaffirming the applicability of the national consensus standard, Quality Systems for Environmental Data and Technology Programs: Requirements with Guidance for Use, ANSI/ASQ E4-2004 (ANSI 2004) developed by the American National Standards Institute (ANSI) and the American Society for Quality (ASQ).
The QAPP document Requirements for Quality Assurance Project Plans, EPA QA/R-5 (EPA 2001a), has been withdrawn in favor of these Internal and External Standards. The QA/R-5 document should not be used for developing QAPPs except where explicitly allowed by the Standards. The guidance that accompanied QA/R-5, Guidance for Quality Assurance Project Plans, EPA QA/G-5, has also been withdrawn as it is superseded by this document, EPA Guidance on Quality Assurance Project Plans (CIO 2106-G-05 QAPP), "QAPP Guidance". It is important to remember that unlike the Standards, which present normative requirements, this document consists of informative (non-mandatory) guidance presented in non-prescriptive language. It is offered as a helpful resource for those who need to prepare QAPPs that comply with EPA Standards. It may be helpful to those creating work plans, sampling and analysis plans, field implementation plans, or other project planning documents.
EPA also allows use of the Uniform Federal Policy for Quality Assurance Project Plans, (EPA-505-B-04-900A), "UFP-QAPP" (EPA 2005a) as an alternative approach to satisfy EPA's requirement for a QAPP if allowed by the external agreement. In 2005, the Intergovernmental Data Quality Task Force (IDQTF) issued the UFP-QAPP to address environmental data collection, production, and use for hazardous waste cleanup activities at federal facilities and installations. Shortly thereafter, EPA's Office of Solid Waste and Emergency Response issued a directive (OSWER Directive 9272.0-17) (EPA 2005c) stating that the UFP-QAPP format should be used for EPA-managed Federal Facility projects where environmental data are collected and used. Both that directive and further guidance issued jointly by EPA's Federal Facilities and Restoration Reuse Office and Quality Staff, Office of Environmental Information (OSWER 9272.0-20) (EPA 2005d) state that QAPPs prepared and approved according to the UFP will be considered consistent with EPA Standards. The UFP-QAPP worksheets are also consistent with the EPA Quality Standards.
Figure 1 depicts the hierarchical relationship of QA documents: ANSI/ASQ E4-2004 national consensus standard, together with the Information Quality Act of 2001 (IQG 2001), establish a basis for the Agency's Quality Policy; the Quality Procedure provides additional explanation about how to carry out the Policy; the two Standards contain requirements for applying the Policy and Procedure to environmental data operations (within and external to the Agency);  Handbook for Preparing Quality Management Plans (EPA 2013c) offers help in preparing Quality Management Plans (QMPs) that meet the requirements of the higher level documents; and this Handbook document offers guidelines, advice, and examples that will help users satisfy provisions of the Standards at the project level.
               Handbook for Preparing Quality Management Plans

                              Quality Procedure
                           CIO Procedure 2106-P-01.0
           Handbook for Developing Quality Assurance Project Plans

                               Internal Standard
                                CIO 2106-S-01.0
                              External Standard
                                CIO 2106-S-02.0
                            Quality Policy CIO 2106
                 National Consensus Standard ANSI/ASQ E4-2004
                            Information Quality Act
          Figure 1.	Relationship among EPA Quality Program Documents
1.2	WHAT IS A QUALITY ASSURANCE PROJECT PLAN?
This section describes QAPPs in a Question-and-Answer format and explains more fully the requirements of Standard 7.5, Quality Assurance Project Plan, and Annex B1, Introduction, B2, QAPP Responsibilities and Approvals, B2.3, Applicability, and B3.2, General Content Requirements.
A QAPP is a formal document describing in comprehensive detail the necessary quality assurance (QA), quality control (QC), and other technical activities that will be implemented to ensure that the results of the work performed will satisfy the stated performance criteria. It describes the activities of a project in the acquisition of environmental data or information from direct measurement activities, existing data, or generated by models.
What is the difference between QA and QC? QA refers to the system of management activities that are designed to ensure quality, whereas QC refers to the system of preventive and corrective technical activities that measure or monitor performance against defined standards and makes adjustments to maintain acceptable performance.
Why is a QAPP important? A QAPP presents the steps that should be taken to ensure that environmental data collected, produced, and used are of adequate and sufficient type and quality required for a specific decision or use. It presents an organized and systematic description of the ways in which QA and QC will be applied to the collection and use of environmental data. It documents the results of the project's planning process thus providing a concise record of the conduct of environmental data operations.
Must I have a QAPP? ANSI/ASQ E4-2004 Section 6 (Part B) (ANSI 2004) and the EPA Internal and External Standards, Clause 7 (EPA 2013a, EPA 2013b) require that a QAPP be approved for all data collection projects. The specific requirements for the content of a QAPP are presented in Annex B of each Standard.
What are the benefits of a QAPP? A QAPP (or equivalent document as stipulated in the organization's QMP) integrates QA and QC aspects of a project throughout its life cycle, including planning, implementation, assessment, and corrective actions. A QAPP is a "roadmap" for ensuring the implementation of quality procedures and practices in a project.  When implemented as prescribed, a QAPP increases efficiency and enables decisions to be made effectively with greater certainty and reduces the potential for possible rework.
To what does a QAPP apply? The ultimate success of an environmental program or project depends on the quality of the environmental data collected and used in decision-making, and this may depend significantly on the adequacy of the QAPP and its effective implementation. The title of the QAPP being developed should include information as to what the QAPP will apply to and bear a unique identifying number (or QAPP Tracking Number) such that it can be readily identified amongst similar QAPPs.
Agency activities involving environmental data required by the Internal and External Standards to have a QAPP include, but are not limited to:

direct and indirect field and/or laboratory measurements;
evaluating the operation and performance of environmental technology;
inspections;
existing environmental data;
questionnaire survey development or application;
development and validation of sampling or analytical methods;
environmental model modification and/or development;
enforcement monitoring or assessments;
application of environmental management systems;
environmental safety and health monitoring;
scientific research;
regulatory development;
statistical or economic analyses of environmental data;
use of information technology (mathematical models);
use of information sources outside of direct EPA management controls or authority; and
use of other data sources (e.g., literature or the Internet).
In some instances an organization may develop a QAPP for field collection activities and another for laboratory analysis activities; a clear distinction between them is advisable. For projects with multiple participants each developing a QAPP, an overarching QAPP which would provide requirements such that individual components may be effectively integrated is advised.
Who is involved in developing a QAPP? Most activities involving environmental data require the coordinated efforts of many individuals, possibly including managers, engineers, scientists, statisticians, information technology (IT) experts, modelers, and others. The QAPP should integrate the contributions and requirements of everyone involved into a clear, concise statement of what needs to be accomplished, how it will be done, and by whom. It should provide understandable instructions to those who must approve or implement the QAPP, including the field sampling team, the analytical laboratory, and data users and reviewers. Beyond the general guidance provided in the QAPP, it should identify and integrate the use of related policies and procedures applicable to the project such as administrative procedures, laboratory procedures, data analysis methodologies, IT policies and procedures, and data handling and analysis policies.
How is the QAPP effective? The QAPP must specify the level or degree of QA and QC activities needed for the particular environmental data or model operations with clear objectives, acceptance criteria, and QA/QC control strategies. The QA and QC technical requirements of a project should be commensurate with the type of work to be done (e.g., monitoring, site characterization, model simulation, and bench level proof of concept), the purpose of the project (e.g., regulatory enforcement, development work for rulemaking, permit approval, research publications, and journal articles), and the scale of the project such as a one-time assessment or a template for multiple assessments. Regardless of the complexity of the project, the QAPP documents how the project team will ensure that the quality of data is suitable for its intended use by documenting acceptance criteria against which assessments may be made.
Why is it important to me? The QAPP is invaluable in not only documenting all aspects of the project serving as a structured resource for writing the final report on the project. In addition, it helps a successful implementation during reassignments of personnel. It is the basis for assuring the integrity of the project's conclusions.
Who approves a QAPP before being put into operation? Each QAPP must be approved by the organization's project manager or director and the QA Manager prior to any data collection, production, or use (with the possible exception of circumstances requiring immediate action to protect human health and the environment, or operations conducted under police powers). If this approval function has been delegated, it must be in accordance with the organization's QMP.
How do I develop a QAPP? A brief summary of the process:
   1. determine the scope and objectives of the project and gather background information;
   2. assemble a project team with necessary project and QA expertise;
   3. conduct planning sessions to determine how to collect the specific type of data, the amount of data, and what the goals or acceptance criteria will be;
   4. using the Standard and Handbook (if necessary) develop a draft QAPP;
   5. circulate the draft QAPP for peer review, comment, and input for improvement;
   6. submit the final QAPP for approval;
   7. after approval, distribute the QAPP to all persons involved with the project; and
   8. proceed to implement the QAPP allowing for documentation of changes, re-approvals, and distribution of updated QAPPs.
 How is a QAPP organized? The QAPP elements are arranged in four sections to reflect the logical flow of the project life cycle (also known as the Plan-Do-Check-Act steps, Figure 2).

                 Figure 2. Plan-Do-Check-Act Project Life Cycle
 The four types of elements in these steps are:
Project Management (Plan) - These elements, covering the basic areas of project management, project objectives, and the roles and responsibilities of the participants, document the development of progress towards the project's goal. The development of technical and quality project objectives, such as Data Quality Objectives (DQOs), are part of this step.
Data Acquisition (Do) - These elements cover all aspects of data measurement, data acquisition designs, and their implementation. The development of performance and acceptance criteria for the collection of data by direct measurement, acquisition from existing sources, and by modeling are integral to this step. This ensures the intended measurements, data collection, or acquisition methods are appropriate for achieving project objectives.
Assessments (Check) - These elements address the activities for assessing the effectiveness of the implementation of the project and associated QA and QC activities. The purpose of assessment is to ensure that the QAPP is implemented as prescribed and that project actions are implemented as expected.
Review, Evaluation of Usability, and Reporting Requirements (Act) - These elements cover the QA activities that occur after the data collection (or use) phase of the project is completed. Implementation of these elements will help in determining that the data conform to the specified criteria, and ensuring that data usability can be documented in a final project report. This step includes any limitations or restrictions on the use of the data, or contingencies for revising project objectives.
 How is this QAPP Handbook organized? This document covers QAPP development for three different types of projects:
studies in which new data will be collected (Chapter 2);
investigations in which existing data will be used (Chapter 3); and
development, modification, and use of models using environmental data (Chapter 4).
These three project types are each addressed in their own chapter and should be used with the relevant appendices of this document. It is not the intent of this Handbook to specifically identify every type of project for which a QAPP will be needed, but rather to describe QAPP elements for general types of projects. If a QAPP element is not applicable, then an indication of why it is not relevant should be given.
If information is available elsewhere should I rewrite this into the QAPP? Reference to existing documents (such as standard operating procedures, sampling and analysis plans, environmental assessments, work plans, literature files, fully documented models) can greatly reduce the size of a QAPP. Documents may be included by reference or appended to the QAPP. It is essential, however, that everyone, including QAPP reviewers, involved with the project has access to the referenced documents.
How long is a QAPP? The QAPP needs to have enough information to adequately describe the project's objectives, implantation, and results. This implies a complex project will demand more than a relatively simple study. The use of a "graded approach" is recommended.
Is guidance written by other organization based on EPA QA/G-5 obsolete? All of the strengths of the now obsolete guidance EPA QA/G-5 (EPA 2002a) have been incorporated into this QAPP guidance. This means that guidance written by other organizations based on the obsolete document will still be very strong, but will not have some of the latest advances in QA practices. Organizations are encouraged to review their guidance to ensure it meets the requirements of the Standards, and also take advantage of some of the advice offered in this QAPP guidance.
1.3	THE GRADED APPROACH
The graded approach is an extremely important concept and the outline given in Standards Annex B3.1, General Requirements, amplified by discussion.
Recognizing that a "one size fits all" approach to quality requirements will not work in organizations as diverse as EPA, implementation of the EPA Quality Program is based on the principle of a graded approach. The graded approach is the process of establishing the project requirements and level of effort commensurate with the intended use of the results and the degree of confidence needed in the quality of the results.
The graded approach allows for the content and level of detail in each QAPP to vary accordingly. For example, the quality expectations of fundamental research may be different from that of regulatory compliance because the purpose or intended use of the data is different. The intent of the graded approach is to achieve sufficient detail in the QAPP to satisfy the objectives of the project, ensure adequate quality for the intended use of the data, and meet the resources allocated to the project.  Cooperation and discussion during the QAPP development process is encouraged, especially on the implementation of the graded approach. The final decision on QAPP content and level of detail belongs to the EPA organization responsible for the work to be done, consistent with the approved QMP, and within the resources available for the project.
1.4	GENERIC QAPPs
Many organizations are engaged in projects or programs collecting data under very similar circumstances. This section offers advice on the use of generic QAPPs using the requirements of Standards Clause 7.5, QAPP, and Annex B3, General Requirements.
A generic QAPP provides an overarching plan that describes the quality objectives and documents a comprehensive set of sampling, analysis, QA/QC, data review, and assessment procedures specific to a large program or long-term project. In contrast to the project specific QAPP, the generic QAPP serves as an umbrella under which multiple data collection, production and use activities may be conducted over an extended period of time.
Generic QAPPs may make sense in situations such as where multiple sites, systems, or projects will be sampled under a common sampling and analysis protocol/plan. A simple way to determine whether a generic QAPP is appropriate for a project is to consider whether there is sufficient consistency across the QA needs of multiple projects within a program that combining the planning into a single, generic QAPP will:
ensure the necessary level of quality for all projects covered by the generic QAPP; and
require less time and resources to manage with a single generic QAPP than with multiple project specific QAPPs.
An approved generic QAPP should be supported by task or project specific addenda, which address the issues unique to each task or project. Project or task specific information that is not covered by the generic QAPP should be documented in detail in these addenda. The generic QAPP should specify the preparation, review, and approval of task or project specific addenda. For external non-EPA projects, EPA may authorize the organization to approve project specific addenda contingent upon a review and approval process that is fully documented in the approved generic QAPP.
Some organizations may find a variant on generic QAPPs, developed by EPA Region 9, more suitable to their needs. Called Region 9 QA Program Plans,
      "...they are intended to establish policies that define and document the type and quantity of data needed for program level environmental decisions and to describe the methods required for collecting, analyzing, and assessing data to support those decisions. For the purposes of this [EPA Region 9] guidance, an environmental program is considered to be a series of activities which are based directly or indirectly on an act of Congress and defined in regulations promulgated by EPA, State, or tribal governments...The measurements under a program generally reflect on-going activities which do not have defined start and end dates (not to be confused with grant cycles). This does not preclude a program implementing or conducting projects. Program activities are usually of a recurring nature although specific activities may not recur.  For example, there may be on-going water monitoring, but sampling locations may change from year to year. EPA and/or a funded organization may sponsor environmental projects. Projects are considered to be activities that have a definable beginning and a definable end. Projects would be expected to have their own QAPPs that would either be approved by EPA or by the other organization.  A QA Program Plan should describe policies regarding what type of grants, cooperative agreements, or activities require a QAPP or other QA documentation, what the nature of that documentation would be, and what activities might be exempted from that requirement.  For example, an investigation of an integrated pest management approach under FIFRA or an evaluation of a new wastewater treatment methodology would be considered projects that included environmental measurements that would be under a broader environmental program. A QA Program Plan should also describe what information the QAPP should include or cite appropriate guidance...Both QA Program Plans and QAPPs should include specific identifiable goals and objectives and discuss the uses of the data. Often, but not always, program goals are more closely tied to environmental regulations, whereas project goals are developed for that event based on a systematic planning " (EPA 2012).
1.5	PHASED QAPPs
Some organizations need QAPPs that are brought on-line as a project is being developed. This section offers advice on the use of a set of QAPPs using Standards 7.5, QAPP.
When there is significant uncertainty surrounding a project and additional clarity is expected as initial data or information are gathered, or for a project that is iterative in nature, it may be appropriate to use a phased approach to QAPP development. The phased QAPP should include a description of the decision points for the project. Examples of when a phased QAPP might be appropriate include:
literature reviews in which the eventual direction and depth of the research is dependent upon the information found in the articles in the first round of the project;
investigations or phased project cycles where the results of the initial assessment will dictate the following steps (e.g., no contaminants found above thresholds leading to site closure vs. contaminants found above thresholds leading to a risk assessment and feasibility study);
analytical method development for a specific purpose (e.g., improved sensitivity) for which an early step in the process may determine the viability of continued effort; or
model development and implementation for which the project overview, objectives, and measurement performance criteria can be defined upfront, but until the model requirements are defined, the requirements for input data can't be clearly established.
If a phased QAPP is developed, it should still contain all relevant elements of a QAPP. The efficiency to be gained from a phased QAPP may be realized by not spending time or resources to specify the detailed project requirements for each "what-if" scenario for the project. If a phased QAPP is developed, it is very important that it be updated as the direction of the project becomes clear. No project work should take place without an appropriate QAPP in place, so moving beyond the phase documented in the QAPP should not occur until the QAPP is updated. These updates should be described in the revision history (see Section 1.8).
1.6	WHEN SHOULD A QMP BE COMBINED WITH A QAPP?
Some organizations have no direct need for a full Quality Management Plan (QMP) but could incorporate needed elements into a combined QMP/QAPP outlined in Standards Clause 7.5.2.
The role of a QMP is to define and describe an organization's overall structure, processes, roles, and responsibilities for planning, implementing, and evaluating environmental data operations. The role of a QAPP is to provide a road map for environmental data production, collection, and use in a project. QAPPs document project objectives along with the QA and QC procedures that will ensure that those objectives are achieved. While the QMP defines the Quality Program at the organizational level, the QAPP defines project-specific quality concerns. There may be, and usually are, multiple QAPPs developed under a QMP. Figure 3 depicts the relationship between QMPs and QAPPs within an organization's Quality Management System (QMS). QAPPs are needed for projects where collection of new data is planned, existing data will be incorporated to meet project objectives, or models will be used to evaluate and interpret data.

                 Figure 3.	Relationship between QMP and QAPPs
In some circumstances, EPA may recommend the preparation of a combined QMP and QAPP as an appropriate means to document organizational level and project-level environmental data activities. The goal of the project, together with its data needs, is the principal driver of the QA/QC activities documented in the combined QMP/AQPP. Examples where this may be appropriate could include research projects, grants, inspection programs, and monitoring networks involving multiple small organizations. In such cases, a combined QMP and QAPP can be an effective means to document organizational level and project-level environmental data activities, particularly when a full QMP is not needed. Combined QMP/QAPPs should not be used for external agreements containing multiple projects that may be determined over the life of the agreement such as mission contracts and program grants). Some specific situations in which a combined QMP/QAPP might be appropriate include (but are not limited to):
   * routine inspection programs where inspections are conducted in the same manner for all visits (e.g., EPA lead or radon programs);
      *       small organizations that do infrequent work with EPA funds (e.g., competitively awarded one-time grants and environmental education grants);
      *       one-time mobilization to a previously unvisited field location such as for monitoring assistance to a Tribal location;
      *       necessary quick-turnaround data analyses that may involve new laboratory facilities, methods, or equipment;
      *       environmental measurements related to technology performance, where QMS planning for the operation of the technology is already addressed by another document;
      *       academic research activities of a one-time nature where an organizational QMP has more detail than needed; or
      *       a monitoring program conducted by a volunteer organization where the size and structure of the organization is limited.
A combined QMP/QAPP incorporates some elements of the QMP and some elements of the QAPP. A combined QMP/QAPP is primarily a QAPP; that is, it pertains to a specific project or program. However, the project may not require the application of a full QMS. As with any QMP, it should document the management controls that should be applied to the planning, implementation, and assessment of quality activities, and what processes will be followed to carry out those activities. These may vary depending upon the scope and objectives of the project and should be defined by the EPA organization.
It is EPA's responsibility to ensure that all elements of a combined QMP/QAPP, as defined by the EPA organization, are addressed by the organization performing the work. The combined QMP/QAPP is subject to the same EPA review and approval as any other QAPP. The QMP elements of the combined QMP/QAPP should be subject to appropriate QMP-level review.
Further guidance for a combined QMP/QAPP is provided in the QMP Guidance.
1.7	DEVELOPING, REVIEWING, AND APPROVING A QAPP
This section offers guidance on Standards Clause 7.5 QAPP, and Annex B2.1, QAPP Preparation Responsibilities and Approvals, and Annex B2.2 QAPP Implementation and Revision.
1.7.1	Developing a QAPP
The QAPP may be prepared by an EPA organization or a non-EPA organization (e.g., a contractor, an assistance agreement holder, another Federal agency under an interagency agreement). All EPA organizations collecting, producing, or using environmental data must have their QAPPs approved by their managers and QA Manager or designee as specified in the organization's QMP. Except where specifically delegated in the QMP of the EPA organization sponsoring the work, all QAPPs prepared by non-EPA organizations must be approved by EPA before implementation, and involvement of EPA and other regulatory agencies, as applicable, in development is encouraged.
QAPP development necessitates the coordinated efforts of many individuals such as those who will gather data and those who will use the data to make decisions. These individuals include:  decision makers, project managers, regulators, stakeholders, and technical staff (e.g., hydrologists, biologists, chemists, samplers, statisticians, engineers, economists, modelers, and risk assessors). In addition, peer reviewers and individuals with varied expertise ensure that technical areas are sufficiently addressed, thus helping to identify important issues during review, approval, and implementation.
For internal EPA projects, the Project Manager (PM) or Principal Investigator (PI) is generally responsible for overseeing the preparation of the QAPP. For external projects, the funded entity is usually responsible for QAPP development (grantees being allowed to develop an independent QAPP), but for other agreements (for example, contractual) the QAPP is subject to review and approval by EPA prior to implementation. In the case of another Federal agency receiving funds from EPA, as through an interagency agreement, the award indicates who is responsible for QAPP preparation. When EPA receives project funds from another Federal agency, EPA personnel usually write the QAPP unless the other organization negotiates specific quality requirements. It is usually advantageous to hold round-table discussions between stake-holders, QA specialists, and Project Manager prior to writing the QAPP. This ensures the coverage of the QAPP is inclusive and has a broad coverage of the project.
Chapters and appendices of greatest utility when creating a QAPP vary with the intent and purpose of the data collection:
   * New data for a new project:  Chapters 1 and 2, Appendices A, B, C, and D
   * New data for an established program: Chapters 1 and 2, Appendices A, B, and E
   * Using existing data: Chapters 1 and 3, Appendices A, B, and D
   * Use in modeling: Chapters 1 and 4, Appendix A
1.7.2	Reviewing a QAPP
A QAPP can be reviewed by several authorized EPA reviewers to ensure it contains the appropriate content and level of detail as defined by the EPA organization's QMP. These reviewers may include the Project Manager, Contracting Officer's Technical Representative, and QA Manager. In some cases, the authority to review and approve QAPPs is delegated to another part of the EPA organization covered by the same QMP. In cases where the authority to review and approve QAPPs is delegated in writing by EPA to another organization (e.g., a Federal agency or a State under an EPA-approved QMP when the environmental data operation has been delegated to that organization for implementation), it is advised that the EPA Project Manager and EPA QA Manager should be involved in the review and approval steps.
Reviewers with expertise in the project specific areas, such as program managers (decision makers), QA staff independent of project management, and field and laboratory technical staff, should review the plan. These reviewers should:
ensure that the information is accurate and complete;
ensure that all appropriate elements are addressed;
ensure that the plan identifies the project's technical and quality objectives, and that the intended measurement and data acquisition methods will satisfy these objectives;
confirm that the planned assessment procedures will be adequate to evaluate the project; and
confirm that there is a process to identify any limitations on the use of the data.
1.7.3	Approving a QAPP
The EPA organization's QMP establishes how, when, and by whom development, review, approval, and effective oversight of QAPPs should occur. This includes processes for external (non-EPA) organizations that prepare QAPPs. For EPA projects, the Project Manager (or Project Officer in the case of external funding), and the QA Manager usually approve the QAPP. For external non-EPA projects, the responsible organization's Project Manager or Principal Investigator and QA Manager may review and approve the QAPP and then submit it for EPA approval (unless that EPA organization has specifically delegated approval in its Agency-approved QMP). It is also beneficial if other key staff such as the laboratory or division directors, and prime contractors and subcontractors, sign the plan to indicate their review and approval.
No work involving environmental data collection, production (or generation), or use shall be started until the QAPP has been approved by the EPA QA Manager (or authorized representative).
Some organizations, according to their QMP, may permit conditional approval of a QAPP in situations where only non-critical deficiencies are unresolved (e.g., missing a final organizational chart). Conditional approval allows project work to start while the QAPP is finalized. The QAPP is then resubmitted for approval when the remaining deficiencies are resolved. Any work performed under conditional approval should be in accordance with the fully approved QAPP, with any deviations being fully documented for resolution.
1.8	DISTRIBUTING, IMPLEMENTING, AND MODIFYING A QAPP
This section offers guidance on Standards Annex B2.2, QA Implementation and Revision.
1.8.1	Distributing a QAPP
All applicable personnel involved in the project should retain or have access to the current version of the QAPP. This may include the Project Manager, QA Manager, modeler, data reviewer, laboratory manager, field team leader, and any essential contractor and subcontractor personnel involved with the project.
1.8.2	Implementing a QAPP
The organization performing the work is responsible for ensuring that the QAPP is implemented as written and approved, whether this work is conducted by EPA or non-EPA personnel. Ultimately the Project Manager is responsible for project activities. A clearly written QAPP will help the Project Manager implement the plan, because all project personnel will understand the specifications before the start of data production or data use activities. However, effort is needed throughout the entire project lifecycle to ensure by direct evidence that the QAPP is being implemented properly as planned.
1.8.3	Modifying a QAPP
The QAPP's effective period of approval will be defined by the parent EPA organization that approves subordinate EPA organizations and non-EPA organization QAPPs. In general, the QAPP approval should encompass the project-specific activities and proposed timeframe to complete the activities. A QAPP approval for an external agreement may not exceed the period of performance for the agreement or no-cost extensions to the period of performance.
The QAPP should be kept current and all appropriate personnel involved in the work should have easy access to a current version of the QAPP. Although the approved QAPP must be implemented as prescribed, it is not inflexible. Due to the complex and diverse nature of environmental data operations, changes to original plans are often needed. When such changes occur, the organization's QA Manager (or designee), together with the Project Manager, determines if the change significantly impacts the technical and quality objectives of the project. When a substantive change is warranted, the originator of the QAPP modifies the QAPP to document the change and submits the revision for approval by the same authorities that performed the original review. The EPA Project Manager should determine whether the need to revise the QAPP merits stopping project activities until the revised QAPP is approved (e.g., if serious safety threats are identified that haven't been addressed in the original QAPP), or if the type of revisions required do not necessitate stopping work (e.g., the data validation firm named in the QAPP went out of business and an equally competent firm was identified to fill that role). All revisions should be documented in the QAPP revision history.
For programs or projects of long duration such as multi-year monitoring programs, the QAPP should be reviewed regularly as specified in the organization's QMP. If it is necessary to modify the QAPP during the timeframe when it is in effect, the same steps should be taken as in the initial phase to ensure that the revised QAPP is reviewed and approved by the original approvers or by appropriate personnel as specified in the organization's QMP.
1.9	PERIOD OF APPLICABILITY
This section offers guidance on Standards Annex B2.3, Applicability of QAPPs.
Consistent with EPA policy, this Handbook is valid for five years from publication and will be subject to review at which time it will be revised and reissued, reaffirmed without change, or withdrawn. If circumstances warrant, this handbook may also be revised during the course of the five-year period.
1.10	SUPERSESSION
This document conforms to the current editions of Quality Policy CIO 2106, and Quality Procedure 2106-P-01.0, as well as the Internal and External Standards. This document supersedes the previous QAPP-related guidance documents EPA QA/G-5 (EPA 2002a) and
EPA QA/G-5M (EPA 2002c).

CHAPTER 2

QAPP ELEMENTS FOR THE
COLLECTION OF DATA BY DIRECT MEASUREMENT
The collection of data by direct measurement describes the activities where the Agency, its contractor, or organization working in agreement with Agency, gathers new data for a project. The internal Standard applies to Agency activities, the External Standard to non-EPA organizations.  The two Standards run in parallel with the exception of Internal Clause 7.11 Management of Quality Requirements for External Agreements, which applies only to Agency personnel involved with organizations having agreements with the Agency.
2.1	OVERVIEW OF QAPP ELEMENTS FOR THE COLLECTION OF DATA BY DIRECT MEASUREMENT
The QAPP integrates all technical and quality aspects for the life cycle of the project including planning, implementation, and assessment. The ultimate success of an environmental program or project depends on the quality of the environmental data collected, produced, and used in decision-making, and this quality depends significantly on the adequacy of the QAPP and on its effective implementation. Probably the most important part of the QAPP is in the planning for obtaining the data or information because an error at this stage may be difficult to rectify once the project is underway.  One of the key outputs of the systematic planning process is the construction of a conceptual model that describes the scientific and engineering process under investigation. The conceptual model is an important tool for organizing information about the current state of knowledge and understanding of the project, as well as for documenting key theoretical and practical assumptions underlying the data and information collection. Further details on systematic planning are to be found in Annex C of the Standards and EPA 2006a.

The format of the QAPP is not critical except that all required elements must be addressed and the content of the QAPP presented in a logical, easily understood manner. In some cases, certain elements will not be appropriate for a particular project. Elements that do not apply can be addressed with a simple statement of why the information is not relevant or with a cross-reference to another approved document in which the information appears. It is acceptable to add sections to a QAPP beyond those identified herein. For example, if a project includes the temporary care of live animals such as laboratory mice, additional sections relevant to activities involving the care and feeding of the animals could be included. In addition, any other applicable policies or procedures (e.g., administrative, IT, or long-term data management) may be referenced or included as beneficial for the quality of the project. If use of a particular information system or hardware/software configuration is vital to project success, the QAPP should address IT quality considerations, including a listing of IT-related procedures that will be applied. EPA organizations should consider the set of CIO policies, procedures, standards, and guidelines for this purpose. For non-EPA organizations, the applicable external agreement may incorporate relevant IT guidelines.

Documentation, such as an approved Work Plan, SOPs, model calibration reports, etc., may be referenced in response to a particular required QAPP element to reduce the size of the QAPP and the time required for preparation and review. All referenced documents should be attached as appendices to the QAPP itself or be placed on file with the appropriate EPA organization and available for routine referencing when needed.
This chapter focuses on QAPPs for projects that involve the generation of new data. The elements are arranged according to the Plan-Do-Check-Act project life cycle.
 2.2	PROJECT MANAGEMENT (Plan)
Planning, the first part of the construction of a QAPP, deals mainly with the management aspects of the project. Most of the structure is simple good management practices, the only QA-related part being the establishment of data or project quality objectives and measurement performance criteria (section 2.2.6). The section offers guidance on Standards Clause 7.5, QAPP, Standards Clause 7.7, Systematic Planning, Annex B3.3 Project Management, and Annex B3.4 Data Acquisition.
The elements in this section address the format and disposition of the QAPP, project administrative functions, project information and goals. These elements document the backbone of the project planning process and lay the groundwork for the more technical elements. The QAPP must describe the project adequately and the elements that can address the basic project management and objectives of the work include:
   * title, version number, and approval/sign-off sheets;
   * document format and table of contents;
   * distribution list;
   * project organization and schedule;
   * project/problem background and description;
   * data or project quality objectives and measurement performance criteria;
   * special training requirements/certification; and
   * documentation and records requirements.

The records for a QAPP should conform to the EPA Records Management Policy (EPA 2009a).

2.2.1	Title, Version, and Approval/Sign-Off
Each QAPP should include a page with the title of the project and the name of the organizations involved in various aspects of that project. The version and number assigned to the QAPP should also be clearly identified along with the title. It is acceptable to create separate title pages and signature pages, as long as the document title, version number, and date appear on the signature page. The names, titles, signatures, and signature dates of those approving the plan are also placed on this page. Individuals responsible for approving the QAPP may include the organization's Technical Project Manager and QA Manager, and the EPA (or other funding agency) Project Manager and QA Manager. Their signatures indicate both their approval of the plan and commitment to follow the procedures noted. Other key personnel that may sign the plan are the laboratory directors, the field operations manager, other QA officers, prime contractors, and subcontractors.
This approval information is typically the first page of the QAPP. Depending on the organization's administrative policy, QAPP approval could also be in a separate memorandum. The signature dates indicate the earliest date when the environmental data operations portion of the project can start (i.e., its effective date). The QA Manager will also determine if digital (electronic) signatures are acceptable for the approval of the QAPP.
In addition to the title, version number, and approval signatures, it is important to include a revision history. Each time the QAPP is revised, as approved by the QA Manager, the version number should be updated and the revision history should be amended to include a brief summary of the change and date.
2.2.2	Document Format and Table of Contents
The QAPP should be organized such that it meets the project's needs, can be reviewed efficiently, and meets the document control requirements of the QMS under which it is developed. A document control format, such as the example shown in Figure 4, or footer created for each page to show revision status may be used to support QAPP development.
                       Project Name/# __________________
Section # _______________________
Revision # ______________________
Date ___________________________
Page ___________ of ____________
                   Figure 4.	Document control format example
The Table of Contents will generally list QAPP elements, as well as any tables, figures, reference sections, and appendices necessary to the project. If the QAPP author prefers to organize the plan differently than how the elements are organized in this Handbook or in the UFP-QAPP worksheets (EPA 2005b), a table may be inserted here to cross-reference where the information for each element may be found to simplify review. Standard operating procedures (SOPs) may be included as appendices to the QAPP. Depending on the type of project, sampling methods, analytical research protocols, or data management procedures may be attached. If SOPs or other data gathering, data analysis, or evaluation protocols are not documented in, or attached to the QAPP, they must be available to the project team and QAPP reviewers through some other means. In the case where proprietary standards (for example, those of ANSI or ISO) are used, reference to the location on the relevant web-site may be sufficient.
2.2.3	Distribution List
The distribution list identifies all individuals who should get a copy of the approved QAPP, either in hard copy or electronic format, as well as any subsequent revisions. Key personnel responsible for project implementation and/or funding, and who should have the currently approved QAPP, should be listed with their project titles or positions, organization names, email addresses, and telephone numbers. Beyond the initial distribution of the QAPP to all personnel who will need access to it, the distribution list also serves as an easy reference of who needs to be alerted and provided with a revised version of the QAPP in the case that modifications are necessary. Some organizations choose to provide the distribution list on the title or approval page, others elect to include this list in the project organization section when listing key personnel and their contact information.
2.2.4	Project Organization and Schedule
It is important that roles and responsibilities are well defined prior to initiating project activities. Those individuals involved with the major aspects of the project are named in this element along with their project responsibilities. For example, the people responsible for maintaining, updating, and overseeing implementation of the QAPP would be named here. The personnel included in this element should include the lead scientists, researchers, modelers, consultants, and contract and subcontract QA representatives.  If the actual personnel cannot be initially identified, then the position description of that person's function should be given.
An organizational chart or table can be very helpful, and should be included if appropriate. It is also helpful to indicate lines of communication among individuals or groups, and this can be shown easily on an organizational chart or an organizational network diagram. While a single individual may have more than one responsibility in a project, the project should be organized such that any person having QA responsibilities is independent of those generating and using project information. If this is not possible, owing to the size of the organization or for some other reason, an alternative method of ensuring effective QA review should be specified in the QAPP.
The level of detail included in the schedule is left to the discretion of the QAPP authors. It may be beneficial to have a very detailed and strongly stated schedule for the project to follow. This increases the risk of requiring QAPP revisions if the schedule needs to be changed during the lifetime of the project unless the QAPP explicitly states it will not be revised simply due to schedule delays. It is useful to include critical points in the project such as expected date of QAPP approval, sub-section start and end points, dates for sampling, dates by which analyses are needed, or dates modeling subroutines need to be completed.  When creating the schedule, allowance should be made for potential delays, down-times, maintenance off-lines, and general inefficiencies inherent in any project. If the project includes regulatory or court-mandated deadlines, these should be highlighted to ensure their importance is noted. For projects in which milestones or deadlines are not well defined, a more generalized work schedule can be formulated.
2.2.5	Project Background, Overview, and Intended Use of Data
This overview should give the reader an understanding of the problem to be solved, along with any pertinent background information for the project. It describes why the project will be done and what goals the project intends to accomplish. The general project goals stated here form the foundation for the entire study. Equally important, the development and documentation of this element will help ensure that all project team members clearly understand and agree on the underlying purpose of the project, increasing the likelihood that the project design will address and accomplish that purpose.
In addition to the general overview, include information on the background and use of previously approved QAPPs relevant to this project.  In addition to information about the project, it is good practice to include an outline of information that is currently not sufficiently developed. Clearly state who needs the information and what the intended use of this information will be. Problems that are more complex will lead to more extensive information in this section. The reader should be able to understand the importance or context of the project. The general project goals stated in this section will be refined in Section 2.2.6.
2.2.6	Data/Project Quality Objectives and Measurement Performance Criteria
EPA requires the use of systematic planning for all projects that involve the collection or use of environmental data, and the QAPP documents the outcomes of this process (see also Annex C of the two Standards).  EPA encourages the use of the Data Quality Objectives (DQOs), which may also be called project quality objectives, depending on the organization's preference. The DQOs are established from total study variability as the overall qualitative and quantitative goals of the project. If the DQO process (EPA 2006a) is implemented, the QAPP houses the outcomes of that process. For example, the project background, the number and type of samples needed, and the designs for sample collection and analytical measurement are all included in the QAPP.
Under any systematic planning process for a project that includes environmental data collection it is desirable to define tolerable levels of uncertainty for components of the total study variability. Total study variability is due to natural (field) variability and measurement (laboratory) variability. The effects of field variability may be minimized by selecting an appropriate sampling design and number of samples collected (see Appendix B for further discussion). Measurement variability can be minimized by careful selection of specific measurement techniques (see Appendix C for further discussion).
DQOs are supported by Data Quality Indicators (DQIs), which are supported by Measurement Quality Objectives (MQOs). These DQIs are linked to field and measurement variability (see Appendix B for further discussion). MQOs are sometimes known as Measurement Performance Criteria (MPCs) depending on the protocols of the organization. See Appendix B for definitions and details of DQIs and MQOs. It is essential that the definition of what is meant by MQOs or DQOs is made clear early in this section. Some organizations use the same acronyms for different operations. For example, some organizations use DQOs in place of DQI performance measures, or use MPCs for these measures.
During the DQO or other systematic planning process, DQIs are considered, and specific MQOs are set to ensure that data are of appropriate quality for their intended use. These MQOs are the criteria against which the data are measured in the "Do" step of the "Plan-Do-Check-Act" cycle. MQOs/MPCs are the cornerstones of the technical sections of the QAPP.
The traditional DQIs considered in project planning include:
precision;
bias;
representativeness;
comparability;
completeness; and
sensitivity.
While historically there has been considerable attention directed to bias, precision, and sensitivity, it is really representativeness that is probably the single most important indicator of data quality. Representativeness is a qualitative measure of the degree to which data accurately and precisely represent a characteristic of a population.
A poor sampling design with very high quality analytical laboratory analyses might give data that meet MQOs for precision, bias, sensitivity, comparability, and completeness while not providing information that is actually representative of the characteristics of the population of interest. Throughout this guidance, the concept of representativeness (as well as the other DQIs) will be used to motivate the need for QA and QC elements. If the data collected fail to be representative of what was intended to be sampled, no amount of statistical manipulation can make the data set valid. Great attention should be given to ensure that what is sampled or collected truly represents the project population of interest. It is axiomatic that a sufficient number of samples should be taken in order to achieve sense of representativeness.
For each DQI there should be an indication of how rigorous the measurement needs to be. Without making this clear, data may be inadequate for final use, or conversely, more expensive to collect or produce than necessary because a higher standard of quality is targeted than is required for the project purposes.
The QAPP should indicate how rigorous the collected data or information should be to answer the question, resolve the problem, or support the decision to the level desired and needed for the project. For projects that include environmental data collection or production, acceptance or performance criteria information should be determined during the project planning process and documented in the QAPP.
For example, in an enforcement decision, this discussion could focus on regulatory or action levels and the quality of the data required to make and enforce decisions in relation to the project action limits. Some organizations, especially those using the UFP-QAPP, distinguish between screening data (data of known quality suitable for use for interim decisions) and definitive data (data of known quality suitable for use for final decisions). Whether for screening data, definitive data, interim data, or count data, the same tenet holds true: it is important to understand the intended use of the data and to set and meet measurement performance criteria that ensure the quality of that data are sufficient to achieve the decision objectives.
It is important to note that the selection of DQOs and MQOs must be grounded in reality. An arbitrary selection of these numerical values, or default to instrument performance characteristics, will rarely lead to the generation of data quality sufficient to meet the needs of the project.  The MQOs must be clearly linked to the DQOs, with these in turn linked to the project's overall objectives.
In addition to planning for the type of data needed to address a particular project, it is important to also consider the way those data will be analyzed. That is, what statistical or other evaluation methods will be applied to the data in order to reach a conclusion for the project? The QAPP should address all aspects of project planning from the problem statement, through data and information needs, through quality criteria to be applied, through laboratory analyses of samples, through statistical analyses of the data, through the final project decisions and reports, to disposition and maintenance of the data and records.
Establishing DQOs and MQOs for exploratory projects (for example, research studies) where little or no information on the quantitative magnitude of what is expected is difficult. The use of information from similar or related studies is useful.
For example, consider an experiment designed to investigate the neutralizing properties of a chemical on a known contaminant mixture. For this experiment, it may be sufficient to use qualified statements such as "the precision (MQO) for measuring reduction of acid is expected to be of the order +/- 10% based on similar experiments as documented in XXXX". A discussion documented in the QAPP of the consequences (or results) of achieving different realized DQOs or MQOs is advisable.
In some circumstances, DQOs may be stated in qualitative terms having definitive reference points. For example, "The project will produce data that will qualify to receive the `A' rating with respect to the rating system described in Section 4.4.2 of the Procedures for Preparing Emission Factor Documents (EPA-454/R-95-015)" (EPA 1997a). Although it is stated in qualitative terms, this DQO is measurable using specific acceptance criteria that are presented in the cited document.  Simplistic descriptors such as "very good" or "acceptable for this project" are to be discouraged unless these are specifically defined in the QAPP.
2.2.7	Special Training Requirements and Certification
Special training or certifications are sometimes necessary for project personnel and laboratories associated with projects. This may include such things as having project personnel complete specialized hazardous material handling training, being skilled in the collection of samples for trace metal analysis, being trained in global positioning technology, or being trained in specific sampling techniques. Similarly, project personnel may require special security clearances such as clearance to enter a military site. Laboratory certification for the analysis of certain types of samples may also be necessary. The QAPP should describe how the project will ensure that competent personnel are available to perform the work. A method for ensuring competency helps ensure project specific requirements are met. For example, the ORD Policies and Procedures Manual, Section 13.4, Minimum QA/QC Practices for ORD Laboratories (EPA 2001b), states "Prior to performing sample analysis with a new method for which proficiency has not been previously demonstrated, the analyst must demonstrate proficiency with the method by completing the following: (1) perform valid initial calibrations, (2) perform method detection limit determination, (3) demonstrate that they can meet all minimum QA/QC acceptance criteria as presented in the method document, e.g. the SOP, and (4) if available, satisfactorily analyze a performance evaluation sample or a second source standard". The QAPP should also specify how this information will be documented, where the records will be kept, and indicate who is responsible for ensuring that these special training and certificate needs are met and documented.
2.2.8	Documentation and Records Requirements
The QAPP should describe the process and responsibilities for ensuring that project personnel will receive the most recently approved project documents such as the QAPP, SOPs, and other documents used throughout the project operation. The QAPP authors may choose to tell how these documents will be updated and how information regarding updates will be communicated.
The QAPP should identify the information to be included in the project data files (or project record) and its format. This might include the following items:
sampling collection and handling records such as
field notebooks or operational records,
Global Positioning System (GPS) data,
chain-of-custody forms, and
sample receipt records, including sample tags and shipping bills;
analytical records such as
analytical log books,
test method raw data and QC sample records,
Standard Reference Material and/or proficiency test sample data, and
instrument, equipment, and model calibration information; and
data assessment records such as
               o             validation of software used,
               o             input and output files as results of code,
               o             statistical methodologies, and
               o             data-base test procedures.
In some cases, access and the final disposition of records may be regulated or required by policy and this status should be included in the QAPP. The QAPP should clearly indicate where all project documents, reports, records (including digital records) will be stored and for how long.
2.3	DATA ACQUISITION (Do)
This is the heart of a QAPP as it documents the requirements for sample identification (where the samples will be taken), sample collection (how to preserve the integrity of each sample), and sample handling (the management of the samples). This section offers advice on the Standards Clause 2.1, Scope, Clause 7.7, Systematic Planning, and Annex B3.4, Data Acquisition.
The elements in this section address all aspects of data production and collection to ensure that appropriate methods for sampling, measurement and analysis, data production, data handling, and QC activities are employed and documented. These elements describe the actual methods or methodology to be used for the collection, handling, analyses of samples, and the management such as compiling and handling of the data. Where applicable, the elements shall be addressed in the QAPP and include:
   * sampling collection design, experimental design, and sampling tasks;
   * sampling procedures and requirements;
   * sample handling and custody requirements;
   * analytical methods requirements;
   * quality control requirements;
   * instrument/equipment testing, calibration and maintenance requirements, supplies and consumables; and
   * data management requirements.

The QAPP should provide detailed information on the methods to be used. It is important to note that if the designated methods are well documented and are readily available to all project participants, citations are adequate.
When procedures or methods are not commonly used or readily available, detailed copies of the methods and SOPs should accompany the QAPP either in the text or as attachments.
2.3.1	Data Collection Procedure, Experimental Design, and Sampling Tasks
This element describes the project's data collection or research experimental design. Keys to this element are the assumptions made and how the data will be obtained. This element explains the "how and why" of the project's information collection design to ensure that the appropriate data are collected for this project. Input for this element will come from the project's systematic planning (e.g., the DQO process) and also see Annex C of the two Standards.
For data collection there are two classes of sampling designs to consider: probability-based and judgmental, and they have very different properties. The former are sometimes called statistical designs, and the latter sometimes called directed sampling designs. Strong statistical conclusions are available with probability-based designs but not with judgmental designs. Use of professional expertise and/or historical knowledge about the site can improve development of statistical or judgmental sampling designs.
 Key questions to be considered include:
is this project to be comparable with previous sampling or analytical efforts, or with a health-based or regulation standard?
can samples or measurements be taken according to a probability-based design?
is the objective of the sample to estimate an average or to find a hot spot?
is there a reference or background population that can be used as a comparison to the target population?
will sampling units be chosen in advance, in the field based on visual evidence, or using other technical evidence?
is there a network of sampling sites that will be visited periodically or where sampling will be performed continuously?
do all the samples need to be taken simultaneously?
is the target population approximately homogeneous or is it heterogeneous in nature needing stratification or division into approximately homogeneous areas?
can samples be composited? and
are there any potentially important sources of variability which affect the sampling such as tidal cycles, seasonal differences, and rain and wind patterns?
The answers to these questions should have been considered during the planning process and help to determine allocation of resources for obtaining samples for analysis.
For example, when the project involves a physical site, this element of the QAPP may involve:
defining the size of the area, shape, volume, or time that is to be represented by a sample (called the scale of representativeness) as part of the justification for how the sampling sites and durations will be selected;
specifying the type and number of samples to be collected, and their locations; and
providing detailed schedules for sampling and analytical activities, test runs, and reviews.
 If, instead, the project is to develop a new analytical measurement method, this step could involve:
defining the procedure to be used for the testing;
specifying the levels at which testing will occur;
setting a timescale for sequential instrument readings; and
clearly stating any extraneous variables (e.g., temperature) that will be controlled.
The QAPP should describe the experimental data production or data collection design for the project, including as appropriate:
the sampling design (e.g., systematic grid if spatial, time-interval if temporal);
the rationale for the design;
the types and numbers of samples required;
restrictions due to resource constraints;
need to collect background samples;
the sampling locations and frequencies;
rationale for ensuring representativeness of data; and
measurement parameters of interest.
 Advice on selecting the appropriate sampling design may be found in Chapter 2 of the Guidance for Choosing a Sampling Design for Environmental Data Collection, EPA QA/G-5S (EPA 2002b). Advice on experimental data collection is available from the standard statistical literature under the title "Design of Experiments" or "Experimental Design".
2.3.2	Sampling Procedures and Requirements
Standardized sampling procedures provide consistency between samplers; facilitate collection of accurate, precise, and representative samples; and help to ensure data comparability and usability. These may be in the form of SOPs established by the organization or may be unique to the investigation in hand. The QAPP should provide a list or table of all field sampling procedures and brief descriptions of sampling procedures. In many instances reference to where the SOPs are located should be sufficient but in some cases may be attached to the QAPP to allow for review and approval. Any SOPs that are modified to meet project-specific needs should be attached to the QAPP.
Sampling procedures should include SOPs, when available, for sampling each matrix and each analytical parameter for each type of equipment and technique. The SOPs provide detail on the appropriate number, size, and type of sample containers to be used for collection of each field sample and field QC sample and the proper temperature, light, and chemical preservation procedure for those samples.
Also included should be information on the techniques to be used for specific locations due to difficulties in sample collection. These can include restrictions on physical sample collection (e.g., inability to collect the sample where planned), availability of trained personnel (e.g., lack of a certified technician), or seasonal influences on the obtaining of a physical sample (e.g., adverse winter conditions).
The QAPP should describe how samples will be collected. The selected sample collection procedures should be appropriate to ensure that project personnel collect representative samples in a consistent manner for all required sample matrices and locations; that contamination is not introduced during collection; and that sample volumes are properly preserved in order to meet project objectives.
The QAPP should include a description of preservation procedures (temperature, light, or chemical) that maintain sample integrity in the field, prior to and during shipment to, and immediately upon receipt by the off-site or mobile on-site laboratory. The QAPP should document requirements for sample volumes, container types, number of containers, and preservation procedures for each analytical group, matrix, and concentration level.
The QAPP should provide details on the procedures for both the initial cleaning of sampling equipment and also subsequent decontamination procedures that will be followed during the sampling event. These procedures will help ensure that collected samples are representative of the sampling location by verifying that sampling equipment is clean and free of target analytes, Chemicals of Concern, or interferences. Cleaning and decontamination procedures should cover all equipment that contacts a sample. If the sampling equipment is disposable ("one use only"), procedures for cleaning and decontamination are not necessary; however, the QAPP should state that disposable equipment will be used and where that equipment will be deposited.
Development of a standardized table customized to the needs of the organization is encouraged as it leads rapid review and ease in tracking adherence to sample analyses requirements.
2.3.3	Sample Handling, Custody Procedures, and Documentation
The QAPP should include all sample collection documentation and sample handling, tracking, and custody procedures needed to ensure that sample integrity and custody are maintained. Without sample integrity, and without clarity regarding the chain-of-custody for all samples, it may be impossible to assert that the data are of adequate quality to meet project objectives. The procedures should address sample collection, packaging, handling, and shipping, as well as records, receipt of laboratory samples, archiving, and disposal. These may be included by reference if an approved SOP is maintained by the organization.
It is important to note that the detail needed for handling, chain-of-custody, and documentation should be commensurate with the goals or objectives of the project (application of the Graded Approach).  There are many sample collection procedures that do not need extensive procedures (for example, use of a sample inventory, or use of an EPA number to preserve some degree of confidentiality).  Reference to the organization's Quality Management Plan and the organization's QA Manager should clarify what is needed to meet the project's objectives.
Sample Handling and Tracking
Proper sample tracking systems support the chain-of-custody procedures, which help to ensure sample authenticity and data defensibility. The QAPP should document the procedures that will be followed to identify and track samples that are collected in the field, analyzed on-site, and delivered or shipped to an off-site laboratory for analysis, as well as samples transferred throughout the laboratory. If samples are shipped to an off-site laboratory, then the laboratory's sample handling and tracking system should also be described.
The sample handling and tracking procedures should include the following descriptions:
the sample numbering system for field sample collection with an example (if applicable, the numbering system should follow specific programmatic requirements that apply to the project. A systematic approach for numbering samples should be used so that each sampling location, matrix type, sample depth or height, and date and time of collection can be uniquely identified and cross-referenced to the programmatic sample number);
the sample container identification information;
the latitude, longitude, and elevation (altitude) for the sample collection location;
the laboratory sample tracking procedures (if laboratory identification numbers will be used to track samples internally, the laboratory procedure must describe how these laboratory identification numbers will be cross-referenced with the sample number assigned in the field); and
the sample storage procedures used by the off-site or mobile on-site laboratory.
To demonstrate the project's sample handling process, the QAPP should include a table that shows the flow of samples from the time of collection to laboratory delivery to final sample disposal. The table should identify each component of the project-specific sample handling system; indicate the personnel (and their organizational affiliations) who are primarily responsible for ensuring proper sample handling, custody, storage, and disposal; and specify the length of time that samples, digestates and extracts, and biological collections will be retained by the laboratory prior to disposal.
The QAPP should also describe how samples will be delivered or shipped to the laboratory. The description should include the name of the carrier service, if applicable, and define how samples will be batched or grouped when sent to the laboratory. The QAPP should include provisions for packaging, marking and labeling, and shipping samples in compliance with the Department of Transportation regulations for shipping hazardous and nonhazardous materials (see also 49 CFR 171). Air carriers that transport hazardous materials require compliance with the current edition of the International Air Transport Association Dangerous Goods Regulations.
Shipment papers, including bills of lading and airbills, should be retained by the laboratory with chain-of-custody records. Examples of all sample shipment forms to be used should be attached as an appendix to the QAPP.
Sample Custody Procedures
A sample is in "custody" if it is in the actual physical possession of authorized personnel or in a secured area that is restricted to authorized personnel. For some projects, an evidentiary paper trail documenting sample custody is required in order to meet project quality objectives. Since it is often difficult to predict what samples or projects will require proof of custody after the fact, all data collection events should employ documented chain-of-custody procedures to ensure data authenticity and defensibility. The evidentiary trail from sample collection through data production and archiving is maintained using sample custody procedures and documented by complete chain-of-custody records. Chain-of-custody procedures ensure accountability for the location and integrity of the sample at all times. The ASTM document D4840-99 Standard Guide for Sampling Chain-of-Custody Procedures contains useful information regarding chain-of-custody procedures (ASTM 1999).
The QAPP should describe (or have attached) the procedures that will be used to maintain sample custody and integrity and to document the implementation of chain-of-custody procedures. Sample custody procedures should include the field sampling team's procedures for maintaining and documenting sample custody from the time samples are collected in the field through packaging, shipment, and delivery to the laboratory. Field sampling documents that describe chain-of-custody procedures, including SOPs, should be attached to or referenced in the QAPP. The laboratory's procedures for maintaining and documenting sample custody from the time the samples are received at the laboratory through archiving and disposal should also be attached to or referenced in the QAPP. The use of identification or tracking software such as FORMS II Lite (TM), should be documented, if applicable. Examples of all chain-of-custody documents that will be used during the project should be provided in the QAPP, including chain-of-custody forms, sample identification, custody seals, laboratory sample receipt forms, and laboratory sample transfer forms.
Sample Documentation
The QAPP should describe sample documentation procedures that will be followed for the project. Proper documentation of field sampling activities helps ensure sample authenticity (i.e., the sample identity is correct) and data integrity.  The QAPP should also indicate what information should be marked on the sample collection container label (e.g. time, date, sample number), the analyses to be performed, preservatives used, and name (with signature) of the field technician completing the sample collection.
Documentation for sample collection includes, descriptors of sample type (duplicate, QC check, etc.), field logbooks, field data collection forms, field calibration forms, chain of custody, and sample container identification. The QAPP should specify the required sample identification information and include an example. An electronic system, such as a bar code or FORMS II Lite (TM) (which retrieves information stored elsewhere), may be used. The QAPP should also describe how the information on the label will be preserved (e.g., by covering the label with clear tape to minimize water damage during transit).
2.3.4	Analytical Methods Requirements and Task Description
The QAPP should identify the analytical methods, sensitivity, precision, bias and acceptance limits for QC results, equipment required, sub-sampling or extraction methods, laboratory decontamination procedures and materials (such as in the case of hazardous or radioactive samples), waste disposal requirements (if any), and any specific performance requirements for the method. Where appropriate, analytical methods may be identified using their number, date, and regulatory citation. If data from laboratory analyses do not form part of the overall project description, then a simple outline of the data production method is necessary. For example, to determine if there have been observable ecological deficits at a study site, plant cover and a diversity study should be included in the project summary. The QAPP should then describe the number and size of the study sites, the procedure for counting the species (e.g., number of unique species or number of specimens of each species), and the procedure for locating the sites.
The QAPP should specify any method performance standards. If a method allows the user to select from various options, then the rationale for selection should be given. For non-standard method applications such as for unusual sample matrices (e.g., explosives testing of sea cucumber tissue), appropriate method performance study information is needed to confirm the performance of the method for the particular matrix. If previous performance studies are not available, they should be developed during the project and included as part of the project results.
The organizations or laboratories that will provide the analytical services (all on-site and off-site laboratory analytical work, including all prime laboratories, subcontractor laboratories, and backup laboratories) should be identified together with the measures taken to ensure laboratory proficiency.  It is recommended that a table of analytical services be included in the QAPP, containing the following information at a minimum:
matrix;
analytical group or class of compounds;
concentration level;
sample location or identity number;
analytical SOP to the analysis requested;
preservation and holding time;
number and type of sample containers;
required data package turnaround time;
the number of QA and QC samples needed;
laboratory/organization (name, address, contact person, and phone number); and
the arrangements made with a backup laboratory/organization.
 If these method requirements are already described in the laboratory's set of SOPs, these should be attached or referenced as appropriate.  In addition, the format of the data package (electronic, hard-copy, etc.) should be identified.  These requirements should be commensurate with those requirements contained in the overall agreement between the organization and laboratory, including the timeframe for delivery of data and information. It is possible the conditions of the agreement will affect the MQOs established earlier and resolution of the differences will be needed (for example, the detection limit MQO set at 0.1ppb, but the agreement reporting limit set at 0.5ppb).
The measures taken to ensure the laboratory will meet the proposed data quality needs of the project should be discussed. When necessary, as determined by application of the graded approach, the review of SOPs and the results from performance evaluations using similar contaminants of concern in matrices similar to those expected may be needed.
It is important to address what to do when a failure in the analytical system occurs, who is responsible for corrective action, and how the effectiveness of the corrective action shall be determined and documented.
2.3.5	Quality Control Requirements
Quality Control (QC) has two important aspects: preventive and corrective. The preventive aspect is designed to prevent quality problems before they are observed.  This is achieved through the use of proactive procedures that minimize variability and bias at each stage of measurement. Examples of proactive QC include the use of check sheets, control charts, and standardized data recording forms to promote and document consistency in performance of specific activities. The corrective aspects of QC can sometimes alleviate situations where there is the potential for an error in interpretation. Some of the retrospective activities of data verification and validation may be considered examples of corrective action, whereas the use of statistical (Shewhart) control charts is an example of proactive corrective activities.
The QAPP should identify QC activities needed for each sampling, analysis, or measurement technique. For each QC activity, the associated method or procedure, acceptance criteria, and corrective action should be listed. As standard methods are often vague or incomplete in specifying QC requirements, simply relying on the cited method to provide this information is usually insufficient. QC activities for the field and the laboratory include, but are not limited to, the use of blanks, duplicates, matrix spikes, laboratory control samples, surrogates, or second column confirmation. A list of commonly used QC samples is included in Appendix C.
In environmental studies, measurement is the process of obtaining a quantitative value describing a chemical or physical property of an individual sampling unit or specimen collected from this unit. Measurement may involve direct field measurements using survey instruments or collection and handling of physical samples followed by analysis in a fixed or mobile laboratory. Choices that the technical team makes regarding sample acquisition, sample handling, preparation, and analysis can influence the quality attributes of the resulting data. Sources of measurement variability include:
within-unit, small-scale variability (influenced by the nature and distribution of the characteristic of interest within the sampling unit and the media assayed);
physical sample acquisition protocol (sample collection tools and procedures such as compositing);
sample handling and transport;
variability associated with the measurement assay (influenced by the capabilities and management of the analytical resources);
inaccurate determination of strata boundaries;
inadvertent bias by the sample selector (identification uncertainty);
homogenization and sub-sampling procedures;
sample preparation, extraction procedures, and sub-sampling extract for analysis; and
analytical determination.
Each MQO should have at least one QC procedure designed to prevent unsatisfactory results for as many sources of variability as possible. An example would be the proper adherence to documented SOPs (a preventative measure) combined with use of Shewhart QC charts (statistical process control) to monitor analytical results (a corrective measure).
The QAPP should describe, reference, or attach the procedures to be used to calculate applicable statistics such as precision and bias. Copies of the formulae are acceptable as long as the accompanying narrative or explanation specifies clearly how the calculations will address potentially difficult situations such as missing data values, "less than" or "greater than" values, "below detection", "not detected" and other common data qualifiers.
One use of routine QC results is assessment of the laboratory's internal operations; for example, the routine analysis of calibration blanks to test for inorganic contamination or a problem in system calibration. In this case, a DQI could be just the actual blank result, expressed in appropriate units, with an MQO equal to the laboratory's internal acceptance criterion. A poor result from a routine calibration blank should result in an immediate corrective action by the laboratory. This action may be limited to simply rerunning a calibration blank to demonstrate that the first result was an aberration. While it is important to verify that the laboratory is monitoring and controlling the quality of their internal operations, results from samples like a calibration blank are less important in project planning from the DQO perspective than other potential sources of error that may not be controlled. The same can be said for overall analytical calibrations. A properly calibrated analytical system is always required and expected for analysis of project samples.
2.3.6	Instrument/Equipment Testing, Calibration and Maintenance Requirements, QA/QC for Supplies and Consumables
It is necessary to recognize the growing importance of field instrumentation and use, and so the same care and attention given laboratory equipment should be given to field equipment.  If the field equipment is intended to give less precise readings, lesser QA/QC requirements may be needed as documented in the organization's QMP.
Many of these requirements are part of a laboratory's quality system and should be documented in the laboratory's quality manual (however named). It may be sufficient to simply reference this quality manual (including revision number and date) or reference the contract arrangement between organization and laboratory. If the project has specific requirements different from the laboratory's quality manual, specificity in activities, actions, analyses, and responsibilities for their successful implementation may be necessary. The depth of specificity should be linked to the project's objective by application of the graded approach or consultation with the organization's Quality Manager.
Instrument/Equipment Testing and Inspection
The QAPP should describe how inspections and acceptance testing of instruments, equipment, and their components affecting quality will be performed and documented to assure their intended use as specified. The QAPP should identify and discuss the procedure by which final acceptance will be performed by independent personnel (e.g., personnel other than those performing the work) and/or by the EPA project manager. The QAPP should describe how non-conformances or deficiencies are to be resolved, when re-inspection will be performed, and how the effectiveness of the corrective action will be determined and documented.
Instrument Calibration and Maintenance
The QAPP should identify all tools, gauges, instruments, and other sampling, measurement and test equipment used for data collection or production activities affecting quality that should be controlled and calibrated at specified periods, to maintain performance within specified limits. The QAPP should include detailed descriptions or references to SOPs describing how calibration will be conducted, the frequency of calibration, the procedures for analyzing calibration data, the acceptance criteria for calibration limits (with references), what certified equipment is needed, how valid traceability to established performance standards will be demonstrated, and how long calibration records will be maintained. If no such established standards exist, the basis for the calibration should be documented.
The principal reason for performing calibration is to reduce bias to a minimum. The decision as to the type of standard the instrument should be calibrated against depends on the overall accuracy expected from the instrument, the project objectives (DQOs or MQOs as appropriate), and the resources allocated for this part of the project's development. The DQOs/MQOs of the project may be badly compromised should an inadequate calibration be performed.  Ineffective calibration will often give the impression of precision which can be substantially overwhelmed by an increase in bias. See Appendix C for a further discussion on bias.
The QAPP should describe or reference the process for how periodic preventive and corrective maintenance of measurement and test equipment or other systems and their components affecting quality shall be performed to ensure availability and satisfactory performance of the systems. The QAPP should identify the equipment or systems requiring periodic maintenance. The QAPP should also discuss how the availability of critical spare parts, identified in the operating guidance and/or design specifications of the systems, will be assured and maintained.
Supplies and Consumables
Not all projects will need supplies and consumables considered "critical" to the project. For those that do, this element documents the system used to ensure appropriate materials are obtained in a timely fashion.
The QAPP should describe how and by whom supplies and consumables (e.g., standard materials and solutions, sample glassware/containers, calibration gases, reagents, hoses, deionized water, potable water, and electronic data storage media) shall be inspected and accepted for use in the project. The QAPP should state acceptance criteria for such supplies and consumables and have criteria for disposable sampling equipment and supplies. Where applicable, information enabling the needed materials to be located and obtained should be listed in the QAPP, including:
   * supply source or vendor;
   * procedures for identifying, tracking, storing, and retrieving these materials;
   * identification of those responsible for maintaining supplies; and
   * any acceptance criteria needed (for example, certificate of testing).
2.3.7	Data Management Requirements
The QAPP should describe the project data management process, tracing the path of the data from their production to their final use or storage (e.g., the field, the office, or the laboratory), which is very important for ensuring data integrity. The QAPP should describe or reference the standard record-keeping procedures, document control system, and the approach used for data storage and retrieval of electronic media. The QAPP should discuss the control mechanism for detecting and correcting errors and for preventing loss of data during data reduction, data reporting, and data entry to forms, reports, and databases. Examples of any forms or checklists to be used should be provided.
The QAPP should identify and describe all data handling equipment and procedures to process, compile, and analyze the data, including any required computer hardware and software, and address any specific performance requirements for the hardware/software configuration used. A description of the procedures that will be followed to demonstrate acceptability of the required hardware/software configuration should be included (for example, after upgrading software, the data should be checked to ensure the migration has occurred successfully).
Finally, the QAPP should discuss, as necessary, the archiving, data migration, and software update changes checks that will be used to ensure no data loss.  This includes backup procedures for any data stored electronically, and protocols for access to, retrieval from, and photocopying of, information archives. In some cases retention, access to, and final disposition of some records may be regulated and require field and laboratory QC results to be captured in electronic data management systems. In those cases, this element should address and comply with all relevant regulations. In other cases, requirements for this element may be stated in the terms and conditions of external agreements and the QAPP should identify any requirements set by statute or policy.
Further assistance on aspects of data management may be found in Brilis et. al., 2004.
2.4	ASSESSMENTS (Check)
The third part of the Plan-Do-Check-Act paradigm considers the technical aspects of QA that apply to ensuring the integrity of the data.  Not all of these different assessments are necessary and should be selected by application of the graded approach that selects the level of assessment commensurate with the intended use of the data. The advice given here is drawn from the Standards Clause 7.8, Assessment of Data and Information, and Annex B3.5 Assessments.
These QAPP elements ensure that planned project activities are implemented as described in the QAPP and that reports are provided to apprise management of the project status and any QA issues that arise during implementation (see also Section 2.5.5). Assessment activities help to ensure that the resultant data quality is adequate for its intended use, and that appropriate responses are in place to address non-conformances and deviations from the QAPP. Several types of assessments that could be included are:
technical systems assessments;
performance audits of measurement and analytical systems;
surveillance operations;
audits of data quality;
qualitative and quantitative comparisons to acceptance and other criteria;
interim assessments of data quality;
evaluation of unconventional measurements; and
evaluation of unconventional monitoring projects.

 If deviations from the QAPP are identified through a formal assessment or audit, corrective actions may be necessary; these are discussed in greater detail in Chapter 3, Section 3.5 of Guidance on Technical Audits and Related Assessments for Environmental Data Operations, EPA QA/G-7 (EPA 2000).  Deviations from the QAPP identified through formal or informal review may also require corrective actions but, using the graded approach, the project manager and QA Manager should confer to ascertain the possible impact on the validity of the data. Their joint decision should be documented in the revised QAPP. In some cases deviations from the QAPP can be identified by project personnel without the benefit of formal, scheduled assessments. This section addresses those situations and describes the process by which the need for corrective action is documented, reported, and implemented and its effectiveness assessed.
2.4.1	Technical Systems Assessments
Technical Systems Assessments (TSAs) can have two forms: field-based and laboratory-based.

A field TSA includes an auditor monitoring of sampling activities to ensure that the QAPP and any field sampling and sample handling SOPs are being followed.  Additionally, the field TSA should evaluate reporting, handling, tracking, management, and review of data generated in the field (e.g. pH, salinity, conductivity, test kits, and GPS data).
A field TSA may be conducted independently of the laboratory TSA and may even use different personnel (auditors). A field TSA can be performed at the start of field sampling activities as a readiness review. The readiness review is a technical assessment that is planned and performed prior to the initiation of a project to verify that project management has brought the facility and applicable measurement systems to a state of readiness to begin the project.
A laboratory TSA is a thorough assessment of laboratory analytical procedures during which the facility (or mobile laboratory), equipment, instrumentation, supplies, personnel, training, analytical methods and procedures, laboratory procedures, sample handling and tracking, data reporting, data handling and management, data tracking and control, and chain-of-custody procedures are checked for conformance with the QAPP.

There are several ways of conducting a laboratory TSA. The technique of tracing the project samples through a laboratory can be used effectively to gain an understanding of the overall analysis process during a TSA. If required by the QAPP, samples should arrive with chain-of-custody papers or appropriate documentation and should be checked for discrepancies needing resolution before further analytical work is done. Typical sample handling steps may be digestion, extraction, and splitting or making an aliquot of the extract for various analyses. One way to trace an assessment trail is to physically follow the trail of the samples through the facility, looking in the records for their fate at each step of the trail.
An laboratory TSA can be performed prior to, at the start of, or at any time during field-sampling activities. A TSA performed during the data collection phase of a project may be particularly helpful in discovering operational problems early that could adversely impact the quality of the project results and the usability of the data.

The decision to conduct a field or laboratory TSA, and the intensity of the TSA, should be linked to the project's objectives and intended use of the data.  It is important that the principle of the graded approach be considered and the extent of the TSA tailored to avoid unnecessary expenditure of resources.
2.4.2	Performance Audits of Measurement and Analytical Systems
This has two components: proficiency evaluation, and split sample analysis.
Proficiency Evaluation
A performance evaluation (PE) sample is a quantitative assessment in which analytical results are generated by a measurement system for a sample that originates outside of a project. PEs may also be known as proficiency testing (PT) samples. Statistical analysis of PE sample results provides information on routine laboratory performance and the overall accuracy and bias of the analytical method.
Ideally, a PE sample mimics routine field samples in all possible aspects, including similarity of container, label, and shipping, such that the analyst is unaware that it is not a routine project sample. When possible, PE samples should be of similar matrices and analytes, in the same approximate concentrations as expected in the field samples, so that the analyst does not have to treat the PE sample differently than typical project samples (e.g., does not have to dilute the PE sample if project samples don't have to be diluted). Sample matrix similarity is especially important for projects involving solids, sediments, biosolids, and tissues. One potential drawback to PE samples is the difficulty in mimicking the complexity of the matrix containing a chemical of concern. If the PE matrix is different from the actual sample matrix, difficulties in interpretation of readings can occur. In the context of a Quality Program, a PE is used to determine if a measurement system's results are within data quality goals specified in the QAPP. PE results are often used to estimate the degree of bias in the measurement system. Although a PE sample can identify a problem quantitatively, it typically cannot be used to directly determine the cause of the problem. The appropriate choices for the PE samples' composition should be specified in the QAPP.
When needed, the QAPP should address the selection of appropriate PE samples. Factors to consider include:
    * analyte selection;
    * whether PE samples are single- or double-blind, native or synthetic matrix, spiked or natively contaminated or both;
    * multiple matrices and concentrations;
    * total number of PE samples;
    * acceptance criteria for PE samples; and
    * analytical methods.
A plan for selection, implementation, and reporting of PE samples should be included in the QAPP.

Split Sample Analysis
One use of split sample analysis is for comparison studies conducted to assess inter-laboratory precision and accuracy. The sampler collects one field sample and then physically splits it into two representative sample aliquots. The samples are then sent to different laboratories for analysis. Split samples quantitatively assess the measurement error introduced by the organization's sample shipment and analysis system. Split sample comparability criteria must be generated prior to sample collection and documented in the QAPP.
Another use for split sample analysis is for projects involving enforcement actions or compliance with a set standard.  Split samples are collected and one sample is analyzed by the EPA, the other is provided to a party representing the site or facility owner.  Here the two samples are independently analyzed although EPA may not know the results of the other sample (including QC results) until much later in the project's development.

For further information on the use of audits see Guidance on Technical Audits and Related Assessments for Environmental Data Operation, EPA QA/G-7, (EPA 2000).  For further information on the use of various QC samples (including split samples), see Appendix C of this guidance.
2.4.3	Surveillance of Operations
Surveillance is the observation of ongoing work for the purpose of verifying and documenting conformance with specified requirements and/or procedures such as those given in the QAPP and/or SOP. Surveillance is focused on a particular technical activity, rather than on the entire measurement system. It is typically less formal than other types of assessments, but it should also include appropriate preparation, conduct, reporting, and follow-up phases. As appropriate, surveillance may be employed as part of a TSA. The objective of surveillance is to provide confidence through real-time observations that an activity has been performed in accordance with approved and specified methods and procedures. It allows for immediate identification of any deficiency and initiation of action to correct the deficiency and its underlying cause. When non-conformances or deficiencies are identified, project management should be notified promptly so that corrective action may be implemented.
The QAPP should also state under what conditions surveillance of operations will be performed, (i.e., what are the triggering events that would require a surveillance assessment). The organization assessed should provide a written response that discusses the action taken to correct any observed deficiencies and to prevent similar deficiencies in the future. The QAPP may allow for immediate notification of the status and performance of the project to management if authorized by the scope of the assessment. It also can include follow-up to verify that corrective action was implemented.
In some cases, activities observed during a surveillance assessment may merit an immediate stop- work order.
The QAPP should discuss the circumstances that would cause a need for such an action and the procedure for issuing an official stop-work order (Internal Standard, Clause A3.12 (EPA 2013a).
2.4.4	Audits of Data Quality
An audit of data quality is an examination of data after they have been collected and verified by project personnel. It is conducted to determine how well the measurement system performed with respect to the performance goals specified in the QAPP and whether the data were accumulated, transferred, reduced, calculated, summarized, and reported correctly. It documents and evaluates the methods by which decisions were made during treatment of the data. The QAPP should specify the questions to be answered in an audit of data quality such as:
is there sufficient documentation of all procedures used in the data collection effort to allow for repetition of the effort by a person or team with technical qualifications similar to those of the original data collector?
can the data be replicated by the original data collector?
is there sufficient documentation to verify that the data have been collected and reported according to these procedures?
is enough information provided to allow a potential user to determine the quality and limitations of the data and whether the intended use of the data is appropriate? and
are the data of sufficient quality with respect to DQI goals and other performance criteria for their intended use?
An audit of data quality entails tracing data through their processing steps and duplicating intermediate calculations. A representative set of the data is traced in detail from raw data and instrument readouts through data transcription or transference through data manipulation (either manually or electronically by commercial or customized software) through data reduction to summary data, data calculations, and final reported data. The focus is on identifying a clear, logical connection between the steps. Particular attention is paid to the use of QC data in evaluating and reporting the data set. For a large project, a statistical approach may be necessary to determine a representative number of datasets to be examined. Often, however, the number of sets is limited by the budget of the audit. An audit of data quality may occur at different stages of the project. For example, it may be performed after field analysis but prior to off-site laboratory analyses. The products of the data quality assessment are a report detailing the results of custody tracing, a study of data transfer and intermediate calculations, a review of QA and QC data, a study of project incidents that resulted in lost data, and a review of study statistics. The audit report typically ends with conclusions about the quality of the data from the project and their fitness for their intended use. The scope and schedule (either by date or triggering event) for any audit of data quality should be set in the QAPP.

2.4.5	Qualitative and Quantitative Comparisons to Acceptance Criteria

The data now need to be assessed against the acceptance criteria already established.  For each DQI or parameter for which acceptance criteria has been established, the data are compared to determine how well they meet the criteria.  This is done by comparing the individual MQOs to the acceptance criteria.
Qualitative comparisons to acceptance criteria are, almost by definition, very subjective and involve the use of expert opinion. In many cases the expert opinion is that of the project manager or principal investigator, who may be subjected to unknown bias in evaluating how good the comparison is. The principal qualitative indicators are comparability and representativeness. For examples on different types of qualitative comparisons, help may be found in Chapter 3, Section 3.4, which discusses various forms of comparisons when considering integrating existing data into a project. Documentation of how the qualitative aspects of comparison of data to acceptance criteria have been met is essential.
Appendix B contains supporting information on DQIs for which MQOs can be set as acceptance criteria for including existing data in a project. The DQIs that may be able to be quantitatively assessed are:
precision;
bias;
accuracy (the combination of precision and bias);
completeness; and
sensitivity.
A typical example would be to determine if the accuracy of the measuring instrument (precision plus bias, the DQI chosen) is at most 10 ppb (the MQO for this DQI). The performance of the measuring instrument (actual data) would be used to determine if this MQO could be achieved.
For quantitative comparisons, graphical and statistical analyses can provide estimates of the strength of the comparison to criteria. The use of statistical analyses when possible is recommended to minimize evaluation bias. The extent of statistical analysis is directly related to the wealth of information available about the project; a poorly documented project being less useful than a well documented one. For this reason, a well documented QAPP should be developed to enable later researchers to evaluate the data for subsequent use. Reference to Data Quality Assessment: Statistical Methods for Practitioners, EPA QA/G-9S, (EPA 2006b), a guidance document on statistical methods written for non-statisticians, is recommended.
2.4.6	Interim Assessments of Data Quality
Data quality is formally assessed in the data review steps, which include data verification, data validation, and data usability. These steps generally don't take place until samples are collected and analyzed. However, there are many opportunities during a project to raise possible concerns with data quality. Possible indicators of data quality issues include (but are not limited to):
observed field sampling practices that potentially impact data;
an anomalous event in the laboratory (e.g., fire or chemical spill) that could potentially impact data;
extreme weather (e.g., stream samples collected through a layer of ice); and
an intuitive sense that something isn't right with how the project is proceeding.
The QAPP should include statements encouraging project team personnel to document any concerns they have over data quality throughout the course of the project. The mechanism for making their concerns known, and the individual responsible for determining appropriate action should be identified. Interim assessments may also be combined with the verification and validation activities for the project (see Section 2.5.1), and with any results from the comparisons to acceptance criteria (Section 2.4.5).
2.4.7	Evaluation of Unconventional Measurements
Sometimes there is a need to extend beyond standard analytical methods into unconventional methods. These can be thought of as any method for which there is not an already existing standard (e.g. EPA, ASTM, or Standard Methods) that has been widely accepted and validated. Unconventional methods should not be confused with modifications to an existing method.
Unconventional methods may include both sampling (e.g. passive sampling for indoor air testing using thin films) and testing (e.g. three dimensional gas chromatography) procedures. Typically they are developed to sample or measure an analyte that is new or for which a regulatory limit or threshold has changed. Many environmental forensic methodologies such as isotope ratio mass spectrometry,  are also considered unconventional because they are not considered to be typical for measuring environmental contaminants.