If the data to be used comes from an EPA source, then the method of data collection, the results, and limitations on use of the results should be available in the data's QAPP. Reference to the QAPP and its location should be made, and important extracts from that data QAPP included in the modeling QAPP.  If the data source is outside EPA attention should be given to the issues raised in Chapter 3.3 of this Handbook (Data Acquisition, Existing Data).

If new data are to be generated for use in modeling, a separate QAPP for the data will probably be needed. If existing data are to be used, the information in Chapter 3, Section 3.4 and 3.5 should be incorporated into the modeling QAPP.

The methods described in these elements should have been summarized earlier in Section 4.2.6. The purpose here is to provide detailed information on the methods. If the designated methods are well documented and are readily available to all project participants, citations are adequate; otherwise, detailed copies of the methods or SOPs should accompany the QAPP either in the text or as attachments.
4.3.1	Data Requirements for Model Input
For projects involving modeling, the data to be used in the model should meet specified objectives, as should the outputs of the model. The QAPP should define how the data quality acceptance criteria will be determined (for all information to be collected including information obtained from previous studies). The QAPP should also explain how the data acceptance criteria relate to the desired quality of model outputs.
<IP_ADDRESS> QAPP Requirements for Model Development or Modification
For model development, Annex B<IP_ADDRESS> indicates the QAPP must define:
   * the data needs/inputs/sources;
   * any boundary or initial condition specifications;
   * data reporting requirements and format for model inputs/outputs; and
   * the data quality objectives for model input and data used for model development.
The QAPP should document the quality and quantity of data necessary for model development, modification, use, or evaluation. DQOs and specified acceptable range of uncertainty for the model input data should be defined as they "provide the specifications for model quality and associated checks. Well-defined DQOs guide the design of monitoring plans and the model development process (e.g., calibration and verification). The DQOs provide guidance on how to state data needs when limiting decision errors (false positives or false negatives) relative to a given decision. The DQOs should include a statement about the acceptable level of total uncertainty that will still enable model results to be used for the intended purpose. Uncertainty describes the lack of knowledge about models, parameters, constants, data, and beliefs. Defining the ranges of acceptable uncertainty  either qualitatively or quantitatively  helps project planners generate "specifications" for QA planning and partially determines the appropriate boundary or initial condition specifications and complexity for [model development]." (EPA 2009b)
For developing or modifying models, any boundary or initial condition specifications should be defined. In applying a model or software, availability of data and knowledge of inputs needed is central. Data quality and quantity should be noted, along with procedures for excluding data.
<IP_ADDRESS> QAPP Requirements for Model Application
For model application, Annex B<IP_ADDRESS> gives the data requirements by focusing on data reporting requirements,  format for model inputs and outputs, and discussion of issues regarding information on how data will be acquired and used in the project:
the need and intended use for each type of data or information to be acquired;
selected source of the data and how it will be differentiated from other sources in the data management system;
how the data will be identified or acquired together with the basis for exclusion of competing datasets;
the method of determining the underlying quality and quantity of the data; and
the criteria established for determining whether the level of quality for a given set of data is acceptable for use on the project.
 <IP_ADDRESS> DQIs for all Types of Modeling Projects
Some of the key criteria for individual data values, or a set of data values, include an examination of their DQIs with respect to potential use in modeling projects:
Precision: Is the estimate of variability small enough to meet the uncertainty objectives of the modeling project?
Bias: Would any characteristics of the dataset directly impact the model output (e.g., unduly high or low process rates)? For example, was bias in result analysis documented? Is there sufficient information to estimate and correct bias? If using data to develop probabilistic distributions, are there adequate data in the upper and lower extremes of the tails to allow for unbiased probabilistic estimates?
Representativeness: Were the data collected from a population sufficiently similar to the population of interest and the model-specified population boundaries? Were the sampling and analytical methods used to generate the collected data acceptable to this project? Are there potentially confounding effects in the data (e.g., season, time of day, location, and scale incompatibilities) that need to be addressed so as to not unduly impact the model output?
Comparability: Are there concerns about comparing the data used in modeling to current conditions with respect to age or changes in modeling practices? Are there possible concerns that the structure of the population from where the data was taken could be different from current conditions?
Completeness: Are there any gaps in the data record that could cause concerns with the modeling output? To what extent was data interpolation or imputation necessary before the data were suitable for modeling purposes?
Sensitivity: Have the data been evaluated in a manner that permits perturbation analysis or estimates made as to the sensitivity of the model output to parameter changes? Is the system of qualifying or flagging data adequately documented to allow data from different sources to be used on the same project (e.g., distinguish actual measurements from estimated values and note differences in detection limits)?
The influence and relative importance of the DQIs depends on the purpose and nature of the modeling project and further information may be found in Appendix B of this guidance.
When input data are derived from existing models care has to be taken to ensure evaluation of all stages of model development: "Some assessments require linking multiple model frameworks such that the output from one model is used as input data to another model. For example, air quality modeling often links meteorological, emissions, and air chemistry/transport models. When employing linked models, the project team should evaluate each component model, as well as the full system of integrated models, at each stage of the model development and evaluation process" (EPA 2009b).
The importance of knowing the nature and characteristics of the input data is crucial to the interpretation of the project's results.
4.3.2	Data Management Tasks
The QAPP should discuss how data will be managed, how the path of data production in the field or laboratory will be traced for the entire project. This includes the following:
describe or reference the standard record-keeping procedures, and discuss the approach to be used for data storage and retrieval of electronic media (see Brilis, et al, 2004);
discuss the plan for detecting and correcting errors from data conversion (e.g. metric/English, units-to-units, and significant figures) as well as for preventing loss of data during reduction, reporting, and entry to forms, reports, databases;
discuss the plan for detecting and correcting errors from data migration to new hardware/software (including checks to verify integrity and non-corruption);
describe all data-handling equipment and procedures to process, compile, analyze, and interpret the model data, including any necessary computer hardware and software;
address any specific data-management performance requirements and describe the procedures that will be followed to demonstrate acceptability of the necessary hardware/software configuration; and
identify who is responsible for each data-management task together with frequency for performing these tasks.
 In addition to describing the data management tasks, the QAPP should also address how these tasks will be documented throughout the project, not already addressed in Section 	4.2.8.

4.4	ASSESSMENTS: MODEL EVALUATION ACTIONS (Check)
The third part of the Plan-Do-Check-Act paradigm assesses the model by considering the output, especially with respect to deviations from what was anticipated. The advice given here is drawn from the Standards Clause 7.8, Assessment of Data and Information, and the assessment parts of Annex B<IP_ADDRESS>, QAPP Requirements for Model Development/Modification, and B<IP_ADDRESS> QAPP Requirements for Model Application.
The elements in this section address project checks to see if the model will conform to the project's objectives, and to estimate the effect of any deviations. These activities may be performed throughout the project to inform mid-course corrections or at the completion of the project.
The roles and responsibilities of project personnel involved in assessing modeling activities should be defined in this section. This should include the procedures for corrective actions and to whom reports are made. To the extent that the modeling activity includes acquiring data or the use of existing data, the assessment methods outlined in the previous two chapters are appropriate. This section is guidance for the evaluation of the modeling activity.
4.4.1 	Model Evaluation Methods and Activities
Model evaluation is defined as the process used to generate information to determine whether a model and its analytical results are of a quality sufficient to serve as the basis for a decision. Model evaluation is conducted over the life cycle of the project, from development through application using the performance criteria established.  The effectiveness of evaluation depend on good documentation of both quantitative and qualitative comparisons to the performance criteria.
Model evaluation provides information to help answer four main questions (Beck 2000):
    How have the principles of documented science been addressed during model development?
    How is the choice of model supported by the quantity and quality of available data?
    How closely does the model approximate the real system of interest? and
    How does the model perform the specified task while meeting the objectives set by QA project planning?
These four factors address two aspects of model quality. The first factor focuses on the intrinsic mechanisms and generic properties of a model, regardless of the particular task to which it is applied. In contrast, the latter three factors are evaluated in the context of the use of a model within a specific set of conditions. Hence, it follows that model quality is an attribute that is meaningful only within the context of a specific model application. A model's quality to support a decision becomes known when information is available to assess these factors. Evaluation of a regulatory model builds upon the four factors described previously and should continue throughout the life of the model.
As recommended by the National Research Council (NRC 2007), the QAPP should contain a detailed model evaluation plan that includes (but is not limited to):
a description of the model and its intended uses;
a description of the model's relationship to data, including the data for inputs, calibration, and corroboration;
a description of how such data and other sources of information will be used to assess the ability of the model to meet its intended task;
a description of all the evaluation plan elements by using an outline or diagram that shows how the elements relate to the model's life cycle;
a description of the factors or events that might trigger the need for major model revisions or the circumstances that might prompt users to seek an alternative model (these can be fairly broad and qualitative); and
identification of the responsibilities, accountabilities, and resources needed to ensure implementation of the evaluation plan.
 The goal of model evaluation is to ensure model quality. Documentation of the plan for model evaluation provides assurance that the desired model quality will be achieved.
4.4.2 Evaluation of the Model
<IP_ADDRESS>	Evaluation of Model Development or Modification
Standard Clause B<IP_ADDRESS> states that the evaluation of a model should focus on documented science, testing strategies and procedures, and documentation of results. The QAPP should describe how the project team will ensure that the following issues will be evaluated and documented during model development or modification, as appropriate:
documentation of the science (including peer review of the theory and equations) underlying the model and appropriateness of model complexity for the problem;
testing strategies including individual module tests, integration tests, systems testing, acceptance testing, and beta testing, as applicable; data quality and quantity objectives to support the model choice;
the procedure for each test shall be provided and the process of confirming the test results included (i.e., evaluation criteria are to be identified during the initial stages of model development);
the use of sensitivity and uncertainty analysis with consistency of model structure with all relevant inputs described in the conceptual model;
implementation process for the software and any applicable development standards with identification of the model code and code verification, if completed;
internal quality checks applied during the development process (e.g., design and code verification, configuration control procedures, and change control procedures);
procedures for controlling, documenting, and archiving all significant changes to the software and hardware; and
requirements for project documentation (e.g., design document, source code, programmer's manual, or user guide).
<IP_ADDRESS>	Evaluation of Model Application
It is important to note the iterative nature of modeling: "Once a model has been accepted for use by decision-makers, it is applied to the problem that was identified in the first stages of the modeling process. Model application commonly involves a shift from the hindcasting (testing the model against past observed conditions) used in the model development and evaluation phases to forecasting (predicting a future change) in the application phase. This may involve a collaborative effort among modelers and program staff to devise management scenarios that represent different regulatory alternatives (EPA 2009b).
The needs (Standards Clause B<IP_ADDRESS>) of a QAPP center on:
   * the quality objectives to be used in the model evaluation;
   * the requirements for qualitative and quantitative model corroboration;
   * use of sensitivity and uncertainty analysis;
   * requirements for model post-auditing;
   * how the model evaluation will be documented;
   * the model parameters and method of estimation; and
   * the model calibration methods.
Some model applications may entail trial-and-error model simulations, where model inputs are changed iteratively until a desired or stable condition is achieved. When a sufficient quantity of data is available, it is common practice to use a portion of their existing data to establish the model, and then use the remaining portion to evaluate the performance of the model. The rationale on what proportion of data is used to establish the model and how that data is selected (randomly or serially across time for example) should be documented. When the model generates new data based on the use of existing data, the data fields (variables) and algorithms should be traceable back to their origin for this helps prevent the misinterpretation of the project results.
The importance of documentation can be seen in EPA 2009b: "Using a model in a proposed decision requires that the model application be transparently incorporated into the public process. This is accomplished by providing written documentation, in the QAPP, of the model's relevant characteristics in a style and format accessible to the interested public, and by sharing specific model files and data with external parties, such as technical consultants and university scientists, upon request."
Finally, it is important to note how the data used to evaluate the model compares to the actual scenario the model will be applied to. A model that has good evaluation properties for statistics such as averages may be quite poor for predicting extremes. The acceptance criteria established should be directly related to the eventual use of the model.
4.4.3	Model Output Sensitivity and Uncertainty Analysis
Sensitivity analysis (Standard Clause B<IP_ADDRESS>) is the study of how a model's response can be attributed to known changes in model inputs. Sensitivity analysis is recommended as the principal evaluation tool for identifying the most and least important sources of uncertainty in environmental models.
From EPA 2009b: "Uncertainty analysis investigates the lack of knowledge about a certain population or the real value of model parameters. Uncertainty can sometimes be reduced through further study and by collecting additional data. EPA guidance (e.g., EPA 1997b) distinguishes uncertainty analysis from methods used to account for variability in input data and model parameters. [V]ariability in model parameters and input data can be better characterized through further study but is usually not reducible." (EPA 2009b)
"Although sensitivity and uncertainty analyses are closely related, sensitivity is algorithm-specific with respect to model "variables" and uncertainty is parameter-specific. Sensitivity analysis assesses the "sensitivity" of the model to specific parameters and uncertainty analysis assesses the "uncertainty" associated with parameter values. Both types of analyses are important to understand the degree of confidence a user can place in the model results" (EPA 2009b)".
Advice on conducting uncertainty analysis and the establishment of acceptable levels of total error (uncertainty) may be found in EPA 2009b, and additional information on uncertainty analysis in Good Modelling Practice Handbook (DDPW 1999), and Uncertainty Analysis Guidance (EU 2007).
4.5	REVIEW, EVALUATION OF USABILITY: MODELING RESULTS USABILITY AND REPORTING REQUIREMENTS (Act)
The final part of the paradigm concerns the reconciliation of the model outputs to the overall project objectives. Peer review is an important part of modeling and must be comprehensively addressed. This section offers guidance on Standards Clause 7.9, the evaluation parts of Annex B<IP_ADDRESS>, QAPP Requirements for Model Development/Modification, B<IP_ADDRESS>, QAPP Requirements for Model Application, and B3.6, Review, Evaluation of Usability, and Reporting Requirements.
The QAPP should state how the results obtained from the project will be reconciled with the project objectives and acceptance criteria. This includes the proposed methods for analyzing the modeling results and for determining possible anomalies or limitations on the use for the intended purpose.
The QAPP should describe how departures from assumptions established in the planning phase of the modeling process will be assessed, how anomalies will be resolved, and how limitations on the use of the data from anomalies and departure from assumptions will be reported to decision makers.
4.5.1	Peer Review
For all models, peer review (Standard Clause B<IP_ADDRESS>) is an important aspect for acceptance of both model development/modification and application as it acts as an independent form of QA to ensure the validity of results. "Peer review provides the main mechanism for independent evaluation and review of environmental models. Its purpose is two-fold:
to evaluate whether the assumptions, methods, and conclusions derived from environmental models are based on documented scientific principles; and
to check the scientific appropriateness of a model for informing a specific regulatory decision (the latter objective is particularly important for secondary applications of existing models).
Information from peer reviews is also helpful for choosing among multiple competing models for a specific regulatory application. Finally, peer review is useful to identify the limitations of existing models. Peer review is not a mechanism to comment on the regulatory decisions or policies that are informed by models (EPA 2006c). Peer review charge questions and corresponding records for peer reviewers to answer those questions should be documented during the planning stage. For example, peer reviews may focus on whether a model meets the objectives or specifications that were established for the project.
All models that inform significant regulatory decisions are candidates for peer review for several reasons:
model results will be used as a basis for regulatory or policy/guidance decision making;
these decisions likely involve significant investment of Agency resources; and
these decisions may have inter-Agency or cross-agency implications/applicability.
It is recommended that "a new model should be scientifically peer-reviewed prior to its first application; for subsequent applications, the program manager should consider the scientific/technical complexity and/or the novelty of the particular circumstances to determine whether additional peer review is needed (EPA 1993)." (EPA 2009b)
"Models used for secondary applications (existing EPA models or proprietary models) will generally undergo a different type of evaluation than those developed with a specific regulatory information need in mind. Specifically, these reviews may deal more with uncertainty about the appropriate application of a model to a specific set of conditions than with the science underlying the model framework." (EPA 2009b)
The following aspects of a model should be peer-reviewed to establish scientific credibility (EPA 1993):
appropriateness of input data;
appropriateness of boundary condition specifications;
documentation of inputs and assumptions;
applicability and appropriateness of selected parameter values;
evaluation of model uncertainty;
documentation and justification for adjusting model inputs to improve model performance (calibration);
model application with respect to the range of its validity; and
supporting empirical data that strengthen or contradict the conclusions that are based on model results.
To be most effective and maximize its value, external peer review should begin as early in the model development phase as possible. The importance of using independent model review is extremely important and the decision whether to utilize the Agency's Peer Review system made as soon as possible (EPA 2006c).
4.5.2	Description of Model Documentation
The importance of documentation can be seen from EPA 2009b: "Documentation enables decision makers and other model users to understand the process by which a model was developed and used. During model development and use, many choices are made and options selected that may bias the model results. Documenting this process and its limitations and uncertainties is essential to increase the utility and acceptability of the model outcomes. Modelers and project teams should document all relevant information about the model to the extent practicable, particularly when a controversial decision is involved. In legal proceedings, the quality and thoroughness of the model's written documentation and the Agency's responses to peer review and public comments on the model can affect the outcome of the legal challenge."
The QAPP should describe how the modeling process, the model framework, the model, the sensitivity and uncertainty analyses, and the model outputs will be documented. The documentation should include a clear explanation of the model's relationship to the scenario of the particular application. This explanation should describe the limitations of the available information when applied to other scenarios. The evaluation, verification, and validation performed on the model should be documented. This documentation should include a clear explanation of the model's relationship to the final goal of the project or task. Disclosure about the state of science used in a model and future plans to update the model can help establish a record of reasoned, evidence-based application to inform decisions.
It is also advisable to include in the QAPP any particular memory requirements, necessity for specific software for model application, and any issues with model portability.
4.5.3	Specifications for Model Maintenance and User Support
The project team should determine and document in the QAPP, during the planning stage, the expected maintenance and user support needed throughout the life cycle of the model.
4.5.4	Reports to Management
The QAPP should identify the frequency and distribution of reports issued to inform management (EPA or otherwise) of the project status. QAPP authors may choose to identify the preparer and the recipients of the reports, and any specific actions recipients are expected to take as a result of the reports.
Periodic QA management reports are intended to ensure that managers and stakeholders are kept informed of project status and results of all QA assessments. Efficient communication of project status and problems will allow project managers to implement timely and effective corrective actions so that data produced for the project can meet the DQOs.
The QAPP should describe the content of each QA management report that will be generated for the project, including an evaluation of measurement error as determined from the assessments. Assessment checklists, reports, requests for corrective action letters, and the corrective response letters should be included as attachments or referenced in the QA management reports.
The following issues may be included in QA management reports:
Graphs, charts, diagrams helping interpreting results;
conformance of project activities to QAPP requirements and procedures;
status of project and schedule delays;
deviations from the approved QAPP and approved amendments to the QAPP;
results of interim model evaluation activities and corrective actions;
model sensitivity and uncertainty analyses; and
limitations on the use of model output.
Although these issues listed may be addressed in QA management reports, they should also be included in the QA/QC section of the final project report. The final project report should, at a minimum, give a reconciliation of project data with project objectives, data summary (including tables, charts, and graphs), a summary of major problems encountered and their resolution, and any additional data quality concerns together with conclusions and recommendations.

GLOSSARY
Several definitions given in CIO Policy 2106.0, CIO Procedure 2106-P-01.0, CIO Standard 2106-S-01, and CIO Standard 2106-S-02 are repeated below for added clarity and utility to this Guidance.

Approved: The documented determination that the proposed quality document is suitable for the intended purpose and meets the requirements is specified in the applicable Quality Standard.

Assessment: The evaluation process used to measure the performance or effectiveness of a system and its elements. As used here, assessment is an all-inclusive term used to denote any of the following: audit, performance evaluation, management review, peer review, inspection, or surveillance.

Conceptual Model: This describes the scientific and engineering process under investigation. It is used as a tool for organizing information about the current state of knowledge and understanding of the project, as well as for documenting key theoretical and practical assumptions underlying the data and information collection.

Corrective Action: An action to eliminate the cause of a nonconformity or undesirable situation and to prevent recurrence.

Data Standard: A documented consensus-based agreement on the format and definition of data.

Data Usability: The process of determining and ensuring that the quality of the data produced meets the intended use of the data.

Document: Recorded information regardless of physical form or characteristics including individual records or items of non-record materials.

Environmental Data: Any data or information pertaining to the environment that describe measured outputs from processes; environmental conditions in a specific location; ecological effects and consequences; health effects and consequences; biological, chemical, and radiological conditions; or the performance of environmental technology. Environmental data include information collected directly from measurements, produced from models, and compiled from other sources such as databases, information systems, literature, or the internet.

Environmental Data Collection: The process of acquiring or gathering environmental data through various means including, but not limited to, sampling and analysis activities, retrieval from information systems and literature sources, and receipt from EPA partners and the regulated community.

Environmental Data Operations: The work performed to collect, produce, or use environmental data.

Environmental Data Production: The process of generating environmental data through various means including, but not limited to, the use of measurement instrumentation, information technology, computer models, and data analysis tools (e.g., statistics, risk assessment methods).

Equivalent Document: A set of documents that contains all the information and management controls (signatures) as the required documents used in the Standard.

Extramural Agreement: Legal agreement between EPA and a non-EPA organization for the acquisition of items or services by EPA or financial assistance to a non-EPA organization. Such agreements include acquisition agreements (e.g., contracts, work assignments, delivery orders, task orders), assistance agreements (e.g., cooperative agreements, research grants, state and local grants), and EPA-funded IAs with other governmental entities.

Graded Approach: The process of basing the level of application of managerial controls applied to an item or work according to the intended use of the results and the degree of confidence needed in the quality of the results.

Guidance: A non-mandatory compilation of advice, examples, best practices, or past experience. Guidance may supplement procedures.

Handbook: A non-mandatory compilation of advice, examples, best practices, or past experiences, may be revised according to the issuing Office's Peer Review Policy.
Information: For purposes of this Standard, information means any communication or representation of knowledge such as facts or data, in any medium or form, including, but not limited to, textual, numerical, graphic, cartographic, narrative, or audiovisual forms.

Information System: An organized collection, storage, and presentation system of data for decision making, progress reporting, and for planning and evaluation of programs. It can be either manual or computerized, or a combination of both.

Information Technology: The study, design, development, implementation, support, or management of computer-based information systems, particularly software applications and computer hardware.

Informative: Non-mandatory advice on how to achieve a goal or meet a requirement.
Integrity (Information): The assurance that the information is protected from unauthorized access or change and is not compromised through corruption or falsification.

Life Cycle: The life span of a product or service from its initial planning and development, to its use and maintenance, and to its final closure or disposal.

Management Controls: A system of management functions to enable managers to determine that the operations of a program or organization satisfy predetermined goals and objectives, that performance is in line with standards and specifications, and to implement any remedial actions needed to ensure that human and other resources are being used in the most effective and efficient way possible in achieving the organization's mission.

Management System: A system to establish policy and objectives and to achieve those objectives (ISO 9001). A management system may describe the policies, objectives, principles, authority, responsibilities, accountability, and implementation plan of an organization for conducting work and producing products and services. Management systems include ISO 9001 on quality management, ISO 14001 on environmental management, and OHSAS 18000 on occupational health and safety.

Metadata: The information about data required to facilitate its use, understanding, and management. For example, metadata should answer questions about data such as why they were collected, how they were collected and by whom, what was done to the data, what they were used for, what were their limitations, and what were the acceptance criteria.

Model: A simplification of reality that is constructed to gain insights into select attributes of a physical, biological, economic, or social system. A formal representation of the behavior of system processes, often in mathematical or statistical terms. The basis can also be physical or conceptual.

Normative: A mandatory set of rules that must be followed.

Organization: A company, corporation, firm, enterprise, or institution, or part thereof, whether incorporated or not, public or private, that has its own functions and administration. In the context of this Standard, an EPA organization may be an Office, Region, National Research Center or Laboratory, or a subunit such as a division, branch, section, or team.

Objectivity (Information): The assurance that information is presented in an accurate, clear, complete, and unbiased manner, and, as a matter of substance, is accurate, reliable, and unbiased.

Peer Review: A review conducted by qualified individuals or organizations who are independent of those who performed the work, but are collectively equivalent in technical expertise to those who performed the original work.

Policy: A high-level statement about an Agency requirement designed to influence and determine decisions, actions, and other matters. It is usually driven by statute, executive order, the mandate of an oversight agency or Congress, or the head of the organization.

Practice:
   * an expected procedure or way of doing of something.
   * the carrying out or exercise of an expected procedure or way of doing of something.

Procedure: The required steps, course of action, or processes needed to accomplish or satisfy a policy.

Process: A set of interrelated resources and activities which transforms inputs into outputs. Examples of processes include analysis, design, data collection, operation, fabrication, and calculation.

Product: The intended result or final output of an activity or process that is disseminated or distributed among EPA organizations or outside of EPA.

Quality: The totality of features and characteristics of a product or service that bear on its ability to meet the stated or implied needs and expectations of the user.

Quality Assurance (QA): A management or oversight function that deals with setting policy and running an administrative system of management controls that cover planning, implementation, review and maintenance to ensure products and services are meeting their intended use.

Quality Assurance Manager (QAM): The individual designated as the principal manager within the organization having management oversight and responsibilities for planning, documenting, coordinating, and assessing the effectiveness of the QMS for the organization. Note: Other personnel having QA or QC duties may be referred to as QA Officer and QA Coordinator.

Quality Assurance Project Plan (QAPP): A document describing in comprehensive detail the necessary QA, QC, and other technical activities that must be implemented to ensure that the results of the work performed will satisfy the stated performance objectives and criteria.

Quality Control (QC): The overall system of technical activities that measure the attributes and performance of a process, item, or service against defined standards to verify that they meet the stated requirements established by the customer; operational techniques and activities that are used to fulfill requirements for quality.

Quality Management: That aspect of the overall quality management system of the organization that determines and implements the quality policy. Quality management typically includes strategic planning, allocation of resources, and other systematic activities (e.g., planning, implementation, and assessment) pertaining to the application of quality practices to the organization's programs.

Quality Management Plan (QMP): A formal document or manual that describes the QMS in terms of the organizational structure, functional responsibilities of management and staff, lines of authority, and required interfaces for those planning, implementing, and assessing all activities conducted.

Quality Management System (QMS): A structured and documented management system describing the policies, objectives, principles, organizational authority, responsibilities, accountability, and implementation plan of an organization for ensuring quality in its work processes, products (items), and services. The QMS provides the framework for planning, implementing, documenting, and assessing work performed by the organization and for carrying out required QA and QC activities.

Quality Program: The totality of management controls, processes, and documentation in planning, implementation, and assessment of applying quality to the creation of Agency products and services.

Record: A document stating results retrieved or providing evidence of activities (ISO 9000:2005). NOTE: A federal record is an information resource in any format that is needed to describe Agency activities (44 U.S.C. § 3301).

Required Element: An element whose presence is obligatory in a Standard.

Requirement: An expression of the content of a Standard conveying a criterion to be fulfilled if compliance is to be claimed and from which no deviation is permitted.

Service: A discrete function that performs operations and returns a set of results to an external requester.

Specification: A document stating requirements and which refers to or includes drawings or other relevant documents.  Specifications should indicate the means and the criteria for determining conformance.

Standard: An accepted, consensus-based specification which defines systems, processes, methodologies, or practices. It provides a basis for assuring consistent and acceptable minimum levels of quality, performance, safety, and reliability. Standards usually are included in or accompany procedures.

Standard Operating Procedure (SOP): A written document that details the method for an operation, analysis, or action with thoroughly prescribed techniques and steps, and that is officially approved as the method for performing certain routine or repetitive tasks.

Usability Assessment: The evaluation of data based upon the results of data validation and verification for the decision(s) being made. Reviewers assess whether the process execution and resulting data meet quality objectives based on the criteria given in the QAPP.

User: An organization, group, or individual that utilizes the results from environmental programs, or the customer for whom the results or products were collected or created.

Utility (Information): The assurance that information is useful for its intended purpose.

Validation (Information): The confirmation by examination and provision of objective evidence that the particular requirement for which the information is intended are fulfilled; the process of determining whether the specifications were appropriate and that the verified results will meet the data user's needs.

Verification (Information): The confirmation by examination and provision of objective evidence that validated information fulfills specified requirements; the process of checking whether the information met the project's specifications as part of an agreement or contract.

REFERENCES
40 CFR 3, Code of Federal Regulations (annual edition). Title 40: Protection of Environment. Part 3: Special rules governing certain information (from regulatory programs). Washington, D.C.
http://ecfr.gpoaccess.gov/

40 CFR 51 Appendix W, Code of Federal Regulations (annual edition). Title 40: Protection of Environment. Part 51: Requirements for Preparation, Adoption, and Submittal of Implementation Plans. Washington, D.C.
http://ecfr.gpoaccess.gov/

40 CFR 403.12(b), Code of Federal Regulations (annual edition). Title 40: Protection of Environment. Subchapter N: Effluent Guidelines and Standards. Part 403: General Pretreatment Regulations for Existing and New Sources of Pollution. Washington, D.C.
http://ecfr.gpoaccess.gov/

40 CFR 763, Code of Federal Regulations (annual edition). Title 40: Protection of Environment. Part 763, Subpart E: Asbestos-Containing Material in Schools. Washington, D.C.
http://ecfr.gpoaccess.gov/

49 CFR 171, Code of Federal Regulations (annual edition). Title 49: Transportation.  Part 171: Definition of Hazardous Material. Washington, D.C.
http://ecfr.gpoaccess.gov/

ANSI 2004, American National Standards Institute and American Society for Quality, Quality Systems for Environmental Data and Technology Programs: Requirements with Guidance for Use (ANSI/ASQ E4-2004).
http://asq.org/quality-press/display

ASTM 1999, ASTM Standard D4840-99, 1999, Standard Guide for Sampling Chain-of-Custody Procedures, ASTM International, West Conshohocken, PA.
http://astm.org

Beck, B, 2002 Model Evaluation and Performance. In: Encyclopedia of Environmetrics. A.H. El-Shaarawi and W.W. Piegorsch, eds., John Wiley & Son, New York.

Box, G. E. P., W. G. Hunter, and J. S. Hunter, 1978. Statistics for Experimenters. John Wiley.
New York.

Brilis, G. M, J. C. Worthington, A. D. Wait, 2000. Quality Science in the Courtroom: U.S. EPA Data Quality and Peer Review Policies and Procedures compared to the Daubert Factors, Environmental Forensics, vol. 1 p197-203.

Brilis, G. M., J. Lyon, J.C. Worthington, and R. Lysakowski, 2004. Auditing Electronic Record-keeping Systems used in Scientific Research and Development, Quality Assurance, vol. 11 p5-24.

Cochran, W. G., and D. R. Cox, 1957. Experimental Designs. John Wiley. New York.
Currie, L. A., 1995. Nomenclature in Evaluation of Analytical Methods Including Detection and Quantification Capabilities. Pure and Applied Chemistry, Vol. 67, No. 10, pp. 1699 - 1723.

DDPW 1999, Good Modelling Practice Handbook, Dutch Department of Public Works.
http://harmoniqua.wau.nl/public/Reports/Existing%20Guidelines/GMP111.pdf

EPA 1997a, Procedures for Preparing Emission Factor Documents (EPA-454/R-95-015)
U.S. Environmental Protection Agency.
http://www.epa.gov/ttnchie1/efdocs/procedur.pdf

EPA 1997b, Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM). NUREG-1575, EPA/402/R-97/016, U.S. Environmental Protection Agency.
www.marssim.com

EPA 1999, Contract Laboratory Program, National Functional Guidelines for Organic Data Review. U.S. Environmental Protection Agency.
www.epa.gov/superfund/programs/clp/guidance.htm

EPA 2000, Guidance on Technical Audits and Related Assessments for Environmental Data Operations, EPA QA/G-7, January 2000, U.S. Environmental Protection Agency.
http://www.epa.gov/quality

EPA 2001a, Requirements for Quality Assurance Project Plans, EPA QA/R-5. EPA/240/B-01/003, U.S. Environmental Protection Agency.
http://www.epa.gov/quality

EPA 2001b, ORD Policies and Procedures Manual, Chapter 13 Quality Assurance, U.S. Environmental Protection Agency.
http://dcordhqapps1.eoa.gov/orma/policies.nsf/webpolicy?OpenView

EPA 2002a, Guidance for Quality Assurance Project Plans, EPA QA/G-5. EPA/240/R-02/009, U.S. Environmental Protection Agency.
http://www.epa.gov/quality

EPA 2002b, Guidance for Choosing a Sampling Design for Environmental Data Collection, EPA QA/G-5S, U.S. Environmental Protection Agency.
http://www.epa.gov/quality

EPA 2002c, Guidance for Quality Assurance Project Plans for Modeling, EPA QA/G-5M U.S. Environmental Protection Agency.
http://www.epa.gov/quality

EPA 2005a, Uniform Federal Policy for Quality Assurance Project Plans, Intergovernmental Data Quality Task Force (EPA-505-B-04-900A) U.S. Environmental Protection Agency. http://www.epa.gov/fedfac/documents/qualityassurance.htm

EPA 2005b, Workbook for Uniform Federal Policy for Quality Assurance Project Plans, Intergovernmental Data Quality Task Force (EPA-505-B-04-900C), U.S. Environmental Protection Agency.
http://www.epa.gov/fedfac/documents/qualityassurance.htm
EPA 2005c, OSWER Directive 9272.0-17 Implementation of the Uniform Federal Policy for Quality Assurance Project Plans (UFP-QAPP) at Federal Facility Hazardous Waste Sites, June 7, 2005, U.S. Environmental Protection Agency.
www.epa.gov/fedfac/pdf/oswer_qapp_directive.pdf

EPA 2005d, OSWER Directive 9272.0-20 Applicability of the Uniform Federal Policy for Quality Assurance Project Plans (EPA 505-04-900A, December 21, 2005, U.S. Environmental Protection Agency.
www.epa.gov/fedfac/pdf/oswer_qapp_directive.pdf

EPA 2006a, Guidance on Systematic Planning Using the Data Quality Objectives Process, EPA QA/G-4, U.S. Environmental Protection Agency.
http://www.epa.gov/quality

EPA 2006b, Data Quality Assessment: Statistical Methods for Practitioners, EPA QA/G-9S, U.S. Environmental Protection Agency.
http://www.epa.gov/quality

EPA 2006c, Peer Review Handbook, U.S. Environmental Protection Agency. http://www.epa.gov/peerreview/pdfs/prhandbk.pdf

EPA 2006d, Pre-Dissemination Review Guidelines, U.S. Environmental Protection Agency.
http://intranet.epa.gov/quality/informationguidelines/pdf/pdr-guidelines.pdf

EPA 2008a, Quality Policy, CIO Policy 2106.0, Oct. 20, 2008 ("Quality Policy"), U.S. Environmental Protection Agency.
http://www.epa.gov/irmpoli8/policies/21060.pdf

EPA 2008b, Procedure for Quality Policy, CIO 2106-P-01.0, Oct. 20, 2008 ("Quality Procedure"), U.S. Environmental Protection Agency.
http://www.epa.gov/irmpoli8/policies/2106p01.pdf

EPA 2008c, Guidance on Environmental Data Verification and Data Validation, EPA QA/G-8, U.S. Environmental Protection Agency.
http://www.epa.gov/quality

EPA 2009a, EPA Records Management Policy, U.S. Environmental Protection Agency.
http://epa.gov/records

EPA 2009b, Guidance on the Development, Evaluation, and Application of Environmental Models, Council for Regulatory Environmental Modeling (EPA/100/K-09), U.S. Environmental Protection Agency.
http://www.epa.gov/crem/library/cred_guidance_0309.pdf.

EPA 2009c, Guidance for Labeling Externally Validated Laboratory Analytical Data for Superfund Use, OSWER No. 9200.1-85 (EPA 540-R-08-005), U.S. Environmental Protection Agency.
www.epa.gov/superfund/policy/pdfs/EPA-540-R-08-005.pdf

EPA 2010, Technical Basis for the Lowest Concentration Minimum Reporting Level (LCMRL) Calculator, Office of Water (EPA 815-R-11-001), U.S. Environmental Protection Agency.
http://water.epa.gov/drink

EPA 2012, EPA Regional Guidance for the Preparation of Quality Assurance Program Plans (R9/3.1), U.S. Environmental Protection Agency.
http://www.epa.gov/region09/qa/pdfs/mngmt-plan_guidance_2012.pdf

EPA 2013a, Quality Standard for Environmental Data Collection, Production, and Use by EPA Organizations, CIO 2106-S-01.0, In Preparation ("Internal Standard"), U.S. Environmental Protection Agency,
http://www.epa.gov/quality

 EPA 2013b, Quality Standard for Environmental Data Collection, Production, and Use by Non-EPA (External) Organizations, CIO 2106-S-02.0, In Preparation ("External Standard"), U.S. Environmental Protection Agency.
http://www.epa.gov/quality

EPA 2013c, Guidance on Quality Management Plans, CIO Guidance QMP Handbook, In Preparation ("QMP Guidance"), U.S. Environmental Protection Agency.
http://www.epa.gov/quality

EU 2007, Uncertainty Analysis Guidance, European Commission 2007
http://www.harmoni-ca.info/toolbox/Model_Uncertainty/index.php

Hubaux, A. and G. Vos, 1970, Decision and Detection Limits for Linear Calibration Curves, Analytical Chemistry. Vol. 42, No. 8, pp. 849-855.

IQG 2001. Information Quality Act Section 515(a) of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554; H.R. 5658).
www.nrc.gov/public-involve/info-quality/pl106-554.pdf

NRC 2007, Models in Environmental Decision Making, Committee on Models in the Regulatory Decision Process, Board on Environmental Studies and Toxicology, Division on Earth and Life Studies.  National Academic Press, Washington D.C.

Manno, J., R.Smardon, J.V. DePinto, E.T. Cloyd, and S. Del Grando 2008. The Use of Models in Great Lakes Decision Making: An Interdisciplinary Synthesis. Randolph G. Pack Environmental Institute, College of Environmental Science and Forestry.  Occasional Paper 16.

Skoog, D. A. 1985, Principles of Instrumental Analysis, third edition. Saunders College Publishing. Philadelphia, Pennsylvania.

Taylor, J. K. 1987, Quality Assurance of Chemical Measurements. Lewis Publishers. Chelsea, Michigan.

Ziliak, S.T. 2011, Matrixx v. Siracusano and Student v. Fisher, Significance, vol 8, p 131-134.

APPENDIX A
CHECKLIST OF QAPP ELEMENTS
QAPPs should, at a minimum, address all elements detailed in this Handbook as it supports the Internal and External Standards. In some cases, certain elements will not be appropriate for a particular project. Elements that do not apply can be addressed with a simple statement of why the information is not relevant or with a cross reference to another approved document in which the information appears.
 This checklist shows the section number and title for each element with columns for:
describing why the element is important;
indicating what might suffice for the element in a simple project;
listing what additional information might be necessary for the element in a more complex project;
noting whether the element was discussed in the QAPP;
noting where in the QAPP it was discussed; and
explaining why an element was not included.
 This checklist is meant to be useful either as a tool for the developer of the QAPP to ensure all elements are included, or as a tool for reviewers who are checking to be sure that all elements are appropriately included.
 This checklist is organized by the four primary steps in the project life cycle: Plan-Do-Check-Act and presented with steps 1 and 3 shaded, steps 2 and 4 un-shaded for clarity. The elements for Chapter 2 are presented first, then Chapter 3, and finally Chapter 4. To facilitate use, the generation of new data (Chapter 2) is shown in black, the use of existing data (Chapter 3) in green, and the application to models (Chapter 4) in blue.
 In order to help discuss the degree to which the QAAP elements need to be addressed, a simple case is contrasted to a more complex case; an example of the graded approach being used. These two cases are, by nature of being examples, fairly simplistic as the detail of documentation depends on the importance of the project, an exploratory study needing significantly less documentation than one involving a proposed regulation.
                                Section Number
                                     Title
                                 Why Important
                                  Simple Case
                                 Complex Case
2.2.1

Title, Version, and Approval/ Sign-off
To meet the requirements of Clause 7.5.1 of the Standards for an approved QAPP prior to initiating work
Single page with title and required signatures
Revisions history tracking all changes made to the QAPP over time
2.2.2

Document Format and Table of Contents
To make it easy for the project team and reviewers to find the information they need
Table of contents and page numbers
Document control format
2.2.3

Distribution List
To be sure that everyone who needs it has access to, and awareness of, the latest version of the QAPP
Included on the title page or in the project organization section
A separate section with contact information and indication of under what conditions each individual needs to be made aware of revisions to the QAPP
2.2.4

Project Organization and Schedule
To ensure all key project personnel are aware of their responsibilities and the timeframe for completion
Names of key project personnel and period of performance for project
Organizational chart of all key project roles with names and Gantt or similar tracking chart of schedule
2.2.5

Project Background, Overview, and Intended Use of Data
To provide sufficient information as a foundation for the project goals and to clarify the expected uses of the data
General overview and statement of how data and models will be used in this project
Detailed project history, references to other background documentation, regulatory basis for data use
2.2.6

Data/Project Quality Objectives and Measurement Performance Criteria
To ensure that the data or models collected or used in the project are of sufficient quality to support project decisions
Project goals, data required to meet those goals, statement of what criteria the data must meet to be acceptable for project use
7-step DQOs, MQOs for all DQIs, statement of tolerable error ranges for project decisions
2.2.7

Special Training Requirements and Certification
To be sure that the project has qualified personnel to perform all necessary functions
List of specialized training required in the project
Matrix of special training needs, personnel who are qualified, and timeframe of qualifications
2.2.8

Documentation and Records Requirements
To document project process, outcomes, and supporting quality information, to meet clause 7.10 of the Standards
Statement declaring where project documentation will be stored, including, but not limited to: QAPP, data packages, assessment records, interim and final reports
Detailed documentation and storage requirements per regulation or contractual obligation. These may be indicated in the project schedule as well.
2.3.1
Sample Collection Procedure, Experimental Design, and Sampling Tasks
To define the way the data will be collected and state whether it is a probability-based or judgmental design
Define population of interest, sample design and rationale for that design, number/time/location and types of samples to be collected
As appropriate, discuss multistage sampling, randomization procedures, stratification rationale, or any details that support the selection of samples
2.3.2
Sampling Procedures and Requirements
To ensure that appropriate sampling methods are selected to meet project needs and produce data of sufficient quality for project decision making
Include or reference SOPs for sample collection, on-site preservation, and cleaning and decontamination of sampling equipment
Include media-specific sample collection procedures, safety issues involved in sample collection, sample container descriptions, etc.
2.3.3
Sample Custody Procedures, and Documentation
To ensure sample authenticity and to avoid sample loss
Describe sample documentation and handling procedures and include a sample chain-of-custody form
List US DOT regulations and how they will be met, explain sample numbering schema, define procedures for introducing PE samples
2.3.4
Analytical Methods Requirements and task Descriptions
To be sure appropriate measurement methods exist to achieve project quality objectives
Describe measurement techniques, such as counting, visual discrimination, or analytical methods
Details of analytes to be measured, analytical method, SOPs, data-package requirements, analytical laboratory contact information, and MQOs
2.3.5
Quality Control Requirements
To provide confidence that the project data are of suitable quality to be used for their intended purpose
Number and types of QC samples to address sources of measurement error
Table of sources of potential measurement error and the corresponding QC samples that will be used to address those uncertainties
2.3.6
Instrument/ Equipment Testing Calibration and Maintenance Requirements, Supplies and Consumables
To avoid poor instrument performance that could impact project data quality
Document roles and responsibilities and procedures for sample collection and measurement instrument inspection, calibration, and maintenance
Include instrument measurement uncertainty, traceability of calibration equipment, and project-specific schedule for inspection, calibration, and maintenance
2.3.7

Data Management Requirements
To ensure data integrity
Describe data handling from generation, to use, to final storage; include copies of data entry forms, reports and description of databases; specify any special requirements for data such as CBI or hardware/software
Include SOPs for data management, specify project personnel's roles and responsibilities regarding data management, describe how metadata will be gathered and stored with project data
2.4.1
Technical Systems Assessments
To determine if the field sampling team, laboratory, or analysts have sufficient technical capabilities to generate data of appropriate quality for project use
If project-specific needs are minimal, noting accreditation to an appropriate technical system standard may be sufficient ; note that if any project-specific needs are more stringent than the standards, then project-specific assessments should be conducted
Describe plans for assessments of field sampling, on-site and off-site laboratory analyses
2.4.2
Performance Audits of Measurement and Analytical Systems
To directly test that the measurement performance is adequate for project purposes

Document plans and acceptance criteria for split samples and PE samples, if appropriate
Specify schedule, analyte selection, traceability, spiking levels, and sample types (matrix, direct, single-blind, or double-blind), and include the sources and estimated costs for these samples
2.4.3
Surveillance of Operations
To verify that project activities are conducted as planned
State when surveillance will occur (under what conditions or by set timeframe), how it will be conducted, how feedback will be provided and incorporated, and if surveillance leads to a temporary or permanent work stoppage, address how that will be handled
Provide details of the triggering events for surveillance assessments, an SOP for their conduct, a list of who will be notified of any non-conformances observed during the assessments, and how the surveillance assessments will be documented
2.4.4
Audits of Data Quality
To be sure that QC data are used to support data quality, and to determine if the data are replicable
Define the schedule (based on timeframe or triggering events) and scope for audits of data quality
Provide an SOP for conducting audits of data quality
2.4.5

Interim Assessments of Data Quality
To be sure data collection is proceeding according to plan
Include statement encouraging project team members to alert management if they sense anything isn't going quite right
Indicate points during the project timeline at which interim assessments of data quality will be conducted; state who will be responsible for their conduct and how they will be documented
2.4.6

Qualitative and Quantitative Comparisons to Acceptance Criteria
To determine if the project quality objectives are met
State how comparisons to qualitative and quantitative MQOs will be evaluated; describe other criteria (e.g., publication in a peer-reviewed journal) that might be important for the project
Set specific comparison methodology for each DQI, such that there will be a definitive "pass/fail" for each; describe contingencies in case of "fail"
2.4.7

Evaluation of Unconventional Measurements
To be sure that any unusual laboratory methods are fully documented and understood
If no unconventional measurement methods will be/were used, just state that here
Describe the reason for using unconventional measurement methods, provide SOPs for their implementation and how efficacy will be assessed
2.4.8

Evaluation of Unconventional Monitoring Projects
To be sure that any unusual sample collection methods are fully documented and understood
If no unconventional sample collection methods will be/were used, just state that here
Describe the reason for using unconventional sample collection methods, provide SOPs for their implementation and how efficacy will be assessed
2.5.1

Data Verification and Validation Targets and Methods
To determine if data have met the project quality objectives
Provide a standard data verification and validation method or procedure that has been reviewed to ensure it meets project needs
Develop project-specific verification and validation schema that incorporate QC data and metadata
2.5.2

Quantitative and Qualitative Evaluations of Usability
To ensure that project decisions are supported by data of sufficient quality for project needs
State who will be part of the evaluation of data usability, how it will be conducted and documented
Define contingencies for any issues that may be identified during the evaluation of data usability
2.5.3

Potential Limitations on Data Interpretation
To ensure that the data are not stretched beyond their appropriate use
Describe what actions will be taken if project data are deemed unusable for their intended project purpose
Describe how any limitations will be documented and stored in the project metadata
2.5.4

Reconciliation with Project Requirements
To determine if project requirements have been met
Clearly state how the data verification, validation, and usability results will be used to determine if project requirements have been met; describe how the five steps of the DQA process will be conducted
State specific exceptions to statistical signficance that will be overturned in favor of "practical signficance" should they occur; define the steps to be taken for contingencies if data do not support requirements
2.5.5

Reports to Management
To document project outcomes
Define schedule and content for reports to management
Provide report templates
3.2.1

Title, Version, and Approval/ Sign-off
To meet the requirements of Clause 7.5.1 of the Standards for an approved QAPP prior to initiating work
Single page with title and required signatures
Revisions history tracking all changes made to the QAPP over time
3.2.2

Document Format and Table of Contents
To make it easy for the project team and reviewers to find the information they need
Table of contents and page numbers
Document control format
3.2.3

Distribution List
To be sure that everyone who needs it has access to, and awareness of, the latest version of the QAPP
Included on the title page or in the project organization section
A separate section with contact information and indication of under what conditions each individual needs to be made aware of revisions to the QAPP
3.2.4

Project Organization and Schedule
To ensure all key project personnel are aware of their responsibilities and the timeframe for completion
Names of key project personnel and period of performance for project
Organizational chart of all key project roles with names and Gantt or similar tracking chart of schedule
3.2.5

Project Background, Overview, and Intended Use of Data
To provide sufficient information as a foundation for project goals and to clarify expected uses of the data
General overview and statement of how data and models will be used in this project
Detailed project history, references to other background documentation, regulatory basis for data use
3.2.6

Data/Project Quality Objectives and Measurement Performance Criteria
To ensure that the data or models collected or used in the project are of sufficient quality to support project decisions
Project goals, data required to meet those goals, statement of what criteria the data must meet to be acceptable for project use
7-step DQOs, MQOs for all DQIs, statement of tolerable error ranges for project decisions
3.2.7

Special Training Requirements and Certification
To be sure that the project has qualified personnel to perform all necessary functions
List of specialized training required in the project
Matrix of special training needs, personnel who are qualified, and timeframe of qualifications
3.2.8

Documentation and Records Requirements
To document project process, outcomes, and supporting quality information, to meet clause 7.10 of the Standards
Statement declaring where project documentation will be stored, including, but not limited to: QAPP, data assessment records, interim and final reports
Detailed documentation and storage requirements per regulation or contractual obligation. These may be indicated in the project schedule as well.
3.3.1
Proposed Data Source Originator and Publication Information
To provide documentation and defensibility of data considered for inclusion in the project
Include name and date of data originator/origination
Publication source and peer review history
3.3.2
Data Format and Accessibility
To determine if the data are accessible for project use
Describe data format, data fields, and accessibility issues, if any
Provide SOPs for accessing data
3.3.3
Establishment of Acceptance Criteria
To set specific quality requirements for data brought in for project use
Establish the MQOs (qualitative or quantitative, when possible) required to meet project objectives
Discuss contingencies and possible trade-offs if MQOs are not directly achievable with existing data
3.3.4
Sample Data Collection Methodology
To be sure data are appropriate for project purposes
State whether data were collected using a probability-based or judgmental sampling design
Explain the sample design and collection procedures and their implications for project use, such as whether it is reasonable to infer wider meaning beyond the original intent of the data collection
3.3.5
Quality Program and Quality Assurance Procedures Used by Data Originator
To document the quality of the existing data and to determine if the quality program is sufficient to support the use of these data for project purposes
Reference the quality program under which data were collected; if unknown, state the limitations on using data of unknown quality
Describe results of QC samples included with  existing data, data originator notes or reports on data collection, analyses, and limitations on interpretation
3.3.6
Documentation of Sample Quality Assurance Procedures
To facilitate the conversion of existing data to a usable format without error.
A simple statement or description of how this was achieved
Documentation of the methods used and procedures taken to ensure accuracy in conversion to suitable format
3.3.7

Data Management Requirements
To ensure data integrity
Describe data handling from generation, to use, to final storage; include copies of data entry forms, reports and description of databases; specify any special requirements for data such as CBI or hardware/software
Include SOPs for data management, specify project personnel's roles and responsibilities regarding data management, describe how metadata will be gathered and stored with project data
3.4.1
Qualitative Comparisons to Acceptance Criteria
To be sure existing data meet the qualitative acceptance criteria for project use
Describe how the project will assess how well data represent the underlying population of interest, and how comparability among data used in the project will be assessed
Include a table with a discussion of all qualitative aspects of data quality with MQOs for representativeness, completeness, and comparability
3.4.2
Quantitative Comparisons to Acceptance Criteria
To be sure the existing data meet the quantitative acceptance criteria for project use
Document the precision, bias, and sensitivity required to meet project objectives
Include a table with specific MQOs for the quantitative DQIs (precision, bias, sensitivity, and perhaps completeness) with discussion of contingencies and possible trade-offs between MQOs if the data do not fully meet the requirements
3.4.3
Assessments to Other Criteria
To be sure there are not other issues that will inhibit the likelihood of a successful project
State any constraints under which the project must function to be successful (e.g., timeframe, cost, attainment of specific goals)
Describe the mechanisms in place to ensure that these constraints are met
3.4.4

Interim Assessments of Data Quality
To be sure data collection is proceeding according to plan
Include statement encouraging project team members to alert management if they sense anything isn't going quite right
Indicate points during the project timeline at which interim assessments of data quality will be conducted; state who will be responsible for their conduct and how they will be documented
3.4.5

Evaluation of Unconventional Measurements
To be sure that any unusual laboratory methods are fully documented and understood
If no unconventional measurement methods will be/were used, just state that here
Describe the reason for using unconventional measurement methods, provide SOPs for their implementation, and describe how their efficacy will be/was assessed
3.4.6

Evaluation of Unconventional Monitoring Projects
To be sure that any unusual sample collection methods are fully documented and understood
If no unconventional sample collection methods will be/were used, just state that here
Describe the reason for using unconventional sample collection methods, provide SOPs for their implementation, and describe how their efficacy will be/was assessed
3.5.1

Data Verification and Validation Targets and Methods
To determine if data have met project quality objectives
Provide a standard data verification and validation method or procedure that has been reviewed to ensure it meets project needs
Develop project-specific verification and validation schema that incorporate QC data and metadata
3.5.2

Quantitative and Qualitative Evaluations of Usability
To ensure that project decisions are supported by data of sufficient quality for project needs
State which members of the project team will be part of the evaluation of data usability, how it will be conducted and documented
Define contingencies for any issues that may be identified during the evaluation of data usability
3.5.3

Potential Limitations on Data Interpretation
To ensure that the data are not stretched beyond their appropriate use
Describe what actions will be taken if project data are deemed unusable for their intended project purpose
Describe how any limitations will be documented and stored in the metadata surrounding the project data
3.5.4

Reconciliation with Project Requirements
To determine if project requirements have been met
Clearly state how data verification, validation, and usability results will be used to determine if the project requirements have been met; describe how the five steps of the DQA process will be conducted
State specific exceptions to statistical signficance that will be overturned in favor of "practical signficance" should they occur; define the steps to be taken for contingencies if the data do not fully support project requirements
3.5.5

Reports to Management
To document project outcomes
Define schedule and content for interim and final reports to management
Provide report templates
4.2.1

Title, Version, and Approval/ Sign-off
To meet the requirements of Clause 7.5.1 of the Standards for an approved QAPP prior to initiating work
Single page with title and required signatures
Revisions history tracking all changes made to the QAPP over time
4.2.2

Document Format and Table of Contents
To make it easy for the project team and reviewers to find the information they need
Table of contents and page numbers
Document control format
4.2.3

Distribution List
To be sure that everyone who needs it has access to, and awareness of, the latest version of the QAPP
Included on the title page or in the project organization section
A separate section with contact information and indication of under what conditions each individual needs to be made aware of revisions to the QAPP
4.2.4

Project Organization and Schedule
To ensure all key project personnel are aware of their responsibilities and the timeframe for completion
Names of key project personnel and period of performance for project
Organizational chart of all key project roles with names and Gantt or similar tracking chart of schedule
4.2.5

Project Background, Overview, and Intended Use of the Model
To provide sufficient information as a foundation for project goals and to clarify the expected uses of the data
General overview and statement of how data and models will be used in this project
Detailed project history, references to other background documentation, regulatory basis for data use
4.2.6

Data/Project Quality Objectives and Acceptance  Criteria
To ensure that data for models collected, or used in the project, are of sufficient quality to support project decisions
Project goals, data required to meet those goals, statement of what criteria the data must meet to be acceptable for project use
7-step DQOs, MQOs for all DQIs, statement of tolerable error ranges for project decisions
4.2.7

Special Training Requirements and Certification
To be sure that the project has qualified personnel to perform all necessary functions
List of specialized training required in the project
Matrix of special training needs, personnel who are qualified, and timeframe of qualifications
4.2.8

Documentation and Records Requirements
To document project process, outcomes, and supporting quality information, to meet clause 7.10 of the Standards
Statement declaring where project documentation will be stored, including, but not limited to: QAPP, data packages, assessment records, interim and final reports
Detailed documentation and storage requirements per regulation or contractual obligation. These may be indicated in the project schedule as well.

4.3.1
Data Requirements for Model Input
To ensure that appropriate data are available for use in the model
Describe the type, quantity, and quality of data required for use in the model, and how it will be identified or collected
Specify data collection or production and analysis requirements for the model input data (per Chapters 2 or 3 of this QAPP Guidance)
4.3.2
Data Management Tasks
To ensure the integrity of data used in the project
Discuss how data will be collected, produced, managed, and stored
Include SOPs for data management
4.4.1
Model Evaluation Methods and Activities
To know whether or not the model and its results are of sufficient quality to meet project objectives
Describe how the project will evaluate that the model is based on documented science, appropriate data, performs adequately, and meets project objectives
Include specific model evaluation plans throughout the project and model's lifecycle, including factors that would trigger model modification
4.4.2
Evaluation of the Model
To be sure the model functions as needed for project purposes
Documentation of how the model has been and will continue to be assessed for its ability to provide output that meets project objectives
Calibration and internal quality check logs, model change/development logs, evaluation criteria and identification of responsible staff
4.4.3
Model Output Sensitivity and Uncertainty Analysis
To determine if the model is providing sufficient certainty to reach a decision with the desired level of quality
List anticipated outputs of sensitivity and uncertainty analyses, how they will be documented, and any criteria for their acceptability
Computer code for the uncertainty and sensitivity analyses, documentation of the criteria that would lead to refinement of the model via the path identified in the sensitivity analysis
4.5.1
Peer Review
To ensure the validity of the science behind the development of the model
Peer Review by colleagues familiar with the problem
Formal Agency Peer Review with documented charge and response
4.5.2

Description of Model Documentation
To provide defensibility and reproducibility
Description of how model selection or development, modifications, input data, results, sensitivity and uncertainty analyses, and evaluation will be documented
SOPs for document control and archiving of programming code, software, and related manuals
4.5.3
Specifications for Model Maintenance and User Support
To ensure the model doesn't cease to function properly during the project lifecycle
Simple statement of the intent to maintain the model throughout the project lifecycle
Specific schedule for model maintenance and SOP or description of user support to be provided throughout the project lifecycle
4.5.4

Reports to Management
To document project outcomes
Define schedule and content for interim and final reports to management
Provide report templates

APPENDIX B

DATA QUALITY INDICATORS

The choice of MQOs for selection of Data Quality Indicators (DQIs) is critical to evaluating and, in some cases, controlling environmental data quality. The criteria that will be used to gauge measurement performance should be determined and documented for each matrix, analytical group, concentration level, and analyte, if applicable. DQIs relate to the parameters of precision, accuracy/bias, representativeness, comparability, sensitivity, and completeness. It is extremely important to recognize that data quality is a function of both the sampling and measurement processes; it is not solely equated to the quality of individual analytical results. To simplify the way data quality is examined, and to facilitate communication about data-quality attributes, the following DQIs important to environmental studies will be defined: precision, bias, representativeness, comparability, completeness, and sensitivity (PBRCCS). These six DQIs used to be referred to by the acronym PARCCS, with the "A" in PARCCS referring to accuracy instead of bias. Accuracy is actually comprised of random error (precision), and systematic error (bias), and these indicators are discussed separately and hence the "A" has been replaced by "B". The goal of this appendix is to familiarize the reader with the types of DQIs, how each can be used to assess data quality and usability, and how they are measured.

DQIs for precision, bias, and sensitivity can be defined and MQOs measured in quantitative terms; MQOs for representativeness, comparability, and completeness can be defined and measured in both qualitative and quantitative terms. Establishing performance criteria for representativeness is a principal element in setting goals for the sampling design. Establishing performance criteria for sensitivity sets quantitative goals for the quality of data generated in the analytical measurement process. Establishing performance criteria for precision and bias potentially sets quantitative goals for the quality of data generated in both the sampling and analytical measurement process.

Table B-1	Relationship Between Quality Terms
DQOs or PQOs

Qualitative and quantitative quality objectives for project conclusions or decisions. The decision to call these Data Quality Objectives (DQOs) or Project Quality Objectives (PQOs) depends on the organization's preferences.
DQIs

These are the indicators of data quality attributes.
MQOs or MPCs
Acceptance thresholds or goals for the data, usually based on individual DQIs. The decision to call these Measurement Quality Objectives (MQOs) or Measurement Project Criteria (MPCs) depends on the organization's preferences.

Measurement Quality Objectives (MQOs) should be developed as an integral part of the sampling and analysis design generated during systematic planning (e.g., in Step 7 of the DQO process) and see also Annex C of the two Standards. MQOs are not synonymous with project DQOs. DQOs establish the full set of specifications for the design of the data collection effort. The design typically incorporates and specifies requirements for total variability. These requirements are used to establish performance criteria, stated as MQOs (MPCs), for significant components of the total variability. It is emphasized that sampling and analytical procedures should be selected after the performance criteria have been established. The relationship among these quality terms (DQOs/PQOs, DQIs, and MQOs/MPCs) is presented in Table B-1. The historical focus of DQIs has been on the laboratory measurement processes, but, as discussed later, DQIs can be chosen to capture the effects of the full measurement system, sampling design, and sample collection processes, all of which affect the overall quality of study data. Sections B.3 through B.9 detail the DQIs and specific issues relating to each. Table B-2 offers a summary of the definitions for data quality attributes commonly monitored with DQIs along with brief descriptions of methods for determination of those attributes.
Table B-2	Data Quality Indicators
                                      DQI
                                  Definition
                           Examples of Determination
Precision
An evaluation of agreement among replicate measurements of the same property under similar conditions; also referred to as random error or measurement variability and usually expressed as standard deviation, variance, percent difference, or range, in either absolute or relative terms
Overall project precision is measured by collecting data from collocated field duplicate (or replicate) samples. Precision specific to the laboratory is measured by analyzing laboratory duplicate (or replicate) samples
Bias
The systematic or persistent distortion of a measurement process resulting in error in one direction
Measurement of materials with a known concentration (e.g. performance evaluation or reference materials), analysis of matrix spikes, or the use of laboratory control samples
Accuracy
A measure of the closeness of an individual measurement to a known or reference value; includes a combination of random error (precision) and systematic error (bias) components of both sampling and analytical operations
Replicate analysis of a reference material or re-analyze a sample to which a material of known concentration or amount of pollutant has been added; usually expressed either as percent recovery or as a percent bias
Representativeness
A qualitative measure of the degree to which data accurately and precisely represent a characteristic of a population parameter
Evaluation of whether a sample that is collected and then processed and sub-sampled by the laboratory is proportionately representative of some predefined population characteristic or property. As such, representativeness is an "objective-defined" parameter (e.g. total concentration versus dissolved concentration versus bioavailable concentration)
Comparability
A qualitative term describing the degree to which different processes, methods, or data agree or can be represented as similar. It describes the confidence that two datasets can contribute to a common analysis and interpolation. Comparability criteria must be determined for each matrix, analytical group, concentration level, and analyte (if possible).
A comparison of the output of two sediment transport models via sensitivity analysis. Or comparison of the sample collection methods, analytical procedures, holding times, stability issues and QA protocols. One study with results in ug/L is not necessarily comparable to another with results in ppb. A similar argument exists between wet and dry weight comparisions
Completeness
An evaluation of the amount of data needed to be obtained from a measurement system; expressed as a percentage of the number of measurements that should have been collected or were planned to be collected
Evaluation of the number of measurements needed to make a determination of the project results and comparison of this to the number of samples planned to be collected
Sensitivity
The capability of a method or instrument to discriminate the parameter of interest at the level of interest. Terms sometimes used to describe sensitivity include Method Detection limit (MDL), Limit of Detection (LOD), and Limit of Quantitation (LOQ)
The measurement responses representing different levels or amounts of the variable of interest, MDL study, and verification of LOD

B.1	Sources of Measurement Uncertainty
Before discussing details of DQI calculations and uses, a presentation of the framework used in this document for examining error sources is necessary. Consideration of all the sources of error is critical as specific DQIs can be established to address specific sources of error. Total study error is a measure of the uncertainty in a metric such as a site mean concentration caused by the combination of all error sources in the study design. The term "total" is relative to the overall boundary conditions (spatial or temporal boundaries) for which a decision will be made (sometimes referred to as the decision unit) or study conclusions drawn, based on the relevant metric. These boundaries are defined in the DQO process and may include the overall population of interest, as well as specific subpopulations for which a decision will be made. A sampling unit is the portion of the physical environment from which one or more samples may be taken, resulting in measurements appropriate for an intended use. The term "error sources" refers to any factors that increase uncertainty in a measured value. Generally speaking, these error sources are the result of natural variability in the sampled media and uncertainty associated with the sampling and measurement process. The framework for examining study errors will also serve as a context for clarifying the relationship among the total study error, DQOs, and MQOs.
The contribution to total error from within-unit sources is largely a function of the boundary conditions for the study at hand, the inherent variability of the characteristic of interest within the unit, the difficulties associated with obtaining a specimen from the unit, and the error associated with the measurement process itself.
B.2	ESTABLISHING MQOs IN THE CONTEXT OF DQOs
One way to employ DQIs is as a means of specifying MQOs which, if achieved, will provide an indication that the resulting data are expected to meet the DQOs. This is not to say that simply achieving MQOs for the measurements ensures that the DQOs are met. Typically it is more important to ensure that an adequate number of samples, with appropriate sample support, are collected to represent the population of interest. If this is the case, then ensuring that MQOs were achieved provides an indication that the measurement quality will be adequate for the intended use. Used in this way, DQIs provide a metric against which the performance of a program can be measured during the implementation and/or assessment time frames.
During the design phase, the type and number of samples required to achieve DQOs and the way in which these samples should be optimally allocated across space and time are developed. Guidance on Choosing a Sampling Design for Environmental Data Collection, EPA QA/G-5S, (EPA 2002a) provides guidance on alternative design approaches. For statistically-based sampling designs, an estimate of total study error is required to determine the appropriate number of samples required to achieve a certain performance goal, In some cases, historical data and information are available, but were collected with older, or different, sampling or analytical methods than are currently considered (comparability). In other cases it may be advisable to derive an estimate of variability using other information on the background of the measured variable. Usually this involves the use of statistical theory combined with practical knowledge.
 To determine the potential impact of using newer methodologies on total study accuracy (precision and bias), it is useful to see how much the error sources of variability contribute to the total study variability. If the newer methods are more accurate, specific MQOs can be established based on the expected improvements in laboratory performance, and these improvements can be factored into the design equation. It is emphasized that the performance goals of precision (and the rest of the MQOs) be considered relative to action levels for each analyte. For example, if the newer methods are less accurate, but the concentrations are well above (or below) their action levels such that a decrease in accuracy or loss of precision will not affect the ability to make an appropriate decision, then it may still be optimal to use them. If heterogeneity is problematic, then obtaining a larger number of samples would increase coverage (representativeness) to better represent the population of interest and account for between-unit variability.
A practical approach to accomplish this is to first determine the performance required for the DQI, based on an analysis of the relative impact of a specific error source or quality attribute on total study design. This will be based on an understanding of what method or procedure precision levels are tolerable, given a particular design (type, number, and allocation of samples) under consideration. The design process involves evaluating tradeoffs between the sampling aspects and measurement aspects and should be aimed at achieving an optimal combination of the two relative to the DQOs or project objectives.
If laboratory or method defaults are not adequate or if alternatives are much less expensive, then MQOs should be specified and a new agreement reached with the laboratory performing the analyses (or a new instrument or method selected) to meet or exceed these more stringent requirements. If aimed at the analytical laboratory, this may take the form of a special analytical request or a performance-based method contract. If aimed at the use of an instrument or method, it may take the form of a new standard operating procedure designed to ensure that the MQO is achieved such as SOPs for replicating X-ray fluorescence (XRF) measurements to achieve a precision requirement. MQOs should be driven by the DQO process and should always be defined before listing laboratory or method "default" limits in a QAPP. This is especially important when the project does not require levels as stringent as the laboratory's default limits to be adequate for the intended use. For example, if default requirements for sensitivity are far better than required to support a particular decision, project-specific, less restrictive requirements can be stated in the QAPP.
B.3	PRECISION
Precision is a DQI upon which statistically designed sampling is based. For this reason, precision is among the most important DQIs in environmental projects. Precision is the measure of agreement among repeated measurements of the same property under identical or substantially similar conditions. High precision refers to close agreement among repeated measures, whereas low precision refers to poor agreement among repeated measures. For environmental studies, the property measured is typically a concentration of a contaminant, but any physical measurement has error; therefore, the precision of that measurement can be examined. Random errors or fluctuations in the measurement process always result in some range of values of these repeated measurements. Precision is a quantitative indicator of the dispersion generated from these random errors.
Precision is estimated by using some form of replication (e.g., collocated field duplicates, field splits, laboratory splits, matrix spike duplicates, instrument replicates) followed by calculation of a DQI based on the replicate measurements. Estimates of precision provide a measure of agreement among replicate analyses; MQOs for precision set tolerable limits of imprecision for a study.
One of the most common precision DQIs are the summary statistics range for duplicate measurements and standard deviation for multiple measurements. One nice feature of these statistics is that their units are the same as the individual measurements, and they are relatively easy to calculate. Another widely used measure is relative percent difference, usually used when only two samples are taken.
B.4	BIAS
Bias is systematic or persistent distortion of a measurement process that causes errors in one direction. Bias may originate from sources such as calibration errors, response factor shifts, unaccounted-for interferences, or sample contamination. The sample itself may generate real or apparent bias caused by a matrix effect or variation in physical properties such as particle size. DQIs for precision are quantitative indicators, which reflect the magnitude of systematic error resulting from these effects. Bias can be in the positive (a false positive or elevated signal/concentration) or negative (a false negative or suppressed signal/concentration) direction from the true value.
The difference between the measured and known or expected result is a DQI for bias. For spikes and reference materials, bias is most conveniently expressed as a fractional or percent comparison of the measured result to the expected result. Relative bias is also a common indicator, which indicates both the magnitude and direction (positive or negative) of the bias. Expected results are based on the known properties of the QC sample.
The expected result is usually the true value derived from a standard or a theoretical value and documented in the project's QAPP.
	measured result - expected result
	relative bias = 	expected result
Relative bias is the ratio of the difference between the measured and expected results to the expected result. Relative bias is sometimes used to describe bias in terms analogous to relative precision. For applications of bias with a known contaminant, a commonly used measure of bias is percent recovery, which is simply the percent of the expected result that is measured.  Percent recovery calculated from matrix spikes is the difference between the spiked and unspiked sample results divided by the spiking amount. A completely unbiased result thus has a recovery of 1 (or 100%).

Minor short-term problems that do not create a systematic bias are best reflected in precision DQIs (random errors). For example, if an analyst makes an error on a dilution as part of the sample preparation step, that mistake will bias the individual result. However, this sort of error is part of the overall precision in the measurement process. Because the analysis of QC samples (see Appendix C) includes the entire measurement step, the precision of the measurement process is also imposed on results intended for estimating bias. For this reason, estimates of bias primarily serve as a quality control function during implementation of a project plan. During the planning phase, procedures put in place to minimize the potential for bias and the use of estimates of bias to verify the procedures should have the desired effect of keeping bias to a minimum and assessing the potential effects on project objectives.
Procedures for estimating bias are also valuable tools for ensuring comparability of data. Completeness of a data set also has the potential to impact bias. If a data set is incomplete, any systematic trend to the missing data may cause bias in estimates of population parameters based on this faulty data set.
B.5	ACCURACY
Accuracy is a measure of the overall agreement of a measurement to a known value. In a limiting case where random errors are very tightly controlled, bias dominates the overall accuracy. In general, however, both precision and bias contribute to accuracy. A measurement result with zero bias may not be accurate if the measurement process is not precise. Figure B-1 demonstrates how different combinations of precision and bias can contribute to accuracy. The true mean value, indicated by the dashed line, is 10.0 and the red line represents a hypothetical action level of 12.0 that might be used as a "bright line" or limit for an action. Diamond symbols indicate the observed measurements. Part A of Figure B-1 shows the least desirable situation of positive bias (a large shift in the positive direction) and low precision (a large spread in the results). The accuracy and precision of these data are poor. Part B of Figure B-1 shows the case of low bias (no apparent shift in a positive or negative direction) and low precision. Any one of the measurements is not accurate, but an average of all the measurements would be accurate relative to the true mean. Part C of Figure B-1 shows data with positive bias and high precision. The accuracy of these data is poor. In this case, random errors are well controlled, but a systematic error limits the accuracy of each individual measurement. Part D of Figure B-1 shows the case of low bias and high precision. Each individual measurement is accurate. Notice that in C and D the DQI of precision is critical for making a correct decision relative to the action level. It is obvious from this visual example that both precision and bias be considered relative to the action level in order to assess whether the DQI has a significant impact on the ability to make a correct decision. Whenever an action level is "close" to the measured data from the site (relative to the dispersion of the data), the more critical the DQIs are.

            Figure B-1	Influence of Bias and Precision on Accuracy
B.6	REPRESENTATIVENESS
Representativeness, as it was defined by ANSI/ASQ E4-2004 (ANSI 2004), is "The measure of the degree to which data accurately and precisely represent a characteristic of a population, parameter variations at a sampling point, a process condition, or an environmental condition." This definition of representativeness encompasses issues at both the micro- and macro-scale by addressing both how well measurements taken within a sampling unit reflect that unit (parameter variations at a sampling point) and the degree to which measurements from a set of sampling units represent (allow you to make inferences about) the population of interest (accurately and precisely represent a characteristic of a population). Central to representativeness is assurance that both the sampling and measurement processes are free from known biases. Developing a clear understanding of the "population" that is the subject of an experiment or investigation is the key to assessing representativeness. The characteristics of the population include the subject's identity or class (e.g., the particular property that is to be measured), the spatial distribution of the property, and in some cases, the temporal characteristics of the property. Step 4 of the DQO process (EPA 2006a) explicitly focuses on establishing a clear definition of the population of interest and any subpopulations that data should adequately represent to support decisions at the desired scale. Representativeness is usually considered a qualitative term that does not lend itself to measurement by an MQO.
Instead, sampling precision and bias indicators related to the desire to represent environmental phenomenon at specified spatial and temporal scales are used to control the quantitative aspects of representativeness.
Representativeness was established as a DQI as a result of the recognition that characteristics of interest to environmental problems are heterogeneously distributed in space and time within the environment, and that careful attention should be paid during planning and implementation of a study to ensure that a set of samples adequately mirrors or reflects the characteristics of interest. Project managers and decision makers want assurance that the results of a particular data collection effort are not biased in any known way by the sampling or analysis design. Such a bias, combined with the inherent heterogeneity in the environment and uncertainty in the measurement process, could result in an incorrect conclusion. Lack of representativeness can have a direct impact on the ability to make the correct decision when relying on data. To ensure representativeness, careful attention during the entire life cycle of a project is required.
The sections on precision and bias covers in detail some of the more quantitative elements of representativeness. Central to representativeness is assurance that both the sampling and measurement processes are free from known biases. Representativeness is widely used in a less precise manner in the environmental community. For example, representativeness is a word commonly used to mean:
there is an absence of biasing forces;
it is a miniature or replica of the population;
it is a typical or ideal case;
there is a wide coverage of a population;
it permits good estimation;
it is good enough for the purposes of the study; or
a statistically-based sampling method was used.
While a number of these are indeed characteristics of a representative study, a more careful definition will better delineate suitable indicators of representativeness.
The concept of representativeness is called out in a number of EPA regulatory programs. For example, the Resource Conservation and Recovery Act (RCRA) talks about "...expected to exhibit the average properties of the universe or whole" (RCRA CFR 260.10). The Toxic Substances Control Act (TSCA) refers to "...at locations representative of the air entering the abatement site" (TSCA 40 CFR 763). In the air programs we see a discussion of representativeness: "...should be selected on the basis of spatial and temporal representativeness" (40 CFR 51 Appendix W). The wastewater program simply states, "...samples should be representative of daily operations" (40 CFR 403.12(b)). While these statements are lacking in a rigorous definition, they provide some understanding of the importance of this indicator to EPA programs.
The process involved in obtaining representative samples includes planning, implementation, and assessment. In addition to sample design, careful attention should be paid to the sample collection, measurement and analysis processes.
Representativeness also has relevance in the world of laboratory research studies and experimental design. For a discussion of these issues, refer to texts on experimental design, including Box, Hunter, and Hunter (1978) and Cochran and Cox (1957).
To understand the importance and scope of this DQI, imagine just these few examples where representativeness is not achieved and consider the potential consequences:
surface water samples are taken from the stagnant water nearest the shore in a calm pool, and are then meant to represent the characteristics of the fast-flowing river;
soil samples are collected using a technique that preferentially selects the smallest particles and omits the larger particles, thus under-representing those with lower surface area proportionately, and therefore lower levels of externally adhering contaminants; or
a survey of public perception to off-shore oil drilling conducted entirely in Ohio, Kansas, and Nevada, and presented as national opinion.
Representativeness, as a DQI, is most relevant when viewed in the context of the data's intended use. Representativeness between sampling units and within sampling units should be considered, and qualitative MQOs for both should be established and documented in the QAPP.
B.7	COMPARABILITY
Comparability is the qualitative term that expresses the confidence that two datasets can contribute to common interpretation and analysis of the parameter or matrix of interest. Quantitative measures of comparability are also available involving statistical tests that measure the similarity or difference between two or more datasets. Comparability should be carefully evaluated in order to establish whether two datasets can be considered equivalent in regard to the measurement of a specific variable or groups of variables (EPA 1997, EPA 2010a).
Comparability is a very important qualitative DQI for analytical assessment and is critical when considering the combination of datasets with the same analytes. The assessment of this DQI determines if analytical results reported are equivalent to data obtained from similar analyses. Only comparable datasets can be readily combined and used for subsequent statistical evaluations. For example, if two separate investigations of the same site utilized different analytical methods with different sensitivities, combining the data may violate the underlying assumptions of which the statistical tests depend. The potential differences in precision and bias could be misinterpreted as within-unit sampling error.
As with any decision about the usability of data, it is important to consider the decision that the data are meant to support. Separate determinations of the comparability of datasets may be necessary for each decision the data are used to support. Examples in the uses of comparability are found in Chapter 3, Section 3.4.1.
B.8	COMPLETENESS
Completeness is a measure of the amount of usable data obtained from a measurement system, expressed as a percentage of the number of measurements that should have been collected according to the study design (i.e., measurements that were planned to be collected). Percent completeness is calculated using the formula:
Percent Completeness = 	(number of usable measurements) x 100 %
	 (total number of measurements planned)
Completeness is one measure of how well a sampling and analysis design was implemented. A data set that is 100% complete is the result of careful planning and precise implementation of the data collection plan. Completeness is not intended to be a measure of representativeness; for example, it does not describe how closely the measured results reflect the actual concentration or distribution of the pollutant in the media sampled, but it may be a contributing factor. A complete data set may or may not achieve the project objective depending on how well the sampling plan reflects the conceptual model, how accurate the conceptual model was prior to sampling, and whether the distribution of the reported data are similar to those anticipated.
When considering completeness, attention should be given not only to simply complete samples but also to the measurements on analytes within a sample. If, for example, measurements on 6 analytes were requested, but only 4 actually made, this sample would be classified as "incomplete" if the missing analyte measurements were part of the MQOs or DQOs. It follows that for some instances, completeness should be defined in terms of analyte/sample combinations rather than samples. For example, suppose measurements of six analytes were requested in 20 samples, but in one sample only four were actually performed or found acceptable readings; the entire sample would be found "incomplete" with a completeness of 19/20 = 95%.  If completeness is defined in terms of analyst/sample combinations, then the completeness would be 118/120 = 98.33%.
The important question for decision makers is whether the number of measurements is sufficient to support the decision to be made. For example, there could be only 70% data completeness (30% lost or found invalid), but, because of the nature of the study design, the results could still be representative of the target population and yield valid estimates. Conversely, a data set with much higher completeness but systematically omitted or rejected data may be insufficient to yield valid estimates of the parameters of interest. Key questions for the DQI of completeness would include:
Was the number of field samples taken for each matrix and analytical group the same as that documented in the sampling plan?
Were all the associated quality control samples taken?
Were all the samples delivered to the lab analyzed?
Were the samples analyzed with the appropriate method as documented in the QAPP? and
Were any of the sample results rejected due to quality concerns?
It is suggested that the QAPP set a quantitative goal for completeness. If that goal is achieved, then no further consideration of this DQI is necessary. If the completeness goal is not achieved, the QAPP should make clear how the project team will proceed and what options might exist. It is important that the completeness goal is potentially attainable (95% when using only 10 samples, for example) and that it is linked to the project's objectives. It is important to distinguish between incompleteness of investigative samples and QC samples, as the loss of the latter can be of less importance than the former.
B.9	SENSITIVITY
Sensitivity generally refers to the capability of a method or instrument to discriminate between small differences in analyte concentration. Both the precision of the instrument and the slope of the calibration curve limit sensitivity. Chemists typically define sensitivity as the slope of the calibration curve at the concentration of interest (Skoog 1985, Currie 1995). If two methods have equal precision, the one having a steeper calibration curve will be the more sensitive. Sensitivity can also be evaluated from the standard deviation of replicate analyses at any concentration level or can be evaluated from the confidence bound on a calibration curve. It represents the minimum difference in two samples that can be distinguished with a defined confidence (Taylor, 1987).
The sensitivity indicators of primary interest relate to limits of detection. The detection limit (DL) is generally considered to be the minimum true concentration of an analyte producing a non-zero signal that can be distinguished from the signals generated when no concentration of the analyte is present, with an adequate degree of certainty. There are a plethora of different approaches to the determination of detection limits that can be broadly categorized into two groups: those that evaluate the statistical variability of instrument responses with and without the analyte present. There are several critical aspects of determining detection limits that have historically been overlooked:
    * the variability of instrument responses are commonly heterogeneous or heteroscedastic (i.e. standard deviation is not linear over the concentration range);
    * most common methods only consider false positives (type I or α error) at the Detection Limit (DL), ignoring the importance of false negatives (type II or β error) (The false negative rate at the DL is 50%. Therefore, there is a 50% chance of error when reporting a sample result as <DL (or <MDL) when nothing is detected);
    * the distribution of measured values around the true detection limit is assumed to be normally distributed. (In reality that assumption at the true DL is likely false. Unfortunately we cannot verify that assumption because we do not have statistical confidence at the DL);
    * instrument responses, variability, and subsequent detection limits are greatly affected by the sample matrix, instrument conditions, and even the analyst; and
    * many analytical methods produce non - zero signals even when a target analyte is not present.
The Limit of Detection (LOD) is the smallest concentration of analyte present in a sample in order to be detected at the 99% confidence level. The LOD differs from the DL in that the probability of a false negative is 1%. The LOD is usually >1 to 2 times the DL.
The quantitation limit (QL) is another concept that is very important for environmental decision making. The QL relates to the lowest concentration at which the method is expected to be able to quantify the amount of analyte present in the sample. The QL is usually 5 to 10 times the DL. The Limit of Quantitation (LOQ) is the smallest concentration with a specified level of precision and bias. That is, samples with concentrations at or above the LOQ can be quantified with a known level of accuracy. Detection and quantitation levels (DLs, QLs, LODs, and LOQs) vary by analyte and by matrix and often vary among laboratories. It is critical that when referring to these levels the specifications are made very clear.
Analytical capabilities are constantly improving, resulting in greater sensitivity and lower detection limits. This improvement in analytical capabilities is frequently the vehicle that drives regulatory and, hence, project requirements. Investigators often base their sensitivity requirements upon analytical method capabilities rather than upon project-specific objectives. The problem for a project team is to determine the levels of sensitivity that will generate data adequate for decision making, to establish MQOs based on this evaluation, and to be sure that the indicator of sensitivity used to evaluate a particular method appropriately reflects the performance of the method in the particular matrix of interest. MQOs tie the required measurement quality to the project DQOs.
The project team should always consider the needed sensitivity of a measurement prior to requesting laboratory analyses. Once the needed sensitivity is determined, the project team can work with laboratory personnel to choose the appropriate analytical method. Sensitivity is dependent on the sample matrix, the analytical method, the instrument used, and the operator conducting the analysis. The sensitivity of the analyses potentially impacts data usability for a given decision. A couple of issues that are frequently encountered in environmental projects relating to sensitivity are described below, along with suggestions for handling them.
Project action levels are numerical values used by the decision-makers as a bright-line for selecting one of two or more alternative actions (e.g. remediate or leave in place). Action levels may be regulatory thresholds such as a Maximum Contaminant Level (MCL), a risk-based or toxicity reference benchmark, or a treatment standard. Some of these action levels may be at concentrations well below the DL of many analytes. However, because of the great uncertainty at the DL, the QL should always be less than project action levels. During project planning, project teams may be faced with the dilemma of targeting action limits at or below QLs.
It is recommended that the laboratory perform project-specific confirmation of action limits in each matrix of concern. Some types of DLs such as MDLs are determined by the laboratory under ideal conditions using a clean reference matrix. Analytical sensitivity is impacted by the presence of interferences. Therefore, the QAPP may include a requirement to perform a sensitivity verification study for each matrix. The laboratory can verify if the project action levels can be met by spiking the appropriate matrix at the action level for each analyte. The verification procedure for establishing the ability to meet the action levels must empirically demonstrate quantifiable estimates of precision and bias at the action level. When precision and bias do not meet project requirements at project action levels, several options should be considered to ensure that the QLs are below the project action levels:
   1. Modification of the proposed method. In some cases, it may be possible to modify the proposed method to achieve improved sensitivity. Possible modifications that could have a beneficial impact on sensitivity include:
            extracting more sample volume or mass, or decreasing the final extract volume (bearing in mind that problems in homogeneity and interferences may result in an increase in bias);
            including additional cleanup steps (sample preparation steps to eliminate interferences); or
            use a different wavelength or ion for quantitation.
      Performing analyses in selected ion monitoring mode rather than full-scan mode during mass spectrometry analysis is an example of a modification to the method that may improve sensitivity.
   2. Use a different method with better sensitivity. For many environmental analysis needs (e.g., measuring the concentration of polycyclic aromatic hydrocarbons in sediment), multiple analysis methods are available. The differences between methods may include cost, availability, and sensitivity. If the project action level is below or near the QL for a particular method, it may be possible to identify and implement a different method for analysis for which the QL is well below the project action level. For example, if running inorganic analyses using Inductively Coupled Plasma Optical Emission Spectroscopy (ICP-OES), is not achieving the desire level of sensitivity, switching to ICP-Mass Spectroscopy (ICP-MS) analyses might provide the additional sensitivity to meet objectives.

   3. Consider alternative action limits when it is reasonable to question the applicability of a pre-set action level for a particular project. For example, taking action at the level of the MCL if there is no potential for the water to be studied (e.g., water in an ephemeral stream) to be used as a drinking water source may be overly conservative, and other, more project-specific action limits could be considered.

   4. Report all analytic results with their associated uncertainties (precision and bias). Although the uncertainty of results below QLs is relatively large, as long as the uncertainty is known a determination can be made regarding the usability of the data in relation to the project action level. If all results are reported with their associated uncertainties, the project team will be able to ascertain the level of confidence that would be achieved if they make a decision regarding an action limit that is below the QL. One way to establish the uncertainty (precision and bias) in the estimated results is using a technique developed by Hubaux and Vos (1970). The Hubaux-Vos method is an extrapolation method that utilizes the linear regression of the calibration curve. The DL is determined by the width of the confidence bands of the calibration process. This approach has the added benefit of considering both false positive and false negative instrument readings. If this approach is utilized, the detection limits should be reported along with the total number of calibration points, the number of replicates, and the associated error rates used to define the precision and bias bounds.
Another issue relating to sensitivity is the way that very low results are reported as there is a great deal of uncertainty associated with very low results. The high uncertainty makes it difficult for chemists to be able to state what concentration of an analyte is present in the sample. One way that chemists handle this is by reporting that the result is less than some detectable or quantifiable amount. There are several different conventions for how this is done such as:
choosing protection against false positive results (incorrectly reporting that an analyte is present in a sample when in reality it is not present) such as:
results below the MDL are reported as undetectable ( < DL),
results between the MDL and the practical quantitation limit (PQL) are reported as estimated), and
results above the PQL are reported directly.
Choosing protection against false negative results (incorrectly reporting that an analyte is not present in a sample when it actually is present) such as:
all results below the laboratory DL are reported as undetectable (as <LOD, where LOD is one or more times the DL),
results between the DL and LOQ are reported as estimated, and
results above the LOQ are reported directly.
With multiple reporting conventions in use, many different methods for setting the DL (EPA 2010), and many different methods for establishing the QL, it is clear that Project Managers need to document in the QAPP which convention they will adopt. If the standard practices of the laboratory are deemed acceptable for the project, it should be stated in the QAPP and the laboratory SOPs relating to their DLs, QLs, and reporting of results should be summarized and/or attached. If the DL is going to be calculated using a matrix-specific method (e.g., Hubaux and Vos 1970), then the methods should be well documented in the QAPP.
Finally, while setting a sensitivity level that must be reached to attain DQOs would appear to be straightforward, it may not be. Is it the DL or the QL, or both that need to be defined in MQOs? Is a reporting convention that protects only against false positives acceptable for the needs of the project? Does the project call for a reporting convention that protects against false negatives? In the arena of DLs, QLs and reporting conventions for data at very low levels, there is not an overall best way to proceed as there are just a few common practices that have become relatively standard for environmental data analyses. If DQOs indicate that these issues might be important (i.e., the concentration levels at which decisions must be made are very low), it is suggested that the project team carefully consider how MQOs for sensitivity are defined, and how the laboratory statement of work is written to ensure that the MQOs can be achieved.
B.10	THE ROLE OF DQIs IN THE PROJECT LIFECYCLE
DQIs play a number of important roles during the project planning phase such as calculating relevant indicators from historical data to support the design of new efforts, establishing MQOs, identifying future DQI needs, and specifying quantitative and qualitative requirements.
The role of DQIs during project planning focuses on assumptions (typically based on analysis of historical performance data) made during the development of DQOs and an associated statistical design of a project. For example, design optimization may involve an analysis of the major sources of sampling and measurement errors, as well as spatial and temporal variability that will contribute to uncertainty. Total study error impacts the ability to achieve the limits on decision error specified during Step 6 of the DQO process (EPA 2006a). Depending on the relative contribution of different components of total study error (especially components of total study precision), different choices may be made to cost-effectively achieve the specified DQOs.
Project DQOs together with individual MQOs for precision, bias, and sensitivity are critical inputs to developing alternative sampling and analysis designs. Alternative designs should be developed that reflect different combinations of statistical sampling and measurement schemes. The expected performance of these designs may be calculated using these indicator DQOs/MQOs as inputs; the output of this effort is only as valid as the assumptions (including the quality and relevance of the indicators) that are made. In addition, once the critical DQIs are identified (those to which the final data quality is expected to be most sensitive), important design choices concerning QA/QC samples that enable estimates of the DQIs can be made. The purpose of such an analysis is to determine the likely output before the actual obligation of resources. Understanding the critical relationships between DQOs and MQOs is essential to the generation of data of known quality.
Decisions about what DQIs are appropriate in a study and what MQOs are to be achieved to meet the project DQOs should be documented in the QAPP.
By conducting interim evaluations of the performance of the measurement system, project staff can identify whether to implement adjustments or corrective action to keep measurement systems in control before the project budget and schedule are expended. This is especially true for ongoing measurement programs such as monitoring efforts required for compliance programs, because mid-course corrections can be made based on feedback from the last sampling effort, or last series of samples. To support these QC measures, specific DQIs should be identified and data collected to support their calculation. In addition, MQOs assess and interpret the QC data results. Based on these types of comparisons, corrective actions can be taken as needed to improve project performance and increase the probability that data will be adequate to support the intended use.
Too often, QC data are generated, but not evaluated at all, or not evaluated in a timely, meaningful way. The DQIs discussed in this document can be used not only in a real-time QC mode, but also in the assessment of the overall performance of a program over some period of time such as a field season. For example, indicators of precision, accuracy, and sensitivity can be evaluated over the field season to determine how well the program was able to measure critical variables.
Following a data collection effort, determination of the adequacy of the data can be made via data verification, validation, and an evaluation of usability. The purpose of those data review steps is to determine whether the data set will support a decision with the desired degree of certainty. It is important to consider the performance and representativeness of the measurement effort prior to reaching conclusions regarding data adequacy. Step B of data validation (see Appendix E) focuses specifically on attainment of the MQOs set as quantitative goals for DQIs. Data usability then revolves around whether an adequate number of samples was obtained, given the observed measurement, spatial and temporal variability, and given the actual magnitude of the measurements made (relative to levels of concern) to meet the DQOs. If a data collection effort fails to generate adequate data to meet DQOs, then interest in DQIs is heightened, especially if there is a desire to "diagnose" which assumptions proved valid or invalid. Having documented the basis for the design based on DQIs, project staff can identify where the system failed to perform adequately and determine what additional data will be required to support the decision.

APPENDIX C

QUALITY CONTROL SAMPLES

Figures C-1 and C-2 depict a measurement process that involves several steps. Each step has the potential to introduce variability or bias that might influence the quality of project data. Estimates of the components of overall study variability and the relative contribution of measurement (within-unit) precision are important inputs to the statistical design process. The QAPP should describe the processes used to estimate and monitor the magnitude of at least the most important potential measurement error sources. Quality planning for measurements simultaneously establishes MQOs appropriate for the project and data use and defines the required DQIs. The data sources for DQIs are often derived from samples inserted in the sampling process by field or laboratory personnel on a frequency specified in the QAPP. Figures C-1 and C-2 suggest the types of QC samples that might be identified as the source of the underlying data for precision (Figure C-1) and bias (Figure C-2) DQI calculations.

Figure C-1.	Total Sampling and Measurement Process Denoting the Use of QC Samples to Measure Components of Total Study Precision