Use of unconventional methods may or may not be allowed depending upon the environmental program under which the project is regulated. For some programs only methods that are codified are allowed and they must be followed verbatim. If unconventional methods are allowed and appropriate for use on the project, the QAPP should include scientifically valid and legally defensible documentation of the procedures relating to those methods.
As with conventional methods, the quality of environmental data obtained via unconventional methods should support the intended use for the project, and ensure that the DQOs are met. Clear MQOs should be established and documented in the QAPP that describe the level of precision, bias, and sensitivity expected and needed to meet project objectives. Evaluation of these unconventional methods should occur during project assessments, and should be clearly documented in the QA Reports and/or final report. It must be remembered, however, that research projects often use new methods for which performance data do not exist. A full documentation of such methods' performance may be premature as these methods are amongst the first being developed.
Evaluation should include review of at least these items:
demonstration that the method can be implemented as described in the QAPP;
documentation that records of the implementation method are created and maintained;
how data generated via this method achieves the MQOs set forth in the QAPP; and
how the data will be used for their intended purpose.
Unconventional measurements come under greater scrutiny than standard measurements and so the level of quality assurance should be commensurate with that added scrutiny.
Note that the evaluation of unconventional methods should not occur during assessments by independent assessors.  This activity should have already been performed by project personnel at an earlier stage in the development of the project and documented for review by independent assessors.
2.4.8	Evaluation of Unconventional Monitoring Projects
Environmental projects span a very wide range of activities and there are some unusual situations where the standard methods for monitoring environmental conditions do not fully meet project needs. Some examples for which nontraditional environmental monitoring might be required include crab counts in the Chesapeake Bay, fish kills in Louisiana rivers, environmental sampling for chemical agents surrounding chemical weapons disposal facilities, or monitoring of radioactivity levels in pools containing spent nuclear fuel rods.
As with unconventional laboratory methods, it is especially important that these nontraditional monitoring methods be well documented in the QAPP. Along with full descriptions of the methods, project-specific MQOs should be stated for representativeness, completeness, and for comparability if these data will be combined for use with any other environmental data. For some monitoring projects, MQOs for precision, bias, and sensitivity may also be appropriate (e.g., the sensitivity of carbon-dating methods for estimating tree age).
Evaluation of nontraditional monitoring methods should include an assessment implementation of the methods against their SOPs or other description in the QAPP. The efficacy of the methods should also be evaluated against the MQOs stated in the QAPP. Finally, the usability of the data generated from any nontraditional methods should be evaluated to determine if it is of appropriate quality to support its intended use on the project.
2.5	REVIEW, EVALUATION OF USABILITY, and REPORTING REQUIREMENTS (Act)
The final part of the paradigm concerns the actual data; are they valid and can they be used for the intended purpose. Verification and validation is followed by an evaluation of usability. Included in the usability section is a small discussion of statistical significance. This section offers guidance on Standards Clause 7.9, Data Review, Verification and Validation, and Data Usability Reporting, Standards Clause 7.10, Documents and Records Management, and Annex B3.6, Review, Evaluation of Usability, and Reporting Requirements.
The elements in this section address project checks to see if the data or product obtained will conform to the project's objectives and to estimate the effect of any deviations. These activities may be performed throughout the project to inform mid-course corrections and/or at the completion of the project.
Three distinct data evaluation steps are used to ensure that project data quality targets are met and should be applied to all data collected and used in environmental projects. These steps apply to all aspects of data production, including field sampling and analytical activities. Although the data verification/validation/usability evaluation process outlined in the following sections is portrayed as a sequential process, it may be beneficial (and more cost effective) for many projects to combine steps. For example, the entity conducting the verification could also conduct the first step of the validation process concurrently. The method by which an organization inspects, reviews, verifies, and validates data should be stated in the QAPP.
Data verification entails confirmation by examination and provision of objective evidence that the validated information fulfils specified requirements (sampling and analytical), or requirements mandated by a contract or agreement.
Data validation means confirmation by examination and provision of objective evidence that the particular requirement for which the data or information was collected is fulfilled.  It includes the process of checking whether the information meets the project's specifications.
Data usability is the determination of the adequacy of data, based on the results of verification and validation meet the QAPP criteria.
See Appendix D for a more complete discussion of the steps involved in data verification, data validation, and data usability.
2.5.1	Data Verification and Validation Targets and Methods
This section describes what information should be included in the QAPP and presents procedures for implementing the first two of the three data evaluation steps: verification and validation.
To perform the data evaluation steps described above, reported analytical data should be supported by a complete set of auxiliary (or meta) data as defined in the QAPP. These auxiliary data include sample receipt and tracking information, chain-of-custody records, tabulated data summary forms, and raw analytical data for all field samples, standards, QC samples, and all other project-specific documents that are generated. If relevant raw data or sample information are not available or adequate to document data quality, then data analysis becomes problematical. Lacking a clear linkage of auxiliary data to the project's objectives creates a potential weakness in assessing the quality of the data and the inferences to be drawn from them.
Verification is performed at the onset of the data review process to determine if the required information (the complete set of data) is available for further review. It involves a review of objective evidence that validated information fulfills specified requirements. The QAPP should contain procedures for how verification of field and analytic activities will be conducted.
Validation occurs after verification and ensures that the data and information used in decision making are of appropriate quality for their intended use. Validation guidance and documents may be attached or referenced in the QAPP and should address the following:
the process that will be used to validate sample collection, handling, field analysis, and analytical laboratory project data;
the specific validation process that will be used for each analytical group, matrix, and concentration level;
the procedures and criteria used to validate data information operations, which may include: the electronic or manual transfer, entry, use, and reporting of data for computer models, algorithms, and databases; correlation studies; and data plotting;
how validation of field sampling, handling, and analysis activities will be documented such as QC signatures in field logs and QC checklists;
the person, identified by title, responsible for data validation.
In some cases the degree of validation is dependent on the organization, program, contract, or grantee and consultation with the organization's QA Manager is advised.
The data validation process is constrained by a number of factors, including contract requirements and client/management expectations. However, in most cases there remains an opportunity for the data validator or project team to exercise professional judgment in order to maximize the benefits of the data validation process. For example, Contract Laboratory Program, National Functional Guidelines for Organic Data Review (EPA 1999) includes a section titled "Overall Assessment," describes the use of a brief narrative in which the reviewer expresses concerns and comments on the quality of the data. If the reviewer has access to the project's MQOs, then the reviewer is encouraged to comment on the usability of the data.
2.5.2	Quantitative and Qualitative Evaluations of Usability
Using information provided by the data validator, the project team considers whether data meet project quality objectives or DQOs. The usability assessment is the final step of data review and can be performed only on data of known quality (i.e., verified and validated data).
Note that this differs from comparisons to acceptance criteria in that here it is the project level objectives (DQOs) that are being considered, previously only the specific MQOs were considered. A typical example would be to see if the estimated mean level of contaminant (one of the project objectives) can be estimated to within 10 ppb (another project level objective) with 95% certainty (the final project objective). To accomplish this step of data review, the project team should do the following and describe the results in the QAPP:
summarize the usability assessment process and all usability assessment procedures, including interim steps, peer reviews, and any statistics, equations, and computer algorithms that will be used to assess data;
describe the documentation that will be generated during usability assessment;
identify the personnel (by title and organizational affiliation) responsible for performing the usability assessment;
describe how usability assessment results will be presented so that they identify trends, relationships (correlations), anomalies; and
describe the evaluative procedures used to assess overall measurement error associated with the project and include the MQOs described in Section 2.2.6.
 Statistical comparisons are very useful and are similar to those used with MQOs discussed in Section 2.2.6. EPA 2006b may be consulted for examples and instructions of how to perform statistical tests.  For the success of any project, team members should be encouraged to contribute freely in the overall evaluation of the project's data.
2.5.3	Potential Limitations on Data Interpretation
The QAPP should address the action to be taken when the MQOs or project-required measurement performance criteria are not achieved, if for various reasons, the project data are not usable to adequately address environmental questions (i.e., to determine if regulatory or project action limits have been exceeded), or the data cannot be used to support project decision making.
In addition to potential data usability issues, there are other considerations that might limit the use of the data within or beyond the original project. Issues that might impact interpretation of data include (but are not limited to):
unique qualities of the sample matrix;
inadequate metadata documentation (e.g., temperature, or barometric pressure);
collection methods that alter the composition of the samples;
use of analytical methods which may nullify comparisons to standard analyses;
differences in implied sample support (e.g., different amount of media collected per sample, or samples representing different sized areas);
extent to which the planned statistical data collection method was compromised using judgmental samples; and
use of a judgmental sampling scheme (which negates the ability to extrapolate results to a larger population of interest).
The QAPP should state that any potential limitations of data interpretation should be documented in the metadata records associated with these data and included in the final project report.
2.5.4	Reconciliation with Project Requirements
The QAPP should describe how results obtained from the project or task will be reconciled with the requirements defined by the data user or decision maker. The proposed methods to analyze the data to identify possible anomalies or departures from assumptions established in the planning phase of data collection should be included in the QAPP. These should be determined prior to data collection to ensure they are driven by project needs. The QAPP should describe how reconciliation with user requirements will be documented, issues resolved, and how limitations on the use of the data will be reported. Specifically, how sample collection, handling, and analysis procedures will be validated against the measurement performance criteria specified in Section 2.2.6 should be described. Also, the evaluative procedures to be used in validation to assess overall measurement error associated with the project, including attainment of the objectives defined in Section 2.2.6 should be described.
Considering how the data will be used to answer the main study questions is an important step in project planning, and should be documented in the QAPP. Without consideration of this, it is possible to gather data according to plan and of high quality, but cannot be used to answer the questions at hand because no statistical methods are available to support the project needs. The QAPP should describe how the data will be summarized or analyzed (e.g., qualitative analysis, and descriptive or inferential statistics) to meet project objectives. If descriptive statistics are proposed, state, if appropriate, what tables, plots, and/or statistics (e.g., mean, median, standard error, or minimum and maximum values) will be used to summarize the data. If an inferential method is proposed, indicate whether the method will be a hypothesis test, confidence interval, or confidence limit, and describe how the method will be performed. Specifically, the QAPP should briefly discuss the five steps of the data quality assessment (DQA) process (see EPA 2006b) which include:
a review of the project's objectives to assure that they are still applicable and a review of the sampling design and data collection documentation for consistency with the project objectives noting any potential discrepancies;
a review of preliminary data from QA reports (when possible) for the validation of data, calculation of basic statistics, and generation of graphs of the data to probe the structure of the data and to identify patterns, relationships, or potential anomalies;
selection of the appropriate statistical procedures for summarizing and analyzing the data, including treatment of censored values (non-detects), based on the review of the performance and acceptance criteria associated with the project's objectives, the sampling design, and the preliminary data review, and identification of the key underlying assumptions associated with the statistical tests;
verification that those underlying assumptions most likely hold, or whether departures could be acceptable, given the actual data and other information about the study; and
determination of conclusions from the data by performing the calculations pertinent to the statistical test, and documentation of the conclusions to be drawn as a result of these calculations. If the design is to be used again, there should be an evaluation of the performance of the sampling design.
 Although the steps of the DQA will not be implemented until data collection is complete, the final DQA will have comparisons to both qualitative and quantitative criteria.  These criteria should be documented in the QAPP.  Quantitative (statistical) evaluations of the data can provide strong evidence in support of, or to refute, project hypotheses or prior assumptions. Even if only qualitative comparisons are planned, semi-quantitative methods such as graphical methods should be planned for in the QAPP.
Statistical significance resulting from quantitative comparisons to planned criteria relies on the concept founded on evaluation that the weight of evidence (data) supporting a hypothesis is valid. It is never possible to have perfect knowledge about a studied population, but it is possible to learn enough about it to be able to say with confidence that a particular hypothesis concerning that population cannot be true. However, one should be very careful not to allow the statistics to dictate decisions without recourse to common sense. In particular, as more and more data are collected, it becomes easier and easier to achieve statistical significance. The concern is that at some point it may be possible to determine statistical significance at levels that are not of practical significance. This can be illustrated through the following example:
      Based on operations at an industrial plant, and its waste-release permit, it is expected that the pH of water leaving the plant will be 5.9. The releases are monitored by weekly collections and each week these data are combined with all previous data and the average pH is compared to 5.9. After the first few months, the average release pH is 5.88, which is not statistically significantly different from 5.9 and the conclusion of no real difference justified. After several years have elapsed, the average release pH is 5.8996 and this is statistically significantly different from the permitted value of 5.9, but is a conclusion of a real difference justified? This is a case where having so much data allows the reviewer to identify very small differences from the expected level, but the statistically significant result may very well not have any practical significance (in this case a difference in pH of 0.0004, which is barely measurable).
While statistics provide a strong and essential tool for environmental decision making, the science of statistics is not a substitute for common sense and can lead to bad decisions if not tempered with practicality. This is particularly true with publications that tend to accept only papers that show a "statistically significant result" (often using 5% or even 1% as the level of significance). It is relatively rare that the publication editor has required the author to include a discussion of the practical meaning of the statistical significance using costs, benefits, and analysis of the magnitude of the observed effects. The QAPP should take care to specify the conditions under which statistical significance may not have practical meaning.
It is worth noting that "statistical significance" should not be the sole indicator of importance of a result or conclusion. This was discussed by the U.S. Supreme Court in Matrixx Initiatives, Inc v. Siracusano (March 22, 2011) (Ziliak 2011). While the case involved security law, the defendant (Matrixx) tried to suggest that its line of demarcation on whether to release data was based on statistical significance.  The Court disagreed unanimously. They said that the presence or absence of statistical significance is not the key factor as to whether an adverse effect is material.  Justice Sotomayor wrote the opinion noting, "A lack of statistically significant data does not mean that medical experts have no reliable basis for inferring a causal link between a drug and adverse events". Statistical significance is only part of the quantitative aspect in the interpretation of a project's results.
Although reconciliation with project requirements represents the final part of the quadripartite Plan-Do-Check-Act, it is important to emphasize the final results should be compared with the planned objectives of Section 2.2.6. Important deviation of results obtained from those planned for can greatly affect the effectiveness of the decisions or inferences made, and can allow the project team to make an inherently intuitive conclusion as to the overall validity of results.
2.5.5	Reports to Management
The QAPP should identify the frequency and distribution of reports issued to inform management (EPA or otherwise) of project status, including reports on the results of performance evaluations and system assessments, results of periodic data quality assessments, and significant QA problems, and recommended solutions. Assessment checklists, reports, requests for corrective action letters, and the corrective response letters should be included as attachments to or referenced in the QA management reports.
Periodic QA management reports are intended to ensure that managers and stakeholders are kept informed of project status and results of all QA assessments. Efficient communication of project status and problems will allow project managers to implement timely and effective corrective actions so that the data produced can meet the project's objectives.
The following issues may be included in QA management reports:
summary of project QA/QC program activities including tables, graphs, and charts;
a summary of training conducted during the project;
status of project and schedule delays;
deviations from the approved QAPP and approved amendments to the QAPP;
description and findings of TSAs and other assessments;
required corrective actions and implementation; and
limitations on the use of measurement data generated.
Although these issues listed may be addressed in QA management reports, they should also be included in the QA/QC section of the final project report. The final project report should, at a minimum, give a reconciliation of project data with project objectives, a data summary (including tables, charts, and graphs), a summary of major problems encountered and their resolution, and any additional data quality concerns together with conclusions and recommendations. Separate from reports to management are reports and publications subject to peer review, or pre-dissemination review.  Formal Agency Peer Review is discussed further in Chapter 3, Section 3.4 and also in EPA 2006c. Pre-dissemination review is discussed in Pre-Dissemination Review Guidelines (EPA 2006d).

CHAPTER 3

QAPP ELEMENTS FOR EVALUATING EXISTING DATA
3.1	OVERVIEW OF QAPP ELEMENTS FOR EVALUATING EXISTING DATA
In an increasing number of projects, recourse must be made to data other than new data produced specifically for this project.  The use of existing data (data collected outside of this project) can be due to resource or timeliness restrictions, convenience, or may be the only potentially suitable data for consideration for this project. Agency Quality Policy CIO 2106.0 mandates the use of a QAPP: this Chapter gives advice on Standards Clause 7.5, QAPP, and Annex B.3 QAPP Elements and Requirements.
Existing data are any data or information available to the project team originally collected for a purpose different from the one for which they are intended to be used within the project. They may be data collected by the same project team previously for another purpose, or they could be data from many years in the past that just happen to relate to some aspect of the current project. For some programs the data were collected for a purpose similar to the proposed project but by a different organization. Existing data (sometimes erroneously known as "secondary data", and occasionally discussed as "secondary use of data") are challenging to use because determining whether they are of appropriate quality can be difficult to ascertain.
Existing data are often used because they may be easily accessible, less resource-intensive than collecting new data, available within a short time-frame, or even collected for a similar project to the one the dataset is being considered for. Some examples include data obtained from www.data.gov; published literature, reports, and handbooks; state and local programs; outputs
from existing models; pilot studies; GIS layers; surveys; and other Government agencies.

Inspection of existing data for potential use can be greatly helped by examining the metadata that accompanies the dataset. Metadata is the information that describes the dataset and its quality criteria. The ultimate success of an environmental program or project depends on the quality of the environmental data used in decision making, and this quality depends significantly on the ability to inspect the metadata and ascertain its applicability to the project. It will be rare that the existing data set will be exactly what is needed for the proposed project, and documentation of the difference between project requirements and existing data characteristics is essential.  It is important to utilize the graded approach which allows the existing data set to be considered in its entirety rather than by individual characteristic unless the objectives of the project demand it.

Probably the most important part of the QAPP is in the planning for obtaining the data or information because an error at this stage may be difficult to rectify once the project is underway.  One of the key outputs of the systematic planning process is the construction of a conceptual model that describes the scientific and engineering process under investigation. The conceptual model is an important tool for organizing information about the current state of knowledge and understanding of the project, as well as for documenting key theoretical and practical assumptions underlying the data and information collection. Further details on systematic planning are to be found in Annex C of the Standards and EPA 2006a.

QAPPs must, at a minimum, address all elements required by the Standards and these elements are described later in more detail with guidance for how to implement them. The QAPP is invaluable in not only documenting all aspects of the project but serves as a structured resource for writing the final report on the project.

The elements arranged according to the Plan-Do-Check-Act project life cycle, are described in the following sections.

3.2	PROJECT MANAGEMENT (Plan)
Planning, the first part of the construction of a QAPP, deals mainly with the management aspects of the project. Most of the structure is simple good management practices, the only QA-related part being the establishment of data or project quality objectives and measurement performance criteria (section 3.2.6). The section offers guidance on Standards Clause 7.5, QAPP, Standards Clause 7.7, Systematic Planning, Standards Clause 7.10, Documents and Records Management, and Annex B3.3 Project Management, and Annex B3.4 Data Acquisition.
The elements in this section address the format and disposition of the QAPP, project administrative functions, project information, and goals. These elements document the backbone of the project-planning process and lay the groundwork for the more technical elements.
The QAPP must describe the project adequately and the elements that can address the basic project management and objectives of the work include:
   * title, version number, and approval/sign-off sheets;
   * document format and table of contents;
   * distribution list;
   * project organization and schedule;
   * project/problem background and description;
   * data or project quality objectives and measurement performance criteria;
   * special training requirements/certification; and
   * documentation and records requirements.

3.2.1	Title, Version, and Approval/Sign-Off

Each QAPP should include a page with the title of the project and the name of the organizations involved in various aspects of that project. The version of the QAPP should also be clearly identified along with the title. It is acceptable to create separate title pages and signature pages, as long as the document title, version number, and date appear on the signature page. The names, titles, signatures, and signature dates of those approving the plan are also placed on this page. Individuals responsible for approving the QAPP may include the organization's Technical Project Manager and QA Manager, and the EPA (or other funding agency) Project Manager and QA Manager. Their signatures indicate both their approval of the plan and commitment to follow the procedures noted. Other key personnel that may sign the plan are other QA officers, prime contractors, and subcontractors.
This approval information is typically the first page of the QAPP. Depending on the organization's administrative policy, QAPP approval could also be in a separate memorandum. The signature dates indicate the earliest date when the environmental data operations portion of the project can start (i.e., its effective date). The QA Manager will also determine if digital (electronic) signatures are acceptable for the approval of the QAPP.
In addition to the title, version number, and approval signatures, it is important to include a revision history. Each time the QAPP is revised, as approved by the QA Manager, the version number should be updated and the revision history should be amended to include a brief summary of the change and date.
3.2.2	Document Format and Table of Contents
The QAPP should be organized such that it meets the project's needs, can be reviewed efficiently, and meets the document control requirements of the QMS under which it is developed. A document control format, such as the example shown in Figure 5, may be used to support QAPP development, or a footer created for each page to show revision status.
                       Project Name/# __________________
Section # _______________________
Revision # ______________________
Date ___________________________
Page ___________ of ____________
                   Figure 5. Document control format example
The Table of Contents will generally list QAPP elements, as well as any tables, figures, reference section, and appendices necessary to the project. If the QAPP author prefers to organize the plan differently than how the elements are organized in this Handbook a table may be inserted here to cross-reference where the information for each element may be found to simplify review. Depending on the type of project, analytical research protocols, or data management procedures may be attached. If the existing data are unaccompanied by SOPs, resort to other indicators of quality may have to be made and documented. In the case where proprietary standards (for example, those of ASTM or ISO) are used, reference to the location on the relevant website may be sufficient.
3.2.3	Distribution List
The distribution list identifies all individuals who should get a copy of the approved QAPP, either in hard copy or electronic format, as well as any subsequent revisions. Key personnel responsible for project implementation and funding, and who should have the currently approved QAPP, should be listed with their project titles or positions, organization names, email addresses, and telephone numbers. Beyond the initial distribution of the QAPP to all personnel who will need access to it, the distribution list also serves as an easy reference of who needs to be alerted and provided with a revised version of the QAPP in the case that modifications are necessary. Some organizations choose to provide the distribution list on the title or approval page, others elect to include this list in the project organization section when listing key personnel and their contact information.
3.2.4	Project Organization and Schedule
It is important that roles and responsibilities are well defined prior to initiating project activities. Those individuals involved with the major aspects of the project are named in this element along with their project responsibilities. For example, the people responsible for maintaining, updating, and overseeing implementation of the QAPP would be named here. The personnel included in this element should include the lead scientists, researchers, modelers, consultants, and contact information for the external organization holding the existing data.  If actual personnel cannot be initially identified, then the position description of that person's function should be given.
An organizational chart or table can be very helpful, and should be included if appropriate. It is also helpful to indicate lines of communication among individuals or groups, and this can be shown easily on an organizational chart or an organizational network diagram. While a single individual may have more than one responsibility in a project, the project should be organized such that any person having QA responsibilities is independent of those generating and using project information. If this is not possible, an alternative method of ensuring effective QA review should be specified in the QAPP.
The level of detail included in the schedule is left to the discretion of the QAPP authors. It may be beneficial to have a very detailed and strongly stated schedule for the project to follow. In this case, there is a risk of requiring QAPP revisions if the schedule needs to be changed during the lifetime of the project unless the QAPP specifically states it will not be revised simply due to schedule delays. It is useful to include critical points in the project such as expected date of QAPP approval, sub-section start and end points, or dates modeling subroutines need to be completed.  When creating the schedule, allowance should be made for potential delays and general inefficiencies inherent in any project. If the project includes regulatory or court-mandated deadlines, these should be highlighted to ensure their importance is noted. For projects in which deadlines or milestones are not well defined, a more generalized work schedule can be formulated.
3.2.5	Project Background, Overview, and Intended Use of Data
This overview should give the reader an understanding of the problem to be solved, along with any pertinent background information for the project. It describes why the project will be done and what goals the project intends to accomplish. The general project goals stated here form the foundation for the entire study. Equally important, the development and documentation of this element will help ensure that all project team members clearly understand and agree on the underlying purpose of the project, increasing the likelihood that the project design will address and accomplish that purpose.
In addition to the general overview, include information on the background and use of previously approved QAPPs relevant to this project.  In addition to information about the project, it is good practice to include an outline of information that is currently not sufficiently developed. Clearly state who needs the information and what the intended use of this information will be. Problems that are more complex will lead to more extensive information in this section. The reader should be able to understand the importance or context of the project. The general project goals stated in this section will be refined in Section 3.2.6.
3.2.6	Data/Project Quality Objectives and Measurement Performance Criteria
In this element, the QAPP should indicate how good the data or information must be to answer the question, resolve the problem, or support the decision to the level desired for this project. For projects that use existing environmental data, acceptance or performance criteria for the data should be determined during the project planning process and documented in the QAPP.
EPA requires the use of systematic planning (see also Annex C of the two Standards) for all projects that involve the collection or use of environmental data and the QAPP documents the outcomes of this process.  When using existing data it is very important that a systematic planning process has been used as it directly influences the quality and representativeness of the data generated. EPA encourages the use of the DQO process, which may also be called project quality objectives, depending on the organization's preference. The DQOs are established from total study variability as the overall qualitative and quantitative goals of the project.
Under any systematic planning process for a project that includes environmental data collection it is desirable to define tolerable levels of uncertainty for components of the total study variability. Total study variability is due to natural (field) variability and measurement (laboratory) variability. The effects of field variability depend on the selecting of an appropriate sampling design and number of samples collected (see Appendix B for further discussion). Measurement variability depends on the specific measurement and analytical techniques (see Appendix C for further discussion).
DQOs are supported by Data Quality Indicators (DQIs), which are supported by Measurement Quality Objectives (MQOs). These DQIs are linked to field and measurement variability (see Appendix B for further discussion). MQOs are sometimes known as Measurement Performance Criteria (MPCs) depending on the protocols of the organization. See Appendix B for definitions and details of DQIs and MQOs. It is essential that the definition of what is meant by MQOs or DQOs is made clear early in this section. Some organizations use the same acronyms for different operations. For example, some organizations use DQOs in place of DQI performance measures, or use MPCs for these measures.
The traditional DQIs considered in project planning include:
precision;
bias;
representativeness;
comparability;
completeness; and
sensitivity.
While historically there has been considerable attention directed to bias, precision, and sensitivity, it is really representativeness that is probably the single most important indicator of data quality. Representativeness is a qualitative measure of the degree to which data accurately and precisely represent a characteristic of a population.
A poor sampling design with very high quality analytical laboratory analyses might give data that meet MQOs for precision, bias, sensitivity, comparability, and completeness while not providing information that is actually representative of the characteristics of the population of interest. If the existing data fail to be representative of what was intended to be sampled, no amount of statistical manipulation can make the data set valid. Great attention should be given to ensure that the existing data are truly representative of the project's population of interest. It is axiomatic that a sufficient number of samples will be needed to assure a good representation of the population has been achieved.
The QAPP should indicate how rigorous the existing data or information has to be to answer the question, resolve the problem, or support the decision to the level desired and needed for the project. Information on acceptance or performance criteria should be determined during the project planning process and documented in the QAPP.
Establishing DQOs and MQOs for exploratory projects (for example, research studies) where little or no information on the quantitative magnitude of what is expected is difficult. The use of information from similar or related studies is useful. A discussion documented in the QAPP of the consequences (or results) of using existing data with unknown MQOs is advisable.
3.2.7	Special Training Requirements and Certification
It is not likely that special training will be required when using existing data, but it is still possible (e.g., handling of Confidential Business Information, or security clearance). The QAPP should also specify how this information will be documented, where the records will be kept, and indicate who is responsible for ensuring that these special training and certificate needs are met and documented.

In some cases, reference to an outside authority or consultant with the skills and training appropriate for the evaluation of a specific scientific, engineering, modeling, or other technical area may be necessary. The name of the consultant together with a summary of their field of expertise should be documented in the QAPP.
3.2.8	Documentation and Records Requirements
The QAPP should describe the process and responsibilities for ensuring that project personnel will receive the most recently approved project documents such as the QAPP, SOPs, and other documents used throughout the project operation. QAPP authors should discuss how these documents will be updated and how information regarding updates will be communicated.
The QAPP should indicate where all project documents such as the QAPP or final report and records such as statutory requests and training records will be stored and for how long. In some cases retention, access to, and final disposition of some records may be regulated. In those cases, this element should address and comply with all relevant regulations.

3.3	DATA ACQUISITION (Do)
This is the heart of a QAPP as it documents the requirements for dataset identification and access, the establishment of acceptance criteria, information on how the data were collected, and how the integrity of the data are assured. This section offers advice on the Standards Clause 2.1, Scope, Clause 7.7, Systematic Planning, and Annex B3.4, Data Acquisition.
3.3.1	Proposed Data Source Originator and Publication Information
Knowing who originated (initially collected, who generated, or was responsible for) the existing data can be very informative. Data originators might be EPA, states, tribes, municipalities, potentially responsible parties (PRPs), or activist groups, for example. Confidence in the data may, to some small extent, be judged according to the data originator. That is not to say that data collected by states, for example, should always be accepted, or that data collected by PRPs, for example, should always be excluded. However, knowing who the data originator was may offer some information as to the purpose of the data collection and reflect on the possible impartiality of the lead author or source of the data.
Whether data are private or publicly accessible, and if they have ever been published, may also be helpful to know. Conversely, if the data are identified without any understanding of who originally collected them or for what purpose, that would be cause for very careful consideration of their appropriateness for use in the project.
The QAPP should discuss how the data source originators will be identified and documented, whether or not the data have previously been made public, and how any decisions regarding the use of data will be dependent on this information. The rationale for the selection of a particular database in preference to others should be discussed with respect to the acceptance criteria established for this project (see Section 3.3.3). If one dataset is considered "higher quality" than another, then the rationale for this decision must be documented in the QAPP.
In some cases, the site where the dataset can be found is of importance when evaluating the suitability of the data for use. As an example, data available from a Government site would have been subjected to some set of quality assurance protocols (see section 3.4.1), whereas a site that does not practice or document their quality assurance protocols would be of lesser possible worth.
3.3.2	Data Format and Accessibility
The QAPP should describe, to the extent possible, how the existing data will be accessed and what format they are in. Data are stored in various ways ranging from national relational databases where access is relatively easy, to stacks of documents in the back corner of a dusty warehouse. Any knowledge regarding data accessibility and format should be described in the QAPP, and if that is unknown, how it will be ascertained may be included. Procedures for transferring the data to a format usable by the project team should be discussed if relevant. For a further discussion see Brilis et al. 2004.
3.3.3	Establishment of Acceptance Criteria
Section 3.2.6 discussed the need to document DQOs that apply to the project as a whole. Acceptance criteria are the specific quality needs for existing data. In many cases these may be universal (e.g., the same requirements may be applicable for all existing data brought into the project). There may also be cases where the acceptance criteria for a particular set of existing data have slightly different acceptance criteria than for other data brought into the project (e.g., the compiled data in full should be representative of the whole watershed, but this particular data set is meant to be representative of only the area at the mouth of one specific creek). The QAPP should document the acceptance criteria for all existing data that will be included in the project.
The use of assumptions to qualify the existing data set as adequate for use should be carefully documented as these can have an immense impact on the validity of the final project's results if unknown bias is present. The use of assumptions such as "presumed to be valid", or "assume this data can be used for this project" is severely discouraged and effort should be made to establish objective acceptance criteria.
It must be acknowledged that when considering existing data for potential use, quantitative acceptance criteria may be difficult to construct. However, some concept of what the acceptable range of acceptance criteria needs to be developed by the project team. Reliance on broad statements such as "we'll use the best we can find" or "whatever is in the literature" should be discouraged.  If the quantitative criteria are poorly defined, then an iterative approach may be appropriate where the characteristics of one existing data set are compared to those of another until the "best" is selected. It is advised that the acceptance criteria be phrased in a comparative fashion by selecting an important DQI and then addressing the MQO for that indicator. For example, "the dataset having the most precise estimate of variance will be selected", the desired indicator, precision, is the acceptance criterion for the different datasets under consideration.
When possible, the existing data should be compared to the MQOs (or MPCs depending on the nomenclature of the project's organization) related to the objective of the project through examination of the metadata that accompanies each project. The principal DQIs (precision, bias, representativeness, comparability, completeness, and sensitivity) are especially useful in establishing an overall set of descriptors of evaluators of existing datasets.  Other contributory indicators such as age of data, temporal representativeness, technological obsolesce should be developed if relevant and documented here. When using data from a variety of sources or sampling events, it is important to be sure that the data are in harmony (e.g., similar sample collection design, sample collection methodology, sample preparation, analysis, and reporting) in the ways useful to the project's objectives. When an existing dataset lacks information on MQOs, careful attention should be given to the overall DQOs of this dataset, because the set of MQOs directly affect DQOs. A very "tight" achievement of the DQOs would be a good augury of the quality of the dataset, a "looser" achievement, less so. In general, the QAPP should foster a "common sense" approach in the absence of the possibility of setting quantitative objectives and consequent reliance on qualitative objectives. When metadata are absent, recourse to statistical analysis or comparisons to similar studies may be necessary and then documented.
In some circumstances, DQOs may be stated in qualitative terms having definitive points of reference.
For example, "The existing data qualifies for the `A' rating with respect to the rating system described in Section 4.4.2 of the Procedures for Preparing Emission Factor Documents (EPA-454/R-95-015)". Although it is stated in qualitative terms, this DQO is measurable using specific acceptance criteria that are presented in the cited document.  Simplistic descriptors such as "very good" or "acceptable for this project" are to be discouraged unless these are specifically defined in the QAPP.
Qualitative acceptance criteria (for example, representativeness, or comparability) may be difficult to describe, but need to be couched in terms that clearly indicate why one set of existing data may be preferred to another.  For example, "the project's objectives require the cohort of data be exposed to defined environmental conditions for a period not exceeding 5 years", the selected indicator, comparability, being linked to the objectives of the project.
3.3.4	Sample Data Collection Methodology
The QAPP should give great attention as to how the existing data set has been collected with particular reference to whether a statistical (probabilistic) or judgmental data collection scheme was used. The inference or conclusions from the data hinge on this distinction.
There can be a tremendous problem if judgmentally sampled data are misclassified as random (of some description) data as the probabilistic statements generated are unlikely to be valid. The same is true if data consisting of composite samples are regarded the same as if they were random samples.  The problem is compounded when data from different collection or analysis are compared or evaluated against acceptance criteria.
For example, are all data from the top six inches of soil, or are some of the data from deeper intervals? Were all data collected according to statistically rigorous sampling designs such that inferences to the greater population can be made, or were some of the data collected based on best professional judgment? Do all data represent the same amount of environmental media, or are some samples simple grabs while others are composite samples? These and many other similar questions shed light on the appropriateness of combining data and using them for a common purpose. The QAPP should identify and document what is important to the project team relating to sample data collection methodology.
3.3.5	Quality Program and Quality Assurance Procedures Used by Data Originator
If possible, the Quality Management Plan (QMP) under which the existing data were collected, and the QA procedures that were used during data collection should be documented in the QAPP. At a minimum, the quality program should be referenced, and some documentation such as a QMP or text extracted from the QMP may be needed as an attachment to the QAPP to provide evidence that the data quality are expected to meet the project's objectives.
When the quality program under which the data were collected is unknown, discussion of how the project will deal with unknown data quality should be documented to avoid unacceptable uncertainties in project decisions that are based on existing data. If the quality program under which the data were collected is unknown when the QAPP is written, but is later identified, it should either be included in a revision to the QAPP or documented in the final report.
It is, however, rare that existing data have the QC information readily at hand although it is quite possible the QC procedures would be documented. The project team should carefully consider the importance of each type of information and determine how its absence will impact the project if this information is not available.  Data from EPA should have this information as part of the original QAPP as QC procedures are an integral part of the QAPP, but is less likely for other organizations. Effort should be made to ascertain the status of the data generator's QMP as it is possible that QC procedures would have been referenced as part of the analytical SOW, thus giving more credence to the reliability of the data.
Regardless of whether the quality program under which the data were collected is known, this section should include documentation of the project's data quality requirements along with statements of how the data quality will be assessed.
The overriding objective in considering the Quality Program under which the existing data was collected is ascertaining the integrity of the data. It is important to ask the question "Can I trust these data?" together with the follow-up question "What leads me to believe these data are reliable?"  Documentation of qualitative (and quantitative if possible) aspects of the dataset that answer these questions is essential in ensuring the dependability of these data.
3.3.6	Documentation of Sample Quality Assurance Procedures
Section 3.3.5 discussed documentation of the QA procedures used when the existing data were originally collected. This section is for QA procedures to be used as existing data are brought into the project. For example, if the existing data are available only in pdf files and need to be converted to spreadsheets for analyses, the procedures used to assure that errors in translation are minimized should be documented here. Some datasets may be subject to regulations such as the special rules governing certain information (40 CFR 3) and may have the needed information. In general, the QAPP should specify what procedures will be used to ensure that existing data maintain their original integrity and quality as they are migrated into current project files.
For existing data to be used in the project, the QAPP should ideally identify the information to be included in the project data files (project record) and its format, if possible.
3.3.7	Data Management Requirements
How the existing data will be stored, who will be responsible for access and maintenance, and how it will be incorporated with other project data are important elements to consider.
The element is meant to provide an opportunity to document the hardware, software, and personnel requirements for managing and incorporating existing data into the project. All aspects of data management, including handling, tracking, processing, and data transfer need QC-type measures to ensure the integrity and dependability of the data is not lost. This would apply equally to both manual and electronic information systems and transfers between them.

The security of the database is one QC-type measure that helps maintain the integrity of the data.  Access to the database with the ability to edit the data is a potential threat to integrity and the measures taken to prevent unauthorized or accidental editing of data should be documented.
Project teams are also encouraged to retain information about the original sources of the data in the actual database so this information is available for future analysis.

3.4	ASSESSMENTS (Check)
The third part of the Plan-Do-Check-Act paradigm considers the qualitative and quantitative aspects of QA that apply to ensuring the integrity of the existing data. The techniques used for generating new data rarely apply as existing data rarely are accompanied by detailed QC information. When making assessment on existing data attention has to be given to careful use of the graded approach that selects the level of assessment commensurate with the intended use of the data. In some cases a "judgment call" will have to be made as the existing data may not have the detail desired to enable a good assessment to be made; documentation of the rationale for making the decision to use the dataset is essential. The advice given here is drawn from the Standards Clause 7.8, Assessment of Data and Information, and Annex B3.5 Assessments.
For existing data, technical systems audits, performance audits of measurement and analytical systems, surveillance of operations, and audits of data quality usually cannot be conducted unless QC information is available.
3.4.1	Qualitative Comparisons to Acceptance Criteria
Qualitative comparisons to acceptance criteria are, almost by definition, very subjective and involve the use of expert opinion. In many cases the expert opinion is that of the project manager or principal investigator, whose judgment may be biased. Documentation of how the qualitative aspects of comparison of data to acceptance criteria have been met is essential. The two principal DQIs, comparability and representativeness, are the most important indicators to consider. The following table shows examples of various aspects of qualitative comparisons:
Table 1. Examples of Qualitative Comparisons
Example Indicator of Comparability
Example Questions and Answers for Comparison to Qualitative Criteria
Samples within datasets should be selected in a similar manner
Are the sample collection designs or methods compatible? If a probabilistic sample design has been used, ensure they are the same type (e.g. random). If stratification has been used make sure the boundaries of the strata (sub-populations) are comparable. If a complex design has been used, assistance of a statistician may be necessary.
Were the sampling locations or frequencies selected in a similar manner? If non-probabilistic (directed, or non-random) samples were taken then there needs to be a justification of why these samples are acceptable for use in the project. Caveats may be necessary as the conclusions from non-probabilistic are difficult to extrapolate to wider situations of even populations of interest. If the non-random samples are arguably similar to what would have been selected if a probabilistic sample had been used then a justification should be made. If a probabilistic sample has been augmented by non-random samples, a minor justification should be made if the augmentation is small, but a more comprehensive justification would be needed if the augmentation is relatively large.
The samples should apply to the same target population
Are they equally representative of the population of interest? Attention must be given to ensure the two sets of samples really apply to the same target population; consideration of the metadata contained in the description of how the samples were obtained is essential. It is advisable to evaluate any differences with respect to potential bias in the conclusions drawn from comparisons of the two datasets or conclusions from combining the two datasets.
Data should be temporally and spatially consistent
Were samples collected in the same sampling event? If there are discrepancies in time, justification of why these time periods are of little importance will be needed. In some cases it may be necessary to argue these time periods are essentially "typical" of the situation.
Are there temporal factors such as seasonality? This may be of importance and a justification of why this can be ignored will be needed. For example, comparing stream flow data from the winter season with that of the summer season may not make sense due to the different seasonal factors influencing stream flow.
Are the samples taken representative of the same population? This may require a discussion on what representativeness really means in context with the project's goals and objectives. There is no easy definition of representativeness and so justification using qualitative measures will be necessary.
Were the samples in comparable matrices? The efficacy of extraction of contaminates from the sample obtained from the field is very dependent on how the contaminant binds or adsorbs to the inert materials of the matrix. Comparison of data obtained from a water sample with that from sediment may be problematical depending on the ease by which the contaminates can be measured. Reference to methods handbooks may be necessary to ascertain comparability.
Datasets should contain the same set of variables of interest
Were the variables of interest reported for all datasets? Sometimes measurements on key variable are not available. If only a few readings are missing it may be possible to justify imputation methods to substitute a value using information from auxiliary variables. This is particularly true when comparing or merging multivariate datasets. It depends on what information is available from the metadata associated with the data set. For example, particle size, total organic carbon, or percent moisture may be useful for determining if the data are comparable, but these variables may not always be reported.
Units in which these variables were measured should convert to a common metric
The units of measurement should be reported for all datasets. This goes beyond a simple Imperial/Metric (ounces to grams) argument but to whether the units all convert to a common metric. For example, some results may be reported in wet weight and some in dry weight, which are not directly comparable without additional information. How this information is used becomes part of the justification.
Field collection methods should be similar

What instrument was used and which procedure was followed? It is important to note the way in which the physical samples were collected as different methods may lead to different results.  For example, documenting the actual collection method from a 6" core could be very different from that obtained from a composite sample of four 6' cores that have been combined to make a single sample. Incremental samples (often between 50 and 200 very small samples combined into a single sample) should be examined with respect to the project objectives. The QA/QC criteria used in making the incremental sample must be documented to ensure the use of this information matches the goal of the project. Inadvertent bias may result from the merging of data from differing sample collection methods and justification for use may have to be comprehensive. The same is true of comparing filtered samples with unfiltered samples, or where the sample handling conditions are markedly dissimilar.
Rules for excluding certain types of observations should be similar for all datasets
Are "outliers" treated in a similar fashion across all datasets? The exclusion of data for being `too large' or `too small' is a potentially biasing situation. The reason for the exclusion of data must be carefully examined by the potential user as incorrect exclusion can create some extreme bias. The rationale for exclusion must be carefully considered with respect to the goals and objectives of the project. If no documented rationale exists, then a justification for its use is essential.
The nomenclature should be consistent with the objectives of the project
Are the definitions used in the existing dataset consistent with those of the project? Convertible units (e.g. Imperial to Metric or g/Kg to ppb) are not the problem but definitions involving the target population and sample technique used could cause problems if too divergent.
Metadata should be adequate to understand the protocols used
Are the metadata (information about the way the actual data were obtained) adequately described? The metadata should describe how the samples (data) were collected and be comprehensive enough for a reviewer to understand the purposes and procedures involved.
Similar analytical methods or procedures should be used to analyze the samples for all datasets.
Was the same analytical method used for all sample results considered and, if not, are the analytical methods comparable? The use of routine methods simplifies the determination of comparability because all laboratories used the same standardized procedures and reporting parameters. However, when reviewing the analytical methods used to produce the data, consideration should also be given to options in methods that may have been used by the laboratory. Although the analytical method may be the same, options such as matrix or concentration level will affect results reported. For difficult matrices, different methods are sometimes used to obtain measurements of different analytes. If different methods have been used, a justification as to why the data are suitable for the project may be necessary. The same is true for the preparation methods used before the actual sample was analyzed. For example, data resulting from a total digestion preparation when compared with data from a partial digestion preparation would not result in directly comparable results.
Measuring devices used for both datasets should have approximately similar detection levels
Are the detection or quantitation levels acceptable for use in the proposed project? Combining datasets having different detection or quantitation levels can lead to difficulties in analytical interpretations. For example, data set A may have a minimum detection of 5.0ppm, but data set B a minimum detection of 2.0ppm. Justification will be necessary to describe how data occurring between 2.0 and 5.0 in data set B will be treated when both datasets are merged together.

 Qualitative evaluations of data having been through a peer review system are important when only partial information is available about a data set. In most cases existing data have been through some form of review and it follows that publication resulting from an incomplete evaluation of data used should be accompanied by a disclaimer. Given that existing data have cleared some pre-requisite before release, there are four cases to consider; data occurring in a peer-reviewed publication, data from other Government organizations, data from within the EPA, and data that have been accepted by the courts.
Peer Review for Publication
In the case of papers, articles, and discussion pieces written for external publication, the peer review procedure used is determined by the individual publication's review policy. This can range from a simple acceptance by an editor for conformance to accepted practices for that publication, to double-blind review with full documentation for extremely rigorous publications.
It is usual for most publications to employ an anonymous reviewer (referee) system (the reviewer does know the identity of the paper's author, hence single-blind review) with at least one level of review by peers competent in the same field as the author.
In many scientific publications the authors make some statement on the QC procedures used to generate the data but rarely give specifics. Additionally, they only occasionally make available the original data to the peer reviewers. A summary of the data or the data summary statistics usually appears in the published article. In many cases this is not due to the author's wish to prevent the release of data (although this could be true in the case of proprietary data or data deemed to be of a sensitive nature), but an economic one based on publication printing and distribution costs. For a publication constrained by internal policy not to exceed a certain size/distribution, this could be the over-riding factor in causing lack of data availability. Some publications make the data accessible to other interested parties for testing for internal consistency and reliability, but that it is not a universal practice.
In some instances a request to the principal author can result in the release of data in sufficient detail that comparison to a set of acceptance criteria may be made. If however, the data fall under proprietary information such as Confidential Business Information, even a request to the principal author may not result in the release of data in a useful format.
Consideration and evaluation of whether the data meet the Acceptance Criteria established by the potential user of the data must still be determined and documented. The caveat that a full investigation of data characteristics could not be done should accompany the evaluation.
Data from other Federal Organizations
When considering the use of data from another Federal agency or organization, there is a minimum set of peer review requirements established. Nearly all Federal organizations have to comply with specific peer review requirements before disseminating influential scientific information and data and make available relevant data. These requirements were published by the Office of Management and Budget (OMB) in a Peer Review Bulletin issued December, 2004 (OMB 2004), and provides the definition of influential scientific information as:
      "Scientific information the agency reasonably can determine will have or does have a clear and substantial impact on important public policies or private sector decisions. ...(T)he term "influential" should be interpreted consistently with OMB's government-wide information quality guidelines and the information quality guidelines of the agency."
The Bulletin provides detailed peer review guidelines and applies a more stringent set of requirements for "highly influential scientific assessments", which are further defined as:
      "...an evaluation of a body of scientific or technical knowledge that typically synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information."
The difference from a traditional peer review for publication is that there is public disclosure of peer reviewers' identities when dealing with highly influential scientific assessments. In addition it is required that the peer reviewers, their reports, and the response of the agency to those reports be made available to the public.
The evaluation of whether the data meet the acceptance criteria established by the potential user of the data is made easier when peer review reports can be considered with the published data and information. The decision of whether the data meet the definition of "influential" (thus triggering the peer review requirements) has to be documented and use of data not defined as influential justified.
Data from EPA
The peer review requirements for Agency data are laid out in the Peer Review Handbook (EPA 2006c). It follows the guidelines established by the OMB Bulletin and requires that influential scientific information be peer-reviewed in accordance with the handbook. The handbook provides a flowchart for conducting a peer review and includes a checklist for managers planning a peer review. The handbook notes "Peer review enhances the credibility and acceptance of the decision based on the work product" and encourages the widespread use of peer review. According to Stephen L. Johnson, EPA Administrator in 2006, "Peer review of all scientific and technical information that is intended to inform or support Agency decisions is encouraged and expected"(EPA 2006c).
Information and data generated by the Agency should be easier to evaluate against the acceptance criteria established by the potential user of the data. Informal collaboration or assistance from the data generator is often available, leading to a more comprehensive evaluation for potential use. A request for information in the form of the QAPP used to collect the data is often available from the data generator and is invaluable in documenting the suitability of the data for project use. The encouragement by the Administrator to apply peer review to a wider set of information increases the potential reliability of the data.
Data Accepted by the Courts; the Daubert Rule
When considering data for a source involving a litigated court case, reference to the Daubert Rules of evidence should be made. Daubert is one of the first of several rulings by the United States Supreme Court giving guidance on the admissibility of scientific theory or evidence (Brilis, Worthington, and Wait 2000). It is usually used in conjunction with Rule 702 of the Federal Rules of Evidence.
Scientific expert testimony (including data) submitted for testimony first has to be admitted by the trial judge, who acts as a gatekeeper in assuring that the expert testimony truly proceeds from documented scientific knowledge. The trial judge uses relevance and reliability (the Daubert factors) as well as Federal Rule 702 to make the determination to admit expert testimony as evidence.
It follows that when data submitted for evidence are considered for possible use in another context, there is some sense that the data are reliable because they meet the Daubert and 702 Rules. The data does, however, needs to be evaluated against the acceptance criteria for the proposed project because data accepted for one purpose may not be suitable for another.
It is axiomatic that basic QA principles involving the generation of the data have been achieved as it is not possible to meaningfully test hypotheses using unreliable data. It also follows that the data should be reliable because they have been peer reviewed.
The Daubert factors would also likely apply to novel situations where a standard approach to performing the operation does not exist or is incomplete in documentation. Attention should be given to the overlying charge that forms the basis for data inclusion for Agency use "to assure that Agency decisions are supported by data of adequate quantity, quality, objectivity, and utility for their intended purpose" (see Clause 4 of the Internal Standard). Note that "intended purpose" is key to the establishment of clear objectives for the project, the associated measurements, and their associated quality control criteria.
However, being accepted for evidence does not automatically imply they are acceptable for other uses. The presiding judge has acted as a "gate-keeper" in allowing these data as relevant and acceptable for consideration for this particular case. Consideration and evaluation of whether the data meet the acceptance criteria established by the potential data user must still be determined and documented.
3.4.2	Quantitative Comparisons to Acceptance Criteria
The QAPP should describe how results obtained from the project or task will be reconciled with the requirements defined by the data user or decision-maker. The proposed methods to analyze the data to identify possible anomalies or departures from assumptions established in the planning phase of data collection should be included in the QAPP. These should be determined prior to data collection to ensure they are driven by project needs. The QAPP should describe how reconciliation with user requirements will be documented, issues will be resolved, and how limitations on the use of the data will be reported to decision makers.
Appendix B contains supporting information on DQIs for which MQOs can be set as acceptance criteria for including existing data in a project. The DQIs that may be able to be quantitatively assessed are:
precision;
bias;
accuracy (the combination of precision and bias);
completeness; and
sensitivity.
For each of these DQIs, the QAPP may include a plan for assessing project data against the acceptance criteria set during project planning and documented within the QAPP. The discussion of comparison of project data to acceptance criteria may include the following aspects:
when in the project life cycle these comparisons will occur;
who will be responsible for these comparisons;
how the comparisons to MQOs will be documented; and
what action will be taken if the project data do not meet the MQOs.
 Statistical comparisons are very useful and are discussed in EPA 2006b which has examples and instructions on how to perform statistical tests.  For the success of any project, team members should be encouraged to contribute freely in the overall evaluation of the project's data.
 There are, however, many instances where comparisons to acceptance criteria for MQOs cannot be achieved.
 This is usually because the published dataset only gives the final DQOs and not the specific MQOs.  In such cases a note or caveat should accompany the description in the QAPP that this aspect of the reliability of the data could not be fully confirmed.
3.4.3	Assessments to Other Criteria
Beyond the DQIs and MQOs defined for the project, there may be other criteria against which the use of data should be assessed. Examples of other aspects that may be included in such assessments include the following:
cost or resource constraints for incorporating existing data;
the timeframe in which the data will be available;
access to the physical site of study; and
environmental justice concerns.
The timing of these types of assessments may be mentioned in the QAPP (e.g., during readiness review, after the first data are incorporated, or at the conclusion of data analyses), as well as responsibility for conducting them and the process for documenting their conclusions.
3.4.4	Interim Assessments of Data Quality
It may be appropriate to conduct interim assessments as part of an iterative sequence of determining the suitability of existing data. The depth to which this can be done depends on the acceptance criteria established for the project.  It is sometimes used to show why a certain data set should not be considered for a particular project.  Possible indicators of data quality issues include:
field sampling practices that potentially impact data quality or reliability;
anomalous events potentially influencing data quality;
extreme weather conditions; and
an intuitive sense that something isn't right with the data.
Of course, the latter indicator is subjective and should be discussed with the entire project team.  Its inclusion here is to act as a warning signal that further investigation into the background of the existing data is warranted. Interim assessments may also be combined with any results from the comparisons to acceptance criteria (Section 3.4.2).
3.4.5	Evaluation of Unconventional Measurements
Unconventional methods can be thought of as any method for which there is not an already existing standard that has been widely accepted and validated. If data or measurements from unconventional methods are used for a project, the QAPP should include scientifically defensible documentation why these procedures are appropriate for this project.
The quality of environmental data obtained via unconventional methods should clearly support the intended use for the project and, when possible, quantitative acceptance criteria for selected DQIs should be developed.
If qualitative acceptance criteria are constructed, then a clear, detailed, explanation of why these measurements are used is needed. Use of data from unconventional methods will come under increased scrutiny by reviewers, hence the need for detailed explanations.
3.4.6	Evaluation of Unconventional Monitoring Projects
As with unconventional laboratory methods, it is especially important that nontraditional monitoring methods be well documented in the QAPP. Along with full descriptions of the methods, acceptance criteria for qualitative indicators will be needed. The use of the data generated from any nontraditional methods can be difficult unless documentation of why the proposed data should be used is included in the QAPP.
3.5	REVIEW, EVALUATION OF USABILITY, AND REPORTING REQUIREMENTS (Act)
The final part of the paradigm concerns the actual data; are they valid and can they be used for the intended purpose. The evaluation of usability must include any perceived limitations on the use of the data and with the reconciliation with project requirements is a small discussion of statistical significance. This section offers guidance on Standards Clause 7.9, Data Review, Verification and Validation, and Data Usability Reporting, and Annex B3.6, Review, Evaluation of Usability, and Reporting Requirements.
The elements in this section address project checks to see if the data or product obtained will conform to the project's objectives and to estimate the effect of any deviations. These activities may be performed throughout the project to inform mid-course corrections and/or at the completion of the project.
3.5.1	Data Verification and Validation Targets and Methods
For existing data it is not common for verification and validation details to be found and recourse has to be made to the provenance of the data. If the generator of the data has a known quality system, there is some assurance that the data have been verified and validated.  This is especially true for data generated by or for the EPA as the Agency's quality system has verification and validation as an integral part of the system.  Data from organizations with less documented quality systems may give sufficient cause for concern that caveats should be attached to any conclusions reached from the data.
3.5.2	Quantitative and Qualitative Evaluations of Usability
Usability of existing data depends on the objectives of the project and the extent to which the existing data meet quantitative and qualitative acceptance criteria. All existing data have use, but the strength of the conclusions reached by analysis of the data varies directly with the representativeness of the data with respect to the project's objectives. Based on information gleaned through qualitative and quantitative comparisons to acceptance criteria, and reconciliation with project requirements, the project team considers the degree to which the data meet project quality objectives. This requires the careful discussion of where the data are inadequate or relatively weak in desired quality.  This might be as simple as a precision estimate, but could be as much as an unintended change of population coverage due to data collection methodology.
Note that this differs from comparisons to acceptance criteria in that here project level objectives  are being considered, previously only the specific MQOs were considered. A typical example would be to see if the estimated mean level of contaminant (one of the project objectives) can be estimated to within 10 ppb (another project level objective) with 95% certainty (the final project objective).
The use of statistical comparisons to quantitative acceptance criteria is encouraged, but with a warning that any degree of statistical significance must be tempered by the extent to which the existing data deviates from a truly random sample.  The statistical analysis can be used to "weigh the evidence" by making comparative conclusions instead of actual levels of significance. Data Quality Assessment: Statistical Methods for Practitioners, EPA QA/G-9S, (EPA 2006b) contains many commonly used statistical procedures and has been written for the non-statistician.
Comparisons to qualitative acceptance criteria such as representativeness and comparability are by definition, subjective, and can be affected by the unintentional bias of the project team. Qualifiers and caveats as to interpretation of the results are to be expected (see also Section 3.5.3).  It is advised that a summary of the project team's collective reasoning be included in the conclusions from the analysis of the data.
3.5.3	Potential Limitations on Data Interpretation
The QAPP should address the action to be taken when project-required MQOs (acceptance criteria) are not achieved or for other reasons existing data may not be immediately usable with respect to the project objectives. In addition to the potential data usability issues, there are other considerations that might limit the interpretation of the data such as:
   * information gaps regarding data collection methodology;
information gaps regarding QA procedures used during data collection;
use of nonstandard analysis methods;
undocumented gaps in supporting metadata;
inclusion of potentially anomalous data not documented as outliers;
sample support (e.g., different amount of media collected per sample, or samples representing different sized areas); and
the use of a judgmental sampling design.
 The QAPP should state that any potential limitations of data interpretation should be documented and included in the final project report and included in the metadata stored with the project data. The use of footnotes and separate appendices to caveat weaknesses in interpretation of data is discouraged as these tend to be omitted when project reports are sent to outside parties.  The potential limitations should be in the text of the summary report.
3.5.4	Reconciliation with Project Requirements
Considering how the data will be used to answer the main study questions is an important step in project planning, and should be documented in the QAPP.
Without consideration of this, it is possible to compile data of high quality that actually cannot be used to answer questions because no statistical methods are available to support the project needs. The QAPP should describe how the data will be summarized or analyzed (e.g., qualitative analysis, descriptive, or inferential statistics) to meet project objectives. If descriptive statistics are proposed, state what tables, plots, and/or statistics (e.g., mean, median, standard error, and minimum and maximum values) will be used to summarize the data. If an inferential method is proposed, indicate whether the method will be a hypothesis test, confidence interval, or confidence limit and describe how the method will be performed. Specifically, the QAPP should briefly discuss the five steps of the DQA process (see EPA 2006b):
a review of the project's objectives to assure that they are still applicable and a review of the sampling design and data collection documentation for consistency with the project objectives noting any potential discrepancies;
a preliminary data review of QA reports (when possible) for the validation of data, calculation of basic statistics, and generation of data graphs to learn about the data structure and to identify patterns, relationships, or potential anomalies;
selection of the appropriate statistical procedures for summarizing and analyzing the data, based on the review of the performance and acceptance criteria associated with the project's objectives, the sampling design, the preliminary data review, and identification of the key underlying assumptions associated with the statistical tests;
verification that those underlying assumptions most likely hold, or whether departures could be acceptable, given the actual data and other information about the study; and
determination of conclusions from the data by performing the calculations pertinent to the statistical test, and documentation of the conclusions to be drawn as a result of these calculations. If the design is to be used again, there should be an evaluation of the performance of the sampling design.
Although the steps of the DQA will not be implemented until data compilation is complete, planning for DQA including selecting likely statistical methods for analyzing the data and understanding the underlying assumptions of those methods is vital to do prior to data compilation to avoid missteps. Quantitative (statistical) evaluations of the data can provide strong evidence in support of (or to refute) project hypotheses.
When considering using published data for purposes other than what it was collected for, attention should be given to the practical significance otherwise misinterpretation of importance may ensue. This is particularly true when considering the use of data concerning a popular area for research or study. It is possible that the significant result was published, while all the non-significant results were deemed not suitable for publication.
Statistical significance is a concept based on the weight of evidence that a hypothesis is valid. It is never possible to have perfect knowledge about a population being studied, but it is possible to learn enough about it to be able to say with confidence that a particular hypothesis concerning that population cannot be true. However, one should be very careful not to allow statistics to dictate decisions without recourse to common sense. In particular, as more data are collected, it becomes easier to achieve statistical significance.
The concern is that at some point it may be possible to determine statistical significance at levels that are not of practical significance. This can be illustrated through the following example:
      Based on operations at an industrial plant, and their waste-release permit, it is expected that the pH of water leaving the plant will be 5.9. The releases are monitored by weekly collections and each week these data are combined with all previous data and the average pH is compared to 5.9. After the first few months, the average release pH is 5.88, which is not statistically significantly different from 5.9 and the conclusion of no real difference justified. After several years have elapsed the average release pH is 5.8996 and this is statistically significantly different from the permitted value of 5.9, but is a conclusion of a real difference justified? This is a case where having so much data allows the reviewer to identify very small differences from the expected level, but the statistically significant result may very well not have any practical significance (in this case a difference in pH of 0.0004, which is barely measurable).
While statistics provide a strong and essential tool for environmental decision making, the science of statistics is not a substitute for common sense and can lead to bad decisions if not tempered with practicality. This is particularly true with publications that tend to accept only papers that show a "statistically significant result" (often using 5% or even 1% as the level of significance). It is relatively rare that the publication editor has required the author to include a discussion of the practical meaning of the statistical significance using costs, benefits, and analysis of the magnitude of the observed effects. The QAPP should take care to specify the conditions under which statistical significance may not have practical meaning.
It is worth noting that "statistical significance" should not be the sole indicator of importance of a result or conclusion. This was discussed by the U.S. Supreme Court in Matrixx Initiatives, Inc v. Siracusano (March 22, 2011) (Ziliak 2011). While the case involved security law, the defendant (Matrixx) tried to suggest that its line of demarcation on whether to release data was based on statistical significance.  The Court disagreed unanimously. They said that the presence or absence of statistical significance is not the key factor as to whether an adverse effect is material.  Justice Sotomayor wrote the opinion noting, "A lack of statistically significant data does not mean that medical experts have no reliable basis for inferring a causal link between a drug and adverse events". Statistical significance is only part of the quantitative aspect in the interpretation of a project's results.
The QAPP should also identify data reporting requirements, including data reduction procedures specific to the project and applicable calculations and equations. In addition, data storage requirements for both hard copy and electronic data, as well as software requirements for the analysis should be specified.
3.5.5	Reports to Management
The QAPP should identify the frequency and distribution of reports issued to inform management (EPA or otherwise) of the project status, including results of periodic data-quality assessments, significant QA problems, and recommended solutions.
Periodic QA management reports are intended to ensure that managers and stakeholders are kept informed of project status and results of all QA assessments.
Efficient communication of project status and problems will allow project managers to implement timely and effective corrective actions so that data produced for the project can meet the project's objectives.
The QAPP should describe the content of each QA management report to be generated for the project, including an evaluation of measurement error (when possible) as determined from the assessments. Assessment checklists, reports, requests for corrective action letters, and the corrective response letters should be included as attachments to or referenced in the QA management reports.
The following issues may be included in QA management reports:
summary of project QA program activities including tables, charts, graphs depicting results;
status of project and schedule delays;
deviations from the approved QAPP and approved amendments to the QAPP;
required corrective actions and implementation; and
limitations on the use of measurement data generated.
Although these issues listed may be addressed in QA management reports, they should also be included in the QA section of the final project report. The final project report should, at a minimum, give a reconciliation of project data with project objectives, data summary (including tables, charts, and graphs), a summary of major problems encountered and their resolution, and any additional data quality concerns together with conclusions and recommendations. As many Agency reports are being read by an ever increasing audience, it is recommended that sufficient detail and explanation is presented in the final report to allow for an independent assessment of whether the existing data used in the project is suitable for reaching the project's objectives.

CHAPTER 4

QAPP ELEMENTS FOR DEVELOPMENT, MODIFICATION, AND USE OF MODELS
4.1	OVERVIEW OF QAPP ELEMENTS FOR MODELS
The use of models is becoming an increasingly important part of the Agency's investigation into environmental phenomena and Agency Quality Policy CIO 2106.0 mandates the use of a QAPP.  The QAPP has some elements very similar to those for QAPPs for new or existing data, but there are important differences to allow effective model development and use.  This Chapter offers advice on Standards Clause 7.6, Use of Information Technology Methods and Sources, and Annex B3.4, Data Acquisition.
This chapter focuses on QAPPs for developing, modifying, applying, and evaluating models, and for project use of data from models. The QAPP, using the elements described in this chapter, should provide the project team with a clear path for determining if use of models will achieve the project objectives or DQOs.
Probably the most important part of the QAPP is in the planning for obtaining the data or information because an error at this stage may be difficult to rectify once the project is underway.  One of the key outputs of the systematic planning process is the construction of a conceptual model that describes the scientific and engineering process under investigation. The conceptual model is an important tool for organizing information about the current state of knowledge and understanding of the project, as well as for documenting key theoretical and practical assumptions underlying the data and information collection. In some modeling projects the conceptual model is the model itself, but in others the model being investigated may be only part of the complete conceptual model. Further details on systematic planning are to be found in Annex C of the Standards and EPA 2006a.

The use of models adds many nuances to traditional EPA QAPPs. The QAPP is meant to provide a description of how the project team will ensure that the project attains a level of quality commensurate with its importance. The project team should document all aspects of the project that could impact project quality. The QAPP is not meant to be burdensome, and any sections of this chapter that do not pertain to a particular project involving the development, modification, or use of models should simply present why those sections do not. The ultimate purposes of the QAPP are for the project team to ensure that using the data, model, or both result in achieving the project objectives, and to provide documentation and defensibility for the quality of the project results.

EPA uses a wide range of probabilistic, fate and transport, and other types of models to inform decisions that support its mission of protecting human health and safeguarding the natural environment  -  air, water, and land  -  upon which life depends. These models include atmospheric and indoor air models, ground-water and surface-water models, multimedia models, chemical-equilibrium models, exposure models, toxicokinetic models, risk-assessment models, and economic models. These models range from simple to complex and may employ a combination of scientific, economic, socio-economic, or other types of data. As defined in this guidance, a model is an application tool, while a model framework is the system of governing equations.
As models become increasingly significant in decision making, it is important that the model development, modification, use and evaluation processes conform to protocols or standards that help ensure the utility, documentation, and defensibility of the models and their outputs for decision making. EPA's Council for Regulatory Environmental Modeling (CREM) is a valuable resource for modelers with Guidance on the Development, Evaluation, and Application of Environmental Models (EPA 2009b) and self-study training modules at www.epa.gov/CREM.
The following figure (Figure 6, The Modeling Process) [EPA 2009b, page 61, figure D.1.1] shows model development, evaluation, and application, and the relationship between them, including the use of data, data quality assessment and peer review. Note the order in the figure, where evaluation is central to both development and application. The development and use of mathematical and electronic models (IT products) should be described in a QAPP and meet the applicable requirements of the Internal and External Standards. Further information is provided in this chapter; the elements are described in more detail in this chapter and organized by the life cycle of the project according to the Plan-Do-Check-Act paradigm.
QAPPs should, at a minimum, address all elements required by the Internal and External Standards as detailed in this guidance. Elements can be addressed with a simple statement of why the information is not relevant, as mentioned previously, or with a cross-reference to another approved document in which the information appears. It is acceptable to add sections to a QAPP beyond those identified. For example, if a project involves heavy use of computers run in parallel to accomplish runs of complex models, the computer requirements might be added as a separate element in the QAPP.
If the project must first set objectives and define what is to be accomplished by the model, it may be most appropriate to develop the QAPP in a phased manner (see Chapter 1, Section 1.5). The first stage might include all of the project planning and project quality objectives, and the latter stages might include model development or model selection and setting a schedule for the appropriate assessments to ensure the model is operating as expected. If a phased approach is used, the project must not move beyond the activities addressed in the QAPP until the next phase of the QAPP is developed and approved.
This chapter focuses on QAPPs for modeling projects with the elements are arranged according to the Plan-Do-Check-Act project life cycle.
 4.2	PROJECT MANAGEMENT (Plan)
Planning, the first part of the construction of a QAPP, deals mainly with the management aspects of the project. Most of the structure is simple good management practices, the only QA-related part being the establishment of data or project quality objectives and measurement performance criteria (section 2.2.6). The section offers guidance on Standards Clause 7.5, QAPP, Standards Clause 7.7, Systematic Planning, Standards Clause 7.10, Documents and Records Management, Annex B3.3 Project Management, and Annex B3.4 Data Acquisition as it applies to models.
The elements in this section address the format and disposition of the QAPP, project administrative functions, project information, and goals. These elements document the backbone of the project planning process and lay the groundwork for the more technical elements.

                                                      Figure 6. The Modeling Process
The QAPP must describe the project adequately and the elements that can address the basic project management and objectives of the work include:
   * title, version number, and approval/sign-off sheets;
   * document format and table of contents;
   * distribution list;
   * project organization and schedule;
   * project/problem background and description;
   * data or project quality objectives and measurement performance criteria;
   * special training requirements/certification; and
   * documentation and records requirements.

4.2.1	Title, Version, and Approval/Sign-Off

Each QAPP should include a page with the title of the project and the name of the organizations involved in various aspects of that project. The version of the QAPP should also be clearly identified along with the title. It is acceptable to create separate title pages and signature pages, as long as the document title, version number, and date appear on the signature page. The names, titles, signatures, and signature dates of those approving the plan are also placed on this page. Individuals responsible for approving the QAPP may include the organization's Technical Project Manager and QA Manager, and the EPA Project Manager and QA Manager. Their signatures indicate both their approval of the plan and commitment to follow the procedures noted. Other key personnel that may sign the plan are upper management, other QA officers, prime contractors, and subcontractors. The QA Manager will also determine if digital (electronic) signatures are acceptable for the approval of the QAPP. This approval information is typically the first page of the QAPP. Depending on the organization's administrative policy, QAPP approval could also be in a separate memorandum. The signature dates indicate the earliest date when the environmental data operations portion of the project can start (i.e., its effective date).

In addition to the title, version number, and approval signatures, it is important to include a revision history. Each time the QAPP is revised, as approved by the QA Manager, the version number should be updated and the revision history should be amended to include a brief summary of the change and date.
4.2.2	Document Format and Table of Contents
The QAPP should be organized such that it meets the project's needs, can be reviewed efficiently, and meets the document control requirements of the QMS under which it is developed. A document control format, such as the example shown in Figure 7, may be used to support QAPP development, or a footnote created for each page to show revision status.
                       Project Name/# __________________
Section # _______________________
Revision # ______________________
Date ___________________________
Page ___________ of ____________
                   Figure 7. Document control format example
The Table of Contents will generally list QAPP elements, as well as any tables, figures, reference section, and appendices necessary to the project. If the QAPP author prefers to organize the plan differently than how the elements are organized in this Handbook a table may be inserted here to cross-reference where the information for each element may be found to simplify review. SOPs may be included as appendices to the QAPP. Depending on the type of project, analytical research protocols, or data management procedures may be attached. If SOPs or other data gathering, data analysis, or evaluation protocols are not documented in, or attached to the QAPP, they must be available through some other means. The location of the SOPs referenced in the QAPP should be accessible to both QAPP users and QAPP reviewers.
4.2.3	Distribution List
The distribution list identifies all individuals who should get a copy of the approved QAPP, either in hard copy or electronic format, as well as any subsequent revisions. Key personnel responsible for project implementation and funding, and who should have the currently approved QAPP, should be listed with their project titles or positions, organization names, email addresses, and telephone numbers. Beyond the initial distribution of the QAPP to all personnel who will need access to it, the distribution list also serves as an easy reference of who needs to be alerted and provided with a revised version of the QAPP in the case that modifications are necessary. Some organizations choose to provide the distribution list on the title or approval page, others elect to include this list in the project organization section when listing key personnel and their contact information.
4.2.4	Project Organization and Schedule
It is important that roles and responsibilities are well defined prior to initiating project activities. Those individuals involved with the major aspects of the project are named in this element along with their project responsibilities. For example, the staff responsible for maintaining, updating, and overseeing implementation of the QAPP would be named here. The personnel included in this element should include the lead scientists, researchers, modelers, consultants, QA personnel, and contact information for involved external organizations (contractors and sub-contractors). If the actual personnel cannot be initially identified, then the position description of that person's function should be given. At a later date, after personnel have been identified, amendments to the QAPP may be made (at the discretion of the QA Manager).
An organizational chart or table can be very helpful, and should be included if appropriate. It is also helpful to indicate lines of communication among individuals or groups, and this can be shown easily on an organizational chart or an organizational network diagram.
While a single individual may have more than one responsibility in a project, the project should be organized such that any person having QA responsibilities is independent of those generating and using project information. If this is not possible, an acceptable alternative method of ensuring effective QA review should be specified in the QAPP.
The level of detail included in the schedule is left to the discretion of the QAPP authors. It may be beneficial to have a very detailed and strongly stated schedule for the project to follow. This increases the a risk of requiring QAPP revisions if the schedule needs to be changed during the lifetime of the project unless the QAPP specifically indicates that such changes need not cause a formal revision of the QAPP. It is useful to include critical points in the project such as expected date of QAPP approval, sub-section start and end points, or dates modeling subroutines need to be completed.  When creating the schedule, allowance should be made for potential delays and general inefficiencies inherent in any project. If the project includes regulatory or court-mandated deadlines, these should be highlighted to ensure their importance is noted. For projects in which deadlines or milestones are not well defined, a more generalized work schedule can be formulated.
The project management elements of a QAPP for modeling activities are parallel to those for a QAPP for a project based on newly collected data or on existing data, as described in Chapters 2 and 3, respectively. These chapters can be helpful for projects combining activities.
4.2.5	Project Background, Overview, and Intended Use of the Model
For modeling projects, more effort is put into this, the planning aspect of project development, than any of the other parts of Plan-Do-Check-Act paradigm. This stems from the fact that if insufficient attention is given to planning, potentially erroneous results may occur (a variation of "garbage-in, garbage-out").
This overview should give the reader an understanding of the problem to be solved, along with any pertinent background information for the project to provide a historical and scientific perspective for future projects. It describes why the project will be done and what goals the project intends to accomplish. The general project goals stated here form the foundation for the entire study. Equally important, the development and documentation of this element will help to ensure that all project team members clearly understand and agree on the underlying purpose of the project, increasing the likelihood that the project design will address and accomplish that purpose. Included in this discussion should be a summary of any previously approved QAPPs relevant to the project or study. This should include both those specific to modeling and those involving the collection or use of data (see also section 4.2.6).
<IP_ADDRESS> QAPP Requirements for Model Development/Modification
This section expands on the demands of Standard B<IP_ADDRESS>, QAPP Requirements for Model Development/Modification and integrates specific advice with the Council for Regulatory Environmental Modeling "Guidance on the Development, Evaluation, and Application of Environmental Models".
After the specification of the project objective and type of model needed to address the problems associated with the project objective is the development of a suitable model framework.
"A model framework is a formal mathematical specification of the concepts, procedures, and behaviors underlying the system, object, or process relevant to the problem of interest, usually translated into computer software" (EPA 2009b). The structure of the model should be clearly stated especially whether it is a stochastic model or a deterministic model as this is of crucial importance in the interpretation of model output. A stochastic model (one founded on probabilistic principles involving random events or fluctuations) may result in an output where the results must be interpreted by a series of probabilistic (statistical) statements. A deterministic (sometimes called a mechanistic) model will usually not carry a probabilistic statement.
 "For mechanistic modeling of common environmental problems, one or more suitable model frameworks may exist. Many existing model frameworks in the public domain can be used in environmental assessments. Several institutions, including EPA, develop and maintain these model frameworks on an ongoing basis. Ideally, more than one model framework will meet the project needs, and the project team can select the best model for the specified problem. Sometimes no model frameworks are appropriate for the task, and the team will develop a new model framework or modify an existing framework to include the additional capabilities needed to address the project needs" (EPA 2009b).
 "During problem specification, the project team defines the regulatory or research objectives, the type and scope of model best suited to meet those objectives, the model performance and data criteria, the model's domain of applicability, and any programmatic constraints. These considerations provide the basis for developing a conceptual model, which depicts or describes the most important behaviors of the system, object, or process relevant to the problem of interest. Problem specification and the resulting conceptual model define the modeling needs that the project team can then determine whether an existing model can be used to meet those needs or whether a new model should be developed."(EPA 2009b)
Of importance to model modification is consideration of the original model origin and purpose. The assumptions used in constructing the original model may be very different than those being used now. The effectiveness of new models built on existing models using different assumptions can lead to erroneous conclusions.
For models to be developed, modified, and evaluated, objectives and performance criteria should be defined for both the inputs and outputs of the models.

Problem specification and identification of model purpose and scope

From Chapter 3.2, EPA 2009b: "Problem specification, culminating in development of the conceptual model, involves an iterative, collaborative effort among model developers, intended users, and decision-makers to specify all aspects of the problem that will inform subsequent selection or development of a model framework. (Manno et al. 2008)."

There are three important planning areas the QAPP needs to cover: objectives, important functions, and model scope.

Objectives:

   * the purpose or general objectives of the project/model, and the expected applicability or scope of the model and this includes general statements such as what questions the model is designed to answer; and
   * sufficient background information to provide a historical and scientific perspective for context setting for this particular project, and to act as a basis for future related projects.
 Important Functions:
   * identification of the key functions or outcomes the software must perform, key decisions to be made, actions that need to be taken, and an outline of the outcomes expected from the information obtained from modeling activities.
 Model Scope:
   * the spatial and temporal limits for the model including any key assumptions on model applicability to the particular and any limitations on model applicability for this modeling process.

 Model development, modification, or selection process

 For model development a clear statement of the basis for the model including its mathematical structure, how software will be used in the development of the new model, system requirements, and procedures for software version control is needed. It is essential that the QAPP addresses:

   * the theory that forms the basis for the model, the mathematical algorithms, and approaches used in executing the model computations;
   * the existing software to be used in development of a new model, how it will be used, why it was selected (subject to the requirements of CIO 2123.0), and model scope (e.g., spatial, temporal and process detail);
   * the computer hardware and operating system requirements for the software application together with the model code development and verification plans, and the most important functions that the software application must perform; and
   * how the model structure reflects all the relevant inputs to the conceptual model and the expected applicability and limitations of model use.

<IP_ADDRESS> QAPP Requirements for Model Application

This section expands on the demands of Standard B<IP_ADDRESS>, QAPP Requirements for Model Application and integrates specific advice with the Council for Regulatory Environmental Modeling "Guidance on the Development, Evaluation, and Application of Environmental Models."

"The scope (i.e., spatial, temporal and process detail) of models that can be used for a particular application can range from very simple to very complex depending on the problem specification and data availability, among other factors. When different types of models may be appropriate for solving different problems, a graded approach should be used to select or develop models that will provide the scope, rigor, and complexity appropriate to the intended use of and confidence needed in the results." (EPA 2009b)

Many of the QAPP requirements for model applications are similar to those needed for model identification, and should be noted in the QAPP. The QAPP should document the "how and why" the model will be useful in meeting the project's objectives. An effective QAPP should be such that a reviewer or interested reader should be able to understand the importance or context of the project. The QAPP has to document the specifications for model application.

Problem Selection, Model Purpose, and Model Selection

There are two key areas the QAPP needs to cover: the basis for selection of model, and purpose of model application and analysis.

Basis for Model Selection:

   * the rationale for choosing this particular model from other competing models; "In all cases, the QAPP should clearly document for the selected model, why and how the model can and will be used, the spatial extent and resolution, and the temporal extent (length of the modeling period)" (EPA 2009b).
Purpose of Model Application:
   * the research or regulatory objective of the application or the purpose for applying the model for this particular project.

Model Application

For model application the two most important areas the QAPP must define are scenarios, and results.

Scenarios:
   * it is rare that one model can fit multiple scenarios and so it important that the QAPP accurately defines the probable simulation scenarios to which this model will be applied. This documentation will help prevent misunderstandings as to what the model applies to.

Modeling Analysis and Results
   * the modeling analysis and discussion of results is very important to establishing the validity of this particular application. Documentation of these results is an essential part of model development and application.

4.2.6	Data/Project Quality Objectives and Acceptance Criteria
In many cases the dataset being considered for use is incomplete in some aspect. Such datasets are often "repaired" using imputation techniques such as nearest neighbor averaging or estimated parameter forecasting. It is essential that the QAPP should document both the extent of imputation and the methodology used.

<IP_ADDRESS> Data Requirements for Model Development or Modification
It is acknowledged that when considering existing data for potential use in a modeling project, quantitative acceptance criteria may be difficult to construct. However, some concept of what the acceptable range of acceptance criteria needs to be developed by the project team. Reliance on broad statements such as "we'll use the best we can find" or "whatever is in the literature" should be discouraged.  If the quantitative criteria are poorly defined, then an iterative approach may be appropriate where the characteristics of one existing data set are compared to those of another until the "best" is selected. It is advised that the acceptance criteria be phrased in a comparative fashion by selecting an important DQI and then addressing the MQO for that indicator. For example, "the dataset having the most precise estimate of variance will be selected", the desired indicator, precision, is the acceptance criterion for the different datasets under consideration.
Qualitative acceptance criteria (for example, representativeness, or comparability) may be difficult to describe, but need to be couched in terms that clearly indicate why one set of existing data may be preferred to another.  For example, "the project's objectives require the cohort of data be exposed to defined environmental conditions for a period not exceeding 5 years", the selected indicator, comparability, being linked to the objectives of the project.
When an existing dataset lacks information on MQOs, careful attention should be given to the overall DQOs of this dataset, because the set of MQOs chosen for the key indicators directly affect the DQOs. A very "tight" achievement of the DQOs would be a good augury of the quality of the dataset, a "looser" achievement, less so.
While historically considerable attention has been directed to the principal DQIs of precision, bias, representativeness, comparability, completeness, and sensitivity, it is really representativeness that is probably the single most important indicator of data quality. Representativeness is a qualitative measure of the degree to which data accurately and precisely represent a characteristic of a population. If the data used in the model fail to be representative of what was intended, no amount of mathematical manipulation or adjustment can make the results valid. For modelers, the DQIs of calibration technique, uncertainty analysis, sensitivity analysis, and qualitative comparisons to other studies become more important.  The QAPP has to set both quantitative and qualitative criteria with what is likely to be available from data analysis or available through qualitative comparisons.  In either case, acceptance criteria need to be documented (see also section 3.4 of this guidance).

<IP_ADDRESS> Criteria for Acceptance for Model Application
The criteria for model application are similar to those encountered for model development or modification and so the opening remarks of section <IP_ADDRESS> apply to model application as well.
In this element, the QAPP should indicate how "good" the input data and the model need be to answer the question, resolve the problem, or support the decision at the level of quality desired. The intended use of the model should be clearly stated and if used to extrapolate beyond the range of data entered into the model, the appropriate caveats should be included.
For each type of data or aspect of the model, indicate how accurate it needs to be and how to ensure the integrity of the information so that it can be relied upon and used for the intended purpose. Without making this clear, data or model outputs may be inadequate for final use, or conversely, more expensive to collect or produce than necessary because a higher standard of quality is targeted than is required for the project purposes.
The QAPP should clearly state the criteria for acceptance of the model performance or model results. Acceptance criteria are the specific limits, standards, goodness-of-fit, or other criteria on which a model will be judged as properly calibrated (e.g., the percentage difference between reference data values from the field or laboratory and predicted results from the model). This includes the types of data and other information that will be necessary to acquire in order to determine that the model is properly calibrated (e.g., field data, laboratory data, and predictions from other accepted models [see Section 4.3.1]). In addition to addressing these questions when establishing acceptance criteria, the QAPP should document the likely consequences (e.g., incorrect decision making) of selecting data that do not satisfy one or more of these areas (e.g., are non-representative or are inaccurate), as well as procedures in place to minimize the likelihood of selecting such data.
 "If the team is evaluating multiple model frameworks, it may be useful to statistically compare the performance of these competing models individually and in aggregate with observational, field, or laboratory data." (EPA 2009b) If that is the case, the QAPP should state how those comparisons will be conducted and evaluated.
More detailed documentation of data acquisition and use procedures is described in Section 4.3.
4.2.7	Special Training Requirements and Certification
Special training or certifications are sometimes necessary for project personnel. For modeling projects, this may include training in the use of certain models, software, or programming languages; or the necessity for Confidential Business Information security clearance. The QAPP should also specify how this information will be documented, where the records will be kept, and indicate who is responsible for ensuring that these special training and certificate needs are met and documented.
4.2.8	Documentation and Records Requirements
The QAPP should describe the process and responsibilities for ensuring that project personnel will receive the most recently approved project documents such as the QAPP, SOPs, and other documents used throughout the project operation. QAPP authors should discuss how these documents will be updated and how information regarding updates will be communicated.
The QAPP should indicate where all project documents such as the QAPP or final report and records such as statutory requests and training records will be stored and for how long. Include backup procedures for any data stored electronically and cite the protocols for access to, retrieval from, and photocopying of information archives. In some cases retention, access to, and final disposition of some records may be regulated. In those cases, this element should address and comply with all relevant regulations.
The QAPP should identify the information to be documented and can include:
a description of the reporting format for model development/modification, evaluation and application, and for model inputs and model outputs;
identification of other project records to be maintained, how/where the records will be stored, and the length of storage time; types of records may include:
technical reviews, model tests, data quality assessments of output data and results, along with signatures of individuals performing review and approval,
candidate model assessments for model selection, including references,
actual input used and databases used,
SOPs for model development and evaluation,
pre- and post-software development information,
spreadsheet data files containing monitoring data,
copies of modeling reports, and
model code, equations, and user's guide (including information on the location of earlier versions).
The QAPP should indicate where all project documents (e.g., QAPP or final report) and records (e.g., statutory requests or training records) will be stored and for how long. The QAPP should identify any requirements set by statute or policy together with information on access, retention, and final disposition.
4.3	DATA ACQUISITION: MODEL DEVELOPMENT, MODIFICATION, AND USE (Do)
For modeling projects there is less of a clear line between "plan" and "do" than for projects involving simply data.  This section offers advice on the Standards Clause 2.1, Scope, Standards Clause 7.7, Systematic Planning, Annex B3.4, Data Acquisition, and Annex B4.4, Development and Use of Mathematical and Electronic Models. Some of the elements listed in the "plan" part could be shifted to the "do" part depending on the intent of the modeling project.  For example, some of Data Requirements for Model Development or Modification (<IP_ADDRESS>) or Criteria for Acceptance for Model Application (<IP_ADDRESS>) could be placed here or even in the "check" section (4.4). The exact placement depends on the objectives and logistics of model development.
The elements in this section address all aspects of data production and collection to ensure that appropriate methods for sampling, measurement and analysis, data production, data handling, and QC activities are employed and documented. The following QAPP elements describe the actual methods or methodology to be used for the collection, handling, and analyses of samples, and the management (i.e., compiling and handling) of the data.
The selection of data to be used in any modeling project is critical to meaningful interpretation of the results of the project. Knowing who originated (collected or was responsible for collecting) the proposed data can be very informative. Data originators might be EPA, states, Tribes, municipalities, potentially responsible parties (PRPs), or activist groups, for example. Confidence in the data may, to some small extent, be judged according to the data originator. That is not to say that data collected by states, for example, should always be accepted, or that data collected by PRPs, for example, should always be excluded. However, knowing who the data originator was may offer some information as to the purpose of the data collection and reflect on the possible impartiality of the lead author or source of the data.