February 2003 3 11 Assessment of Detection and Quantitation Approaches Development and Implementation of Water Quality based Controls States designate water quality standards for various bodies of water within their boundaries. Each standard consists of a designated use, criteria to support that designated use, and an anti degradation policy. Examples of designated uses include public water supply, recreation, and propagation of fish and wildlife. When the water quality standard is not met, waste load allocations are developed to indicate the maximum amount of a substance that can be discharged to a particular water body without impairing the designated use. EPA and authorized states calculate water quality based effluent limits based on the waste load allocation and the variability of the substance in the wastewater discharge. The concept is to prohibit discharge of a substance beyond the level at which a designated use would be impaired. Water quality based permits generally specify the use of measurement methods promulgated at 40 CFR part 136 under the Clean Water Act Section 304( h).

A special case occurs when the water quality based effluent limit is less than the detection limit of the most sensitive analytical method. This case is addressed in Section 3.2.3 below, on compliance evaluation thresholds.

Permit Compliance Monitoring Under Clean Water Act Sections 318, 402, and 405, NPDES permits are issued to owners of facilities that discharge wastewater to waters of the United States ( coastal areas, lakes, rivers, streams, certain wetlands, etc.). Specific discharge limits are established either for individual facilities or for classes of facilities. Individual permits are established for industries with many site specific issues that determine the substances discharged, such as the pharmaceutical industry in which the specific drugs produced could influence the water quality. General permits are issued when the substances discharged do not vary widely among facilities ( e. g., coastal oil and gas extraction industry facilities). The permit limits are typically established using technology based effluent guidelines, unless the facility is discharging into a water body that does not meet its designated use or that will not meet the designated use if a technology based limit is permitted.

Detection plays a role in compliance monitoring because of concerns with measurement results at the low end of any measurement method. All measurement results are variable. At the low end of most measurement methods, there comes a point at which a particular measurement result is unacceptably likely ( a policy decision) to have come from a sample in which the substance of interest is absent ( zero concentration). Such a measurement result would be below the critical value defined by Currie ( 1995) and in common usage it would be called below detection. In practice, the reporting limit may be set equal to a critical value, detection limit, or quantitation limit. Assuming that the reporting limit is a detection limit of 1 mg/ L of oil and grease, the measurement result would be reported as   less than 1 mg/ L of oil and grease.

<IP_ADDRESS> Non Regulatory Studies and Monitoring EPA conducts a variety of non regulatory studies and monitoring activities to support its Clean Water Act programs. These activities range from long term surveys, such as the Great Lakes Water Quality Surveys that are conducted each spring and summer to monitor trends in water quality against established baselines, to short term studies that are used to establish baselines, model pollutant cycles, and guide direction for future study and policy. Examples of such studies include the National Study of Chemical Residues in Fish that was conducted in the late 1980s ( a follow up to that study is currently underway), and the Lake Michigan Mass Balance Study conducted in the early 1990s.

3 12 February 2003 Chapter 3 When designing a study or monitoring program, EPA uses information about detection and quantitation limits, along with information on the risks associated with the pollutant( s) of interest and the cost of measurement, to select an appropriate method for measuring the pollutant. Accepting all positively valued measurement results and selecting a measurement method with a detection limit lower than the level of concern for the substance being measured would provide some assurance that measurement results associated with that concentration would be positively valued. Selecting a measurement method with a quantitation limit lower than the level of concern for the substance being measured would generate measurement results that are easier to explain to the data user and the general public.

3.2.2 Descriptive versus Prescriptive Uses of Lower Limits to Measurement The literature on detection and quantitation generally assumes that these procedures are descriptive, as opposed to prescriptive. In other words, detection and quantitation studies are described as characterizing the current performance of a laboratory or laboratories using a method to measure a substance. Two possible reasons for this treatment are: ( 1) the intended audience includes laboratory staff and measurement methods developers who wish to make new methods useable to as many laboratories as possible, and ( 2) the author may have an institutional reason for not attempting to control variability and thus lower detection and quantitation limits. On the other hand, the technology based and water quality based effluent limitations programs administered by EPA s Office of Water have an institutional goal of protecting human health and the environment. Providing this protection requires that the Agency be able to measure pollutants at ever lower concentrations. Establishing stringent standards and a compliance scheme for laboratories is one way to more rapidly develop the ability to measure at these concentrations. A prescriptive strategy concerning detection and quantitation limits would be to:

  Determine the detection and quantitation limits at multiple laboratories.   Establish a detection limit and a quantitation limit for the method that is based on some performance of these laboratories. These limits could be established as the limits reported by the mean or median laboratory, or by some other criterion, such as the pooled value of the limits achieved by all laboratories, or the limits that are met by a certain percentage of the laboratories.   Use the established detection and quantitation limits as performance standards that must be demonstrated by laboratories that practice the method.

Such an approach is consistent with other performance standards included in EPA methods, such as standards for instrument calibration, recovery of spiked reference and matrix samples, etc.

The use of such an approach would help ensure that prescriptive detection and quantitation limits ( i. e., performance standards) reflect the capabilities of multiple laboratories, rather than a single state of the art research laboratory. Of course, it is possible that even when multiple laboratories are used to establish performance standards for detection and quantitation, some laboratories may not be able to achieve these standards using their current operations. However, most laboratories facing this problem should be able to achieve these standards by investing in staff training, improved equipment, a stronger quality assurance program, or higher quality maintenance and operations.

There is of course, a risk that some members of the laboratory community will not be able to meet the standard, either because they are not willing to invest the resources necessary to do so, or for other reasons. That risk should be considered when using a prescriptive approach to detection and quantitation ( i. e., establishing limits that act as performance standards). Conversely, the risk of using a descriptive approach is that it can result in detection and quantitation limits that reflect a broad community of laboratories, including those that have made little if any effort to control variability at these levels, thus raising detection and quantitation limits to a level that is higher than desired.

February 2003 3 13 Assessment of Detection and Quantitation Approaches 3.2.3 Compliance Evaluation Thresholds <IP_ADDRESS> Compliance Evaluation Thresholds When technology based effluent limitations are developed, the limits are expressed as being at or above the quantitative measurement capabilities ( e. g., the ML) of one or more analytical methods that are available to support compliance monitoring. Therefore, it is possible to monitor and evaluate permit compliance at concentrations with an accepted degree of measurement certainty.

A situation that arises frequently in addressing water quality based limits is the setting of the permit limit below the detection or quantitation limit of the most sensitive, approved analytical method. This subject was addressed in EPA's draft National Guidance for the Permitting, Monitoring, and Enforcement of Water Quality based Effluent Limitations Set Below Analytical Detection/ Quantitation Levels ( WQBEL guidance). The WQBEL guidance suggested use of the minimum level of quantitation ( ML) as the compliance evaluation threshold ( CET) when the water quality based effluent limit ( WQBEL) is below the detection or quantitation limit of the most sensitive, approved analytical method. In comments on the WQBEL guidance, the regulated industry objected to the CET, claiming that it did not include interlaboratory variability and other sources of variability. States objected to the CET, claiming that it would not allow them to be as protective as if the detection limit were used. ( This 1994 draft guidance document was never finalized due to the controversy.)

From a technical standpoint, a one sided limit that addresses false positives only, such as Currie's critical value or EPA's MDL, is the most appropriate approach for producing a CET for the situation in which the WQBEL is less than detection limit in the most sensitive analytical method because the one sided limit allows measurement to the lowest possible level while protecting a discharger from the risk of a false violation. For example, consider the situation in which 2,3,7,8 tetrachlorodibenzo p dioxin ( dioxin) is to be evaluated against the ambient water quality criterion of 13 parts per quintillion ( ppqt). The most sensitive analytical method approved at 40 CFR part 136 is EPA Method 1613, with an MDL of 4 parts per quadrillion ( ppq) and an ML of 10 ppq. The MDL is more than 300 times greater than the ambient criterion. Therefore, if dioxin is detected in the receiving water as a result of a discharge ( i. e., the measurement result is greater than the MDL of 4 ppq), there must have been an exceedance of the ambient criterion. In the WQBEL guidance, EPA suggested use of the ML because it was the point at which the measurement could be considered reliable. However, from a purely technical standpoint, the MDL ( or Currie's critical value) is most appropriate if the goal is to protect the receiving water. It is important to note, however, that because individual states are responsible for implementation and enforcement of NPDES permits, use of the MDL and ML as regulatory reporting and compliance evaluation thresholds varies among the states.

Detection and quantitation limits have been used to establish CETs and permit limits. For example, see application of the ML to establishment of permit limits in Procedure 8 of Appendix F of Water Quality Guidance for the Great Lakes System at 40 CFR part 132. However, EPA believes that the decision to establish CETs based on a detection or quantitation limit is a separate issue from the question of which detection and quantitation limit approach is most valid. The objective of the assessment described in this document is to evaluate the merits of each approach.

3.2.4 Accepting the Procedures of Voluntary Consensus Standards Bodies In February 1996, Congress enacted Public Law 104 113 ( 15 USC 3701), the National Technology Transfer and Advancement Act ( NTTAA). This act directs " federal agencies to focus upon increasing their use of ( voluntary consensus) standards whenever possible, thus reducing federal procurement and operating costs." The Act gives Federal agencies discretion to use other standards 3 14 February 2003 Chapter 3 where the use of voluntary consensus standards would be " inconsistent with applicable law or otherwise impractical."

The NTTAA is implemented by Federal agencies based on the policies described in Circular A 119 from the Office of Management and Budget ( OMB). The current version of this OMB circular was published in the Federal Register on February 19, 1998 ( 63 FR 8546).

Neither the NTTAA nor Circular A 119 require that agencies replace existing government standards with standards from a voluntary consensus standard body ( VCSB). In other words, if EPA already has standards in place for detection and quantitation approaches, EPA is not obligated by NTTAA to replace these with VCSB standards.

Circular A 119 also discusses the effect of the policy on the regulatory authorities and responsibilities of Federal agencies. The circular states that:

" This policy does not preempt or restrict agencies' authorities and responsibilities to make regulatory decisions authorized by statute. Such regulatory authorities and responsibilities include determining the level of acceptable risk; setting the level of protection; and balancing risk, cost, and availability of technology in establishing regulatory standards. However, to determine whether established regulatory limits or targets have been met, agencies should use voluntary consensus standards for test methods, sampling procedures, or protocols."

Thus, EPA is responsible for establishing the levels of risk and protection, not only for the regulatory limits applied to discharges, but also to the risks of decision errors ( e. g., false negatives or false positives) in the detection and quantitation approaches applicable under the Clean Water Act.

Finally, Circular A 119 describes two types of technical standards: performance standards and prescriptive standards. A performance standard is defined as:

" a standard ... that states requirements in terms of required results with criteria for verifying compliance but without stating the methods for achieving required results." In contrast, a prescriptive standard is one " which may specify design requirements, such as materials to be used, how a requirement is to be achieved, or how an item is to be fabricated or constructed."

Neither the NTTAA nor Circular A 119 direct agencies to favor performance standards over prescriptive standards, or vice versa. EPA believes that the current MDL procedure is a prescriptive standard, in that it specifies both the design of the MDL study and how the requirement to establish method sensitivity be achieved. There is some obvious flexibility or opportunity for judgement in employing the MDL procedure, and much of the historical debate over the utility of the MDL procedure would suggest that it may not be prescriptive enough. The alternative approaches for establishing detection and quantitation are also prescriptive standards, rather than performance standards.

One option that EPA may consider is to employ a performance based approach to establishing detection and quantitation limits in which method developers, laboratories, and others would be free to use any one of a variety of approaches to establishing these limits, including the existing MDL procedure or a VCSB standard. Thus, establishing method sensitivity could be considered a performance standard under NTTAA and Circular A 119, rather than a prescriptive standard. The fact that different approaches ( prescriptive standards) yield different answers would be immaterial if EPA evaluates the answers relative February 2003 3 15 Assessment of Detection and Quantitation Approaches to a specific decision. That evaluation should not be divorced from knowledge of the decision to be made ( e. g., the regulatory limit for a given pollutant).

3.2.5 National versus Local Standards for Measurement In accordance with the Settlement Agreement, EPA is re examining the approaches of detection and quantitation used with methods approved for use at 40 CFR part 136. The Clean Water Act authorizes states and local governments to implement permits, with the requirement that they be at least as protective ( stringent) as the national standards established by EPA. Thus, EPA must take into account the impact of any revised or new detection/ quantitation limit approaches and procedures on state and local governments, as well as on those affected by state and local requirements. EPA also is aware that some states have implemented approaches to detection and quantitation that are either specific to that state, result in lower numeric limits in discharge permits, or both. Given the ability of state and local governments to use more stringent approaches, any decision by EPA with regard to this re evaluation of detection and quantitation approaches may not have an effect on those states and local governments.

3.2.6 Cost and Implementation Issues Detection and quantitation limit procedures are typically employed by organizations that develop methods and by laboratories that use the methods. Method developers typically include governmental organizations such as EPA, NOAA, USGS, and DOE, or voluntary consensus standards bodies ( VCSBs) such as the American Public Health Association ( APHA), ASTM International, AOAC International, and ISO/ IUPAC. Method developers also may include manufacturers of instruments or supplies used in testing. Methods users generally are the laboratories performing tests to assess and assure product quality, to support regulatory compliance monitoring, or to support scientific studies.

Method development requires a more diverse set of skills than method use because such development generally demands an understanding of quality systems, statistics, and analytical technologies. Staff working for the method developer will usually include a project manager, measurement analysts, and statisticians. Method use requires a focus on obtaining reliable results in the analysis of a given sample. Staff working for laboratory typically include the manager and measurement analysts.

<IP_ADDRESS> Implementation of a Detection/ Quantitation Limit Procedure by a Method Developer The basic resources available to the method developer are time, money, and the technical skills of its staff. The fundamental decision for implementing a detection or quantitation procedure is whether that procedure is intended to characterize the performance of the method at a well performing laboratory or if it is intended to characterize the performance of the method across a group of laboratories. If the procedure is intended to characterize the performance of the method across a group of laboratories, it is also necessary to decide if there will be some way to compare the performance of individual laboratories to the group performance standard. There are serious time, cost, and skill issues with each of these decisions. Ordering these decisions from the least resource intensive to the most, they are characterizing the performance of the method: ( 1) at a well performing laboratory, ( 2) at a group of laboratories, or ( 3) at a group of laboratories with comparisons of individual laboratories. Other costs for the method developer could include planning, data management, reference laboratory services, and whether laboratories are willing to volunteer for the study or if their services must be purchased.

An independent decision is whether to assume a simple model for measurement variability and limit the number of test concentrations, iterate assuming a simple model, or to design a study of the relationship between measurement variation and the concentrations of the substances measured by the 3 16 February 2003 Chapter 3 method. This decision will greatly influence the number of samples measured in the study. If the laboratories do not volunteer for the study, then the direct cost for measuring these samples or blanks ranges from a few dollars per sample to more than $ 1,000 per sample for some analytes. Until such time as the relationship between measurement results and standard concentrations becomes well known, such studies will require the active participation of professional statisticians in design, implementation, and analysis.

<IP_ADDRESS> Implementation of a Detection/ Quantitation Limit Procedure by a Laboratory A laboratory may implement detection or quantitation procedures for its own quality control purposes, because of regulatory requirements, or as part of the study of a method by some other organization. When participating in the study of another organization, the laboratory may voluntarily accept some cost of the study for marketing purposes, professional development, or to benchmark the performance of the laboratory.

In each case, a detection or quantitation limit approach will be of little utility if it is not capable of being implemented by the laboratory. An advantage of straightforward approaches such as the EPA MDL, the ACS limit of detection, and the ISO/ IUPAC critical value is that they can, in principle, be understood by analysts expected to use the approach. Likewise, the procedures described for implementing the MDL approach are straightforward and have been implemented by thousands of laboratories. In contrast, the ASTM IDE and IQE procedures are highly complex and, as a consequence, are beyond the capability of nearly all environmental testing laboratories.

Another disadvantage of highly complex procedures is that they are usually more costly to implement than simple procedures. As noted in Section 3.1.5, method performance generally improves over time, and EPA believes that a detection and quantitation limit approach should be supported by procedures that will allow individual laboratories and other organizations to affordably characterize such improvement. A significant shortcoming of interlaboratory procedures is that small laboratories that develop new techniques or modify existing techniques to achieve improved measurement sensitivity would have to rely on, and perhaps even pay, other laboratories to demonstrate the sensitivity of their procedures. Such a limitation has the effect of hindering method development and improvement.

3.2.7 Use of a pair of related detection and quantitation procedures in all Clean Water Act applications.

In Section 3.2.1, we discussed several different applications for detection and quantitation limits under the Clean Water Act. To review, these applications are:

  Method development and promulgation,   Method performance verification at a laboratory,   Technology based effluent guidelines development,   Water quality based effluent limits development,   Permit compliance monitoring, and   Non regulatory studies and monitoring.

Although EPA could develop a separate detection and quantitation approach for each of these applications and attempt to define and evaluate each of these approaches in our re examination of detection and quantitation approaches, the resulting matrix of applications and approaches would cause confusion for regulators, permittees, and the laboratory community. Further, when proposed, each element of the matrix of approaches and applications would, individually, be subject to contention and second guessing, and it is likely that the outcome would be nearly the same as if a single pair of February 2003 3 17 Assessment of Detection and Quantitation Approaches approaches is selected. EPA prefers to avoid this confusion by using a single pair of related detection and quantitation procedures to meet any the needs of all Clean Water Act applications.

3.2.8 Alternative Procedures One of the peer reviewers who evaluated a draft version of this assessment document noted that:

" EPA has stated in the TSD that one primary procedure is needed for clarity and to avoid confusion among stakeholders. If alternate procedures are needed, the EPA Clean Air Act system of reference and equivalent methods has worked well, and could be a model for EPA to follow under the Clean Water Act." ( Cooke, 2002)

The system of reference methods used under the Clean Air Act is similar to the existing " alternate test procedure" ( ATP) program for analytical methods currently used within the Office of Water. The difference between the ATP program and the case of the procedures for establishing detection and quantitation limits is that in an ATP program, the goal is clear and agreed upon, whereas there remain fundamental theoretical issues surrounding detection and quantitation.

For example, when a test procedure is developed for use in the Clean Air Act or Clean Water Act programs, the reference method is designed to measure Analyte X, in Matrix Y, at some concentration related to a regulatory need ( i. e., a compliance limit). Alternative procedures may be capable of making measurements of Analyte X in Matrix Y, at the level of concern, using completely different instrumentation. Thus, the demonstration of equivalency between the reference method and a possible alternative method is judged using a metric that consists of Analyte X, Matrix Y, and the level of concern, as well as other aspects of method performance.

In contrast, the primary differences between the EPA MDL/ ML approaches and potential alternatives such as the ASTM IDE and IQE are related to the philosophical differences of how detection and quantitation limits should be derived and applied. These differences are described at length in this chapter and the rest of the Assessment Document. Therefore, EPA does not believe that a variant of existing ATP programs is likely to be an effective model for assessing other detection and quantitation procedures.

What EPA would be willing to consider is that an analytical method from a VCSB or other source may be acceptable for approval at 40 CFR Part 136 and use in Clean Water Act programs even if it employs an alternative procedure for establishing method sensitivity. For example, consider the theoretical situation of an ASTM method for the determination of an analyte regulated under the NPDES program that uses the IDE or IQE to describe method sensitivity and for which the value of the IDE or IQE was below the relevant regulatory limit. EPA would evaluate the overall performance of such a method for approval at 40 CFR Part 136, despite the fact that the method did not contain an MDL determined using the Appendix B procedure.

3.3 Statistical Issues The goal of this section is to provide a brief explanation of the key statistical issues involved in the development of detection and quantitation limits.

3 18 February 2003 Chapter 3 3.3.1 Sources of Variability Various known and unknown sources of variability will influence measurements made by a laboratory using a specific method. These sources may include random measurement error, differences in analysts, variations between different equipment manufacturers and models, variations in analytical standards, routine fluctuations in equipment performance, and variations in facility conditions ( e. g., varying levels of background contributions).

There are a number of ways in which variability can be controlled. One is a strong quality assurance ( QA) program that includes use of: 1) trained and qualified staff, 2) properly maintained equipment, 3) standards that are fresh and properly prepared and stored, 4) written standard operating procedures and methods for all sample handling, analysis, and data reduction/ reporting activities, 5) procedures for monitoring ongoing laboratory performance and 6) quality control ( QC) samples and QC acceptance criteria to ensure that the laboratory systems are in control. The EPA methods promulgated at 40 CFR part 136 require the use of qualified staff, appropriately cleaned and calibrated equipment, and properly prepared standards. Each method also provides detailed steps for performing all sampling handling and analysis activities.

Even when prescribed EPA requirements are implemented, however, it is not possible to completely eliminate all variability within or between laboratories. The potential effects of sources of variability should be considered when establishing detection and quantitation limits. Even with procedures in place to control quality and reduce variability, it should be recognized that some laboratories may achieve lower detection and quantitation limits than others. Ultimately, some laboratories may not be capable of meeting low level measurement requirements without some effort to improve operations.

One of the peer reviewers asked to evaluate EPA's reassessment pointed out that EPA should consider added sources of variability, such as matrix interferences, when developing detection and quantitation limits ( Wait, 2002). Such sources of variability are considered in establishing detection and quantitation limits for analytical methods under EPA's Clean Water Act programs because these detection and quantitation limits are established in a single laboratory study, verified in multiple single laboratories, and, where necessary, further verified in an interlaboratory study. If matrix effects surface in these studies, either procedures are put in place to overcome matrix interferences, or the detection and quantitation limits are adjusted upward to account for these matrix effects. However, it has been EPA's experience that concern over matrix effects may be somewhat overblown. For example, in EPA's interlaboratory validation studies of the 600 series wastewater methods, the recoveries of some organic analytes from some real world matrices were closer to 100% than from a reagent water matrix. This effect is thought to be attributable to dissolved solids in the real world matrix that, in effect, " salt out" the organic compounds.

What EPA has not done, and does not believe is appropriate, is to aggressively pursue matrix effects in method development or in establishing detection and quantitation limits ( i. e., EPA has not attempted to find worst case matrices in order to maximally exacerbate matrix effects). EPA has not pursued this approach because, for any method, it is possible to contrive and synthesize a sample matrix that would render the method unusable ( e. g., by saturating an sample in which organic analytes are to be determined with a wide range of polymeric materials, both water soluble and water insoluble). Rather, EPA considers the type of matrix that would be encountered in a wastewater discharge and that would be regulated under the Clean Water Act ( e. g., the effluents that are discharged from properly designed and operated secondary treatment plants).

February 2003 3 19 Assessment of Detection and Quantitation Approaches A source of variability that is not considered in any of the detection and quantitation limits is the variability that is associated with the sample itself. Detection and quantitation limits focus exclusively on the capabilities of the measurement process. However, measurements are only as reliable as the sample being measured. If the sample is not truly representative of the population from which it was collected, then the variability associated with measurements made in the region of detection or quantitation may be immaterial.

For example, EPA s Technology Innovation Office conducted a study to characterize the effects of sampling variability on measured results. In that study, results from seven discrete samples collected within a two foot distance of one another were evaluated. Each sample was analyzed for the explosive TNT on site, using a colorimetric test kit, and in a traditional laboratory using EPA SW 846 Method 8330 ( high performance liquid chromatography). Analysis of the results from these measurements indicated that 95% of the total variability was due to sampling location and only 5% was due to differences between the analytical methods. Put another way, differences in sampling location caused 19 times more uncertainty in the data results than did the choice of analytical method, over a distance of only 2 feet ( Crumbling, 2002). While EPA does not wish to diminish the importance of understanding measurement error in the region of detection and quantitation, EPA believes it is equally important to understand it in the context of the overall sampling and analysis error.

3.3.2 Censoring Measurement Results Measurement results are often reported as less than some detection, quantitation, or reporting limit ( see Section <IP_ADDRESS>, Permit Compliance Monitoring) without providing a single best estimate for the numeric result. For example, if a direct reading of the measurement results would indicate a concentration of 3 mg/ L and the reporting limit for the substance is 5 mg/ L, the laboratory may only report that the measurement result is less than 5 mg/ L. Statisticians call this process of suppression of results less than a specified amount   censoring.  Reasons for the practice of censoring relate directly to issues surrounding the development of detection and quantitation limits ( i. e., the premise that measurement results below certain low levels may not useable for certain purposes).

In order to evaluate low level variability, EPA conducted a comprehensive study of results from 1/ 10th of the MDL to concentrations into the usual quantitation range. Ten different analytical techniques were evaluated in the study ( see Appendix B, Characterizing Measurement Variability as a Function of Analyte Concentration for a Variety of Analytical Techniques). Data from this study indicate that measurement results may be generated at low concentrations which are quite variable in relation to the true concentration in the sample. While this observation has not been demonstrated with every substance measured, it is suggested by plots of data from most of the measurement techniques observed in the study. An example is Ammonia as Nitrogen by Method 350.3. Plotting measurement results versus concentrations spiked into reagent water samples ( Figure 3 1), we see the strong relationship between measurement results and spike concentrations that would be expected. However, it is difficult to see what is going on at low concentrations in a graphic that covers measurement results over several orders of magnitude.

3 20 February 2003 Chapter 3 1.2 1.0 0.8 0.6 0.4 0.2 0.0 Classicals, Method 350.3Classicals, 350.3 AMMONIA AS NITROGEN, MG/ LAMMONIA L Log 10 of Measured Concentration Measured Concentration 0.0 0.2 0.4 0.6 0.8 1.0 Spike Concentration Figure 3 1 By plotting the measurement results versus the spike concentrations on log scales ( Figure 3 2), we would expect to see an expansion of variability at low concentrations, a contraction of variability at high concentrations and points mostly plotted along a 45 ° angle to indicate that measurement results are approximately equal to the spike concentrations. However, we only see the 45 ° angle at higher concentrations. Measurement results in the lowest order of magnitude appear to have reached a plateau below which they do not go. ASTM Committee E 1 has termed the model that describes the general pattern displayed by these data as the " General Analytical Error Model."

1.00 0.50 0.10 0.05 0.001 0.005 0.010 0.050 0.100 0.500 1.000 Classicals, Method 350.3Classicals, 350.3 AMMONIA AS NITROGEN, MG/ LAMMONIA L Log 10 of Spike Concentration Figure 3 2 February 2003 3 21 Assessment of Detection and Quantitation Approaches Despite such evidence that results at low concentrations can be quite variable, some data users prefer to use the actual measurement results ( even if they are negative values), rather than to censor the results at a reporting or detection limit, because censoring data at such a limit can introduce bias into the data set. If all low values are eliminated, then the mean of the remaining data would have a positive bias. In other words, while negative or extremely low values may be considered problematic by some, they are of value to statisticians and modelers, especially when dealing with analytes that are highly toxic and/ or environmentally significant.

Some programs, such as EPA's Superfund Contract Laboratory Program, require laboratories to report the measurement result obtained from the analysis in conjunction with a qualifier that the result is below a specified detection, quantitation, or reporting level. Going back to the example in the first paragraph, the laboratory might report both a measured value of 3 mg/ L and a reporting limit of 5 mg/ L. Under certain assumptions, measurement results below the specified level could then be used to calculate averages and statistical estimates that would be superior to estimates calculated using censored data.

EPA believes that such an approach provides the greatest degree of flexibility for data users, but also believes that it should be used with care. First, data users who choose to use values reported below a detection or quantitation limit need to have a firm understanding of the limitations of those data. Second, and as noted in Section <IP_ADDRESS>, Permit Compliance Monitoring, reporting data below a detection or quantitation limit can lead to misinterpretation.

One of the peer reviewers that evaluated EPA s assessment of detection and quantitation limit approaches noted that European Union ( EU) has adopted another variant for reporting or censoring data.

" In this case, the EU has adopted EPA Method 1613B ( for analysis of dioxins and furans) as well as EPA s MDL approach. However, the EU has further specified that the MDL be used as an Upper Bound reporting limit where all non detects are found in the analysis of human or animal foodstuff. This forces laboratories to achieve levels available with modern instrumentation, otherwise, the Upper Bound reporting level is above the regulatory compliance level, and the data ( or foodstuffs) are rejected" ( Cooke, 2002).

EPA agrees that this approach, which yields a ? worst case  ( or highest possible) estimate of the pollutant concentration, can serve as a useful regulatory tool for encouraging the analytical and regulated community to pursue measurements at the lowest levels necessary to protect human and ecological health. However, EPA also cautions that this approach also should be recognized as a regulatory strategy that effectively censors measurements made below the MDL.

EPA believes that while the issue of censoring is important, it should not be a consideration when selecting a detection and quantitation limit approach. The decision to censor data is a data reporting and data use issue, rather than a detection and quantitation issue. This issue will apply regardless of what detection or quantitation limit approach is used. The EU approach reflects a similar point of view, in that it relies on the MDL as a detection approach, and also establishes this limit as the reporting level for non detects in order to encourage development of lower MDLs.

3.3.3 Outliers Outliers are extreme or aberrant measurement values that, on inspection, do not follow the characteristics of a set of data. Outliers may be generated by a number of causes, such as errors in following an analytical procedure, errors in recording numerical results, or the result of extreme random variation in a properly operating process. For example, if a new measurement method is being tested but 3 22 February 2003 Chapter 3 the laboratory fails to follow the procedure correctly with some samples, the associated measurement results may stand out as outliers. A graphic example is provided in Figure 3 3, which shows measurement results for aluminum, determined using EPA Method 1620, versus concentration. At a spike concentration of 250 : g/ L, one of the measured values is about 750 : g/ L, and visually stands out from the rest of the values. This result may turn out to be an outlier.

A common process for identifying potential outliers is to apply one or more statistical procedures for identifying values far from the mean ( average) of the data. An example of such a procedure is ASTM Practice D 2777. Because extreme values can be expected to occur on occasion, it is not necessarily appropriate to exclude them from the measurement results used to develop detection or quantitation values. As recommended in the ASTM procedure, a review of the analyst's records associated with the measurement may establish whether the extreme value was caused by failure to follow the method or by some rare event associated with the method. If the method under study was not followed, it is appropriate to exclude the measurement result from the detection or quantitation analysis. If the measurement result is a rare event associated with the method under study it may also be appropriate to exclude the measurement result from the results in the study.

Measured Concentration 1500 1000 500 0 Metals, Method 1620Metals, 1620 ALUMINUM, UG/ LALUMINUM, L 0 200 400 600 800 1000 1200 Spike Concentration Figure 3 3 Influential early work in using ranking procedures to help identify outlying laboratories in studies was conducted by Youden ( Youden, W. J. and E. H. Steiner, Statistical Manual of the Association of Official Analytical Chemists, 1975).

February 2003 3 23 Assessment of Detection and Quantitation Approaches 3.3.4 Criteria for the Selection and Appropriate Use of Statistical Models Detection and quantitation limits may be based on statistical models of the relationship between measurement variation and the concentration of a substance in the sample. Results are produced by adding varying known amounts of the substance to the sample (  spiking ), making replicate measurements at each concentration, and modeling the variability of the results as a function of concentration. This section summarizes the history of modeling variability versus concentration, considers criteria for selecting models, and discusses current practices with regard to available data.

<IP_ADDRESS> Short History of Modeling Measurement Results Over time, a number of different models have been used to estimate measurement variation. Currie ( 1968) modeled variation in radiochemical measurement methods using a procedure associated with counting large numbers of distinct objects which are appropriately modeled with the Poisson distribution. However, he relied on large sample sizes and standard normal distributions to describe all other types of measurement methods. Hubaux and Vos ( 1970) developed a procedure based on an estimated calibration relationship that uses smaller sample sizes to estimate Currie s detection and quantitation limits. Again, measurement results were assumed to follow standard normal distributions, but it was also assumed that measurement variation was constant throughout the range of interest. Similarly, Glaser et al. ( 1981) suggested that measurement variation increases linearly with concentration, but they did not provide estimators under this theory because they believed that measurement variation is usually approximately constant in the range of detection. Glaser et al. ( 1981) did suggest that, when appropriate data were available, a linear regression analysis of the relationship over the analytical range be performed. Clayton et al. ( 1987) discussed transforming the measurement results ( using logarithms or square root functions). Gibbons et al. ( 1991) suggested that measurement variability may be proportional to concentration. Rocke and Lorenzato ( 1995) proposed a model motivated by physical characteristics of measurement processes, in which measurement variability is approximately constant at low concentrations, but changes in a continuous mathematical manner to a relationship where variability increases as concentration increases.

Figure 3 4 illustrates the fundamental analytical measurement models in linear and logarithmic domains. The models are applicable to nearly all analytical measurements; we will not deal with the exceptions because they represent a small percentage of cases. As can be seen from the top two graphs, response is a linear function of concentration in both the linear and log domains. The middle two graphs and the bottom two graphs are those most pertinent to the discussion detection and quantitation.

3 24 February 2003 Chapter 3 Response vs. Concentration Response vs. Concentration Linear Domain Log Log Domain 100 100 80 10 60 1 40 0.1 20 0.01 0 0.001 0 20 40 60 80 100 0.01 0.1 1 10 100 1000 Concentration Concentration SD vs. Concentration SD vs. Concentration Linear Domain Log Log Domain 10 100 8 10 6 1 4 2 0.1 0 0.01 0 20 40 60 80 100 0.01 0.1 1 10 100 1000 Concentration Concentration RSD vs. Concentration RSD vs. Concentration Linear Domain Log Log Domain 100 1000 80 100 60 40 10 20 0 1 0 20 40 60 80 100 0.01 0.1 1 10 100 1000 Concentration Concentration Figure 3 4 <IP_ADDRESS>.1 Detection Limits Using Variability at Low Concentrations The middle two graphs in Figure 3 4 show variability versus concentration and show the model postulated by Rocke and Lorenzato. The flat ( constant) portion of the graph in the linear domain is difficult to see because it occurs near the origin, but it can be seen easily in the log domain. Most detection approaches ( e. g., Currie's critical value and detection limit; EPA's MDL; the ACS LOD) are constructed assuming that the flat ( constant) region of the variability versus concentration relationship in the graph holds true, although the graph is rarely displayed ( a horizontal line would be singularly uninteresting). Detection approaches such as Currie's critical value, detection limit, LOD, and MDL are constructed by multiplying the standard deviation in the flat region by some constant.

February 2003 3 25 RSD SD Response RSD SD Response Assessment of Detection and Quantitation Approaches Contention and differences of opinion occur in determining how to arrive at an " appropriate" standard deviation and what to do with the standard deviation when you have it. Currie's critical value and EPA's MDL use a multiple of the standard deviation in a similar manner ( a t statistic adjusted for the number of replicates used for Currie's critical value; 3.14 for 7 replicates in EPA's MDL). The IDE uses an additional upward adjustment based on a statistical tolerance limit calculation.

<IP_ADDRESS>.2 Quantitation Limits Using Standard Deviation Multiples and Models of Standard Deviation versus Concentration and RSD versus Concentration The limit of quantitation ( LOQ) advanced by Currie and the American Chemical Society's Committee on Environmental Improvement, and EPA's minimum level of quantitation ( ML) result from multiplication of the standard deviation by a factor of 10, again assuming a flat portion of the variability versus concentration graph. This factor of 10 is directed at achieving a relative standard deviation ( RSD) of 10 percent. An advantage of this approach is that a quantitation limit is produced, regardless of what the RSD turns out to be.

For example, it is known that the determination of 2,4 dinitrophenol by EPA Method 625 produces highly variable results and that 10 percent RSD cannot be achieved for this compound. Multiplying the standard deviation of replicate measurements of low level samples results in a quantitation limit that is considerably higher than the quantitation limits for other compounds analyzed by Method 625. The RSD at this quantitation limit could be 30, 50, or 70 percent. Arbitrarily limiting the quantitation limit to some value ( e. g., 30%, as with the ASTM IQE) could prohibit the use of EPA Method 625 for determination of 2,4 dinitrophenol. If 2,4 dinitrophenol were present at high concentration in a discharge, it would not be reported. Although it could be argued that a more precise method should be used for determination of 2,4 dinitrophenol, determination of pollutants by a large suite of different methods would be quite costly with little meaningful benefit. Increasing precision ( i. e., decreasing measurement error) would be critical only if the concentration at issue was near a compliance limit.

Another means of arriving at a limiting RSD is to graph RSD versus concentration, as shown in the bottom two graphs of Figure 3 3. This approach is used by the ASTM IQE. It has the advantage that a model is fit to data, rather than using a point estimate such as the Currie and ACS LOD or the EPA ML. However, this approach requires considerably more data than are necessary for approaches based on point estimates. In addition, how a model is selected can play a major role in the outcome.

<IP_ADDRESS> Criteria for Selecting Models Both statistical and graphical procedures have been proposed for selecting between models for measurement results versus spike concentrations.

Statistical Criteria While statistical criteria are available for choosing between models of similar types, the currently available criteria are not satisfactory for choosing between the wide variety of models considered for the relationship between measurement variation and spike concentration, based on EPA's studies. More technically, statistical criteria include using: ( 1) the simplest model to obtain statistical significance, ( 2) the model with the smallest estimated variability, and ( 3) the model with the smallest likelihood ratio. Given the wide variety of models considered for detection and quantitation, there are problems associated with each of these procedures. Data that obviously do not follow the model may produce statistically significant results, variability may be estimated with weights that make the various estimates incomparable, and the likelihood function may not be comparable between models.

3 26 February 2003 Chapter 3 Graphical Criteria Graphical criteria may be susceptible to some subjectivity in their application, but they are currently the best available method for choosing between models. At the most basic level, the primary graphical criteria is for the form of the model to be suggested by the available data. To consider the quality of the graphical analysis, it is useful to see if some small number of data are overly influential in determining if a model does or does not fit. Given the ability of the human eye to discern deviations from a straight line rather than from a curved line, a useful technique is to plot the data so that they will indicate a straight line if they follow the model of interest.

<IP_ADDRESS> Assessment of Current Models EPA graphed variability versus concentration data with regard to how real data from measurement methods used under the Clean Water Act would conform to a number of different models. For details of how data sets were selected and how data were collected within the data sets, see Appendix B, Characterizing Measurement Variability as a Function of Analyte Concentration for a Variety of Analytical Techniques. Four sets of composite scatter plots for all combinations of analytical technique by analyte by study were produced. These sets include:

1. Measurement versus Spike Concentration, 2. Log Measurement versus Log Spike Concentration, 3. Observed Standard Deviation versus Spike Concentration, 4. Log Standard Deviation versus Log Spike Concentration, and 5. Relative Standard Deviation ( RSD) versus Log Spike Concentration.

There are hundreds of scatter plots in each set, sorted by the source, measurement technique, and study. The first set of scatter plots can be used to evaluate how well measurement results match the spiked concentration in the water. If the assumed straight line model is true, then the relationship outlined by the plotted data will be approximately linear. These relationships are plotted using log log plots so that small deviations from the line can be easily visualized. All the graphs are contained in attachments to Appendix B of this Assessment Document.

The plot of observed standard deviations versus spike concentrations can be used to evaluate the reasonableness of the constant variation and/ or linearly increasing variability models ( Currie, 1968, Hubaux and Vos, 1970, and Glaser et al., 1981). If the constant model for standard deviation is true, there would be no apparent relationship between the standard deviation and spike concentration. If the straight line model for standard deviation is true, plots are expected to indicate an approximately linear relationship. Analogously, the standard deviation/ spike concentration versus spike concentration is expected to show a straight line relationship when variability is proportional to the spike concentration ( Gibbons et al., 1991). The log log plots of standard deviation versus spike concentration are expected to indicate if log or square root transformations may be appropriate ( Clayton et al., 1987) or to display a shape that approximates a " hockey stick" when it is appropriate to use the model proposed by Rocke and Lorenzato ( 1995). With the Rocke and Lorenzato model, variability near zero will be approximately constant, but variability will increase proportionally with concentration in the higher concentration range.

The large number of plots make it difficult to draw general conclusions. For the most part, conclusions must be considered on a case by case basis. One somewhat general observation is that measurement variability over low concentrations does not appear to fit a particular curvilinear shape, and thus may be considered to be approximately constant in this range for a large number of analytical techniques.

February 2003 3 27 Assessment of Detection and Quantitation Approaches 3.3.5 Methodology for Parameter Estimation Along with various approaches of detection and quantitation and models for measurement, a number of specific procedures have been suggested for estimating model parameters. Maximum likelihood and least squares are two generally applicable statistical methods that can be used in estimating model parameters. There are advantages and disadvantages to both that must be weighed in particular cases. A standard statistical practice for evaluating the quality of an estimation procedure is to calculate the precision and bias, usually best understood by examining a plot of residuals from a fit to a function. All else being equal, the estimation procedure with the greatest precision and least bias is preferred. In some cases, precision and bias can be calculated based on the assumptions behind the estimation procedure. In other cases, it is either necessary or convenient to estimate precision and bias using simulations. From a general theoretical perspective, the maximum likelihood estimation methodology is preferable because it generates estimates that are generally best with regard to properties of precision and bias ( especially for larger sample sizes), while also being approximately normally distributed. Unfortunately, maximum likelihood methodology sometimes can be problematic because the method requires the solution of complex equations. Least squares estimation is generally more tractable, and thus is more generally applicable, although the estimates that result may not be as desirable from a theoretical statistical perspective.

What can sometimes be overlooked in considering estimating for model fitting is that direct measurement of variation of the blank or low level concentration may be the most cost effective and least difficult method to implement. The loss in statistical efficiency in comparison to more elaborate estimation and model fitting methodology would be offset by the relative ease and lower cost.

3.3.6 False Positives and False Negatives In this section, we discuss the impact of detection, quantitation, and reporting levels on false positive measurement results and false negative measurement results. The definitions of false positives and false negatives are directly related to the concepts of critical value and detection limit used by Currie ( 1995). These terms were adapted from statistical decision theory to establish the framework for decision making with regard to detection of analytes. The critical value, as defined by Currie, is the point at which the detection decision is made. That is, measured values that are less than the critical value are judged to be " not detected." Measured values that exceed the critical value are judged to be " detected."

The critical value is defined such that when the analyte is not present in a sample, there is a small possibility that a measurement will exceed the critical value. A measurement that indicates the critical value has been exceeded is, therefore, the result of one of two circumstances: ( i) the analyte is present in the sample; or ( ii) the analyte is not present in the sample and, by chance, the measurement has exceeded the critical value. The occurrence of ( ii) is defined as the " false positive" situation. A measurement that is less than the critical value occurs when: ( iii) the analyte is not present in the sample; or ( iv) the analyte is contained in the sample but the measurement procedure fails to indicate its presence. The occurrence of ( iv) is defined as the " false negative" situation.

3 28 February 2003 Chapter 3 The following table summarizes the two possible outcomes for each decision: " detected" or " not detected."

Decision State of the Sample Analyte Present Analyte Not Present Detected Correct ( i) False Positive ( ii)

Not Detected False Negative ( iv) Correct ( iii)

As formulated by Currie, the Detection Limit is a value greater than the Critical Value that is used to evaluate the capabilities of analytical procedures. In the terminology of statistical decision theory, the Detection Limit corresponds to a true value referred to as the ? Alternative  ( see, e. g. Introduction to Mathematical Statistics, by Hogg and Craig, 5th edition, [ 1995]). The Detection Limit is not a part of the detection decision process that is applied to individual sample results. The Detection Limit is defined such that when the analyte is present in the sample at a value equal to the Detection Limit, there is a small probability that a measured value will be less than the Critical Value, and thereby result in the false negative decision of ? Not Detected.

A common error in many published discussions of false negatives in relation to detection and quantitation ( such as the ASTM IDE) is the claim that using Currie's detection limit as a reporting limit or action level will somehow ? control  false negatives. That claim is both false and counter productive. To illustrate the problem with this error, consider the scenario in which the true concentration for the substance of interest in a sample is equal to the critical value. Also assume that measurement variability is approximately normal in distribution throughout the region of concern. The critical level ( alpha level) is set to 0.01 or 1% throughout the remainder of this discussion.

If the reporting limit is set equal to the critical value, then given a large number of measurements on the sample, about half of the results will be reported as being measured above the reporting limit, and about half of the measurement results will be reported as being measured below the reporting limit. Measurement results below the reporting limit are treated as if there is no analyte in the sample. These are false negative measurements, and the false negative rate is 50%.

Now set the reporting limit to Currie's detection limit. Recall that the true concentration in the sample is equal to the critical value. Given a large number of measurements on the sample, about 1% of the measurement results will be reported as being measured above the reporting limit, and 99% of the measurement results will be reported as being measured below the reporting limit. This is antithetical to Currie's formulation of detection.

To illustrate the intent of Currie's detection limit, set the reporting limit equal to Currie's critical value, and create a sample with a true concentration equal to Currie's detection limit. Given a large number of measurements on this sample, about 99% of the measurement results will be reported as being measured above the reporting limit, and 1% of the measurement results will be reported as being measured below the reporting limit. Knowledge of Currie's detection limit can be used to determine if the measurement method meets the needs of a study. For instance, a study concerned with a wastewater treatment technology that is not expected to be effective at concentrations below 10 mg/ L may call for a relatively inexpensive measurement method capable of detecting the analyte at 10 mg/ L, rather than a more expensive measurement method capable of measuring a hundred times lower.

February 2003 3 29 Assessment of Detection and Quantitation Approaches 3.3.7 Statistical Prediction and Tolerance When we define a critical value, detection limit, or quantitation limit, different descriptive terminology will dictate differences in the numerical value of the limit. We will use a critical value as an example, but the questions motivating detection and quantitation limits can be phrased in similar fashion. Do we want a critical value that tells us how likely it is that:

1. A measurement result was produced by measuring a blank sample, 2. The next measurement result will be produced by measuring a blank sample, or 3. The next [ pick any number] of measurement results will be produced by measuring a blank sample?

In statistical terms, these three objectives may be addressed, respectively, by application of methodology for determining:

1. Percentiles; 2. Prediction intervals; and 3. Tolerance intervals.

Percentiles are fairly straight forward to interpret, i. e., they specify the percentage of a distribution that falls below a given percentile value. Prediction and tolerance intervals are, in effect, confidence intervals on percentiles and can be somewhat more difficult to understand and apply. There are many excellent textbook and literature references that present the theory and application of prediction and tolerance intervals such as Hahn and Meeker, Statistical Intervals, Wiley, 1991, and Pratt and Gibbons, Concepts of Non parametric Theory, Springer Verlag, 1981. Hahn and Meeker describe at length the different statistical intervals including their properties, applications, methodology for constructing the intervals. Pratt and Gibbons have an excellent discussion of tolerance intervals that is general in application due to the non parametric perspective, i. e., no distributional assumptions are required for the results to be valid.

One of the peer reviewers of EPA's reassessment states:

  Tolerance intervals are inappropriate for environmental monitoring. The main issues here are 1) is the true concentration greater than some specified safe of action level, with sufficient confidence, and 2) what interval of possible concentrations is consistent with one or a series of measurements, with a specified degree of confidence? Both are statements about a given sample or series of samples, and not about the hypothetical variability of future estimates. Suppose that one has a sample of 10 observations with mean concentration of 1 ppb and standard deviation of 0.5 ppb. Then the estimated 99% critical level is ( 2.326)( 0.5) = 1.2 ppb. One may choose to use a t score instead of a normal score so that the chance that a future observation will exceed this level is in fact 99%. In this case, the critical level estimate would be ( 3.250)( 0.5) = 1.6 ppb. This does actually correspond to a prediction interval for future observations from a zero concentration sample.

  If one asked instead for a 95% confidence interval for the .99 percentage point of the true distribution of measurements ( assuming normality) when the true quantity is zero, this can be calculated approximately using a chi squared distribution and covers the interval ( 0.9 ppb, 2.4 ppb). It does not, however, make sense to use 2.4 ppb as a threshold, since the chance of a future observation exceeding 2.4 ppb when the true mean concentration is 0 is about .0005, far smaller than the intended false positive limit of .01.  ( Rocke, 2002)

3 30 February 2003 Chapter 3 Another of the peer reviewers of EPA's reassessment states:

" the operational definition as taken from pp. 5 2/ 5 3 of MDL = t 0.99 ( df) S does not correspond to a confidence statement that I can interpret.... replaced, although I agree that a number of statistical quantities could be used; where the   fray  seems to be most boisterous. more careful in the use of statistical terminology. e both refer often to confidence   intervals,  when in fact the quantity of interest is a confidence limit   or tolerance limit, etc.   on some underlying parametric quantity.)...

" If we accept the TSD s argument on p. 3 25 that the practical value of tolerance limits is limited, then the MDL should be viewed as a prediction limit. an additional term as per Gibbons ( 1994, p. 98):

" Also, to reemphasize, the single most problematic issue when developing a detection limit is correction for false negatives. emphasis on LC type values such as the MDL [ when correctly calculated, as in ( 1)], as motivated by an underlying sort of practical/ environmental conservatism that essentially removes false negatives from the estimator s development. interpretation. ontinue, however, since there seems to be a fair amount of confusion on the issue in the analytical chemistry literature. from my reading of the TSD is that, in effect, we are calculating an LC, but using terminology that makes some readers think it s an LD. false negative errors are not the critical issue here, and hence that the approach is reasonable ( once correct calculations are undertaken). an effort to overcome this confusion in terminology. in reply I d suggest emphasizing that an LC calculation is a form of decision limit, not a detection limit. sers will still confuse the terms, or reverse their meaning, or not see the difference, or who knows what else? this battle is...)

" One caveat: on limit argument is acceptable, if the use of tolerance limits rather than prediction limits is in fact desired, then Gibbons  ( 1994, p. 99) presentation or an equivalent approach should be used instead to correct the MDL calculation." ( Piegorsch, 2002)

Similarly, Hahn and Meeker describe situations in which the various intervals or limits are appropriate to use. the peer reviewer, the terms ? intervals  and ? limits  are sometimes used interchangeably). also give examples of the sort of applications that are suitable for each type of This should be this is ( By the way, the TSD, and I, should be W And if so, it must contain I took from the TSD ( in § 3.3.6) an implicit I am willing to accept this I suspect the fray will c The bottom line I can accept the argument that But, the Agency should put forth ( I expect they will ask me how, and But here I suspect many u I don t know how winnable although I think the predicti ( As noted by They limit although the decision to use a particular type of limit in a given application is not determined strictly by theoretical considerations but is also a matter of judgment.

February 2003 3 31 Assessment of Detection and Quantitation Approaches Prediction intervals contain results of future samples from a previously sampled population with a specified level of confidence. Prediction limits are not estimators of parameters such as means or percentiles. For example, a prediction interval may be constructed to contain future sampling results expressed as a mean or standard deviation of a future sample or all of a certain number of individual future sampling results.

Therefore, EPA agrees with the first peer reviewer that the use of tolerance intervals in environmental monitoring is inappropriate. EPA also agrees with the second peer reviewer that there is considerable confusion in the terminology.

<IP_ADDRESS> Tolerance Intervals Tolerance intervals contain a specified proportion of a population of measured values with a given statistical confidence level. For example, we say that a proportion, P, of a population is contained within the intervals ( L1 , L2 ) with ( 1") 100% confidence. The lower and upper ends of the interval, L1 and L2 , respectively, are referred to as tolerance intervals. A tolerance interval is therefore the endpoint of an interval of random length that is determined on the basis of having a specified probability of 1" that its coverage of the population is at least equal to a specified value P. The quantity 1" is referred to as the confidence level for the interval and P is the minimum proportion of the population contained in the interval. Tolerance intervals are not estimators of values such as a mean or a percentile but rather values that are always guaranteed to be either greater than or less than the desired value at some level of statistical confidence. Pratt and Gibbons discuss this and other properties that affect the utility of tolerance intervals and create difficulties in the interpretation and application of tolerance intervals.

In effect, the determination of what, if any, interval to use is a policy decision. The choice of which kind of interval to use should consider how easy it is to estimate the interval you want under the conditions that exist. As Pratt and Gibbons point out, the interpretation of tolerance intervals ( and analogously, prediction intervals) can be problematic, especially when issues of sample size and the choice of confidence level come into play. Pratt and Gibbons cite examples where the interplay of sample size and high percentile and confidence level make tolerance intervals useless.

<IP_ADDRESS> Use of Tolerance and Prediction in Setting Detection and Quantitation Limits Statistical intervals can, and have by a number of authors, be adapted for use in setting detection and quantitation limits. The basic approach requires a functional definition of detection or quantitation that includes a statistical term or terms. An interval could then be constructed about the statistical term which could be used to assess the detection or quantitation limit, or make an adjustment to a calculated value that would result in the detection or quantitation limit. For example, most detection limit estimators are functionally dependent on an estimate of standard deviation of measurement error. A statistical interval could be constructed about the standard deviation and the length of the interval could be used to assess the detection limit. The end points of the interval could be used as the basis for an adjustment ( upward or downward) in the calculated limit.

However, the use of prediction and/ or tolerance limits in setting detection and quantitation limits is not an absolute requirement and should be evaluated in the context of specific applications and policy considerations. In practice, the effect of adjustment of detection and quantitation limits by use of prediction and tolerance intervals can be quite large, depending on the amount of data available and the choices of percentiles and confidence levels.

3 32 February 2003 Chapter 3 3.3.8 Design of Detection and Quantitation Studies The issues associated with the design of detection and quantitation studies include: how well a selection of spike concentrations can be used to differentiate between different models for the relationship between measurement results and spike concentrations, how the distance between spike concentrations can impact estimates of detection and quantitation limits, how to reduce the influence of uncontrollable factors in the measurement process ( probability design), how complete to make the design factors in terms of the physical measurement process, and how flexible to make the design factors in terms of the physical measurement process.

<IP_ADDRESS> Spike Concentrations and Modeling If a model under consideration cannot be described by the number of spike concentrations in the design, then it is not possible to tell if the model is appropriate. To take the simplest example, it is not possible to describe the slope of a line associated with linearly increasing variation from a single spike concentration. Two well spaced spike concentrations would allow you to estimate a slope, but provide you with no idea of the variability of the estimate. Three well spaced spike concentrations represent the minimum requirement for estimating the linear relationship and the variability of that relationship.

Clayton et al. ( 1987) describe the relationship between the spread of the spike concentrations, the number of spike concentrations, and the number of replicate measurements with regard to estimated variability when a linear model is used. While the specific equation used in their paper does not apply to all models, it indicates principles that do apply. Increasing the number of replicate measurements, increasing the number of spike concentrations, and reducing the spread of the spike concentrations are all expected to reduce estimated variability along with the associated detection and quantitation limits. However, one of the components of variability associated with detection and quantitation is that associated with estimating the calibration relationship. To account for this source of variation, it may be appropriate to cover the entire calibration range. On the other hand, many replicates at a high concentration may improperly weight the data in favor of high detection and quantitation estimates.

<IP_ADDRESS> Probability Design The process known as randomization is an important statistical consideration in the design and interpretation of experimental studies. Randomization involves the allocation of experimental units to factors and treatments under study according a design determined by probability. Randomization avoids bias and systematic errors that can occur in studies where randomization is not used. Randomization is discussed in classic texts such as Statistics for Experimenters, by Box, Hunter, and Hunter, Wiley, 1978.

In studies of measurement methods, randomization can be used in the process of creating spike concentration solutions and the ordering of analyses. However, randomization has practical drawbacks, particularly with regard to studies designed to establish detection or quantitation limits. For example, consider a simple study involving the analyses of samples spiked at five concentrations of the analyte of interest, with five replicates of each sample analyzed. A total of 25 analyses are required for the study, and the analyses of the samples can be organized in a 5 by 5 matrix. A random number is assigned to each block in the matrix, as a means of randomizing the order of the replicates at each concentration.

By virtue of this randomized design, a sample with a high concentration of the analyte of interest may end up being analyzed immediately prior to a sample with a very low concentration of the analyte. Unfortunately, this can lead to problems that result from the " carry over" of analyte within the instrumentation from one analysis to the next. When carry over occurs, the apparent concentration of the low concentration sample can be inflated because some of the high concentration sample l may be carried February 2003 3 33 Assessment of Detection and Quantitation Approaches into the low concentration sample 2. In the context of a study designed to establish ? how low you can go  ( i. e., establishing a detection limit), carry over of the analyte into a low concentration sample may compromise the results by inflating the result for low concentration sample 2, but not inflating the results for other low concentration samples because the randomized design did not cause them to be analyzed immediately following a high concentration sample.

Analysts are aware of the potential for carry over and generally take steps during routine analyses to minimize the chance that it will occur. Examples of steps that can minimize carry over problems include analyzing ? cleaner  samples before ? dirtier  samples, and interspersing ? blanks  between samples when possible. Obviously, the intentional segregation of low and high concentration samples defeats the purpose of the randomized design. Interspersing blanks between the samples can be effective, as well as blocking similar concentrations together and randomizing blocks. But in order to ensure that the blanks do not have other effects on the results, blanks would be needed between each sample or block analysis, and this would greatly increase the cost of the study ( e. g., 25 samples and 24 blanks would be required in case of pure randomization). Therefore, despite the statistical benefits, in practice, randomization of the sample analysis sequence can be difficult to apply in detection and quantitation limit studies.

In the Agency s studies of variability as a function of concentration discussed in Sections <IP_ADDRESS>  <IP_ADDRESS> of this document, EPA chose to use a non random design to avoid carry over problems and to limit the potential difficulties with measurements at very low concentrations. For example, if there was no instrument response at concentration X, then it would be unlikely that there would be a response at a concentration of X/ 2. In the non random design, EPA permitted the analyst to stop analyses of ever lower concentrations, whereas a randomized design would have required that all the samples be analyzed, even when there was no instrumental response for many of those samples.

One of the peer reviewers evaluating EPA s draft version of this Assessment Document commented that the effects of carry over could have been mitigated by studying variability around the calibration line rather than the mean of the replicates. However, carry over affects subsequent samples differently. The effect of the carry over cannot be mitigated, regardless of whether variability is studied around the calibration line or the mean of the replicates, unless the amount of carry over is known and can be subtracted from the affected ( low concentration) sample. This subtraction has limitations because of error accumulation and because the amount of carry over cannot be determined precisely without extensive studies at multiple concentrations.

<IP_ADDRESS> Completeness The physical measurement process can be studied using rough approximations or it can be studied more rigorously. A rough approximation could use the available components of a method as applied to convenient samples. A more rigorous study would use a complete, specific, and well defined measurement method with all sample processing steps. The appropriate level of study will probably depend on the purpose of the study.

Measurement procedures ( methods) may be more or less strictly designed. Variability in what is allowed in the procedures may add to variability in the measurement results. To the extent that permutations of a method s procedures are not expected to be used in a particular detection or quantitation study, EPA recommends that this information be included in the report on the study results. While there may be physical/ chemical reasons for extrapolating the results of a variability study on one set of procedures to permutations of those procedures, there is no statistical basis for making such an extrapolation. Statistical theory by itself is only able to describe conditions that have been observed. On the other hand, a knowledge of the underlying physics of the measurement process can guide the completeness of the modeling process when statistical procedures fail. For example, the Rocke and 3 34 February 2003 Chapter 3 Lorenzato model in the linear or log log domain may be the best general characterization of a physical measurement process. Therefore, this model can be applied to data to produce a complete answer when statistical procedures fail to deduce the " correct" model.

February 2003 3 35 Table 3 1. Summary of Issues Considered Issue Considered Description Assessment/ Relevance to Detection and Quantitation Level Approaches Blank versus zero concentration It is not possible to measure   zero  concentration of a substance.

As chemists push detection capabilities closer to   zero ,

analysis of blanks is needed establish if the result measured is a true concentration or background contamination. Useful approaches should address the potential contribution of the blank, both through the design of studies that generate detection and quantitation limits, and through evaluation of the study results.

Lack of instrument response Instruments do not always produce a response ( i. e., sometimes they are not capable of yielding a result at the known concentration of a pollutant.) Operational systems for detection and quantitation need to take instrument non

response into account.

Matrix effects If not properly controlled, substances in a particular matrix can interfere with measurements of pollutants, resulting in high or low bias in the measurement. Procedures for determining detection and quantitation limits should allow for evaluation of data in specific matrices of concern when all efforts to resolve the matrix interferences have been exhausted.

Bias ( recovery)

correction Some detection and quantitation approaches apply a correction for bias based on the slope of recovery versus concentration line. The typical relationships between recovery, precision, and concentration in analytical chemistry show that precision worsens as recovery falls. Therefore,

increasing detection and quantitation limits to allow for lowered recovery may, in fact, provide a double correction. Preliminary tests have shown that negative detection limits can be produced when a recovery model has a negative slope.

Measurement quality over the life of a method Measurements made by a given method may improve in quality over time as analysts gain more experience and/ or as improvements are made in the underlying analytical technology. Detection and quantitation limit approaches should be supported by procedures that will allow individual laboratories and other organizations to affordably characterize the improvements.

Method development and promulgation CWA Section 304( h) requires EPA to promulgate test methods that are used in certain certification and permitting activities under the Act. If necessary, EPA must develop these methods.

EPA also can promulgate methods developed by other organizations if they meet EPA needs. However, organizations presently use a variety of differing approaches for detection and quantitation. There are real benefits to standardization of detection and quantitation limit approaches, but it is impractical to force outside organizations that develop methods to accept standardization adopted by EPA.

Laboratory specific performance verification Laboratories that use a particular method must be capable of demonstrating they can achieve the required detection limit. This objective can be achieved by establishing a reasonable detection limit based on tests in an experienced laboratory and requiring individual laboratories to confirm they achieve this detection limit.

3 36 3 37 Table 3 1. Summary of Issues Considered Issue Considered Description Assessment/ Relevance to Detection and Quantitation Level Approaches Effluent guideline development CWA requires EPA to develop technology based guidelines and standards concerning the treatment and discharge of pollutants into US waters. Effluent guidelines and standards are based on existing technological capabilities.

Because pollutant limits are derived from statistical analysis of data that captures all sources of variability in the industrial treatment process, including analytical variability, facilities employing well designed and operated treatment should be capable of achieving technology based limits.

Development of water quality based controls CWA requires EPA and authorized States to implement water quality based controls when technology based controls will not sufficiently protect designated uses for a water body. Detection and quantitation limit capabilities are not a consideration when water quality criteria are established, meaning that water quality criteria may be lower than levels that can be reliably measured using available technology. Ideally, detection and quantitation limit approaches will include procedures that encourage the lowering of detection and quantitation limits so that human health and the environment are protected.

Compliance monitoring CWA requires EPA and authorized States to issue NPDES permits to facilities that discharge into waters of the U. S., and requires municipalities to issue pre treatment permits to facilities that discharge into publicly operated treatment systems. Legal issues must be considered when establishing permit limits. Compliance limits must reflect realistic measurement capabilities.

Non regulatory studies and monitoring EPA conducts a variety of non regulatory studies and monitoring activities the goals of the CWA. Ideally, detection and quantitation limits should allow reliable detection and measurement of pollutants at levels that could become of environmental concern.

Descriptive versus prescriptive uses of detection and quantitation limits Descriptive approaches characterize the current performance of a laboratory or laboratories that might use a method to analyze pollutants. Prescriptive approaches define a limit that laboratories must demonstrate they can achieve before practicing a method. The prescriptive approach to detection and quantitation is consistent with EPA s use of other prescriptive performance standards that laboratories must achieve. The prescriptive approach requires specification of reasonably attainable detection limits. Even so, a prescriptive approach could result in exclusion of laboratories that cannot achieve the limit.

A descriptive approach can reflect capabilities of poor laboratories and can,

therefore, have the effect of raising detection and quantitation limits to levels that are higher than desired and not consistent with the performance of laboratories that make reasonable efforts to control variability.

Compliance evaluation thresholds Water quality based permitting can suggest permit limits that are below the detection or quantitation limits of the most sensitive, approved method. EPA s draft guidance suggested that the permit limit be established from the water quality standard but that compliance be evaluated at the quantitation limit of the most sensitive method. Permit writers should have the flexibility to use the detection limit, quantitation limit, or other limit as the compliance evaluation threshold so that the environment is protected. 3 38 Table 3 1. Summary of Issues Considered Issue Considered Description Assessment/ Relevance to Detection and Quantitation Level Approaches Accepting the procedures of VCSBs The National Technology Transfer Advancement Act directs EPA to focus on increasing their use of standards published by VCSBs such as ASTM, ISO, etc., when it is consistent with the Agency s mission. VCSBs use differing approaches to detection and quantitation needs to be considered. EPA would continue to accept analytical methods from VCSBs that may use other detection and quantitation approaches when the methods meet EPA s regulatory needs.

National versus local standards for measurement CWA authorizes State and local governments to implement provisions of the Act, as long as they do so in a way that is at least as protective of the environment as the national standards established by EPA. This assessment must recognize the impact of any new or revised detection or quantitation approaches on State and local governments and requirements. Given that state and local governments may use more stringent approaches, adoption of new or modified approaches by EPA as a result of this assessment may have no practical impact if states choose to continue use of their existing approaches.

Cost and implementation issues The financial and technical resources required to determine detection limit approaches vary widely according the complexity of the procedures involved. Organizations that develop methods typically have greater resources available for determining limits than do organizations that use the methods. EPA must be sensitive the capabilities of the organizations that develop and that use methods. Data from EPA studies indicate that the true detection/ quantitation limits at a given point in time can only be arrived at by running hundreds of replicates. A better alternative would be to identify a simple procedure that yields a reproducible estimate and to allow laboratory specific adjustment based on actual conditions in the laboratory.

Use of multiple approaches or a single pair of approaches Analytical methods are used to support multiple applications under CWA, including development of regulatory requirements,

compliance monitoring and enforcement, and non regulatory studies and monitoring. EPA could adopt multiple detection and quantitation level approaches, choosing for each application an approach that is best suited to that need. Use of a single pair of approaches that meets the needs of all CWA programs is preferable to the adoption of multiple, application specific approaches. Selection of multiple approaches would likely yield a matrix of approaches and applications that would cause confusion and frustration among regulators, permittees, and the laboratory community. It is also likely that the suitability of each approach to each designated application would be subject to the same level of contention that has been applied to nearly all existing approaches.

Sources of variability Various known and unknown sources of variability can impact laboratory results. Steps can be taken to control known sources of variability. When such steps are taken, some variability will still exist, and it can be expected that interlaboratory variability will be greater than intralaboratory variability. The potential impacts of interlaboratory variability must be considered when selecting detection and quantitation limit approaches. Even if prescriptive measures are used to control variability, it should be recognized that some laboratories may achieve lower detection and quantitation limits than others. 3 39 Table 3 1. Summary of Issues Considered Issue Considered Description Assessment/ Relevance to Detection and Quantitation Level Approaches Censoring measurement results Measurement results are often reported as less than some detection, quantitation, or reporting limit ( i. e., they are censored below a designated reporting threshold). The primary reason for censoring is to avoid reporting results with a higher degree of uncertainty, based on a policy decision. Although such results may not have the desired level of certainty for most applications, they may be of value to statisticians and modelers who handle large volumes of data. Although the issue of censoring is important, it should not be a consideration when selecting a detection and quantitation limit approach. The decision to censor data is a data reporting and data use issue, and it will apply regardless of how the detection or quantitation limit is established.

Outliers Outliers are measurement results that are inconsistent with the vast majority of results. They may arise from random variation or some deviation in the measurement process. Data sets used for development of detection and quantitation limits, for QC acceptance criteria, and for other purposes, should continue to be screened for outliers.

Selection of statistical models Method sensitivity is usually established based on measurement variation. Nearly all analytical techniques produce results that can generally be classified according to one of three basic models. Approaches that rely on collection of large data sets and selection of appropriate models based on graphical analysis of the results have been proposed as a more accurate means of determining detection and quantitation limit approaches than simple models based on the collection of small data sets. Regardless of which model is selected, it should yield a reasonable and reproducible estimates of detection and quantitation. Differences in the cost and complexity of these approaches must be weighed against differences in observed outcome using either approach.

Methodology for parameter estimation Methods for estimating the goodness of fit of a set of data to a particular model include use of statistical estimation procedures for precision and bias and graphical displays. However, the exact tests to be used may not be detailed well enough in the detection/ quantitation limit approach. Details of estimating goodness of fit must be specified and must be based on actual tests of real world variability versus concentration data.

False positives and false negatives If a pollutant is present in a sample, but not measured, the reported result is a   false negative.  If a pollutant is not present in a sample, but a positive result is measured and reported, the reported result is a   false positive.  The issue of allowing for false negatives is contentious. A common error in discussions of detection and quantitation limit approaches is that reporting limits can be used to control both the false positive and false negative rates. As long as the reporting limit is the only tool for controlling false positive and false negatives, setting the reporting limit higher only reduces the probability of a false positive at the expense of the false negative rate. 3 40 Table 3 1. Summary of Issues Considered Issue Considered Description Assessment/ Relevance to Detection and Quantitation Level Approaches Statistical prediction and tolerance Currie s original approaches are based on confidence limits.

More recently, others have suggested that detection and quantitation approaches should be based on statistical prediction or tolerance intervals because compliance measurements are future samples, whereas the limits are based on available measurements. Statistical intervals can be, and have been, adapted for use in setting detection and quantitation limits by a number of authors. However, the use of prediction and/ or tolerance limits in setting detection and quantitation limits is not a requirement and should be evaluated in the context of specific applications and policy considerations.

Design of detection and quantitation studies Studies designed to characterize detection and quantitation limits can be affected by the selection of concentrations studied,

how well uncontrollable factors in the measurement process are reduced, the degree to which entire measurement process is studied, and the flexibility of the design factors in terms of the physical measurement process. Resources may be insufficient to support detection/ quantitation limit approaches that model variability versus concentration because the selection of concentrations may require iteration when results do not meet their respective criteria.