<IP_ADDRESS> Implementation of a Detection/ Quantitation Limit Procedure by a Laboratory A laboratory may implement detection or quantitation procedures for its own quality control purposes, because of regulatory requirements, or as part of the study of a method by some other organization. When participating in the study of another organization, the laboratory may voluntarily accept some cost of the study for marketing purposes, professional development, or to benchmark the performance of the laboratory. Studies designed to characterize method performance: ( 1) at a well performing laboratory or ( 2) at a group of laboratories with comparisons of individual laboratories may be used to benchmark the performance of a participating laboratory. Studies designed to characterize method performance at a group of laboratories are unlikely to provide useful feedback to participating laboratories.

3.2.7 Use of a pair of related detection and quantitation procedures in all Clean Water Act applications.

In Section 3.2.1, we discussed several different applications for detection and quantitation limits under the Clean Water Act. To review, these applications are:

° Method development and promulgation, ° Method performance verification at a laboratory ° Technology based effluent guidelines development ° Water quality based effluent limits development ° Permit compliance monitoring, and ° Non regulatory studies and monitoring. Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 3 14 Although we could develop a separate detection and quantitation concept for each of these applications and attempt to define and evaluate each of these concepts in our re examination of detection and quantitation concepts, the resulting matrix of concepts would cause confusion to regulators, permittees, and the laboratory community. Further, when proposed, each member of the matrix of concepts and applications would, individually, be subject to contention and second guessing, and it is likely that the outcome would be nearly the same as if a single pair of concepts is selected. To avoid this confusion, it is therefore desirable to use a single pair of related detection and quantitation procedures to meet needs where they exist in all Clean Water Act applications.

3.3 Statistical Issues The goal of this section is to provide a brief explanation of the key statistical issues involved in the development of detection and quantitation limits.

3.3.1 Sources of Variability Various known and unknown sources of variability will influence measurements made by a laboratory using a specific method. These sources may include, random measurement error, differences in analysts, variations between different equipment manufacturers and models, variations in analytical standards, routine fluctuations in equipment performance, and variations in facility conditions ( e. g., varying levels of background contributions).

There are a number of ways in which variability can be controlled. One is a strong quality assurance ( QA) program that includes use of: 1) trained and qualified staff, 2) properly maintained equipment, 3) fresh and properly prepared and stored standards, 4) adherence to written standard operating procedures and methods for all sample handling, analysis, and data reduction/ reporting activities, 4) ongoing monitoring of laboratory performance and 5) quality control ( QC) samples and QC acceptance criteria to ensure that the laboratory systems are in control. The EPA methods promulgated at 40 CFR part 136 require the use of qualified staff, appropriately cleaned and calibrated equipment, and properly prepared standards. Each method also provides detailed steps for performing all sampling handling and analysis activities.

Even when prescribed EPA requirements are implemented, however, it is not possible to completely eliminate all variability within or between laboratories. The potential effects of sources of variability should be considered when establishing detection and quantitation limit concepts. Even with quality control and variability control procedures in place, it should be recognized that some laboratories may achieve lower detection and quantitation limits than others. Ultimately, some laboratories may not be capable of meeting low level measurement requirements without some effort to improve operations.

3.3.2 Censoring Measurement Results Measurement results are often reported as less than some detection, quantitation, or reporting limit ( see Section <IP_ADDRESS>, Permit Compliance Monitoring) without providing a single best estimate for the numeric result. For example, if a direct reading of the measurement results would indicate a concentration of 3 mg/ L and the reporting limit for the substance is 5 mg/ L, the laboratory may only report that the measurement result is less than 5 mg/ L. Statisticians call this process of suppression of results less than a specified amount " censoring." Reasons for the practice of censoring relate directly to issues surrounding the development of detection and quantitation limits; i. e. the premise that measurement results below certain low levels may not useable for certain purposes. Chapter 3 Draft Document for Peer Review  Do Not Circulate  August 2002 3 15 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Classicals, Method 350.3 AMMONIA AS NITROGEN, MG/ L Measured Concentration Spike Concentration Figure 3 1 In order to evaluate low level variability, EPA conducted a comprehensive study of results from 1/ 10th the MDL to concentrations into the usual quantitation range. Ten different analytical techniques were evaluated in the study ( see Appendix B, Characterizing Measurement Variability as a Function of Analyte Concentration for a Variety of Analytical Techniques). Data from this study indicate that measurement results may be generated at low concentrations which are quite variable in relation to the true concentration in the sample. While this observation has not been demonstrated with every substance measured, it is suggested by plots of data from most of the measurement techniques observed in the study. An example is Ammonia as Nitrogen by Method 350.3. Plotting measurement results versus concentrations spiked into reagent water samples ( Figure 3 1), we see the strong relationship between measurement results and spike concentrations that would be expected. However, it is difficult to see what is going on at low concentrations in a graphic that covers measurement results over several orders of magnitude. By plotting the log of the measurement results versus the log of the spike concentrations ( Figure 3 2), we would expect to see an expansion of variability at low concentrations, a contraction of variability at high concentrations and points mostly plotted along a 45 degree angle to indicate that measurement results are approximately equal to the spike concentrations. However, we only see the 45 degree angle at higher concentrations. Measurement results in the lowest order of magnitude appear to have reached a plateau below which they do not go. ASTM Committee E 1 has termed the model that describes the general pattern displayed by these data the " General Analytical Error Model." Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 3 16 0.001 0.005 0.010 0.050 0.100 0.500 1.000 0.05 0.10 0.50 1.00 Classicals, Method 350.3 AMMONIA AS NITROGEN, MG/ L Log 10 of Measured Concentration Log 10 of Spike Concentration Figure 3 2 Despite such evidence that results at low concentrations can be quite variable, some data users, such as modelers, prefer to use the actual measurement results ( even if they are negative values), rather than reporting limits, because they believe that censoring data at a detection limit also can introduce bias into the data set. If Currie's model is true, and the true value of a sample is zero, then the frequency distribution of observations would have a mean at zero, with values above or below that mean due to the inherent uncertainty of the analytical process. If all negative or low values were eliminated, the mean would have a positive bias. In other words, while negative or extremely low values may have no meaning in the real world, they may be of value to statisticians and modelers who are handling large volumes of data.

Some programs, such as EPA's Superfund Contract Laboratory Program require laboratories to report the measurement result obtained from the analysis in conjunction with a qualifier that the result is below a specified detection, quantitation, or reporting level. Going back to the example in the first paragraph, the laboratory would report both a measured value of 3 mg/ L and a reporting limit of 5 mg/ L. Under certain assumptions, measurement results below the specified level could then be used to calculate averages and extreme value estimates that would be superior to estimates calculated using censored data. The primary assumption is that measurement results are, on average ( i. e., in expectation) approximately equal to the true concentration in the average.

EPA believes that such an approach provides the greatest degree of flexibility for data users, but also believes that it should be used with care. First, data users who choose to use values reported below a detection or quantitation limit need to have a firm understanding of the limitations of those data. Second, and as noted in Section <IP_ADDRESS>, Permit Compliance Monitoring, reporting data below a detection or quantitation limit can lead to misinterpretation. Chapter 3 Draft Document for Peer Review  Do Not Circulate  August 2002 3 17 0 200 400 600 800 1000 1200 0 500 1000 1500 Metals, Method 1620 ALUMINUM, UG/ L Measured Concentration Spike Concentration Figure 3 3 3.3.3 Outliers Outliers are extreme or aberrant measurement values that, on inspection, are not consistent with a set of data. Outliers are generated by a number of causes such as errors in following an analytical procedure, errors in recording numerical results or they may be the result of extreme random variation in a properly operating process. For example, if a new measurement method is being tested but the laboratory fails to follow the procedure correctly with some samples then the associated measurement results may stand out as outliers. A graphic example is shown in Figure 3 4, which shows measurement results for aluminum by Method 1620 versus concentration. At spike concentration of 250 : g/ L, one of the measured values is about 750 : g/ L and stands out visually from the rest of the values. This may be considered an outlier.

A common process for identifying potential outliers is to apply one or more statistical procedures for identifying extremely large or extremely small measurement values. An example of such a procedure is ASTM Practice D 2777. Because extreme values are expected to occur, it is not necessarily appropriate to exclude them from measurement results used to develop detection or quantitation values. As recommended in the ASTM procedure, a review of the analysts records associated with the measurement may establish whether the extreme value was caused by failure to follow the method or by some rare event associated with the method. If the method under study was not followed then it is appropriate to exclude the measurement result from the detection or quantitation analysis. If the measurement result is a rare event associated with the method under study then may also be appropriate to exclude the measurement result from the results in the study.

In large studies of detection and quantitation, it may not be economically or technically feasible to review all extreme values to determine if they are outliers. In such cases, removing all extreme values as if Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 3 18 they were outliers may be acceptable. We recommend that the study documentation clearly state this is the case and that the percentage of data removed be stated. Removing large percentages of extreme values may cause variability estimates to be understated, indicate that there are systematic problems with following the method, or indicate that there are problems with the procedure for determining extreme values.

Influential early work in using ranking to help identify outlying laboratories in studies was conducted by Youden ( Youden, W. J. and E. H. Steiner Statistical Manual of the Association of Official Analytical Chemists).

3.3.4 Criteria for the Selection and Appropriate Use of Statistical Models Detection and quantitation limits may be based on statistical models of the relationship between measurement variation and the concentration of a substance in the water. Results are produced by adding varying known amounts of the substance to the water (" spiking"), making replicate measurements at each concentration, and modeling the variability of the results as a function of concentration. This section summarizes the history of modeling variability versus concentration, considers criteria for selecting models, and discusses current practices with regard to available data.

<IP_ADDRESS> Short History of Modeling Measurement Results Over time, a number of different models have been used to estimate measurement variation. Currie ( 1968) modeled variation in radiochemical measurement methods using a procedure associated with counting large numbers of distinct objects which are appropriately modeled with the Poisson distribution . However, he relied on large sample sizes and standard normal distributions to describe all other types of measurement methods. Hubaux and Vos ( 1970) developed a procedure based on an estimated calibration relationship using smaller sample sizes to estimate Currie's detection and quantitation limits. Again, measurement results were assumed to follow standard normal distributions but it was also assumed that measurement variation was constant throughout the range of interest. Similarly, Glaser et al. ( 1981) suggested that measurement variation increases linearly with concentration but they did not provide estimators under this theory because they believed that measurement variation is usually approximately constant in the range of detection. Glaser et al. ( 1981) did suggest that when appropriate data were available that a linear regression analysis of the relationship over the analytical range be performed. Clayton et al. ( 1987) discusses transforming the measurement results ( using logarithms or square root functions). Gibbons et al. ( 1991) suggests that measurement variability may be proportional to concentration. Rocke and Lorenzato ( 1995) propose a model motivated by physical characteristics of measurement processes in which measurement variability is approximately constant at low concentrations but changes in a continuous mathematical manner to increasing variability as concentration increases.

Figure 3 4 shows the fundamental analytical measurement models in linear and logarithmic domains. The models are applicable to nearly all analytical measurements; we will not deal with the exceptions because they represent a small percentage of cases. As can be seen from the top two graphs, response is linear as a function of concentration in both the linear and log domains. The middle two graphs and the bottom two graphs are those most pertinent to detection and quantitation. Chapter 3 Draft Document for Peer Review  Do Not Circulate  August 2002 3 19 Response vs. Concentration Linear Domain 0 20 40 60 80 100 0 20 40 60 80 100 Concentration Response SD vs. Concentration Linear Domain 0 2 4 6 8 10 0 20 40 60 80 100 Concentration SD Response vs. Concentration Log Log Domain 0.001 0.01 0.1 1 10 100 0.01 0.1 1 10 100 1000 Concentration Response SD vs. Concentration Log Log Domain 0.01 0.1 1 10 100 0.01 0.1 1 10 100 1000 Concentration SD RSD vs. Concentration Linear Domain 0 20 40 60 80 100 0 20 40 60 80 100 Concentration RSD RSD vs. Concentration Log Log Domain 1 10 100 1000 0.01 0.1 1 10 100 1000 Concentration RSD Figure 3 4 <IP_ADDRESS>.1 Detection Limits Using Variability at Low Concentrations The middle graphs shows variability versus concentration and shows the model postulated by Rocke and Lorenzato. The flat ( constant) portion of the graph in the linear domain is difficult to see because it occurs near the origin, but it can be seen easily in the log domain. Most detection concepts ( e. g., Currie's critical level and detection limit; EPA's MDL; the ACS LOD) are constructed assuming the flat ( constant) region of the variability versus concentration graph, although the graph is rarely displayed ( a horizontal line would be singularly uninteresting). Detection concepts such as the critical level, detection limit, LOD, and MDL are constructed by multiplying the standard deviation in the flat region by some constant. Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 3 20 Contention and differences of opinion occur in how to arrive at an " appropriate" standard deviation and what to do with the standard deviation when you have it. Currie's critical level and EPA's MDL use a multiple of the standard deviation in a similar manner. ( a t statistic adjusted for the number of replicates used for Currie's critical level; 3.14 for 7 replicates in EPA's MDL); the IDE uses an additional upward adjustment based on a statistical tolerance limit calculation.

<IP_ADDRESS>.2 Quantitation Limits Using Standard Deviation Multiples and Models of Standard Deviation versus Concentration and RSD versus Concentration The limit of quantitation ( LOQ) advanced by Currie and the American Chemical Society's Committee on Environmental Improvement, and EPA's minimum level of quantitation ( ML) result from multiplication of the standard deviation by a factor of 10, again assuming a flat portion of the variability versus concentration graph. This factor of 10 is directed at achieving a relative standard deviation ( RSD) of 10 percent. An advantage of this approach is that a quantitation limit is produced, regardless of what the RSD turns out to be. For example, it is known that the determination of 2,4 dinitrophenol by EPA Method 625 produces highly variable results and that 10 percent RSD cannot be achieved for this compound. Multiplying the standard deviation of replicate measurements at low levels results in a quantitation limit considerably higher than quantitation limits for other compounds analyzed by Method 625. The RSD at this quantitation limit could be 30, 50, or 70 percent. Arbitrarily limiting the quantitation limit to some value ( e. g., 30% as with the ASTM IQE) could prohibit the use of EPA Method 625 for determination of 2,4 dinitrophenol. If 2,4 dinitrophenol were present at high concentration in a discharge, it would not be reported. Although it could be argued that a more precise method should be used for determination of 2,4 dinitrophenol, determination of pollutants by a large suite of multiple methods would be quite costly with little meaningful benefit. Increasing precision ( i. e., decreasing measurement error) would be critical only if the concentration at issue was near a compliance limit.

Another means of arriving at a limiting RSD is to graph RSD versus concentration, as shown in the bottom two graphs of Figure3 3. This approach is used by the ASTM IQE. It has the advantage that a model is fit to data, rather than a point estimate such as the Currie and ACS LOD or the EPA ML. However, it requires considerably more data than concepts based on point estimates. In addition, how a model is selected can play a major role in the outcome.

<IP_ADDRESS> Criteria for Selecting Models Both mathematical and graphical procedures have been proposed for selecting between models for measurement results versus spike concentrations.

Mathematical Criteria While mathematical criteria are available for choosing between models of similar types, currently available mathematical criteria are not satisfactory for choosing between the wide variety of models considered for the relationship between measurement variation and spike concentration, based on EPA's studies of measurement variation versus spike concentration. More technically, mathematical criteria include ( 1) the simplest model to obtain statistical significance, ( 2) the model with the smallest estimated variability, and ( 3) the model with the smallest likelihood ratio. Given the wide variety of models considered for detection and quantitation, there are problems associated with each of these procedures. Data that obviously do not follow the model may produce statistically significant results, variability may be estimated with weights that make the various estimates incomparable, and the likelihood function may not be comparable between models. Chapter 3 Draft Document for Peer Review  Do Not Circulate  August 2002 3 21 Graphical Criteria Graphical criteria may be susceptible to some subjectivity in their application but they are currently the best available method for choosing between models. At the most basic level, the primary graphical criteria is for the form of the model to be suggested by the available data. To consider the quality of the graphical analysis, it is useful to see if some small number of data are overly influential in determining if a model does or does not fit. Given the ability of the human eye to discern deviations from a straight line than from a curved line, a useful technique is to plot the data so that they will indicate a straight line if they follow the model of interest.

<IP_ADDRESS> Current Practices with Available Data EPA graphed variability versus concentration data with regard to how real data from measurement methods used under the Clean Water Act would conform to a number of different models. For details of how datasets were selected and how data were collected within the datasets, see Appendix B, Characterizing Measurement Variability as a Function of Analyte Concentration for a Variety of Analytical Techniques Four sets of composite scatter plots for all combinations of analytical technique by analyte by study were produced. These sets include:

1. Measurement versus Spike Concentration 2. Log Measurement versus Log Spike Concentration 3. Observed Standard Deviation versus Spike Concentration 4. Log Standard Deviation versus Log Spike Concentration 5. Relative Standard Deviation ( RSD) versus Log Spike Concentration There are hundreds of scatter plots in each set, sorted by the source, measurement technique and study. The first set of scatter plots can be used to evaluate how well measurement results match the spiked concentration in the water. If the assumed straight line model were true then the relationship outlined by the plotted data would be approximately linear. These relationships are plotted using log log plots so that small deviations away from the line can be easily visualized. All the graphs are contained in attachments to Appendix B which are printed on compact disk.

The plot of observed standard deviations versus spike concentrations can be used to evaluate the reasonableness of the constant variation and/ or linearly increasing variability models ( Currie [ 1968], Hubaux and Vos [ 1970], Glaser et al. [ 1981]). If the constant variability model for standard deviation were true, there would be no apparent relationship between the standard deviation and spike concentration. If the straight line model for standard deviation were true, plots are expected to indicate an approximately linear relationship. Analogously, the standard deviation/ spike concentration versus spike concentration is expected to show a straight line relationship when variability is proportional to the spike concentration ( Gibbons et al. 1991). The log log plots of standard deviation versus spike concentration are expected to indicate if log or square root transformations may be appropriate ( Clayton et al., 1987) or to display a hockey stick relationship when it is appropriate to use the model proposed by Rocke Lorenzato ( 1995). Variability near zero will be approximately constant but variability will increase more strongly with spike concentration as you move to higher concentrations with a Rocke and Lorenzato model.

The large number of plots make it difficult to draw general conclusions. For the most part, conclusions must be considered on a case by case basis. One somewhat general observation is that measurement variability over low concentrations does not appear to fit a pattern and thus may be considered to be approximately constant in this range for a large number of analytic techniques. Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 3 22 3.3.5 Methodology for Parameter Estimation Along with proposing various concepts of detection and quantitation and models for measurement, a number of specific estimation procedures have been proposed. Maximum likelihood and least squares are two generally applicable statistical methods that can be used in estimating model parameters. There are advantages and disadvantages to both that must be weighed in particular cases. A standard statistical practice for evaluating the quality of an estimation procedure is to calculate the precision and bias, usually best understood by examining a plot of residuals from a fit to a function. All else being equal, the estimation procedure with the best precision and least bias is preferred. In some cases, precision and bias can be calculated based on the assumptions behind the estimation procedure. In other cases, it is either necessary or convenient to estimate precision and bias using simulations. From a general theoretical perspective, the maximum likelihood estimation methodology is preferable because it generates estimates that are generally best with regard to properties of precision and bias ( especially for larger sample sizes) while also being approximately normally distributed. Unfortunately, maximum likelihood can sometimes be problematic because the method requires the solution of complex equations. Least squares estimation is generally more tractable and thus is more generally applicable although the estimates that result may not be as desirable from a theoretical statistical perspective.

What can sometimes be overlooked in considering estimating for model fitting is that direct measurement of variation of the blank or low level concentration may be the most cost effective and least difficult method to implement. The loss in statistical efficiency in comparison to more elaborate estimation and model fitting methodology would be offset by the relative ease and lower cost.

3.3.6 False Positives and False Negatives In this section, we discuss the impact of detection, quantitation, and reporting levels on false positive measurement results and false negative measurement results. The definitions of false positives and false negatives are easily tied into the Currie ( 1995) concepts of critical value and detection limit. The definitions will also be explained in terms of reporting limits, along with common errors in their usage.

Assume that a critical value was selected such that higher measurements have less than a 1% chance of being associated with a sample that does not contain the substance of interest. When a measurement result associated with a sample that does not contain the substance of interest is higher than the critical value it is treated as if the sample does contain the substance of interest. That is a false positive measurement result or a " false positive."

When a measurement result that contains the substance of interest is lower than the critical value, it is treated as if the sample did not contain the substance of interest. That is a false negative measurement result or a " false negative." Where a false positive is generated from a known concentration ( zero) with a selected probability, false negatives are usually generated from unknown concentrations with probabilities that depend on both the true unknown concentration and the critical value or reporting level. When the unknown true concentration is less than the critical value, the probability of obtaining a false negative is greater than 50%. When the unknown true concentration is equal to the critical value, the probability of obtaining a false negative is approximately 50%. When the unknown true concentration is greater than the critical value, the probability of obtaining a false negative is less than 50%. Currie ( 1995) defined his detection limit to be a true concentration value such that true value concentrations greater than or equal to the detection limit are likely to be associated with measured concentrations that are greater than the critical value. His recommended values for " likely" are 95% or 99%. Chapter 3 Draft Document for Peer Review  Do Not Circulate  August 2002 3 23 The reporting limit functions a lot like a critical value. When a measurement result that contains the substance of interest is lower than either the critical value or the reporting limit, it is treated as if the sample did not contain the substance of interest. The primary difference is that a critical value is explicitly tied to the probability of generating a false positive but the reporting limit is generally not. However, the probability of generating a false positive with a reporting limit can often be back calculated. For example, if the reporting limit is set equal to the detection limit used in the earlier examples, the probability of a false positive becomes somewhat less than 2 in a million. If the reporting limit is set equal to the quantitation limit used in the earlier examples, then the probability of a false positive becomes too small to be easily calculated ( one estimate has been less than 1 in 108).

A common error in many published discussions of false negatives in relation to detection and quantitation ( such as the ASTM IDE) is the claim that using Currie's detection limit as a reporting limit or action level will somehow " control" false negatives. That claim is both false and counter productive. To illustrate the problem with this error, consider the classic scenario in which the true concentration for the substance of interest in a sample is equal to the critical value. Include the simplifying assumptions that variability is approximately normal in distribution and approximately constant throughout the region of concern. The critical level ( alpha level) is set to 0.01 or 1% throughout the remainder of this section.

The first part of this illustration goes much the same as it has been written in a number of different papers. Set the reporting limit equal to the critical value. Given a large number of measurements on the sample, about half of the measurement results will be reported as measured above the reporting limit and about half of the measurement results will be reported as measured below the reporting limit. Measurement results reported below the reporting limit are treated as if there is no substance of interest in the sample; these are false negative measurements. Many authors assert that the 50% level of false negatives is not acceptable.

We will significantly change the second part of the usual discussion by: ( a) explicitly including the concept of the reporting limit and ( b) not changing the reference point ( the true concentration in the sample). Set the reporting limit to Currie's detection limit. Given a large number of measurements on the sample, about one percent of the measurement results will be reported as measured above the reporting limit and ninety nine percent of the measurement results will be reported as measured below the reporting limit. This makes the problem of false negatives worse.

Now, to see what Currie was getting at when he defined his detection limit, set the reporting limit equal to Currie's critical value and create a sample with a true concentration equal to Currie's detection limit. Given a large number of measurements on this sample, about ninety nine percent of the measurement results will be reported as measured above the reporting limit and one percent of the measurement results will be reported as measured below the reporting limit. Knowledge of the lowest true concentration that will routinely produce acceptable measurement results can then be used to determine if the measurement method meets the needs of a study. A study concerned with a wastewater treatment technology not expected to be effective at concentrations below 10 mg/ L may call for a relatively inexpensive measurement method capable of detecting at that level rather than a more expensive measurement method capable of measuring a hundred times lower.

To emphasize the previous point about how counter productive it is to move the reporting limit higher based on concerns associated with false negatives consider the following. Wherever the reporting limit is set, a large number of measurements on a sample with a true concentration equal to the reporting limit will produce false negatives about fifty percent of the time. This is true if the reporting limit is set equal to the critical value, the detection limit, a quantitation limit, or any other positively valued limit within the range of the measurement method. As the reporting limit increases in value, the probability increases that a measurement result associated with any fixed true concentration will be a false negative. This means that Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 3 24 concerns regarding false positive measurement results and false negative measurement results are directly competing goals. As long as the only tool for setting requirements for false positive and false negative measurement results is the reporting limit, setting the reporting limit higher reduces the probability of a false positive at the expense of increasing the probability of a false negative.

To push the argument to the extreme, setting the limit to zero guarantees no false negatives because no negatives will be reported ( assuming the analytical system will not produce a result less than zero). Therefore, to adjust the limit upward to account for false negatives can only exacerbate the false negative problem. The solution to the dilemma is to set the limit to some acceptable false positive rate at which false negative results will be minimized. If there is a concern that a pollutant will not be reported if the false negative rate tied to the false positive limit, repetitive testing over time will eventually detect a true ( not false) positive.

3.3.7 Statistical Prediction and Tolerance When we define a critical value, detection limit, or quantitation limit, different descriptive terminology may suggest changes in the numerical value of the limit. We will use a critical value as an example, but the questions motivating detection and quantitation limits can be phrased in similar fashion. Do we want a critical value that tells us how likely it is that:

1. A measurement result was produced by measuring a blank sample, 2. The next measurement result will be produced by measuring a blank sample, or 3. The next [ pick any number] of measurement results will be produced by measuring a blank sample?

The statistical procedures for finding these answers are called:

1. Percentiles; 2. Prediction intervals; and 3. Tolerance intervals.

A more basic way to divide these procedures is between describing something that already exists ( percentiles) and describing a future occurrence ( prediction and tolerance limits). Percentiles are fairly straight forward to interpret, i. e., they specify the percentage of a distribution that falls below a given percentile value. Prediction and tolerance limits are, in effect, confidence limits on percentiles and can be somewhat more difficult to understand and apply. There are many excellent textbook and literature references that present the theory and application of tolerance and prediction limits such as Hahn and Meeker, Statistical Intervals, Wiley, 1991, and Pratt and Gibbons, Concepts of Non parametric Theory, Springer Verlag, 1981. Hahn and Meeker describe at length the different statistical intervals including their properties, applications and methodology for constructing the intervals and situations in which the various intervals are appropriate for use. Hahn and Meeker also give examples of the sort of applications that are suitable for each type of interval although the decision to use a particular type of interval in a given application is not determined strictly by theoretical considerations but is also a matter of judgment. Pratt and Gibbons have an excellent discussion of tolerance intervals that is general in application due to the non parametric perspective, i. e., no distributional assumptions are required for the results to be valid.

<IP_ADDRESS> Prediction Intervals Prediction intervals are used to specify intervals that contain the results of future samples from a previously sampled population. Prediction intervals are not estimators of parameters such as means or Chapter 3 Draft Document for Peer Review  Do Not Circulate  August 2002 3 25 percentiles. For example, prediction intervals may be constructed to contain future sampling results expressed as a mean or standard deviation of a future sample or all of a certain number of individual future sampling results.

<IP_ADDRESS> Tolerance Intervals Tolerance intervals are used primarily to construct intervals that are intended to enclose or contain a specified proportion of a population. In statistical terms, tolerance intervals are intervals that contain a specified proportion of a population of measured values with a given statistical confidence level. For example, we say that a proportion P of a population is contained within the interval ( L1, L2) with ( 1") 100% confidence. The lower and upper ends of the interval, L1 and L2 , respectively, are referred to as tolerance limits. A tolerance interval is therefore an interval of random length that is determined on the basis of having a specified probability of 1" that its coverage of the population is at least equal to a specified value P. The quantity 1" is referred to as the confidence level for the interval and P is the minimum proportion of the population contained in the interval. Tolerance limits are not estimators of values such as a mean or a percentile but rather values that are always guaranteed to be either greater than or less than the desired value at some level of statistical confidence. Pratt and Gibbons discuss this and other properties that affect the utility of tolerance limits and create difficulties in the interpretation and application of tolerance limits.

In effect, the determination of what, if any, interval to use is a policy decision. The choice of which kind of interval to use should consider how easy it is to estimate the interval you want under the conditions that exist. As PG point out, the interpretation of tolerance intervals ( and analogously, prediction intervals) can be problematic especially when issues of sample size and choice of confidence level come into play. PG cite examples where the interplay of sample size and high percentile and confidence level make tolerance limits useless.

Use of Tolerance and Prediction in Setting Detection and Quantitation Levels Statistical intervals can, and have by a number of authors, be adapted for use in setting detection and quantitation levels. The basic approach requires a functional definition of detection or quantitation that includes a statistical term or terms. An interval could then be constructed about the statistical term which could be used to assess the detection or quantitation level or make an adjustment to a calculated value that would result in the detection or quantitation level. For example, most detection level estimators are functionally dependent on an estimate of standard deviation of measurement error. A statistical interval could be constructed about the standard deviation and the length of the interval could be used to assess the detection level. The end points of the interval could be used as the basis for an adjustment ( upward or downward) in the calculated level.

However, the use of prediction and/ or tolerance limits in setting detection and quantitation limits is not an absolute requirement and should be evaluated in the context of specific applications and policy considerations. In practice, the effect of adjustment of detection and quantitation limits by use of prediction and tolerance intervals can be quite large, depending on the amount of available data and the choices of percentiles and confidence levels.

3.3.8 Design of Detection and Quantitation Studies The issues associated with the design of detection and quantitation studies include: how well a selection of spike concentrations can be used to differentiate between different models for the relationship between measurement results and spike concentrations, how the distance between spike concentrations can Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 3 26 impact estimates of detection and quantitation limits, how to reduce the influence of uncontrollable factors in the measurement process ( probability design), how complete to make the design factors in terms of the physical measurement process, and how flexible to make the design factors in terms of the physical measurement process.

<IP_ADDRESS> Spike Concentrations and Modeling If a model under consideration cannot be described by the number of spike concentrations in the design then it is not possible to tell if the model is appropriate. To take the simplest example, it is not possible to describe the slope of a line associated with linearly increasing variation from a single spike concentration. Two well spaced spike concentrations would allow you to estimate a slope but it provides you with no idea of the variability of the estimate. Three well spaced spike concentrations is the minimum requirement for estimating the linear relationship and the variability of that relationship.

Clayton et al. ( 1987) describes the relationship between the spread of the spike concentrations, the number of spike concentrations, and the number of replicate measurements with regard to estimated variability when a linear model is used. While the specific equation used here does not apply to all models, it indicates principles that do apply. Increasing the number of replicate measurements, increasing the number of spike concentrations, and reducing the spread of the spike concentrations are all expected to reduce estimated variability along with the associated detection and quantitation limits. However, one of the components of variability associated with detection and quantitation is that associated with estimating the calibration relationship. To account for this source of variation, it may be appropriate to cover the entire calibration range. On the other hand, many replicates at a high concentration may improperly weight the data in favor of high detection and quantitation estimates.

<IP_ADDRESS> Probability Design The process known as randomization is fundamental to the design and proper interpretation of experimental studies. This involves the allocation of experimental units to factors and treatments under study according a design determined by probability. Randomization insures the inferential validity of the results by avoiding bias and systematic errors that can occur in studies where proper randomization is not used. In studies of measurement methods, randomization should be used in the process of creating spike concentration solutions and the ordering of analyses. A classic text is Statistics for Experimenters by Box, Hunter and Hunter, Wiley, 1978.

<IP_ADDRESS> Completeness The physical measurement process can be studied using rough approximations or it can be studied more rigorously. A rough approximation could physically use the available components of a method as applied to convenient samples. A more rigorous study would use a complete, specific, and well defined measurement method with all sample processing steps. The appropriate level of study will probably depend on the purpose of the study.

Measurement procedures ( methods) may be more or less strictly designed. Variability in what is allowed in the procedures may add to variability in the measurement results. To the extent that permutations of a method's procedures are not expected to be used in a particular detection or quantitation study, EPA recommends that this information be included in the report on the study results. While there may be physical reasons for extrapolating the results of a variability study on one set of procedures to permutations of those procedures, there is no statistical basis for making such an extrapolation. Statistical theory by itself is only able to describe conditions that have been observed. On the other hand, a knowledge of the underlying Chapter 3 Draft Document for Peer Review  Do Not Circulate  August 2002 3 27 physics of the measurement process can guide the completeness of the modeling process when statistical procedures fail. For example, the Rocke and Lorenzato model in the linear of log log domain is the best characterization of a physical measurement process. Therefore, this model can be forced through data to produce a complete answer when statistical procedures fail to deduce the " correct" model. 2Daubert v. Merrell Dow Pharmaceuticals, 509 U. S. 579 ( 1993)

Draft Document for Peer Review  Do Not Circulate  August 2002 4 1 Chapter 4 Evaluation Criteria This chapter presents criteria developed by EPA as a means for selecting acceptable detection and quantitation limit concepts for use in Clean Water Act ( CWA) programs. These criteria reflect EPA's careful consideration of the issues identified and discussed in Chapter 3.

4.1 Criterion 1 Criterion 1: The detection and quantitation limit concepts must be scientifically valid.

Scientific validity is widely accepted but loosely defined. For the purposes of this evaluation, a detection/ quantitation concept or methodology will be considered scientifically valid if it meets the following conditions:

° It can be ( and has been) tested ° It has been subjected to peer review and publication ° The error rate associated with the concept or methodology is either known or can be estimated ° Standards exist and can be maintained to control its operation ( i. e., it is supported by well defined procedures for use) ° It has attracted ( i. e., achieved) widespread acceptance within a relevant scientific community.

While EPA acknowledges that other measures could be established to demonstrate scientific validity, EPA has adopted the conditions cited because they reflect those established by the U. S. Supreme Court as considerations pertaining to assessments of scientific validity when considering the admissibility of expert scientific testimony. 2 EPA believes that considerations established by the Court as necessary to demonstrate the scientific validity of an expert's reasoning or methodology are equally valid for demonstrating the scientific validity of a detection/ quantitation concept.

4.2 Criterion 2 Criterion 2: The concepts must address demonstrated expectations of laboratory and method performance, including routine variability.

As discussed in Chapter 3 of this TSD, the detection and quantitation limit( s) for an analyte in an analytical method can be established from a single laboratory study, multiple single laboratory studies, or an interlaboratory study. Historical methods developed by EPA under the Clean Water Act programs, and nearly all methods developed by EPA under the Safe Drinking Water Act programs, were developed by EPA's research laboratory in Cincinnati, Ohio. In the course of method development, this single laboratory established detection and quantitation limits. In many instances, these detection and quantitation limits were unrealistic, in that they were unachievable in many non research laboratories. However, with time, laboratory, method, and analytical instrumentation improved, making detection and quantitation limits more easily achievable in nearly all laboratories. Therefore, the difficulty created was in initial application of the research methods. Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 4 2 In recent years, EPA's Office of Science and Technology has used single laboratory studies to develop an initial estimate of detection and quantitation limit for a new or modified method, and has verified this limit in interlaboratory studies or by conducting additional single laboratory studies in other laboratories.

Voluntary consensus standards bodies ( VCSBs) such as the American Society for Testing and Materials ( ASTM) have historically used interlaboratory studies to establish method performance. Over the past 5 to 10 years, ASTM has been developing interlaboratory and single laboratory concepts for detection and quantitation. Whereas the single laboratory studies at EPA's research laboratory in Cincinnati produce the lowest detection and quantitation limits, concepts such as those published by ASTM gather all sources of variability to produce the highest detection and quantitation limits. A realistic expectation of method and laboratory performance likely lies somewhere in between.

As noted in Section 3.2.2 of this TSD, laboratory and method performance can be affected by the use of performance criteria that serve as prediction or tolerance levels. Examples of such criteria include measures to demonstrate that a laboratory is producing accurate results at a concentration of interest ( i. e., analysis of reference standards or spiked samples), measures to demonstrate that results are not biased by contamination ( i. e., analysis of blanks), and measures to demonstrate that the laboratory can achieve the sensitivity required to reliably detect pollutants at low concentrations ( i. e., at the detection limit). It is likely that laboratory performance will be better ( and variability will be lower) when laboratories are required to meet specified performance criteria in order to report results.

A further consideration concerning routine variability is the means for rejection of outliers. Mathematicians and statisticians are very reluctant to remove any data; chemists performing routing analyses know that outliers occur. Therefore, if a statistical procedure is used to establish a detection of quantitation limit, the issue of outliers must be addressed, and some means of resolving outlier issues must be included.

In examining each concept against this criterion, EPA will evaluate if the concept can be used to provide a realistic expectation of laboratory performance. As part of this assessment, EPA will examine the sources of variability captured by the concept, and the degree to which the statistics that underlie the concept realistically reflect these sources of variability.

4.3 Criterion 3 Criterion 3: The concepts must be supported by a practical and affordable procedure that a single laboratory can use to evaluate method performance.

Any concept or procedure should be simple, complete, and cost effective to implement. The laboratories that can be expected to use detection/ quantitation procedures will range from large laboratories and laboratory chains with a wide range of technical capability to " mom and pop" laboratories operated by one or a few people with a limited set of statistical skills. If a procedure is complicated it will be error prone in its use. Similarly, if a procedure requires investment of extensive resources that cannot be billed to the client, laboratories will have a disincentive to use the procedure. Therefore, if the Agency wishes to encourage the development and use of innovative techniques that improve measurement performance or lower measurement cost, the Agency must consider practicality and affordability as significant, if not co equal, considerations to scientific validity.

After evaluating each of the issues discussed in Chapter 3 of this document, EPA has concluded that successful implementation of CWA programs depends on the ability of laboratories to easily and affordably 1) demonstrate that a method works in a particular matrix at levels of concern, 2) characterize improvements Chapter 4 Draft Document for Peer Review  Do Not Circulate  August 2002 4 3 in measurement capabilities in terms of measurement sensitivity, and 3) characterize the sensitivity of new methods.

A matrix effect is an interference in a measurement caused by substances or materials in the sample other than the analyte of interest and that are not removed using the procedures in the method or other commonly applied procedures. In the context of detection and quantitation, matrix effects manifest themselves by precluding measurements at levels as low as could be measured were the interference not present. From a practical perspective, it is not possible to test the sensitivity of new methods in every possible matrix in which it may be used. At a minimum, it is unlikely that EPA or any other organization could possibly identify and obtain samples of every matrix to which the method might be applied, and even if such a feat were possible, the cost and logistics of doing so, would be prohibitive. Therefore, EPA prefers to identify a concept that allows for characterization of measurement sensitivity in representative matrices and is supported by a simple, cost effective procedure would allow individual laboratories to evaluate, on an asneeded basis, the effects of specific matrices on measurement sensitivity.

The reality of environmental analysis is that measurement capabilities improve over time. This is attributable to a variety of factors, including ( 1) increased staff experience with a given technique, ( 2) technological upgrades or improvements in the instrumentation used for analysis, and ( 3) development of new instrumentation or techniques that improves sensitivity, precision, or bias. In each case, the improvements may not be observed across the entire laboratory community. In the case of increased staff experience, for example, it is obvious that a laboratory that specializes in one type of analysis, such as low level mercury measurements, will develop greater experience than a laboratory that rarely performs this measurement. Likewise, it is easy to see how one or a few laboratories that concentrate their business on a particular type of analysis might be willing to invest significant resources in new or upgraded equipment to improve performance whereas laboratories that rarely perform such analyses would not find such upgrades to be costeffective

Improvements in measurement capability, including the development of new methods, may create a dynamic decision making process, in that measurements at lower levels may allow EPA and States to identify previously undetected pollutants that are impairing the water body. Such situations offer a means for monitoring and controlling ( i. e., regulating) the discharge of previously unregulated but harmful pollutants. It is in the best interest of the environment for EPA to encourage the development and use of improved environmental analysis procedures and equipment.

In evaluating this criterion, EPA will favor affordable and easy to use procedures that allow analysts in a single laboratory to 1) determine matrix specific variations based on realistic data and 2) demonstrate lower detection and quantitation limits associated with improvements in their measurement capabilities. Procedures for establishing the sensitivity of new methods or improved measurement capabilities must be practical enough to encourage such development. These procedures should specify the spiking level at which measurements are to be made and the corrective action to be taken if the resulting detection or quantitation limit is inconsistent with the data from which it is derived.

4.4 Criterion 4 Criterion 4: The detection level concept should identify the concentration at which there is 99% confidence that a substance will be detected when the analytical method is performed by experienced staff in a well operated laboratory. Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 4 4 Any detection limit concept should be capable of providing regulators, the regulated, and data users with confidence that a pollutant reported as being present really is present. Historically, nearly every detection limit concept has set the criterion for detection at 99 percent confidence; i. e., the lowest level at which a pollutant will be detected with a probability of 99 percent. This criterion results in the probability of a false positive; i. e., that a pollutant will be stated as being present when it actually is not ( a Type I error), of one percent; and the probability of a false negative; i. e., that a pollutant will be stated as being not present when it actually is ( a Type II error), of 50 percent.

A common error in some detection limit concepts, such as the IUPAC detection limit and the ASTM interlaboratory detection estimate ( IDE) is the claim that a detection limit can somehow be used to " control" false negatives. That claim is untrue in the context in which detection limit is used as a reporting level, as it is in reporting data to the Agency. If you want to find a procedure with the lowest possible probability of a false negative, then select the critical value or reporting limit based on what you can accept for false positives and that limit will have the lowest probability of a false negative associated with your selected false positive rate.

In evaluating this criterion, EPA will favor procedures that reflect routine analytical conditions in a well operated laboratory. For example, the procedure must be capable of arriving at a detection limit when the substance of interest is not found in a blank and/ or when instrument thresholds are adjusted for routine operation.

4.5 Criterion 5 Criterion 5: The quantitation limit concept should identify a concentration at which the reliability of the measured results is consistent with the capabilities of the method when a method is performed by experienced staff in a well operated laboratory.

Measurement capabilities among laboratories vary depending on a number of factors, including, but not limited to, instrumentation, training, and experience. Similarly, measurement capabilities among different analytical methods vary depending on a number of factors, including the techniques and instrumentation employed and the clarity of the method itself.

Historical approaches to recognizing laboratory capabilities in establishing detection and quantitation limits have varied between two extremes of establishing the limit in a state of the art research laboratory to reflect the lowest possible limit that can be achieved, and establishing the limit based on statistical prediction intervals calculated from a large number of laboratories with varying levels of experience, instrumentation and competence. Generally, use of the former has been employed to serve as a goal or performance standard to be met by other laboratories, whereas use of the latter treats the limit, not as a performance standard that needs to be met by each laboratory, but rather as a characterization of the future performance of the entire universe of laboratory capabilities at the time of method development.

Historical approaches to recognizing method capabilities also have varied between those that allow the error ( relative standard deviation, or RSD) among low level measurements to vary, depending on the capabilities of the method and those that fix this error ( RSD) at a specific level.

EPA will evaluate various concepts against this criterion by examining the ease of adjustment of the RSD or other performance measure in the context of the measurement capability of the laboratory or the need to adjust the measurement error to allow for environmental decisions. In evaluating the concepts, EPA will Chapter 4 Draft Document for Peer Review  Do Not Circulate  August 2002 4 5 give preference to those concepts that strike a reasonable balance between using state of the art laboratories and a highly varied community of laboratories to establish quantitation limits.

4.6 Criterion 6 Criterion 6: Detection and quantitation concepts must be applicable to the variety of decisions made under the Clean Water Act, and should support state and local obligations to implement measurement requirements that are at least as stringent as those set by the Federal Government.

The Clean Water Act requires EPA to conduct, implement, and oversee a variety of data gathering programs. As noted in Section 3.2 of this TSD, these programs include, but are not limited to:

° Survey programs to establish baselines and monitor changes in ambient water quality; ° Screening studies to identify emerging concerns and establish the need for more in depth assessment; ° Effluent guideline studies to establish technology based standards for the control of pollutants in wastewater discharges; ° Toxicity and environmental assessment studies to establish water quality based standards for the control of pollutants in wastewater, and ° Risk assessment studies designed to characterize and evaluate human health and environmental risks associated with various water body uses.

In addition, EPA needs to apply detection to permitting, compliance monitoring, and other uses of the 40 CFR part 136 methods. These applications include:

C Permitting; C Ambient and effluent compliance monitoring under NPDES and the pretreatment program; C Ambient and effluent compliance monitoring under State and local programs; C Quality control in analytical laboratories; and C Method promulgation In theory, EPA could evaluate each of these applications independently and identify a detection and quantitation limit concept that is best suited to each application. Such an approach could potentially result in the need for up to 10 different detection and/ or quantitation limit concepts. EPA believes that such an approach would increase confusion, increase record keeping burdens, and increase laboratory testing burdens. For these reasons, EPA believes it is desirable to adopt a single pair of related detection and quantitation procedures that can be used to address all Clean Water Act applications.

EPA also believes that 1) it is unrealistic to expect other organizations, such as the U. S. Geological Survey, the Food and Drug Administration, ASTM International, AOAC, etc., to adopt and standardize on the concept selected by EPA for it's use in CWA programs, and 2) it is desirable to allow use of concepts and methods developed by these and other organizations to be used in CWA programs. The inclusion of such concepts and methods provides the stakeholder community with increased measurement options that may help reduce measurement costs or improve measurement performance for specific situations. This approach is consistent with EPA's movement to a performance based measurement system ( PBMS) and with the intent of the National Technology Transfer and Advancement Act ( NTTAA). So, although EPA prefers to identify and adopt a single pair of detection and quantitation limit concepts that can meet CWA needs, EPA also believes that any concept should be acceptable for use if it meets all of the criteria established above and fulfills the needs of the specific CWA application in which it should be used. Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 4 6 The Clean Water Act authorizes State or local governments to implement specific aspects of the Act, with the proviso that they do so in a way that is at least as protective ( i. e., stringent) as the national standards put forth by EPA. Therefore, this criterion is intended to ensure that any detection and quantitation limit concept adopted by the Federal Government is sufficiently clear and defined that it allows for comparison with concepts adopted by State or local governments. It is important to note that this criterion does not establish the need for a concept or procedure that is less stringent than those already in use by State or local governments.

Finally, it is important to differentiate between detection and quantitation limit concepts and compliance evaluation thresholds. Detection and quantitation limit concepts pertain to measurement process thresholds. More specifically, a detection limit describes the lowest concentration at which it possible to reliably determine that a substance is present, and a quantitation limit describes the lowest concentration at which it is possible to reliably quantify the amount of a substance that is present. In contrast, compliance evaluation thresholds are used to support wastewater discharge limits established in National Pollutant Discharge Elimination System ( NPDES) or pretreatment program permits. Such limits are usually expressed as either a maximum concentration of pollutant allowed in the discharge or a maximum mass of pollutant allowed to be discharged in a specific time period.

Ideally, analytical methods are available to allow for detection and quantitation of pollutants at concentrations that are lower than the discharge levels needed to protect or restore the quality of the receiving water. When such measurement capability does not exist, permitting authorities must decide how to incorporate detection and quantitation limits into the discharge permit. Historically, EPA has recommended that in such cases, the permitting authority include the water quality based limit in the permit, but establish the compliance evaluation threshold at the quantitation limit of the most sensitive available method. However, as with other aspects of the Clean Water Act, State and local governments may adopt permitting and compliance evaluation approaches that are at least as stringent as those put forth by EPA, and some States have preferred to use the detection limit as the compliance evaluation threshold.

This criterion will be evaluated by studying 1) the applicability of various detection/ quantitation concepts to the variety of data gathering decisions that must be made under the CWA, including those that do and those that do not involve compliance monitoring, and 2) the ability of the concepts to support state and local obligations for implementing CWA. Draft Document for Peer Review  Do Not Circulate  August 2002 5 1 Chapter 5 Assessment This chapter summarizes EPA's assessment of various detection and quantitation limit concepts against the evaluation criteria established in Chapter 4. Assessments of detection limit concepts are presented in Section 5.1 and include an assessment of:

° the EPA method detection limit ( MDL; Section 5.1.1), ° the ASTM International interlaboratory detection estimate ( IDE; Section 5.1.2), ° the American Chemical Society ( ACS) limit of detection ( LOD; Section 5.1.3), ° the International Standards Organization/ International Union of Pure and Applied Chemistry ( ISO/ IUPAC) critical value ( CRV; Section 5.1.4), and ° the ISO/ IUPAC minimum detectable value ( MDV; Section 5.1.5).

Assessments of quantitation limit concepts are presented in Section 5.2 and include an assessment of:

° the EPA minimum level of quantitation ( ML; Section 5.2.1), ° the ASTM International interlaboratory quantitation estimate ( IQE; Section 5.2.2), ° the ACS limit of quantitation ( LOQ; Section 5.2.3), and ° the ISO/ IUPAC LOQ ( section 5.2.4).

A brief summary of the evaluation is presented in Tables 5 1 ( detection limit concepts) and 5 2 ( quantitation limit concepts).

EPA limited the assessment to detection and quantitation limit concepts advanced by ASTMInternational by ACS, by ISO/ IUPAC, and by EPA, for use in EPA's Clean Water Act ( CWA) programs, because these concepts are the most widely published and pertinent.

5.1 Detection Limit Concepts Sections 5.1.1 through 5.1.5 describe EPA's assessment of five detection limit concepts. Each discussion is divided into two major subsections. The first subsection describes the concept and, where applicable, the procedure that supports the concept, and the second subsection details EPA's assessment of the concept based on the five criteria established in Chapter 4 for evaluating detection limit concepts. ( Six criteria are given in Chapter 4; four of these pertain to both detection and quantitation limit concepts, one pertains only to detection limit concepts, and one pertains only to quantitation limit concepts.).

5.1.1 Evaluation of the MDL Section <IP_ADDRESS> provides an overview of the MDL concept and the procedures used to implement the concept. Section <IP_ADDRESS> describes EPA's assessment of the MDL against the five evaluation criteria that concern detection limit concepts.

<IP_ADDRESS> Description of the MDL Concept and Procedure As promulgated at 40 CFR Part 136, Appendix B, the MDL is defined as:

" the minimum concentration of a substance that can be measured and reported with 99% confidence that the analyte concentration is greater than zero and is determined from analysis of a sample in a given matrix containing the analyte." Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 5 2 A six step procedure is given in Appendix B, with an optional seventh step to verify the reasonableness of the MDL determined in the first six steps. The procedure is intended for use by experienced analytical chemists. A brief summary of the MDL procedure is as follows:

1. The analyst makes an estimate of the detection limit based on one of four options: the instrument signal to noise ratio; three times the standard deviation of replicate blank measurements; a noted break in the slope of an instrument calibration curve; or known instrument limitations.

2. The analyst prepares a volume of reagent water that is as free of the target analyte as possible ( if the MDL is to be determined in reagent water).

3. The analyst prepares a sufficient volume of spiked reagent water ( or of an alternate matrix) to yield seven replicate aliquots that have a concentration of the target analyte that is at least equal to or in the same concentration range as the estimated detection limit ( it is recommended that the concentration of the replicate aliquots be between 1 and 5 times the estimated detection limit).

4. All of the replicate aliquots are processed through the entire analytical method.

5. The variance ( S2) and standard deviation ( S) of the replicate measurements, as follows:

where:

Xi; from i= 1 to n, are the analytical results in the final method reporting units obtained from the n sample aliquots and E refers to the sum of the X values from i= l to n.

6. The MDL is then determined by multiplying the standard deviation ( S) by the Student's t statistic at a 99% percentile for n 1 degrees of freedom. If seven replicates are used, the Student's t value is 3.143. This information is used to calculate the MDL as follows:

where:

MDL = the method detection limit t( n 1,1" = .99) = the Student's t value appropriate for a 99% confidence level and a standard deviation estimate with n 1 degrees of freedom, and Chapter 5 Draft Document for Peer Review  Do Not Circulate  August 2002 5 3 S = standard deviation of the replicate analyses.

A 95% confidence interval for the determined MDL may be calculated from percentiles of the Chisquared over degrees of freedom distribution ( P2/ df).

7. The optional iterative procedure to verify the reasonableness of the MDL involves spiking the matrix at the MDL that was determined in Step 6, and analyzing another seven replicates spiked at this level. The F ratio of the variances ( S2) is determined and compared with the F ratio found in the table, which is 3.05. If S2 A/ S2 B> 3.05, the analyst is instructed to respike at the most recently calculated MDL and process the samples through the procedure starting with Step 4. If S2 A/ S2 B> 3.05, then the pooled standard deviation is determined. The pooled standard deviation is then used to calculate the final MDL as follows:

where 2.681 is equal to t( 12, 1" =. 99).

The 95% confidence limits around the final MDL may be determined using the Chi squared distribution.

The MDL procedure given at 40 CFR 136, Appendix B is described as being applicable 1) to a wide variety of sample types, ranging from reagent water containing the analyte of interest to wastewater containing the analyte of interest, and 2) a broad variety of physical and chemical measurements. To accomplish this, the procedure was made device or instrument dependent.

<IP_ADDRESS> Assessment of the MDL Against the Evaluation Criteria The following five subsections discuss the MDL concept and procedure in the context of the five evaluation criteria that concern detection limit concepts.

<IP_ADDRESS>.1 Criterion 1: The detection limit concept must be scientifically valid.

For the purposes of evaluating scientific validity, EPA is using the conditions established by the Supreme Court in Daubert v. Merrell Dow Pharmaceuticals ( see Chapter 4, Criterion 1).

Condition 1: It can be ( and has been) tested. The MDL procedure meets this condition. The MDL has been used experimentally since 1980 and in a regulatory context since 1984. The MDL procedure is the most widely used and, therefore, the most widely tested detection limit procedure in the history of concepts of detection.

Critics of the MDL have noted that the detection limit produced with the MDL procedure can vary depending on the spike levels used. This would suggest, on the surface, that the MDL procedure can be used to obtain results that do not support the MDL concept. This is a misinterpretation of the MDL based on the mistaken assumption that spike levels may be arbitrarily selected. In fact, step 1) of the MDL procedure specifies a number of criteria, based on chemical analytical considerations, that must be met in selecting the spike levels ( see Section <IP_ADDRESS>, Step 1).

In preparation for the assessment of detection and quantitation concepts, EPA exhaustively tested the MDL procedure with 10 different techniques, at decreasing spike concentrations, to evaluate this concern and determine how well the MDL procedure characterized the region of interest. Results of the study suggest that, Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 5 4 although the calculated MDL could vary depending on the spike level used, the procedure was capable of reasonably estimating a detection limit when the full iterative procedure was employed.

Condition 2: It has been subjected to peer review and publication. The MDL meets this condition. Prior to promulgation by EPA, the MDL concept and supporting procedure was published by Glaser et al in a peer reviewed journal ( Glaser et al., 1981).

Condition 3: The error rate associated with the procedure is either known or can be estimated. It is possible to estimate error rates associated with the MDL procedure. It is also possible to calculate confidence intervals about estimated MDLs that are expressions of uncertainty in the estimates. Clarification is in order because the promulgated MDL definition may be somewhat confusing in some respects. In particular, the definition is confusing with regard to whether the MDL is a true concentration or a value estimated from measured data. EPA believes that the concept of MDL can be clarified by slightly revising the definition as follows:

" The method detection limit ( MDL) is the measured concentration at which there is 99% confidence that a given analyte is present in a given sample matrix. The MDL is estimated from replicate analyses of a matrix containing the analyte."

Condition 4: Standards exist and can be maintained to control its operation. The MDL concept is supported by a clearly defined, published procedure to control its operation. The procedure gives the steps to be followed and instructs the analyst to use the entire measurement process. Hundreds, if not thousands, of laboratories have successfully implemented the MDL procedure since its promulgation in 1984. EPA has found that when laboratories are required to perform MDL studies as part of an interlaboratory study, the results reported by the laboratories are generally consistent ( i. e., within the expected variability). EPA has observed similar consistency in use of the MDL by laboratories required to perform the procedure to demonstrate proficiency with a method. Therefore, the MDL meets this condition.

That said, however, EPA believes that additional guidance can be provided to clarify certain aspects of the MDL procedure, particularly with respect to handling of outliers, the optional reasonableness step, and multi analyte test methods. The MDL procedure contains no discussion of outliers. It may be helpful to clarify that 1) results should be discarded only if the results are associated with a known error that occurred during analysis ( e. g., the replicate was spiked twice) or through a statistically accepted analysis of outliers, and 2) that laboratories should not run more than seven replicates and simply pick the best of the seven results. The optional step involves iterative testing to verify that the determined MDL is reasonable; EPA has observed that few organizations bother to perform this step. EPA also has observed that when a method involves a large number of analytes, it can be difficult to get all analytes to pass the iterative test in the same run. Additional guidance on this issue may be needed if the iterative test is to become a required component of the MDL, as suggested above in the discussion of Condition 3.

Condition 5: It has attracted widespread acceptance within a relevant scientific community. The MDL meets this condition. Within EPA, the MDL has been used by the Office of Research and Development, Office of Science and Technology, Office of Ground Water and Drinking Water, Office of Solid Waste, Office of Emergency and Remedial Response, and other offices. The MDL also has been used outside of EPA in methods published by ASTM International, in Standard Methods for the Examination of Water and Wastewater, published by the American Public Health Association ( APHA, the American Water Works Association ( AWWA), and the Water Environment Federation ( WEF), and elsewhere. Although the MDL has been criticized by some, EPA believes that it is the most widely used concept of detection within the environmental chemistry community. Many States incorporate the MDL into NPDES permits, for example, and laboratories often advertise MDLs in their sales literature. Chapter 5 Draft Document for Peer Review  Do Not Circulate  August 2002 5 5 <IP_ADDRESS>.2 Criterion 2: The concepts must address demonstrated expectations of laboratory and method performance, including routine variability.

The MDL procedure is designed to demonstrate laboratory performance with a given method, and can be applied to a broad variety of physical and chemical methods. To accomplish this, the procedure was made device or instrument independent. The procedure also recognizes the importance of analyst experience, and explicitly directs the analyst to employ all sample processing and computation steps given in the analytical method when determining the MDL. ( All of these aspects are addressed in the MDL procedure published at 40 CFR 136, Appendix B).

When the MDL procedure is followed as intended, i. e., the MDL is determined by an experienced analyst on each device or instrument used for a given method, the demonstrated MDL will include routine variability associated with the laboratory and the method.

EPA recognizes, however, that one laboratory may obtain detection limits that are lower or higher than those in another laboratory. If the MDL is being determined during method development, it is important to determine the MDL at more than one laboratory to ensure the MDL published in the method reflects demonstrated expectations of method performance in a community of laboratories. EPA does not believe that this community should be so broad as to include the entire universe of possible laboratories that might desire to practice the method. Rather, EPA believes this community should include well operated laboratories that are experienced with the techniques used in the method and that have some familiarity with the method.

In recent years, EPA's Office of Science and Technology has used single laboratory studies to develop an initial estimate of the MDL for a new or modified method, and has verified these limits in interlaboratory studies or by conducting additional single laboratory studies in other laboratories. For example, when EPA initially drafted Method 1631 for measurement of mercury, EPA estimated the MDL to be 0.05 ng/ L based on results produced by a contract research laboratory. Additional single laboratory MDL studies conducted in other laboratories suggested that the MDL should be raised to 0.2 ng/ L to better reflect existing capabilities of the measurement community. During EPA's interlaboratory study, each laboratory was asked to conduct an MDL study. Every laboratory in the interlaboratory study met the MDL of 0.2 ng/ L, the value published in the promulgated version of Method 1631.

EPA believes that 1) the MDL procedure does address demonstrated expectations of laboratory and method performance, including routine variability, and 2) if the MDL procedure is being employed for method development purposes, it should be performed in multiple laboratories to ensure that it adequately demonstrates expectations in a community of qualified laboratories.

<IP_ADDRESS>.3 Criterion 3: The detection limit concept must be supported by a practical and affordable procedure that a single laboratory can use to evaluate method performance.

The MDL is designed for use by a single laboratory. The promulgated version of the MDL procedure can be performed with as few as seven analyses. If the method is to be performed in a matrix other than reagent water, additional analyses may be needed to determine the MDL in that matrix.

Use of the optional iterative procedure would increase the number of analyses by seven each time the procedure is implemented. If the procedure is implemented two times in reagent water, a total of 14 analyses are required. If the procedure is implemented two times in an alternate matrix, EPA estimates that 17 20 analyses may be required. In any of these scenarios, the entire MDL determination can be performed in a single analytical batch ( most EPA methods specify batch sizes of 20 samples). Based on this, EPA believes that the MDL is among the most affordable of the procedures that have been suggested for Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 5 6 determining detection limits. In terms of cost, the only concept that compares favorably with the MDL is the instrument detection limit ( IDL). Although most versions of the IDL compare favorably in terms of the number of samples analyzed, the requirement to perform the test on three non consecutive days has the potential to disrupt routine laboratory operations on three days instead of one. In addition, the IDL does not include sample preparation steps and, therefore, does not completely characterize a method.

<IP_ADDRESS>.4 Criterion 4: The detection limit concept should result in a measured concentration at which there is 99% confidence that a substance will be detected when the analytical method is performed by experienced staff in a well operated laboratory.

EPA believes the MDL meets this condition and refers the reader to the discussion of this subject under Section <IP_ADDRESS>.1, Condition 3.

<IP_ADDRESS>.5 Criterion 5: The detection limit concept must be applicable to the variety of decisions made under the Clean Water Act and should support State and local obligations to implement measurement requirements that are at least as stringent as those set by the Federal Government.

The MDL meets this criterion. The MDL has been successfully applied to a variety of decisions under the CWA since 1984. In addition, many States and others have adopted the MDL in their own programs.

5.1.2 Evaluation of the ASTM International Interlaboratory Detection Estimate ( IDE)

The interlaboratory detection estimate ( IDE) was developed by ASTM with support from members of the regulated industry in an attempt to provide a scientifically sound, comprehensive detection limit procedure that addresses the concerns of the regulated industry, of statisticians, and of analysts involved in ASTM Committee D 19 on water.

A brief summary of the procedure is given in Section <IP_ADDRESS> and Section <IP_ADDRESS> presents EPA's assessment of the IDE against the five criteria established for evaluating detection limit concepts.

<IP_ADDRESS> Description of the IDE Concept and Procedure ASTM Designation D 6091 is the Standard Practice for 99 %/ 95 % Interlaboratory Detection Estimate ( IDE) for Analytical Methods with Negligible Calibration Error. As stated in the practice:

" The IDE is computed to be the lowest concentration at which there is 90 % confidence that a single measurement from a laboratory selected from the population of qualified laboratories represented in an interlaboratory study will have a true detection probability of at least 95 % and a true nondetection probability of at least 99 % ( when measuring a blank sample)." Chapter 5 Draft Document for Peer Review  Do Not Circulate  August 2002 5 7 The IDE is determined and verified using a procedure containing 5 major steps with approximately 53 substeps and conditions. The full text of the IDE procedure is available from ASTM International. The five major steps and their functions are given in Section 6 of the IDE procedure and are as follows:

1. Overview of the procedure.

2. IDE Study Plan, Design, and Protocol  in this section, the task manager ( study supervisor) chooses the analyte, matrix, and analytical method. Details are given for range finding; the concentrations to be used in the study; the study protocol ( ASTM Practice D 2777 is suggested); the allowable sources of variation; and the number of laboratories, analysts, and days, over which the study will be conducted.

3. Conduct the IDE Study, Screen the Data, and Choose a Model  after the study data are collected and screened according to ASTM Practice D 2777, interlaboratory standard deviation ( ILSD) versus concentration data are tabulated and one of three models is fit to the data. The first attempt is at fitting a constant model. If the attempt fails, a straight line model is attempted. If the straight line model fails, an exponential model is fitted. After fitting, the model is evaluated for reasonableness and lack of fit. If the model fails, the study supervisor determines if a subset of the data should be analyzed or if more data are needed.

4. Compute the IDE  the IDE is computed using the ILSD model selected in Step 3 to estimate the interlaboratory standard deviation at a true concentration of zero and at the IDE, using a mean recovery model to transform measured and true concentrations. The IDE is computed as a one sided 90 % confidence upper statistical tolerance limit or interval.

5. Nontrivial Amount of Censored Data  this section addresses the effect of " non detects" or " lessthans Suggestions are given to see if uncensored data can be obtained from the laboratories or if the study needs to be augmented with additional data. Suggestions are given for fitting a model to data that contain less than 10 % non detects or less thans to produce an IDE.

<IP_ADDRESS> Assessment of the IDE Against the Evaluation Criteria The following five subsections discuss the IDE concept and procedure in the context of the five evaluation criteria that concern detection limit concepts.

<IP_ADDRESS>.1 Criterion 1: The detection limit concept must be scientifically valid ( validity is comprised of five conditions)

Condition 1: It can be ( and has been) tested. EPA is not aware of any organization, including ASTM, that has conducted a study to test the procedure as written ( i. e., designed and implemented an interlaboratory study that involves estimating an initial IDE ( IDE0) and multilaboratory analyses of multiple concentrations of each matrix of interest surrounding IDE0). Developers of the concept performed limited testing of the concept on 1) simulated datasets and 2) real world data sets generated for other purposes. These real world data sets, however, are of limited value for testing the IDE because the concentrations ranges associated with the data are above the low level region of interest. As part of this reassessment, EPA tested a variant of the IDE procedure on single laboratory datasets designed for characterization of an analytical method in the region of detection. Despite the lack of comprehensive testing, EPA believes that the procedure can be tested, and therefore meets part of this condition. Specifically, the IDE meets the condition that it can be tested, but it only partially meets the condition that it has been tested. Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 5 8 Condition 2: It has been subjected to peer review and publication. Although the IDE has not been published in the peer reviewed scientific literature, the IDE has undergone extensive review and ballot by members of ASTM Committee D 19, many of whom are qualified peer reviewers. Therefore, although the IDE does not meet this condition in the sense of formal peer review and publication, EPA believes it does meet the intent of this condition ( i. e., submission to scrutiny of the scientific community).

Condition 3: The error rate associated with the procedure is either known or can be estimated. In theory, expert statisticians could estimate the error rate of the IDE. However, the IDE procedure is extremely complex from an analytical chemistry and statistical perspective. As a result, it is unlikely that the error rate could be estimated by the typical users of the analytical method to which it is applied, or even to the typical developers of analytical methods. Moreover, EPA found the model selection procedure to be highly subjective, a situation likely to yield different IDEs from the same dataset, depending on the staff involved in performing the calculations. In practice, such conditions make it impossible to estimate the actual error associated with the IDE. Therefore, the IDE fails this condition.

Condition 4: Standards exist and can be maintained to control its operation. The IDE concept and procedure is supported by a published procedure ( standard) to control its operation. The procedure gives the steps to be followed in determining the IDE and instructs the study supervisor how to gather the data and compute an IDE.

However, there are several " gray areas" in the published procedure. The most significant gray area is in the description of model selection. The procedure provides insufficient guidance on use of residual plots to evaluate and select models and, as a result, selection of the model may be very subjective, especially if the number of concentrations is low. The discussion of what model to use after rejecting the exponential and linear model is also very vague. The Rocke and Lorenzato ( hybrid) model is mentioned, as well as models with more than one coefficient. Much of the data evaluated by EPA have tended to suggest the exponential model, based on the statistical tests discussed, but those data have almost always shown residual " patterns" when using this model, which would then lead to consideration of other models. In addition, fitting the constant model is never discussed in detail. Most likely, this is simply done by calculating a mean ( weighted if necessary) of the variances from the different concentrations; however, this is never explicitly stated.

Another concern with the standard is that it gives procedures that are inconsistent with procedures given in the IQE standard, even though the two concepts should be consistent for a given analyte with a given method. For example, the exponential model figures prominently in the IDE procedure, where it is one of the three main models discussed. The Rocke and Lorenzato model is not discussed in the IDE procedure, but it figures prominently in the IQE procedure. In theory, a single model should support the definition of both the detection and quantitation limits for a given analyte by a given method. As another example, the IDE procedure includes a multiplier to account for bias in estimating the true standard deviation with the sample standard deviation, but the IQE does not.

Finally, the procedure contains statistical errors that, if followed as written, could produce inaccurate IDE values. For example, Table 1 of the procedure contains " Computations to Estimate Straight Line Model Coefficients by Means of Least Squares   Ordinary and Weighted," but the weighted least squares formulas given in the table are incorrect. The formulas for the weighted means of the spike values and results given in Table 1 of D6091 would only be appropriate if the weighting were done based on the number of replicates per spike level, rather than on the estimated variance calculated using the chosen standard deviation model.

In conclusion, EPA believes that although the IDE is supported by a published procedure, that procedure will not control its operation because of the degree of subjectivity involved implementing the Chapter 5 Draft Document for Peer Review  Do Not Circulate  August 2002 5 9 procedures, errors in the procedure, and internal inconsistencies with its IQE counterpart. Therefore, the IDE fails this condition.

Condition 5: It has attracted widespread acceptance within a relevant scientific community. The IDE fails this condition because it is only familiar to, and has been accepted by, a very narrow segment of the scientific community. Although the IDE has been approved by ASTM for more than 5 years, EPA is not aware of an IDE that has been published in the open literature or in an analytical method, including an ASTM method.

<IP_ADDRESS>.2 Criterion 2: The detection limit concept must address demonstrated expectations of laboratory and method performance, including routine variability.

The IDE procedure is designed to reflect expectations of interlaboratory performance, including routine variability. The procedure contains extensive instructions for dealing with unusual conditions, including sources of variability. However, EPA studies of a single laboratory variant of the procedure, suggested that the procedure may not always work as intended. For example, model selection based upon hypothesis tests ( as described in D6091, Section <IP_ADDRESS>) almost always indicated that the exponential model should be used even when the data seemed to be show constant or approximately linear error, while examination of residual plot indicated " systematic behavior" ( i. e., non random deviations from the model) for the exponential and linear models. Another concern with the IDE procedure is that use of the nonmandatory appendices in ASTM D 6512 to determine the fit of a model may produce results that differ from those that would be obtained by using the default procedures for testing model fit that are built into off theshelf statistical software. Such observations, along with the concerns described in Section <IP_ADDRESS>.1, condition 4, lead EPA to believe that, while the IDE concept addresses demonstrated expectations of laboratory and method performance, the IDE procedure does not adequately do so. Therefore, the IDE only partially meets this criterion.

<IP_ADDRESS>.3 Criterion 3: The detection limit concept must be supported by a practical and affordable procedure that a single laboratory can use to evaluate method performance.

The IDE procedure is designed for use by an ASTM study supervisor or task manager and not as a procedure that a single laboratory can use to evaluate method performance. EPA is aware that ASTM Committee D 19 is developing a Within laboratory Detection Estimate ( WDE), but the WDE is presently only in the formative stages. The WDE may meet this criterion, but the IDE does not.

Regarding cost, the IDE procedure would be the most costly of the procedures that EPA has evaluated because of time it would take to understand and implement the procedure, and requirements for: ( 1) estimation of IDE0, ( 2) interlaboratory data, ( 3) extensive statistical intervention in determining the correct model, and ( 4) possible reanalyses if the resulting IDE do not meet the criteria in the procedure.

<IP_ADDRESS>.4 Criterion 4: The detection limit concept should result in a measured concentration at which there is 99% confidence that a substance will be detected when the analytical method is performed by experienced staff in a well operated laboratory.

The IDE definition states that the IDE provides a 95% probability of detection. However, setting aside the use of a 95% figure and the fact that the IDE is an interlaboratory concept that cannot be applied in a single laboratory, the IDE procedure does not satisfy this criterion because it includes an allowance for false negatives and a tolerance interval that inappropriately increase the detection limit. For a discussion of this issue, see sections 3.3.6 ( false positives and false negatives) and 3.3.7 ( prediction and tolerance intervals) in Chapter 3 of this document. When the allowance for false negatives and the prediction and tolerance intervals are taken into account, the resulting detection limit ( IDE) is raised to the point at which the Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 5 10 probability of a false positive is less than 0.00000001 ( 10 8). This false positive rate is excessive and would yield numerical values of little practical value. The IDE therefore fails this criterion.

<IP_ADDRESS>.5 Criterion 5: The detection limit concept must be applicable to the variety of decisions made under the Clean Water Act and should support State and local obligations to implement measurement requirements that are at least as stringent as those set by the Federal Government.

EPA's comparison of detection limits produced by various detection limit concepts shows that the median IDE is considerably higher than ACS, ISO/ IUPAC, and EPA detection limits. Although the IDE could be applied to some decisions to be made under CWA, it may not support decisions when pollutant levels need to be protective of human health and the environment because the IDE may be considerably higher than these levels. At best, the IDE only partially meets this criterion.

5.1.3 Evaluation of the ACS Limit of Detection The limit of detection ( LOD) was developed by the Committee on Environmental Improvement ( CEI) of the American Chemical Society ( ACS). ACS is a professional society for chemists and other scientists and the publisher of a number of scientific journals. It is not a voluntary consensus standards body ( VCSB), nor does it develop or publish analytical methods. In 1978, the ACS/ CEI began addressing concerns about the lack of useful standards for interlaboratory comparisons. In 1980, the Committee published its " Guidelines for Data Acquisition and Data Quality Evaluation in Environmental Chemistry" ( Analytical Chemistry, 52 2242 2249), which included the concepts of the LOD and the limit of quantitation ( LOQ).

<IP_ADDRESS> Description of the ACS LOD The 1980 " Guidelines" define the LOD as:

"... the lowest concentration of an analyte that the analytical process can reliably detect. ... The LOD in most instrumental methods is based on the relationship between the gross analyte signal St, the field blank Sb, and the variability in the field blank F b ."

and construct the formal relations using the equation:

Where Kd is a constant. ACS recommended a minimal value of 3 for Kd. Thus, the LOD is 3F above the gross blank signal, Sb. In the 1980 publication, the ACS stated that at Kd = 3, there is a 7% risk of false negatives and false positives. Given that the LOD is 3F above the blank, however, EPA believes that the risk of false positives is somewhat less than 1%.

In 1983, the ACS Committee published " Principles of Environmental Analysis" ( Analytical Chemistry, 55 2210 2218). That publication occurred after the 1981 paper on the Method Detection Limit ( MDL), and ACS/ CEI stated that the LOD is numerically equivalent to the MDL as Sb approaches zero. However, neither the 1980 nor 1983 ACS publications provide a specific procedure for estimating the LOD, nor do they provide a minimum number of observations needed to estimate the gross blank signal or the variability term Fb. Chapter 5 Draft Document for Peer Review  Do Not Circulate  August 2002 5 11 <IP_ADDRESS> Assessment of the LOD Against the Evaluation Criteria The following five subsections discuss the LOD concept and procedure in the context of the five evaluation criteria that concern detection limit concepts.

<IP_ADDRESS>.1 Criterion 1: The detection limit concept must be scientifically valid ( validity is comprised of five conditions)

Condition 1: It can be ( and has been) tested. Testing of the ACS LOD is hampered by 1) the lack of a supporting procedure for establishing an LOD, and 2) it's conceptual dependence on the variability associated with measuring blanks. For example, there is no procedure to govern the minimum number of analyses needed to characterize the variability of a blank sample. Because many environmental chemistry techniques yield a zero, or possibly even negative, value when a blank sample is analyzed, and because the LOD concept is based on the standard deviation of these results, directly testing the LOD in such techniques will yield a zero or negative value. One solution for testing is to rely on ACS' 1983 statement that the LOD is conceptually equivalent to the MDL as the blank signal approaches zero, and employ the MDL procedure as a means for indirectly testing the LOD concept. EPA believes that use of the MDL procedure is a viable means for testing the concept; therefore, the LOD meets this condition.

Condition 2: It has been subjected to peer review and publication. The LOD definition was published in the peer reviewed journal Analytical Chemistry in 1980 and 1983. Therefore, the LOD meets this condition.

Condition 3: The error rate associated with the procedure is either known or can be estimated. The error rates can be estimated, so the LOD meets this condition. The error rate for both false positives and false negatives is stated to be 7 % in the 1980 Analytical Chemistry article. However, EPA believes that, because the LOD is stated to be 3 times the standard deviation of replicate measurements of a blank, the false positive rate is overstated and is actually somewhat less than 1 % whereas the false negative rate depends on the true concentration in the sample.

Condition 4: Standards exist and can be maintained to control its operation. The LOD lacks a clearly defined procedure for estimating the important terms required to derive it. Although it may be possible to derive LOD values from data used to derive EPA MDL values, there is no procedure giving explicit instructions on the use of replicate blanks, replicate spiked samples, or a minimum recommendation for the number of replicates. Therefore, the LOD fails this condition.

Condition 5: It has attracted widespread acceptance within a relevant scientific community. Because ACS does not develop and publish analytical methods, it is difficult to determine the degree of acceptance of the LOD. EPA has not specifically investigated the numbers of papers published in ACS journals that include LOD values, and EPA's literature search for detection and quantitation concepts did not uncover a large number of citations that promote the LOD in particular. However, ACS LOD values have appeared in the technical literature. Given that ACS is a relevant scientific community, and that use of the LOD has appeared in the technical literature, EPA believes the LOD meets this condition.

<IP_ADDRESS>.2 Criterion 2: The detection limit concept must address demonstrated expectations of laboratory and method performance, including routine variability.

The LOD concept is designed to address demonstrated expectations of laboratory and method performance, including routine variability, so it appears to meet this criterion. Unfortunately, ACS has not published a procedure to implement the concept. In other words, the LOD addresses demonstrated Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 5 12 expectations of laboratory and method performance in theory, but in practice, provides no direct means for performing these demonstrations. Therefore, EPA believes the ACS LOD only partially meets this criterion.

<IP_ADDRESS>.3 Criterion 3: The detection limit concept must be supported by a practical and affordable procedure that a single laboratory can use to evaluate method performance.

The ACS LOD concept is not supported by a clearly defined procedure for establishing the LOD. Therefore, it fails this criterion.

<IP_ADDRESS>.4 Criterion 4: The detection limit concept should result in a measured concentration at which there is 99% confidence that a substance will be detected when the analytical method is performed by experienced staff in a well operated laboratory.

The 1983 publication associated the LOD with the " 99% confidence level when the difference ( St  Sb) > 3F." Therefore, the LOD satisfies this criterion.

<IP_ADDRESS>.5 Criterion 5: The detection limit concept must be applicable to the variety of decisions made under the Clean Water Act and should support State and local obligations to implement measurement requirements that are at least as stringent as those set by the Federal Government.

In the absence of a procedure for determining LOD values, the ACS LOD fails to meet this criterion because it cannot be used in a regulatory context. The LOD passes only if it is assumed to be functionally equivalent to the MDL ( i. e., the MDL procedure is used to establish an LOD).

5.1.4 Evaluation of the IUPAC/ ISO Critical Value ( CRV)

The critical value ( CRV) was developed by the International Union of Pure and Applied Chemistry ( IUPAC) and the International Organization for Standardization ( ISO). IUPAC and ISO are professional societies for chemists and other scientists. ISO develops and publishes analytical methods through its Task Groups. In 1995, Lloyd Currie of the National Institute for Standards and Technology ( NIST; formerly the National Bureau of Standards) published a signature discussion of IUPAC concepts for detection and quantitation ( Pure and Appl. Chem. 67: 10, 1699 1722). Although refined during the intervening years ( see Currie, L. A., J. Radiochem. And Nuclear Chem. 245: 1, 145 156, 2000), the CRV concept remains basically as described in 1995.

<IP_ADDRESS> Description of the ISO/ IUPAC Critical Value ( CRV) Concept and Procedure The 1995 article states that the critical value ( Lc) is:

"... the minimum significant value of an estimated net signal or concentration, applied as a discriminator against background noise. This corresponds to a 1 sided significance test."

For a normal distribution with known variance, Lc reduces to:

Lc = z( 1") F0 where: 1" is the false positive error rate, recommended at 5 % (" = 0.05), and Chapter 5 Draft Document for Peer Review  Do Not Circulate  August 2002 5 13 F0 is the standard deviation at zero concentration If F0 is estimated by s0 ( replicate measurements of a blank), z( 1") is replaced by the Student's t value. For 7 replicates ( 6 degrees of freedom), the Student's t value is 1.943, where " = 0.05.

<IP_ADDRESS> Assessment of the CRV Against the Evaluation Criteria The following five subsections discuss the CRV concept and procedure in the context of the five evaluation criteria that concern detection limit concepts.

<IP_ADDRESS>.1 Criterion 1: The detection limit concept must be scientifically valid. ( Validity is comprised of five conditions.)

Condition 1: It can be ( and has been) tested. The lack of a supporting procedure for establishing the CRV, coupled with it's conceptual dependence on the variability of blank measurements makes testing of the concept difficult. For example, if blank measurements fail to produce a response, it is impossible to calculate a CRV because the standard deviation of zero is zero. One solution for testing the concept is to assume that the CRV is functionally equivalent to the MDL as the blank signal approaches zero, and use a slightly modified version of the MDL procedure to test the CRV concept. The slight modification involves selecting a Student's t value based on " = 0.05 instead of " = 0.01, for n 1 degrees of freedom. EPA believes this is a reasonable assumption, and therefore, that the MDL procedure is a viable means for testing the CRV concept. Therefore, the CRV meets this condition.

Condition 2: It has been subjected to peer review and publication. The IUPAC/ ISO definitions meet this criterion. Moreover, it is likely that these definitions have received greater peer review than any of the other concepts.

Condition 3: The error rate associated with the procedure is either known or can be estimated. The error rate is specified by ", suggested at 0.05 ( 5%). Therefore, the CRV meets this condition.

Condition 4: Standards exist and can be maintained to control its operation. The CRV is defined in the various publications by Currie. However, EPA's search of the ISO web site found no standard for control of the concept. Therefore, the CRV fails this condition.

Condition 5: It has attracted widespread acceptance within a relevant scientific community. Because IUPAC and ISO are international bodies, it is difficult to determine the degree of acceptance of the CRV in the U. S. and the world community. EPA has not specifically investigated the number of papers in published journals that include CRV values, but EPA's literature search for detection and quantitation concepts did not uncover a large number of citations that promote the CRV in particular. Therefore, it is difficult to determine if the CRV meets this condition.

<IP_ADDRESS>.2 Criterion 2: The detection limit concept must address demonstrated expectations of laboratory and method performance, including routine variability.

The CRV concept is designed to account for the variability of measurements of the blank in the context of a " chemical measurement process" ( method). Unfortunately, neither ISO, IUPAC, nor Currie have published a procedure to implement the concept. As a result, the CRV addresses demonstrated expectations of laboratory and method performance in theory, but in practice, provides no direct means for performing these demonstrations. Therefore, EPA believes the CRV partially meets this criterion. Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 5 14 <IP_ADDRESS>.3 Criterion 3: The detection limit concept must be supported by a practical and affordable procedure that a single laboratory can use to evaluate method performance.

The CRV concept is not supported by a clearly defined procedure for establishing a CRV. Therefore, the CRV fails this criterion.

<IP_ADDRESS>.4 Criterion 4: The detection limit concept should result in a measured concentration at which there is 99% confidence that a substance will be detected when the analytical method is performed by experienced staff in a well operated laboratory.

Although the CRV suggests " = 0.05, resulting in 1" of 0.95 or 95 % probability of detection, the concept allows specification of probabilities. Therefore, the CRV satisfies this criterion.

<IP_ADDRESS>.5 Criterion 5: The detection limit concept must be applicable to the variety of decisions made under the Clean Water Act and should support State and local obligations to implement measurement requirements that are at least as stringent as those set by the Federal Government.

In the absence of a procedure for establishing CRVs, the CRV concept fails to meet this criterion because it cannot be used in a regulatory context. The CRV passes only if it is assumed to be functionally equivalent to an MDL determined with alpha set at 0.05 instead of 0.01 ( i. e., if the MDL procedure, with "= 0.05, is used to establish a CRV).

5.1.5 Evaluation of the IUPAC/ ISO Detection Limit The detection limit or minimum detectable value ( MDV) was developed by IUPAC/ ISO and published in the same papers as the CRV ( Section 5.1.4)

<IP_ADDRESS> Description of the IUPAC/ ISO Detection Limit Procedure The 1995 publications define the minimum detectable value ( detection limit) as follows:

" The Minimum Detectable Value ( MDV) ... [ is] ... the net signal ( or concentration) of that value ( LD) for which the false negative error is $, given LC ( or ")." ( see the CRV for LC)

For a normal distribution with known variance, LD reduces to:

LD = z( 1$) FD where: 1$ is the false negative error rate, recommended at 5 % ($ = 0.05), and FD is the standard deviation at the detection limit Later publications refer to the minimum detectable value as the Detection Limit. To avoid confusion in terminology and to help distinguish the ISO/ IUPAC concept from the MDL, LOD, and CRV, EPA will refer to the ISO/ IUPAC detection limit as the Minimum Detectable Value, abbreviated as MDV. Chapter 5 Draft Document for Peer Review  Do Not Circulate  August 2002 5 15 <IP_ADDRESS> Assessment of the ISO/ IUPAC MDV Against the Evaluation Criteria The following five subsections discuss the ISO/ IUPAC MDV concept and procedure in the context of the five evaluation criteria that concern detection limit concepts.

<IP_ADDRESS>.1 Criterion 1: The detection limit concept must be scientifically valid. ( Validity is comprised of five conditions.)

Condition 1: It can be ( and has been) tested. The lack of a supporting procedure for establishing the MDV makes testing of the concept difficult. However, EPA believes that the MDV can be tested using data similar to those used to generate MDL values. Therefore, the MDV meets this condition.

Condition 2: It has been subjected to peer review and publication. The IUPAC/ ISO definitions meet this condition; moreover, it is likely that this definition has received greater peer review than any of the other concepts.

Condition 3: The error rate associated with the procedure is either known or can be estimated. The error rates are specified by " and $, both suggested at 0.05 ( 5 %), so the error rate is known.

Condition 4: Standards exist and can be maintained to control its operation. The MDV is defined in the various publications by Currie. However, EPA's search of the ISO web site found no standard for control of the concept. Therefore, the MDV fails this criterion.

Condition 5: It has attracted widespread acceptance within a relevant scientific community. Because IUPAC and ISO are international bodies, it is difficult to determine the degree of acceptance of the MDV in the U. S. and the world community. EPA has not specifically investigated the numbers of papers in published journals that include MDV values, but EPA's literature search for detection and quantitation concepts did not uncover a large number of citations that promote the MDV in particular. Therefore, it is difficult to determine if the CRV meets this criterion.

<IP_ADDRESS>.2 Criterion 2: The detection limit concept must address demonstrated expectations of laboratory and method performance, including routine variability.

The MDV concept is designed to account for the variability of measurements of the blank in the context of a " chemical measurement process" in the sense that it is used in concert with a critical value that is based on blank measurement variability. The MDV is the true concentration that is used in the planning of method evaluation and development. The actual detection decision is made at the critical value ( CRV) which is determined from measured values. The concept of a true concentration MDV and its associated allowance for false negatives is of little practical value in making the actual detection decision. Therefore, the MDV fails this criterion. The allowance for false negatives in a regulatory context is discussed in greater detail in Chapter 3.

<IP_ADDRESS>.3 Criterion 3: The detection limit concept must be supported by a practical and affordable procedure that a single laboratory can use to evaluate method performance.

The MDV concept is not supported by a clearly defined procedure for establishing MDV values. Therefore, the MDV fails this criterion. Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 5 16 <IP_ADDRESS>.4 Criterion 4: The detection limit concept should result in a measured concentration at which there is 99% confidence that a substance will be detected when the analytical method is performed by experienced staff in a well operated laboratory.

The allowance for false negatives raises the probability of detection to a value estimated to be greater than 99.999999 % ( probability of a false positive less than 10 8). This false positive rate is excessive and would yield " detection limits" that are of little practical value. Therefore, the MDV fails this criterion.

<IP_ADDRESS>.5 Criterion 5: The detection limit concept must be applicable to the variety of decisions made under the Clean Water Act and should support State and local obligations to implement measurement requirements that are at least as stringent as those set by the Federal Government.

In the absence of a procedure for establishing MDV values, the MDV concept fails to meet this criterion because it cannot be used in a regulatory context.

5.2 Quantitation Limit Concepts Sections 5.2.1 through 5.2.4 describe EPA's assessment of four quantitation limit concepts. Each discussion is divided into two major subsections. The first subsection describes the concept and, where applicable, the procedure that supports the concept, and the second subsection details EPA's assessment of the concept based on the five criteria established in Chapter 4 for evaluating quantitation limit concepts. ( Six criteria are given in Chapter 4; four of these pertain to both detection and quantitation limit concepts, one pertains only to detection limit concepts, and one pertains only to quantitation limit concepts.)

5.2.1 Assessment of the EPA Minimum level of Quantitation ( ML)

Section <IP_ADDRESS> provides an overview of the ML concept and the procedures used to implement the concept. Section <IP_ADDRESS> contains EPA's assessment of the ML against the five evaluation criteria that concern quantitation limit concepts.

<IP_ADDRESS> Description of the ML Concept and Procedures The present definition of the ML includes a statement of the concept and the procedures used to establish the ML. This definition states that the ML is:

" the lowest level at which the entire analytical system must give a recognizable signal and acceptable calibration point for the analyte. It is equivalent to the concentration of the lowest calibration standard, assuming that all method specified sample weights, volumes, and clean up procedures have been employed. The ML is calculated by multiplying the MDL by 3.18 and rounding the results to the number nearest to ( 1, 2, or 5) x 10n, where n is an integer. "

The ML is designed to provide a practical embodiment of the quantification level proposed by Currie and adopted by IUPAC. It is functionally analogous to the American Chemical Society's Limit of Quantitation ( LOQ). The LOQ is discussed in Section 5.2.3 of this chapter. Chapter 2 ( Section 2.2.2) describes the ML concept in detail. Chapter 5 Draft Document for Peer Review  Do Not Circulate  August 2002 5 17 The first part of the ML definition ( i. e., the lowest level at which the system gives a recognizable signal and acceptable calibration point for the analyte) ties the quantification limit to the capabilities of the measurement system. The second part of the ML definition provides a procedural means for establishing the ML.

The procedural component of the definition is designed to yield an ML value that equals approximately 10 times the standard deviation of replicate analyses used to determine the MDL. ( The exact value corresponding to 10 times the standard deviation is rounded to avoid error that would arise from preparation of calibration standards at exact, unrounded concentrations.) The procedure given in the above definition assumes that exactly seven replicates are used to determine the MDL. EPA has observed, however, that laboratories occasionally perform MDL studies with more than the required minimum of seven replicates. When this is done, the students t value used to calculate the MDL should be adjusted accordingly. Therefore, EPA believes that the portion of ML definition that addresses calculation of the ML should be revised. An improved definition is as follows:

" The ML is the lowest level at which the entire analytical system must give a recognizable signal and acceptable calibration point for the analyte. It is equivalent to the concentration of the lowest calibration standard, assuming that all method specified sample weights, volumes, and clean up procedures have been employed. The ML is calculated by identifying the standard deviation used to calculate the MDL, multiplying this standard deviation by 10, and rounding the resulting number the nearest ( 1, 2, or 5) x 10n, where n is an integer. If the MDL is determined with seven replicates, the ML is calculated by multiplying the MDL by 3.18 and rounding the result to the number nearest to ( 1, 2, or 5) x 10n, where n is an integer."

<IP_ADDRESS> Assessment of the ML against the Evaluation Criteria The following five subsections discuss the ML concept and procedure in the context of the five evaluation criteria that concern quantitation limit concepts.

<IP_ADDRESS>.1 Criterion 1: The quantitation limit concept must be scientifically valid. ( Validity is comprised of five conditions.)

For the purposes of evaluating scientific validity, EPA is using the conditions established by the Supreme Court in Daubert v. Merrell Dow Pharmaceuticals ( see Chapter 4).

Condition 1: It can be ( and has been) tested. The ML meets this condition. The ML has been used experimentally since 1979 and in the regulatory context since 1984. The ML is tested each time a laboratory calibrates its instrument because methods that employ the ML require that it be included as the lowest nonzero standard in these calibrations.

Moreover, EPA exhaustively tested the MDL and ML procedure with 10 different techniques at decreasing spike concentrations to evaluate how well the MDL and ML procedures characterized the region of interest in preparation for this reassessment of detection and quantitation limit concepts. Results of the study suggest that, 1) although the calculated MDL and ML could vary depending on the spike level used, the procedure was capable of reasonably estimating detection and quantitation limits when the full iterative procedure in the MDL was employed, and 2) the rounding process employed to determine the ML generally yielded consistent MLs even with slight variations in the calculated MDL. Assessment of Detection and Quantitation Concepts Draft Document for Peer Review  Do Not Circulate  August 2002 5 18 In other words, if the procedure for establishing an ML is properly implemented for a given method, it will yield an ML value that is consistent with the concept, and this ML value will be verified ( tested) by a laboratory each and every time it calibrates the instrument used to analyze samples by the method.

Condition 2: It has been subjected to peer review and publication. The ML has not been published in a peer reviewed journal. However, the present definition of the ML describes the concept and the procedures used to establish the ML. This definition is included in EPA Method 1631, which was extensively peer reviewed in accordance with EPA policies on peer review prior to publication and promulgation. Given that EPA's policies on peer review are as stringent as or more stringent than those used by many published journals, EPA believes that the ML has met a high standard of scientific review and scrutiny, and therefore, meets the intent of this condition.

Condition 3: The error rate associated with the procedure is either known or can be estimated. The uncertainty associated with any ML value can be calculated. EPA performed such calculations during this re assessment and found that, on average across all techniques tested, the relative standard deviation of replicate measurements at the ML was approximately 7%. Median RSD values calculated for each multianalyte method tested ranged from 6 to 14 percent. RSD values calculated for each single analyte method tested ranged from 4 to 16 percent. ( See Appendix C to this TSD for a detailed discussion and presentation of results.)

Condition 4: Standards exist and can be maintained to control its operation. The ML meets this criterion. Detailed procedures ( i. e., standards) for establishing the ML are given in the definition itself, although, as noted above, minor revision of this definition is recommended to ensure that the ML is properly calculated when more than 7 replicates are used to determine the MDL.

Condition 5: It has attracted widespread acceptance within a relevant scientific community. EPA believes the ML meets this condition. The ML is functionally analogous to the American Chemical Society's LOQ and to the ISO/ IUPAC quantification limit, suggesting widespread acceptance.

<IP_ADDRESS>.2 Criterion 2: The quantitation limit concept must address demonstrated expectations of laboratory and method performance, including routine variability.