MEMORANDUM TO: Chuck White, EPA WAM FROM: Raphael Kuznetsovski, SAIC WAM DATE: February 27, 1998 REFERENCE: EPA Contract No. 68 C4 0046; Work Assignment No. 3 17 SAIC Project No. 01 0813 08 2108 170 SUBJECT: Updated Annotated Bibliography Attached please find an updated version of the " Annotated Bibliography on Measurement Limitations and Calibration." Since the last report on 12/ 19/ 97 SAIC has added annotated reviews of four articles to the calibration section of the document. The new articles are

Buonaccorsi, J. P. ( 1986) Design considerations for calibration. Technometrics, 28, 149 155.

Naszodi, L. J. ( 1978) Estimation of the bias in the course of calibration. Technometrics, 20, 201 205.

Spiegelman, C. H. and Studden, W. J. ( 1980) Design aspects of Scheffe's calibration theory using linear splines. Journal of Research of the National Bureau of Standards, 85, 295 304.

Thomas, M. A. and Myers, R. H. ( 1973) Optimal designs for the inverse regression method of calibration. Communications in Statistics, 2, 419 433.

Per your January 15, 1998 request, hard copies of these and other articles on calibration included in the annotated bibliography were delivered to you on January 26, 1998. Please provide us with further technical directions and future deadlines on this and related issues. 1 EPA Contract N. 68 C4 0046; WA 3 17 2 SAIC Project No. 01 0813 08 2108 170 1 Annotated Bibliography on Measurement Limitations and Calibration12 ( Draft Report)

Updated: February 27, 1997 Introduction This bibliography summarizes recent publications on statistical issues related to the measurement limitation and calibration. These subjects have received extensive consideration in the literature over the past few decades. The bibliography covers both theoretical and applied aspects of the field and primarily concentrates on material which has been published in the last thirty years from peer reviewed journals, books, and manuscripts. Papers in languages other than English are not included. Although the coverage of the bibliography is not comprehensive, it does include core materials on the subjects of interest and related topics. The bibliography includes 41 entries by 34 authors. The bibliography is classified by topics, and within a topic, the articles are organized alphabetically by author. Annotations in the bibliography are intended only to summarize the views and claims of authors writing in relation to the subject of concern. There is, in the bibliography, no intent to endorse any views expressed by any author.

Annotated bibliography of method detection limits and quantification Bayley, C. J., Cox, E. A., and Springer, J. A. ( 1978) High pressure liquid chromatographic determination of the intermediates side reaction products in FD& C Red No. 2 and FD& C Yellow No. 5: Statistical analysis of instrumental response. J. Assoc. Off. Anal. Chem., 61, 1404 1414.

The authors apply the statistical methods of regression analysis and detection limits to calibration data obtained by high pressure liquid chromatographic analysis for the intermediates and side reaction products of FD& C Yellow No. 5 and the former FD& C Red No. 2. The calibration line is calculated Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 2 from concentrations and corresponding peak area response of the calibration data by the method of least squares. The formulas for prediction intervals are also provided. The Hubaux & Vos ( 1970) approach is applied by the authors to calculate the limit of detection, . That is, is defined as X LD Y Q the response estimate calculated from the calibration line which corresponds to . The authors X LD accent importance of value since it defines the response above which quantitation is to be Y Q performed. is the lowest amount in units of area that can be measured above the blank. Y Q Clayton, C. A., Hines, J. W., and Elkins, P. D. ( 1987) Detection limits with specified assurance probabilities. Analytical Chemistry, 59, 2506 2514.

Clayton and co workers discuss the following topics: ( 1) calibration models; ( 2) detection limits for linear calibration with known parameters; ( 3) detection limits for linear calibration with unknown parameters; ( 4) estimation of detection limits and detection rates for linear calibration. First, the authors consider the straight line calibration and assume that . In ( 2) they consider Y x

  x    > 0 the concept of detectability within the context of a linear calibration model in which it is assumed ( a) that the slope and intercept of the straight line calibration are known and ( b) that the variation of measurement errors ( presumed homogeneous over the range of concentrations of interest) is known. They state that the assumption of constant error variance can often be accomplished by transforming the response Y. The following detection rule is adopted: " Assert ( i. e., the analyte is present) if X> 0 " where is a threshold response value chosen so that the rate of false positives ( in repeated Y> y p y p applications of the rule to blank samples) is fixed at a desired probability level, . In ( 3) detection p limits and detection rates are formally defined for the case of a linear calibration model for which the model parameters and error variance are unknown and must be estimated. Clayton et al. first pointed out that in the case in which is replaced by its sample based estimate s, the appropriate distribution  under the alternative hypothesis ( i. e., ) is the non central t distribution. For this situation, in ( 4) X> 0 tables and a four step procedure for generating detection limit estimates are provided. Experiments that illustrate and validate the approach are then described in subsequent sections.

Coleman, D., Auses, J., and Grams, N. ( 1997) Regulations  From an industry perspective or Relationships between detection limits, quantitation limits, and significant digits, Chemometrics and Intelligent Laboratory Systems, 37, 71 80 Coleman et al. interpret and extend the calibration based definition of detection limit due to Hubaux and Vos ( 1970). For these purposes the authors use a definition of significant digits to determine relative measurement error ( RME) and signal to noise ratio ( SNR). A number expressed in scientific notation is said to have w significant digits if the value has an error interval of width . It can be shown that a number with w significant digits ( 1/ 2)  10

w  1 satisfies the following inequality: Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 3 1 2  10

w  RME < 1 2  10

w  1.

SNR

log 10 signal noise

log 10 measurement total measuremt err

 log 10( 2  RME). The authors then define SNR as Therefore, w  SNR > w  1. Further, Coleman et al. show that the Hubaux Vos detection limit, , is approximately DL H

V equal to the lowest concentration with RME= 50%, SNR= 0, or, equivalently, at least zero significant digits in the measurement. They also define the quantitation limit based on Hubaux Vos ( ) as QL H

V the lowest concentration with RME= 5%, or equivalently, SNR= 1. At this concentration, at least one significant digit can be ensured in the measurement at a specified level of confidence of . If measurement error were constant, the authors expect to have 100( 1

  ) % , since RME= 50% at , and RME= 50%/ 10 = 5% at . This does not QL H

V  10  DL H

V DL H

V QL H

V quite hold true due to the ` flaring' of the prediction intervals away from the calibration line as the concentration moves further from the value of in the calibration design. x Currie, L. A. ( 1968) Limits for quantative detection and quantitative determination, Analytical Chemistry, 40, 586 593 Three limiting levels are defined and discussed in this article: ( 1) a " decision limit" at which one may decide whether or not the result of an analysis indicates detection, ( 2) a " detection limit" at which a given analytical procedure may be relied upon to lead to detection, and ( 3) a " determination limit" at which a given procedure will be sufficiently precise to yield a satisfactory quantitative estimate. ( Critical Level): The net signal level ( instrument response) above which an observed signal L C may be reliably recognized as " detected". Mathematically, the critical level is given as , L C

z    0 where is the abscissa of the standardized normal distribution corresponding to probability level z   and represents the variance of the blank. 1

 2 0 ( Detection Limit): The " true" net signal level which may be a priori expected to lead to L D detection. is defined so that the probability distribution of possible outcomes intersects such L D L C that the fraction, , will correspond to the ( correct) decision, " detected". This may be written as 1   , where is the abscissa of the standardized normal distribution corresponding to L D

L C  z    D z   probability level and is the variance of the distribution of the ( net) signal when its true value 1    2 D is equal to . L D ( Determination Limit): The level at which the measurement precision will be satisfactory for L Q Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 4 quantitative determination. In other words, a result must be satisfactory close to the true value ( limiting mean). Therefore, for the Determinination Limit, , the standard deviation, , must be L Q  Q but a small fraction of the true value. The Determination Limit so defined may be written as , where is the true value of the net signal having a standard deviation, , and is the L Q

k Q  Q L Q  Q 1 k Q requisite relative standard deviation. In order to make the significance of definitions clearer, a number of specific choices for ,

,  and ` s may be helpful. If the risk of making both kinds of mistake are set equal, i. e. , and, in

 addition, if is approximately constant, then and . Making the additional  L C

z  L D

2L C requirement that , we find that . k Q

10 L Q

10  The principles are illustrated by examples of spectrophotometry and radioactivity.

Currie, L. A. ( 1988) Detection: Overview of historical, societal, and technical issues, In Detection in Analytical Chemistry: Importance, Theory and Practice. American Chemical Society, Washington, DC.

This overview presents some of the societal, historical, and broad conceptual issues relating to detection and chemical measurements. Balanced coverage was the intent, but special attention has been given to the following new topics not covered elsewhere: multiple decisions and probabilistic pattern detection, utilization of physical constraints ( on variance), and some effects of varying probability density function as related to experimental design and variation.  The first part of this overview introduces the basic concept of ( chemical) detection, together with applicability to selected societal problems. Part two comprises a brief historical review, highlighting major contributions to the concept and realization of detection in chemical applications. Part three seeks to expose most of the technical issues involved in deriving meaningful detection decisions and detection limits, considering the overall Chemical Measurement Process: issues affecting the validity of detection decisions ( null hypothesis testing); the type II error and (  ) detection for the alternative hypothesis; multiple detection decision and Discrimination Limits for chemical species and chemical patterns.

Currie, L. A. ( 1992) In pursuit of accuracy: Nomenclature, assumptions, and standards. Pure & Applied Chemistry, 64, 455 472.

In this paper, among other topics, Currie briefly discusses issues related to terminology, assumptions, and basic concepts of the decision and detection limit theory used by the scientific community. False positive and false negative errors and homogeneity of variance are among these issues. Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 5 Currie, L. A. and Svehla, G. ( 1994) Recommendations for the presentation of results of chemical analysis. Pure & Applied Chemistry, 66, 595 608.

Among other general terminologies relating to the precision and accuracy of experimental results, Currie & Svehla provide definitions of the minimum significant signal ( Critical Level) and minimum detectable quantity ( Detection Limit). These definitions are equivalent to those used by Currie in his earlier works. Expressions for calculating the estimates are also provided.

Currie, L. A. ( 1995) Nomenclature in evaluation of analytical methods including detection and quantification capabilities. Pure & Applied Chemistry, 67, 1699 1723.

Currie concentrates mainly on establishing uniform terminology, notation, and formulation for performance characteristics of the chemical measurement process ( CMP) in concurrence with ISO and IUPAC scientists. The paper describes the fundamental quantities related to the observed response, calibration and the evaluation function. Precision and accuracy related to performance characteristics of a CMP are discussed at length. The latter part of the paper concentrates on terminology and concepts underlying detection and quantification capabilities in chemical metrology. The author also points out the distinction between the sampled and target population in an interlaboratory environment. The following terminologies are recommended by the author. The critical value, of the L C appropriate chemical variable represents the detection decision. The minimum detectable ( true) value, of a chemical variable is a measure of the inherent detection capability of a CMP. The L D minimum quantifiable ( true) value , , measures the inherent quantification capability of a CMP and L Q L is used as the generic symbol for the quantity of interest. This is replaced by when treating net S analyte signals and when treating analyte concentrations or amounts. x The statistical theory of hypothesis testing forms the basis for the determination of detection and quantification capabilities. To calculate , , and , two kinds of errors ( Type I and Type L C L D L Q II) are considered. The Type I error is committed when a false alternative hypothesis ( analyte present) is accepted and the Type II error is committed when a false null hypothesis ( analyte absent) is accepted. The probability of committing Type I and Type II errors are usually fixed at 0.05 each ( default values). The decision " detected" or " not detected" is made by comparison of estimated quantity with the critical value ( ). Currie points out that, with the exception of some

L L C nonparametric techniques, detection limits cannot be derived in the absence of known or assumed distributions.

Currie, L. A. ( 1997) Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 6 Detection: International update, and some emerging di lemmas involving calibration, the blank, and multiple detection decisions, Chemometrics and Intelligent Laboratory Systems, 37, 151 181.

Currie first presents a brief resume of the history of detection and quantification capabilities of the chemical measurement process. Following that, he reviews key aspects of detection and quantification concepts and their realization, together with open questions that have not been resolved and hence provide important opportunities for further research. Among those questions are detection and quantification capabilities in the signal and concentration domains, respectively; and the link between calibration and detection and quantification limits, together with the blank intercept dichotomy. Also included are special treatments and approximations involving the non central t, and the exact ( nonnormal distribution of the estimated concentration. The author also introduces issues and approaches to multiple independent and multivariate detection decisions and limits, and concludes with a glimpse at some challenges involving the multivariate blank and non monotonic calibration functions.

Davis, C. B. ( 1994) Environmental regulatory statistics. In Handbook of Statistics, Vol. 12, G. P. Patil and C. R. Rao, ( eds.)

A few crucial issues and methodologies have been outlined in some detail, including in particular ` detection limit' definition problems. Section 5.3 of the paper contains an illuminating discussion about the nature of detection limits and the statistical issues associated with it. In the author's opinion, one of the more pressing areas where research is needed involves the definitions of ` detection limits' and other limits. Conceptual frameworks by Currie ( 1968), Glaser et al.( 1981), Clayton et al.( 1987), and Gibbons et al.( 1991,1992) are discussed and critically evaluated. Suggestions for practice and for research are also given.

Garner, F. C. and Robertson, G. L. ( 1988) Evaluation of detection limit estimators. Chemometrics and Intelligent Laboratory Systems, 3, 53 59.

Garner & Robertson offer a general discussion on definitions and methods developed for estimating the minimum detectable level of a measurement process. Their attention is mainly focused on the decision limit, which is also known from Currie ( 1968) as critical level, and the detection limit. The authors discuss assumptions and some characteristics of analytical methods that are typically employed in deriving measurement limitations. Among the assumptions mentioned in the article are assumptions on distribution, independence, and homogeneity of variance of the instrumental responses. The authors also point out that at the moment there is a lack of models developed for Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 7 multivariate instrumental responses and incorporating detection limit estimators.

Gibbons, R. D., Jarke, F. H., and Stoub, K. P. ( 1991) Detection limits: For linear calibration curves with increasing variance and multiple future detection decisions. Waste Testing and Quality Assurance: ASTM STP 1075, D. Friedman, Ed., American Society for Testing and Materials, Philadelphia, 3, 337 390.

In this paper, the authors propose a detection limit estimator which is a generalization of the method of Clayton et al. ( 1987). Clayton and co workers have provided a test for the null hypothesis that the concentration in the solution is zero ( i. e., X= 0) with fixed type I and type II error rates. The potential limitations of their method are: ( 1) The estimator assumes a constant variance across the calibration line and, ( 2) the estimator applies only to the next single detection decision. The authors of this paper propose a new detection limit estimator, which allows variance of the response signal ratio ( i. e., peak area of the analyte to peak area of the internal standard) to be proportional to the concentration. The new detection limit estimator can also provide multiple future detection decisions; with the new detection limit, 99% of all future detection decisions and not just the next signal detection can be made with 99% confidence. The detection limit for non constant variance calibration designs can be obtained by using the weighted least squares method. Calibration data for 10 volatile organic priority pollutant compounds is used by the authors to illustrate the new detection limit estimators.

Gibbons, R. D., Grams, N. E., Jarke, F. H., and Stoub, K. P. ( 1992) Practical quantitation limits. Chemometrics and Intelligent Laboratory Systems, 12, 225 235.

Gibbons et al. develop an operational definition for the practical quantitation limits ( PQL) and a corresponding statistical methodology for obtaining facility specific PQL estimates and regions of confidence. The authors define the PQL as the concentration at which the instrument response signal is times its standard deviation. This definition is similar to that used by Currie ( 1968) who 1 0 0 /

defined the determination limit ( ) as the level of concentration " at which a given procedure will L Q be sufficiently precise to yield a satisfactory quantitative estimate." Gibbons et al. suggest estimating the PQL of a given compound in a given laboratory using calibration data obtained for a series of concentrations in the range 0 to 2 5 times the hypothesized PQL. Least squares regression techniques are used to fit the calibration line for the relationship between actual concentration and instrument response. A variance stabilizing transformation is proposed in cases when the variability is not constant throughout the range of the calibration function. The PQL is then set to a level of concentration corresponding to the response value , which is a solution of the following x

  y

  equation Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 8

100

s(

) ,

where is the predicted response, and is the estimated standard deviation. Confidence limits

s(

) for are obtained to incorporate the uncertainty in the PQL estimator due to the fact that estimates x

  of standard deviation and calibration line parameters are based on a finite sample of observations. The proposed statistical methods are illustrated using 199 calibration samples for 10 volatile organic priority pollutant compounds.

Gibbons, R. D. ( 1994) Statistical methods for groundwater monitoring. John Wiley & Sons, Inc.

Chapter 5 and 6 of this book present statistical methods for computing method detection limits ( MDLs) and practical quantitation limits ( PQLs). In Chapter 5, the historical literature on MDLs is reviewed, and the strengths and weaknesses of various estimators are compared and contrasted. Estimators based on the entire calibration function and not a single concentration are shown to be the methods of choice. In Chapter 6, the ideas developed in Chapter 5 are expanded to the case of estimating the PQL. It is shown that while both MDLs and PQLs can be estimated from similar analytical data, the estimators have little to do with one another, the former being a test of the null hypothesis that the concentration is zero and the latter being a point estimate of analytical precision.

Gibbons, R. D. ( 1995) Some statistical and conceptual issues in the detection of low level environmental pollutants. Environmental and Ecological Statistics, 2, 125 167.

Traditional detection limit estimators are described and critically evaluated. The methods are categorized into single concentration design versus calibration design methodology. This work demonstrates that the most critical source of bias in estimating limits of detection is the effect of nonconstant variance. The solution proposed is to base limits of detection on multiple concentration calibration data and either directly model association between variability and concentration or use a variance stabilizing transformation. The various calibration based methods are illustrated using real data and experimental design issues for detection limit studies are discussed.

Gibbons, R. D., Coleman, D. E., and Maddalone, R. F. ( 1997) An Alternative Minimum Level definition for analytical quantification. Environ. Sci. Technol., 31, 2071 2077. Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 9 y C

K 0.95,0.99 s 0  b 0w

K 0.95,0.99 a 0  b 0w,

L C

y C  b 0w b 1w .

x Q  t b 1w V( y Q),

AML

x Q

ts yQ b 1w . The authors present a new approach to determining the quantification limit in analytical chemistry. The proposed procedure is as follows: First, Gibbons et al. model the relationship between the standard deviation ( ), and the

concentration ( ) using calibration data. They consider the following two models: ( 1) exponential, x ; and ( 2) the Rocke Lorenzato model, . Weighted least squares

( x)  a 0 exp( a 1 x)

( x)  a 0

a 1 x 2 ( WLS) method is then used to obtain a calibration line , I= 1,..., n, by regressing the

y i  b 0w

b 1w x i measured concentration y on the true concentration x. The limit for measured concentrations when the true concentration is zero is given by where is the 95% confidence 99% coverage one sided tolerance limit factor for n K 0.95,0.99 observations. The corresponding critical level is The standard deviation at the , , is computed using the previously fitted model for . L C s LC

( x) Therefore, is the measured concentration at 10 times the standard deviation at the . y Q  10s LC

b 0w L C Finally, the AML is computed as the 99% prediction limit where and t is the upper 99th percentile of Student's t distribution. Alternatively x Q  ( y Q  b 0w)/ b 1w when n is large enough ( n> 25), the AML can be approximated by The proposed procedure is illustrated using U. S. EPA Method 1638 ICPMS data for cadmium at mass 114.

Glaser, J. A., Foerst, D. L., McKee, G. D., Quane, S. A. and Budde, W. L. ( 1981) Trace analysis for wastewaters. Environmental Science & Technology, 15, 1426 1435. Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 10 MDL  2.681  S pooled, The purpose for developing a procedure to evaluate detection limits was to design a methodology not limited by instrumentation or analytical methodology. For pragmatic reasons, authors focused on an operational definition of detection limit. The fundamental difference between the proposed approach to detection limits and former efforts is the emphasis on the operational characteristics of the definition. The method detection limit is defined as the minimum concentration of a substance that can be identified, measured, and reported with 99% confidence that the analyte concentration is greater than zero and is determined from replicate analyses of a sample of given matrix containing analyte. The single step procedure for determining MDL is based on the analysis of seven samples of the matrix containing analyte. The standard deviation, , of the seven replicate measurements is S C calculated and the MDL is computed as , where is the MDL  t( n  1,1    .99) S C t( n  1,1    .99) student's t value at 99% confidence level with degrees of freedom. n  1 An additional iterative procedure is presented to test the reasonableness of the MDL estimate and subsequent MDL determinations on the same matrix. At each iteration of the MDL calculation, the variance from the current MDL calculation and the preceding MDL calculation are compare by computing the F ratio, which is compared with the tabulated F ratio, . If the computed F. 95( 6,6)  3.05 F ratio is less than 3.05, then the pooled standard deviation is calculated using the standard deviation of the current MDL determination and the preceding iteration. The MDL is then calculated as where 2.681 is equal to and . Convergence of the iterative t( 12,1    .99) S pooled  ( S 2 A

S 2 B)/ 2 procedure will depend on the closeness of the estimated MDL, or the background level of analyte present in the matrix, to the calculated MDL. Both theory and applications are discussed for reliable wastewater analyses of priority pollutants.

Hubaux, A. and Vos, G. ( 1970) Decision and detection limits for linear calibration curves, Analytical Chemistry, 42, 849 855.

Hubaux and Vos were the first to apply the theory of statistical prediction to the problem of detection limit estimation. For linear calibration curves, two kinds of detection limits may be connected to the notion of confidence limits around the regression line. Here, the variance of the error distribution of the signals around their expectation is supposed to remain constant. On either side of the regression line, two confidence limits may be drawn, with an a priori chosen level of an overall 100( 1     ) % confidence. The lowest measurable signal is defined as the point at which the upper y C ( 1   ) confidence limit intersects the y axis. This definition corresponds to the definition of critical level L C by L. A. Currie. The lowest content ( or, " detection limit", in Currie's notation) is then defined as x D the point at which one can have % confidence that the measurable signal is greater than . ( 1   ) y C Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 11  ( c)

p( m) f( m| c) dm. It can be obtained graphically by locating the abscissa corresponding to on the lower confidence y C limit. Algebraic solutions for and are also provided. The influence of the precision of the y C x D analytical method, the number of standards, the range of their contents, the various modes of their repartition, and the replication of measurements on the unknown sample are studied from a statistical point of view. Several practical rules such as improving the precision of the method, increasing the number of standards, increasing the range of the contents of these standards, optimizing the repartition of these standards within this range are deduced to lower the measurement limits.

Ingle, J. D., Jr. ( 1974) Sensitivity and limit of detection in quantitative spectrometric methods. Journal of Chemical Education, 51, 100 105.

Ingle discusses some the many existing mathematical definitions of the limit of detection which are based on the use of the z statistic or the t statistic and are concerned with errors of the first type ( false positive). Definitions based on other test statistics or those concerned with errors of the second type ( false negative) are stated to be too complex for usual routine procedures and not discussed in this article.

Lambert, D., Peterson, B. and Terpenning, I. ( 1991) Nondetects, detection limits, and the probability of detection, JASA, 86, 266 277.

When chemists cannot quantify the concentration in a sample, they report a nondetect. There are two widely accepted procedures dealing with nondetects. One is to assume that all nondetects are smaller than all detects. Another is to assume that nondetects are below a threshold such as the detection limit. This article shows that these assumptions can be wrong. Furthermore, new methods for describing which measurements were detectable in a given study are provided. Two new concepts are introduced. The probability of acceptance curve shows how likely a measurement of is to be p( m) m accepted by chemists and reported as a detect. The 90th percentile of is called the censoring p( m) limit and recommended as a bound for nondetects. The probability of detection curve describes how likely a field concentration of is to  ( c) c give a measurement reported as a detect. Suppose that the measurements for field samples with a true concentration of a pollutant c have a density and the probability that a positive measurement passes the identification tests f(  | c) m is . Then p( m) Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 12 Finally, the minimum reliably detected concentration is defined as the 90th percentile of .  ( c) To estimate , and therefore, the minimum reliably detected concentration, the estimates  ( c) of and are needed. Local logistics regression, developed by Hastie & Tibshirani, is used p( m) f(  | c) to estimate the probability of acceptance curve, . Local logistics regression relaxes the linearity p( m) assumption of traditional logistic regression by allowing the intercept and slope of the regression function to vary over the range of measurements. might be estimated, for example, from f(  | c) samples with known concentration. The statements are illustrated with examples from the 1988 Love Canal study.

Liteanu, C. and Rîc

, I. ( 1980) Statistical Theory and Methodology of Trace Analysis. Halsted, Chicago.

In this work, general applications of mathematical statistics in chemical analyses are extended by the introduction of knowledge on stability of analytical systems, and also elements of the informational treatment of analytical measurements. In fact, specific statistical problems in trace analysis, such as limits of detection and determination, are treated in details. For these purposes, the authors use the statistical theory of signal detection. In the case of detection, the analytical measurement is translated into a detection decision by means of statistical decision rules. In determination, the result is obtained from the measurement by use of the calibration function. The authors adopt a statistically based concept of ` detection limit' similar to one introduced by Currie ( 1968). In particular, they make the following statement: the detection limit is the particular value of the amount of component at which the probability of correct detection evaluated with c d respect to the decision threshold ( defined on the basis of a probability of false detection) is equal y k to a preselected value. The authors then describe various methods for estimating the detection limit and the statistical confidence interval for its true value using a calibration approach. They also address the problem of evaluation of the detection limit when the standard deviation of the signals varies with concentration. They introduce a frequentometric principle for estimating the detection limit based on interpolating the probability of correct detection in the detection characteristics given the probability of false detections fixed. Linking together the theory and the problems of analytical practice, the authors discuss the detection capacity and detection limits of the most frequently used methods of trace analysis.

Oppenheimer, L., Capizzi, T. P., Weppelman, R. M., and Mehta, H. ( 1983) Determining the lowest limit of reliable assay measurement. Analytical Chemistry, 55, 638 643.

The " lowest limit of reliable assay measurement" is usually used to evaluate the limitations of an assay and make meaningful comparisons between different assays. This paper describes the derivation of Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 13

2

µ 2e   2

( e   2

1). appropriate expressions for obtaining various types of assay limits under heterogeneous variances. These results are illustrated by using data from an assay developed to monitor tissue residues of Ivermectin, an antiparasitic agent. The paper also discusses related issues which also affect assay limit determinations, such as, the method used to select the variance weights, alternative models which transform the variance structure of the responses, and experimental design considerations used in obtaining the standard curve. These issues are also illustrated by using the tissue residue data. Dramatic differences were observed by the authors depending on whether a valid weighted or an inappropriate analysis without weights was used.

Rocke, D. M. and Lorenzato, S. ( 1995) A two component model for measurement error in analytical chemistry. Technometrics, 37, 176 184.

A new model for measurement error in analytical chemistry is proposed. The model can be written as , where there are two analytical error, and . Here represents x  µ e

N( 0,  2  )

N( 0,  2  )  the proportional error that is exhibited at ( relatively) high concentrations and represents the additive

error that is shown primarily at small concentrations. The standard deviation of the model can be written as This formula illustrates the key feature of the new model  when is small, the error is nearly  constant in size; when is large, the size of the error is roughly proportional to . The new model   allows the error at large concentrations to be lognormal, rather than normal. If viewed in terms of the coefficient of variation, the picture is that large concentrations have a constant CV, whereas small concentrations have an increasing CV that tends to infinity as the concentration approaches 0. The computational approach used is based on maximum likelihood estimation. Maximizing the likelihood function of the model leads to estimates of the necessary parameters , and . Once

estimates and have been derived, the precision of any measured value can be estimated from

the formula for standard deviation by substituting estimated values for the parameters. Discussion is provided on the application of the new model to some common issues including determination of detection limits. In fact, the new model provides a reliable way to estimate the precision of measurements that are near the detection limits so that they can be used in inference and regulation.

Singh, A. ( 1993) Multivariate decision and detection limits. Analytica Chimica Acta, 277, 205 214. Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 14

( Y)

Y, if Y

c any fixed constant, if Y< c L( d,

)

( 1

p)( d

) , if d

p(

d), if d

Principal component analysis is used to develop approaches for estimating multivariate decision and detection limits for gas chromatography mass spectrometry studies where the instrument response is multivariate in nature. When the first principal component explains most of the variation contained in the data, it may be used to express the multicomponent instrumental response as a univariate composite signal representing all of the monitored ions associated with the analyte of interest. Least squares regression is used to obtain a calibration function for the composite instrument response onto the analyte concentration. When the original response variables are Gaussian, the principal components are also Gaussian as a linear combinations of normally distributed random variables. When the original response variables are not Gaussian, Singh appeals to the central limit theorem to justify the normality assumption for the principal components. The first principal component is used to derive decision and detection limits by using two approaches; those of Currie ( 1968) and of Hubaux and Vos ( 1970). The scores for the first principal component are then transformed back to the original responses to give rise to multivariate decision and detection limits. Calibration data of 2,3,7,8 tetrachlorodibenzo p dioxin are used to numerically illustrate the proposed approaches.

Spiegelman, C. H. ( 1997) A discussion of issues raised by Lloyd Currie and a cross disciplinary view of detection limits and estimating parameters that are often at or near zero. Chemometrics and Intelligent Laboratory Systems, 37, 183 188.

Spiegelman applies methods of statistical decision theory to the problem of estimating the detection limits. The definition of the detection limit used by the author is taken from Currie ( 1988). Let be a measurement such that . For some constant , define the critical value Y Y  N( µ ,1) c rule as follows

( Y)

The author then proves that given the mean squared error loss, the detection limit estimators

( Y) are inadmissible from the decision theory point of view. However, given an asymmetric family of loss functions of the form where , is a uniform density on the interval [ 0,1], , and is p(

)    ( 0)  ( 1

 ) q(

) q(

)   ( 0,1)  ( 0) the Dirac delta function with point mass 0, some estimates of the form Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 15

( Y)  Y, if Y

1  ( Y), if c  Y< 1 0, if 0  Y< c S L  S 2 Y  S 2 B 10 .

L C  t( 18,  ) S 2 Y  S 2 B 10 , where and is an increasing function, are admissible and also Bayes rules. 0   ( Y)  1  ( Y)

Spiegelman, C. H. and Tarlow, P. ( 1997) A mock trial for critical values ( detection limits). STATS: The Magazine for Students of Statistics, 20, 13 16.

Based on Currie's definition of the critical value ( see Currie ( 1995)), , the authors provide an L C example of how this definition could be used in practice. They denote the means and standard deviations of ten background and ten chemical sample measurements as , ; , ¯ B S B/ 10 ¯ Y S Y/ 10 respectively. The measurement is and its estimated standard error is

L  ¯ Y

¯ B Assuming homogeneity of variance for background and chemical sample, has 18 degrees of S L freedom. The critical value ( or detection limit) is then defined as where is the upper  quantile of a t distribution with df degrees of freedom. The authors t( df,  )  say that is above the detection limit if and only if .  Y ¯ Y

¯ B  L C The rest of the article is dedicated to the discussion on whether actual values or zeroes should be reported as the result of an experiment given that the raw measurements are below the detection limit.

Zorn, M. E., Gibbons, R. D., and Sonzogni, W. C. ( 1997) Weighted least squares approach to calculating limits of detection and quantification by modeling variability as a function of concentration. Analytical Chemistry, 69, 3069 3075.

Zorn et al. use a weighted least squares regression analysis of replicates spiked at a series of Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 16 concentrations for estimating detection and quantification limits. In addition, models for estimating the weights used in calculating weighted prediction intervals and weighted tolerance intervals are presented. To calculate detection and quantification limits using weighted prediction and tolerance intervals, the weight at zero ( ), the weight at the limit of detection ( ), and the weight at limit w 0 w LD of determination ( ) must be estimated. In this study, the weights have been set equal to the w LQ inverse variance, . Therefore, can be estimated by modeling the standard deviation ( or s i

2 w j variance) as a function of concentration x. The following models of standard deviation were s x examined: a quadratic model, ; an exponential model, ; and a two s x  a 0  a 1 x  a 2 x 2 s x  a 0 e a1x component model proposed by Rocke and Lorenzato ( 1995), . In presented examples, s x  a 0  a 1 x 2 the quadratic model provides the best overall fit to the data.

Annotated bibliography of calibration and related regression techniques.

Berkson, J. ( 1969) Estimation of a linear function for a calibration line; consideration of a recent proposal. Technometrics, 11, 649 660.

Berkson briefly reviews the results of the Krutchkoff ( 1967) article. The remainder of the paper is dedicated to pointing out that the inverse calibration method is inconsistent, whereas the classical calibration method is consistent. Since the work of Krutchkoff ( 1967) was based on relatively small sample sizes, Berkson is interested in how these methods perform for large sample sizes ( i. e., ). N   Controlling the calibration range as in Krutchkoff's article, the ratio of mean squared errors were determined for both and . For this, asymptotic formulas were used to determine N  6 N   the asymptotic variance for each of the calibration methods. For , the ratios of the mean X  1.2 squares are larger than unity for both samples sizes state previously. However, for large samples ( ) and , the mean squared errors for the classical calibration approach are smaller than N   X> 1.2 those found using the inverse calibration approach. Berkson notes that the asymptotic formulas for variance are accurate enough for and N  20 states that since the m. s. e. is not smaller for the inverse estimator for all values of , it undermines X the basis of Krutchkoff's argument that the inverse calibration method is superior to the classical calibration method. Also noted is the fact that the inverse estimator, though inconsistent, shows smaller m. s. e. than the classical estimator in a large region of the scale around the mean of . An X example using the relationship between Farenheit and Centigrade with both the calibration methods is used to show how the inverse calibration method's inconsistent results depends on . The author X completes his paper by stating that the m. s. e. of the classical calibration estimator will actually be smaller than the inverse calibration method estimator when ( 1) sufficient number of samples are take Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 17  2( x)   2( m

1  n

1  ( x

¯ X ) 2T

1)

 2 1 . to determine the calibration line and ( 2) the calibration range is restricted to [ 0,1] as in Krutchkoff's article.

Buonaccorsi, J. P. ( 1986) Design considerations for calibration. Technometrics, 28, 149 155.

Buonaccorsi investigates the effects of the choice of design points on calibration ( X 1, X 2,  , X n) in the presense of a linear regression model, , where X's are under the Y i   0   1 X i   i control of experimenter and . It is assumed that , belong to the  i  iid( 0,  2) X i, i  1,  , n calibration interval , . As an approach to finding optimal design the author considers [ a, b] a< b minimization of the asymptotic variance Several criteria are considered by the author, which include ( i) optimality that minimizes V x ; ( ii) AV optimality that minimizes an average asymptotic variance with respect to some  2( x) prior distribution for ; and ( iii) M optimality that minimizes the maximum asymptotic variance x over certain values of . If , then an optimal design by criteria is one so that the x x  ( a, b) V x mean of two calibration points is the prior " guess" of . By AV criteria, an optimim design is an x endpoint design so that the mean of the prior distribution is . M optimal when the ( a  b)/ 2 maximum variance is considered over an interval symmetric with respect to . In summary, [ a, b] Buonaccorsi suggests that if one has a prior guess at , say then choose so that it is x x 0 [ a, b] symmetric about and to make as wide as possible. Then take n 1 observations at , n 2 x 0 [ a, b] a observations at and n 3 observations at , so that n 1 = n 2 and n 1 + n 2 + n 3 = n. b ( a  b)/ 2 Using accuracy curves, the probability that a confidence region contains when the true x 0 value is , to develop optimality criteria, Buonaccorsi concludes that the equally weighted end x point design is optimal for extreme values of . However, it is generally known a priori that x x will fall in some finite interval. Thus, Buonaccorsi suggests that a confidence region be truncated over that finite range of . x Eberhardt, K. R. and Mee, R. W. ( 1994) Constant width calibration intervals for linear regression. Journal of Quality Technology, 26, 21 29 Eberhardt and Mee develop constant width simultaneous tolerance intervals for which the bound on the nominal coverage probabilities is exact under normality. Their method allows one to control the Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 18

X  Y

a b , ( conditional) probability that a confidence interval captures the true value of X to be greater or equal to any chosen value P, provided that the calibration data set belongs to a set G of " good" outcomes. The set G is implicitly defined in the procedure when the user chooses an appropriate high probability that the calibration data set will belong to G. The proposed method is a modification of the  procedure of Mee, Eberhardt, and Reeve ( 1991). Constant width intervals have a " minimax" appeal in that they are narrower than the maximum width obtained by the other calibration interval methods having this same interpretation.

Eisenhart, C. ( 1939) The interpretation of certain regression methods and their use in biological and industrial research. The Annals of Mathematical Statistics, 10, 162 186.

Eisenhart's work is one of the first publications in statistical literature on regression analysis discussing the problem of predicting X corresponding to a future value of Y, which is known as statistical calibration problem. This paper presents a review of some of the ideas involved in curve fitting practices related to the problem of estimating from . The author considers some of the X Y instances, when it is necessary to follow curve fitting practices with as the dependent variable and Y then using the inverse of the relation found to get an estimate of . The paper also points out the X types of problems to which this method of inverse regression provides a solution and the confidence interval nature of the estimates it gives. Eisenhart briefly discusses effects of limiting the ranges of both independent and dependent variables in the sampling process. The method of inverse regression is demonstrated by working out a problem arising in the manufacture of cheese, and a problem concerned with the biological assay of a hormone substance.

Krutchkoff, R. G. ( 1967) Classical and inverse regression methods of calibration. Technometrics, 9, 425 439.

Krutchkoff considers the classical and inverse methods of the statistical calibration problem and compares them by Monte Carlo simulations. Suppose that the relationship between a controlled variable x and a measured variable y can be written as , where and obtained by the y     x     ordinary least squares method. The classical estimate of an unknown X corresponding to a future reading Y is where Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 19 b

( x i

¯ x) ( y i

¯ y)

( x i

¯ x) 2 and a  ¯ y

b ¯ x .

X  c  dY ,

d

( x i

¯ x) ( y i

¯ y)

( y i

¯ y) 2 and c  ¯ x

d ¯ y . In the inverse approach, Krutchkoff considers the regression of x on y rather than y on x, which leads to a different estimate of the unknown X,

where The classical approach is then compared with the inverse approach by Monte Carlo simulations. Krutchkoff considers an interval [ 0,1] as the calibration range for x and designs the calibration experiment so that there are three observations at each of the end points ( x= 0, x= 1). The experiment is repeated 10,000 times. Based on the results of the simulation study the author concludes that the inverse estimation has a uniformly smaller mean squared error than the classical estimator.

Krutchkoff, R. G. ( 1969) Classical and inverse regression methods of calibration in extrapolation. Technometrics, 11, 605 608.

The author, in an earlier paper, compared the inverse method with the calibration method by Monte Carlo simulations and found that the inverse method has a uniformly smaller average squared error in the range of controlled variable. In this paper, the author extends his previous results by including extra points outside the calibration range. Author also points out that results quoted for X= 5 and X= 10 in the original paper were incorrectly labeled and were independent repetitions of X= 2 column. Based on the results of the new simulation study, Krutckoff demonstrates that, for a sufficiently large number of observations, the classical method gives a smaller average squared error outside the range of the calibration, however the conclusions drawn in the previous paper remain unchanged for X values within the calibration range.

Lechner, J. A., Reeve, C. P. and Spiegelman, C. H. ( 1982) An implementation of the Scheffé approach to calibration using spline functions, illustrated by a Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 20 p  p( v)

 g 1( v),

0 < v<

1  g 2( v),

1 < v<

2  g k

1 ( v),

k < v<

k

1 ,

v   h 0 A( x) dx, pressure volume calibration. Technometrics, 24, 229 234.

This paper presents a Scheffé type calibration procedure to the pressure volume relationship for nuclear materials processing tanks. Piecewise polynomials ( splines) are used to fit this relationship for nuclear materials processing tanks. Since differential pressure can be simply and quickly measured, the relationship between volume and pressure is used to estimate the volume indirectly by measuring the pressure. The settings for the calibration experiment are as follows. The nuclearmaterials tanks are considered to be composed of segments for which an idealized model could be considered as a good representation. The authors assume that a tank is composed of distinct k  1 and known regions, where the idealized relationship between the two variables pressure, and p volume, is given by v where is a polynomial function of . At height the volume in the container is g i( v) v h where A( x) is the cross sectional area at height . When liquid height is , where is the x hp  h  g,  density of the homogeneous liquid and is the acceleration due to gravity. In the areas of the tank g where is constant, the volume pressure is a straight line. Hence A( x)

p  g i( v)   i   i v, for

i  1 < v<

i , i  1,......, k  1.

The authors assume that each pressure measurement has a random error associated with it and the errors are independently and normally distributed, with mean zero and constant variance The  2. calibration experiments obtain estimates of and , and these coefficient estimates are then used  i  i to obtain the estimates of the volume of the tank, using the inverse of the relationship given in ( 1). The authors also obtain the interval estimates of volume for measured values of pressure.

Mee, R. W. and Eberhardt, K. R. ( 1996) A comparison of uncertainty criteria for calibration. Technometrics, 38, 221 229. Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 21 Mee and Eberhardt briefly review the literature for constructing confidence intervals for multiple use calibration, in which a single fitted regression line is used repeatedly to estimate many x's. In addition, they derive probability expressions for computing exact prediction intervals that enable the construction of tighter limits than are currently available based on that criterion. As an example, the authors examine the performance of simultaneous prediction intervals, simultaneous tolerance intervals, and tolerance intervals procedures using the context of a linewidth data. The widths of 10 reference lines were used to calibrate optical imaging systems measuring features in the size range 0.5 to 12 µ m on photomasks for integrated circuits. The range of the 10 certified linewidths on the standard is 0.74 to 10.56 µ m. The authors simulate estimates corresponding to 10,000 samples at x= 0.0, 0.1, 0.2, ... , 12.9 and provide summary statistics for each procedure.

Naes, T. ( 1985) Multivariate calibration when the error covariance matrix is structured. Technometrics, 27, 301 311.

In this paper, multivariate calibration in linear models is considered. The author assumes the error covariance matrix to have a linear factor structure. Parameters of the distribution of the variable to be predicted ( ) is estimated from the s in the calibration set and the distribution of is x x

x incorporated in the predictor. The approach to calibration in this paper involves estimating the parameters of the best linear predictor in the assumed model. The author compares his approach with the ordinary multiple linear regression approach to natural calibration. The predictor is tested on two ( meat and fish) data sets. The first data set contains observations on 60 samples of meat and for each sample water concentration is determined in a laboratory and the spectra are obtained on NIR spectrophotometer. The data set is divided randomly into two subsets, called A and B, each containing 30 samples. The data set A was first used for calibration, and the constructed predictor was tested on the set B. Afterwards the two sets were used in opposite order. Similar technique was used for the second data set ( fish data). Computations on NIR data sets show that the linear factor structure may be adequate and may give improved predictions.

Naszódi, L. J. ( 1978) Elimination of the bias in the course of calibration. Technometrics, 20, 201 205.

Naszódi considers the exact relation between and to be , , where is the x y y

p 1( x

¯ x)  p 2 x

H H domain of . The unknown value of an object is estimated with the help of a measured value x x of the object and of the fitted calibration line by inserting the value to the y m

y

p 1( x

¯ x)

p 2 y m inversion of the function ,

y Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 22

x  ¯ x  ( y m

p 2)

p 1 .

1 b

a

b a | T( x)| dx  > min xi .

a  x 1  x 2    x n/ 2, x n/ 2

1

x n/ 2

2

x n

b The author shows that this point estimate is biased even if is an unbiased estimate of . A new

y y estimate is suggested which can be used with a slight restriction for the distribution of the

x errors. The new is practically unbiased and can easily be derived from . Naszódi then

x  x considers a problem of finding an optimal experimental design by minimizing the average of absolute value of the bias of on the calibration domain , i. e. x H

[ a, b]

If is even, then the solution to this problem is the end point plan n and the reached minimum is . If instead of minimizing the average of the absolute

2/( np 2 1 ( b

a)) value of the bias one considers important the optimum criterion that , one max x

H | T( x)|  > min xi obtains the same plan. Simulations with equidistant and optimal calibration designs were performed to compare the classical and inverse estimates to the new estimate from the viewpoint of bias. The simulation study shows that the new estimate is more efficient then the classical estimate and has the

x  x advantage of consistency over the inverse estimate .  x Ott, R. L. and Myers, R. H. ( 1968) Optimal experimental designs for estimating the independent variable in regression. Technometrics, 10, 811 823.

The problem of optimal calibration design is studied under correct classification and misclassification of a regression model. The design criterion considered by Ott and Myers is minimization of the integral of over the calibration range of x with respect to design. It is assumed that the E(

x

x) 2 calibration range of interest is of finite length. Therefore, without loss of generality, the authors consider that of . They evaluate the optimal calibration design for linear estimation when

1

x

1 the function is truly linear and also when the function is truly quadratic. They found that when the function is linear, regardless of whether x or y is estimated, that the optimal calibration is that half Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 23 of the calibration observations are at the lower end of the calibration range of x and half of the observations are at the upper end of the calibration range. If an odd number of samples is used for calibration, one observation should be at the mid point of the range and the other observations should be evenly split between the two end points. In the case where the function is quadratic in nature, but a linear approximation is used for calibration, the problem is more complex. Thus, if a quadratic function of the form is approximated by , the optimal calibration y  c 0  c 1 P 1( x)  c 2 P 2( x)

y  a  bx design is found to be a function of , where and involves solving an equation f  |

/ c 2|

2  2  2 n of the fourth order. Numerical solutions for various sets of parameters were obtained by the authors in this case.

Scheffé, H. ( 1973) A statistical theory of calibration. The Annals of Statistics, 1, 1 37.

Scheffé considers two related quantities U and V such that U is relatively easy to measure and V relatively difficult, requiring more effort or expense. If a measurement u of U is made when V has the value v, it is assumed for v in an interval that u is a normal random variable with mean [ v ( 1), v ( 2)] and variance , where the { } are unknown parameters and { } are m( v,  )

p j

1

j g j( v)  2

j g j( v) known functions. In a calibration experiment, measurements are made at known n U 1, U 2,

, U n values . Let , . The closed interval V 1, V 2,

, V n v ( 1)

min( V i, i

1,

, n) v ( 2)

max( V i, i

1,

, n) is called the calibration interval of ; it is a known interval, assumed to be T

[ v ( 1), v ( 2)] V nondegenerate. The author makes no assumption about the " fine structure" of the distribution of u i for outside the calibration interval T. The estimates of the { } are calculated from the calibration v i

j experiment by the method of least squares. After the calibration experiment, measurements of U are made at unknown values of u i v i V, for , and it is assumed that the random variables are independent. For i

1,2,

U 1, U 2,

, U n, u 1, u 2,

each measurement , a lower limit, a point estimate, and an upper limit are made for the u i corresponding unknown . Associated with these intervals are two probabilities: and . By the v i   Scheffé procedure, the probability is that at least of the intervals calculated will contain 1

 1

 the true value of . v i The procedure for finding the interval estimates is as follows. First, the numerical values of the least squares estimates { }, the estimate , and the coefficients { } are calculated, where

j

 2 b jk b jk is the covariance between and divided by . The calibration chart is constructed by plotting

j

k  2 the calibration curve , the lower calibration curve , and the upper u

m( v,

) u

m( v,

)

 [ c 1

c 2 S( v)] calibration curve . The function S( v) is defined as for u  m( v,

)

 [ c 1

c 2 S( v)] S( v)  STD( m( v,

) )/  . Constants and are functions of . Corresponding to v

[ v ( 1), v ( 2)] c 1 c 2  ,  , p, minS( v), maxS( v) an observation , falling in the calibration interval of U, a point estimate of may be read as the u i

v i v i ordinate of the calibration curve at . In this case the confidence interval for is determined by u  u i v i the ordinates read from the lower and upper calibration curves at . u  u i As an illustration Scheffé considers how general method applies in the case of linear Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 24 regression with p= 2. In this case the author present an explicit solution of the calibration curve equation without resorting to numerical or graphical methods. In fact, the upper and lower calibration curves are found to be the two branches of hyperbolas. The upper and lower calibration curves will be closest to the calibration line if has its minimum, which is attained for even n if half 1/

( V i  ¯ V) 2 the observations in the calibration experiment are taken at the value and half at . However, v ( 1) v ( 2)

Scheffé does not recommend using this calibration design, but rather a more or less uniform distribution of the within the calibration range . V i [ v ( 1), v ( 2)]

Shukla, G. K. ( 1972) On the problem of calibration. Technometrics, 14, 547 553.

Shukla compares Krutchkoff's ( 1967) classical and inverse estimators using MSE's when the number of calibration samples ( n) is large and the number of replicates of the unknown quantity ( n') is a positive integer. Shukla shows that the classical estimator is consistent, but the inverse estimator is not. He demonstrated that when n'= 1 and n>= 8, the MSE of the inverse estimator is always less than the classical estimator. When the number of observations used in calibration is small, the inverse estimator yields a smaller MSE. However, the inverse estimator is unlikely to be advantageous when a large number of observations are used in calibration and n' is also large. Quite often, only one observation can be made on the unknown . In that situation, the inverse estimator is preferable X 0 when lies close to the mean of the known X's. Shukla suggests that due to its consistency, the X 0 classical estimator be used with large sample sizes when there is no prior information about the unknown X. Shukla mentions that the optimum calibration design for estimating parameters and prediction is the end point design such that a< X< b and where half of the observations are taken at a and half of the observations are taken at b. He mentions that the endpoint design is optimum for classical estimation, but it is not obvious if it is optimum for the inverse estimation.

Spiegelman, C. H. and Studden, W. J. ( 1980) Design aspects of Scheffe's calibration theory using linear splines. Journal of Research of the National Bureau of Standards, 85, 295 304.

Spiegelman and Studden discuss the design of the calibration experiment in the context of Scheffe's approach ( 1973) to the uncertainties of a calibration curve and in particular for the case in which the calibration curve is a linear spline. Their goal is to minimize the measuremnt error caused by calibration by using splines to determine a separate calibration curve for certain intervals. As an example the authors consider the design problem for the pressure volume tank calibration curve based on splines ( see Lechner, Reeve, and Spiegelman ( 1982) for details.) The pressure volume relationship can be represented as a spline Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 25 m( v)

k

1 i

0

i N i( v) ,

N 0( v)

 1

v  1

 0 ,  0

v <  1 0,  1

v N i( v)  v

 i  1  i

 i  1 ,  i  1

v <  i  i  1

v  i  1

 i ,  i

v <  i  1 0, elsewhere i  1,2,

, k N k  1 ( v)  v

 k  k  1

 k ,  k  v <  k  1 0, v >  k  1 where The quantities are called " knots" and these will occur wherever the cross sectional  0 <  1 <

<  k  1 area changes. The design problem is to see what quantative statements can be made about where the calibration points should be chosen to minimize the width of the confidence intervals v 1, v 2,

, v n . A class of appropriate designs is given, which depend on the location of the knots and the I( u) slopes of the segments. Based on these results, a calibration plan is suggested, which incorporates a two phase procedure for collecting observations. A numerical example is presented in which the authors show that in the area where the knots are concentrated the proposed design is superior to the equally spaced calibration design.

Thomas, M. A. and Myers, R. H. ( 1973) Optimal designs for the inverse regression method of calibration. Communications in Statistics 2, Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 26 1 n

x 2 i  1/ 3 S Q

( x 2 i

¯ x 2)  n 10 . 419 433 Whereas Ott and Myers look for optimal calibration designs using the " classical" estimator, Thomas and Myers look at optimal designs using the inverse estimator and an average mean square error criteria. Designs are developed for a linear approximation when the true model is linear and when it is quadratic. Let first the relation between and be linear and given by , where x y y

x   . They found that the optimal designs of the linear relationship which exist rely on   N( 0,  2) n and the unknown parameters, and and are not realistically usable.

 When evaluating calibration designs when the relationship between and is truly x y quadratic ( ), optimal designs rely on and unknown parameters, , , . y

0

1 x

2 x 2   n

1

2  However, using methods of numerical integration and minimization, they found nearly optimal designs when the following conditions were satisfied:

where is limited to the calibration range of [ 1, 1] and when x There is no optimal two point design for a quadratic relationship using linear calibration and a three point design is optimal only when is a multiple of 19. Nearly optimal designs were found n for four and five point designs when the selection of the calibration points are symmetric about the center and the number of observations are also symmetric about the center. Using the range of as [ 1, 1], 0 is the central value ( ), and refers to the number of observations at 0. x x 0 n 0 through refer to the number of observations at , respectively, such that , n 1 n 4 x 1

x 4 x 1

x 4 , , and . Thomas and Myers found nearly optimal design when is x 2

x 3 n 1  n 4 n 2  n 3 2 ( n 3  n 4)/ n greater than or equal to 0.526. They went on to explore six different designs when and n  10 determined the number of observations at each value of and what the optimum values of x x were. In summary, since optimum calibration designs rely on unknown parameters for underlying linear or quadratic relationships, they suggest using the near optimum designs found when the relationship was truly quadratic.

Vecchia, D. F., Iyer, H. K. and Chapman, P. L. ( 1989) Calibration with randomly changing standard curves. Technometrics, 31, 83 90. Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 27 This paper concerns with changing calibration curves from one time to the next due to various factors, such as changing environmental conditions, instrument aging and several other causes. The usual practice in such situations is to recalibrate the measuring devices at frequent intervals and obtain estimate of the unknown values of the test samples using only data from the corresponding calibration period. The authors consider a random coefficient regression model for the different calibration curves and combine the data from all calibration periods to estimate the unknowns. A simulation study was performed to compare the two inverse estimators where the calibration curve is a straight line through the origin. The calibration design was as follows: number of periods, T= 4, 8, 36; number of standards, n= 2, 4; design points, ; slope/ error ratio, = 5, 10, 20, 50, 100; x 1  x 2

 x n  1/ n

/  between/ within ratio, = 1.01, 2, 5; and unknowns, = 0.25, 0.50, 0.75, 1.00. The authors  2 0 u n mention that the parameter values chosen for the Monte Carlo study cover regions that correspond to many practical situations. Using the Pitman closeness criteria, the authors show that it is possible to obtain more efficient estimate of the unknown values by this method as compared to the usual estimator. The authors also propose a large sample combined periods confidence interval for the value of an unknown test sample.

Williams, E. J. ( 1969) A note on regression methods in calibration. Technometrics, 11, 189 192.

In this paper, a brief review of the classical and the inverse calibration methods is performed to set up a response to the questions raised by the Krutchkoff ( 1967) article which simulates the classical and the inverse calibration approaches using the calibration of [ 0,1]. The author firsts examines the classical calibration method and determines that the classical estimator has undefined expectation and infinite variance, and hence infinite mean squared deviation ( MSD). The inverse estimator is then examined and is determined to have finite mean squared deviation for . Therefore, the inverse N  4 estimator is " better" than the classical estimator in terms of MSD. However, the author points out that showing the MSD of an estimator is less than infinity is not proving very much, and thus brings into question the use of MSD as a criterion for goodness of fit of the estimators. To further weaken the argument of Krutchkoff ( 1967), the author uses Blackwell ( 1947) to show that no unbiased estimator ( i. e. the classical estimator) will have finite variance. This determination, combined with the fact that the inverse estimator examined in the Krutchkoff ( 1967) article has constant variance, provides the author with the determination that the minimum variance or minimum MSD criterion is not suitable for this type of problem. Suggested references for using confidence interval are then given as an alternative for what is needed for the calibration type problems.

Yao, Y. C., Vecchia, D. F. and Iyer, H. K. ( 1988) Linear Calibration when the coefficient of variation is constant. Probability and Statistics: Essays in Honor of Franklin A. Graybill, 297 309. Annotated Bibliography on Measurement Limitations and Calibration Updated: 2/ 27/ 98 28 The authors consider point estimation of the unknown parameters in the calibration problem. They assume the calibration curve to be a straight line through the origin and the responses are normal with variances proportional to the square of their mean . The problem is stated in the following manner: Let be the values of the standards and the corresponding responses. Let X 1, X 2,

, X n Y 1, Y 2,

, Y n u be the true unknown value of a test sample and the corresponding response be . The observations z are mutually independent and normally distributed and the unknown is assumed to be nonzero. u Also, it is assumed that , , where E[ Y i]

X i Var( Y i)   2X 2 i , i  1,2....., n, E[ z]

u, Var( z)   2u 2 are unknown. The main objective is to obtain an efficient point estimator of The authors  2,

, u u. consider weighted least square estimation, maximum likelihood estimation, equivariant estimation and inverse estimation. They compare the estimators using the Pitman's closeness criterion. Three different sample sizes were considered. In each case a 2 point calibration design with equal number of observations at each point was used. Asymptotic results indicate that the classical estimator ( weighted least squares) is superior to the maximum likelihood, equivariant and inverse estimator and the same conclusions can be drawn for small samples. The classical estimator is preferable to the other estimators, if the sensitivity ( ) of the calibration procedure is greater than or equal to 1.4 in

/  absolute value, which is usually the case in practice.


[Federal Register Volume 77, Number 23 (Friday, February 3, 2012)]
[Notices]
[Pages 5507-5508]
[FR Doc No: 2012-2379]

-----------------------------------------------------------------------

DEPARTMENT OF ENERGY

Federal Energy Regulatory Commission

[Project No. 2299-075]

Turlock Irrigation District, Modesto Irrigation District; Notice
of Proposed Restricted Service List for a Programmatic Agreement for
Managing Properties Included in or Eligible for Inclusion in the
National Register of Historic Places

    Rule 2010 of the Federal Energy Regulatory Commission's
(Commission) Rules of Practice and Procedure provides that, to
eliminate unnecessary expense or improve administrative efficiency, the
Secretary may establish a restricted service list for a particular
phase or issue in a proceeding.\1\ The restricted service list should
contain the names of persons on the service list who, in the judgment
of the decisional authority establishing the list, are active
participants with respect to the phase or issue in the proceeding for
which the list is established.
---------------------------------------------------------------------------

    \1\ 18 CFR 385.2010.
---------------------------------------------------------------------------

    The Commission staff is consulting with the California State
Historic Preservation Officer (hereinafter, California SHPO), and the
Advisory Council on Historic Preservation (hereinafter, Council)
pursuant to the Council's regulations, 36 CFR part 800, implementing
section 106 of the National Historic Preservation Act, as amended, (16
U.S.C. 470 f), to prepare and execute a programmatic agreement for
managing properties included in, or eligible for inclusion in, the
National Register of Historic Places at the Don Pedro Hydroelectric
Project No. 2299.
    The programmatic agreement, when executed by the Commission and the
California SHPO would satisfy the Commission's section 106
responsibilities for all individual undertakings carried out in
accordance with the license until the license expires or is terminated
(36 CFR 800.13[e]). The Commission's responsibilities pursuant to
section 106 for the Don Pedro Hydroelectric Project would be fulfilled
through the programmatic agreement, which the Commission proposes to
draft in consultation with certain parties listed below. The executed
programmatic agreement would be incorporated into any Order issuing a
license.
    Turlock Irrigation District and Modesto Irrigation District, as the
licensees for the Don Pedro Hydroelectric Project No. 2299, and the
Central Sierra Me-Wuk, Tuolumne Band of Me-Wuk Indians, North Fork Mono
Tribe, Southern Sierra Miwuk Nation, Chicken Ranch Rancheria of Me-Wuk
Indians, Buena Vista Rancheria, California Valley Miwok Tribe, Picayune
Rancheria of the Chukchansi Indians, National Park Service, and Bureau
of Land Management have expressed an interest in this preceding and are
invited to participate in consultations to develop the programmatic
agreement.
    For purposes of commenting on the programmatic agreement, we
propose to restrict the service list for the aforementioned project as
follows:

John Eddins or Representative, Office of Planning and Review, Advisory
Council on Historic Preservation, 1100 Pennsylvania Ave. NW, Suite 809,
Washington, D.C. 20004.
Kevin Day or Representative, Tuolumne Band of Me-Wuk Indians, P.O. Box
699, Tuolumne, CA 95379.
Sandy Vasquez or Representative, Southern Sierra Miwuk Nation, P.O. Box
1200, Mariposa, CA 95338.
Amanda Blosser or Representative, Office of Historic Preservation,
Department of Parks and Recreation, 1725 23rd Street, Suite 100,
Sacramento, CA 95816-7100.
Rhonda Morningstar Pope or Representative, Buena Vista Rancheria, P.O.
Box 162283, Sacramento, CA 95816.
Robert Nees, or Representative, Turlock Irrigation District, P.O. Box
949, Turlock, CA 95381.
James Barnes or Representative, Bureau of Land Management, Mother Load
Field Office, 5152 Hillsdale Circle, El Dorado Hills, CA 95762
Reba Fuller or Representative, Central Sierra Me-Wuk Cultural and
Historic Preservation Committee, P.O. Box 699, Tuolumne, CA 95379.
Ron Goode or Representative, North Fork Mono Tribe, 13396 Tollhouse
Road, Clovis, CA 93611.
Stephen Bowes or Representative, National Park Service, 111 Jackson
Street, Suite 700, Oakland, CA 94607.
Lloyd Mathiesen or Representative, Chicken Ranch Rancheria of Me-Wuk
Indians, P.O. Box 1159, Jamestown, CA 95327.
Silvia Burley or Representative, California Valley Miwok Tribe, 10601
N. Escondido Place, Stockton, CA 95212.
Greg Dias, or Representative, Modesto Irrigation District, P.O. Box
4060, Modesto, CA 95352.
Reggie Lewis or Representative, Picayune Rancheria of the Chukchansi
Indians, 46575 Road, 417A, Coarsegold, CA 93614.

    Any person on the official service list for the above-captioned
proceeding may request inclusion on the restricted service list, or may
request that a restricted service list not be established, by filing a
motion to that effect within 15 days of this notice date. In a request
for inclusion, please identify the reason(s) why there is an interest
to be included. Also please identify any concerns about historic
properties, including Traditional Cultural Properties. If historic
properties are to be identified within the motion, please use a
separate page, and label it NON-PUBLIC Information.
    Any such motions may be filed electronically via the Internet. See
18 CFR 385.2001(a)(1)(iii) and the instructions on the Commission's Web
site (http://www.ferc.gov/docs-filing/ferconline.asp) under the
``eFiling'' link. For a simpler method of submitting text only
comments, click on ``eComment.''

For assistance, please contact FERC Online Support at
<EMAIL_ADDRESS>call toll-free at<PHONE_NUMBER>; or, for
TTY, contact<PHONE_NUMBER>. Although the Commission strongly
encourages electronic filing, documents may also be paper-filed. To
paper-file, mail an original and seven copies to: Kimberly D. Bose,
Secretary, Federal Energy Regulatory Commission, 888 First Street NE.,
Washington, DC 20426. Please put the project number (P-2299-075) on the
first page of the filing.
    If no such motions are filed, the restricted service list will be
effective at the end of the 15 day period. Otherwise, a further notice
will be issued ruling on any motion or motions filed within the 15 day
period.

     Dated: January 27, 2012.
Kimberly D. Bose,
 Secretary.
[FR Doc. 2012-2379 Filed 2-2-12; 8:45 am]
BILLING CODE 6717-01-P