IL t I, AN ASME REPORT CRTD Vol. 60 REFERENCE METHOD ACCURACY AND PRECISION ( ReMAP): PHASE 1 Precision of Manual Stack Emission Measurements Prepared by:

W. Steven Lanier GE Energy and Environmental Research Corporation Charles D. Hendrix Statistical Consultant under the auspices of: American Society of Mechanical Engineers Research Committee on Industrial and Municipal Waste February 200 1 Disclaimer ( hereinafter referred to f, make any warranty,

owned rights.

Reference herein to any preparation or review of this report, or any agency thereof.

Statement from the by laws of th statements or opinions advanced Authorization to photocopy falling within the fair use provisions libraries and other users registered that the applicable fee is paid direct1 01923. [ Telephone: ( 978) 750 8400] Requests for special permission or bulk reproduction shouId be addressed to the ASME Technical Publishing Department. Society shall not be responsible for printed in publications ( 7.1.3).

or personal use under circumstances not Copyright Act is granted by ASME to Copyright Clearance Center ( CCC) provided CCC, 222 Rosewood Drive, Danvers, MA r ' 1 ,

6" I I.% Section 1 . o 2.0 3.0 4.0 5.0 6.0 7.0 TABLE OF CONTENTS Page Executive Summary ................................................................................................................ S I Introduction ........................................................................................................................ 1 Background ....................................................................................................................... .3 . . The Analysis Procedure  A Layman's Descnphon .............................................................. 1 1 Measures of Precision ................................................................................................ 1 1 Estimating Standard Deviation .................................................................................. 13 The Relationship Between S and C ..... 1 ..................................................................... 17 Summary of ReMAP Analysis Procedure ................................................................. 21 3.1 3.2 3.3 3.4 Confidence Intervals .................................................................................................. 18 3.5 EPA Particulate Matter Methods  Method 5 and 5i ............................................................. 25 4.1 Method 5 Data and Precision Analysis ...................................................................... 28 4.2 Method 5i Data .......................................................................................................... 50 4.3 Discussion of Particulate Matter Measurement Results ............................................ 61 EPA Method 23 for Measuring Dioxin and Furan Emissions ............................................... 67 5.1 5.2 Available Multi Train Data for Method 23 as Total PCDDPCDF ........................... 69 Analysis of Method 23 Data for Total Dioxin and Furan .......................................... 70 5.3 Available Multi Train Data for Method 23 as ITEQ ................................................. 8 1 EPA Method 26 for Hydrochloric Acid ................................................................................. 9 1 EPA Methods 29, 10 1 a and 10 1 b for Mercury ...................................................................... 103 TABLE OF CONTENTS ( Cont.)

8.0 EPA Method 29 for Multi Metals .......................................................................................... 1 19 8.1 EPA Method 29 Data for Antimony, Arsenic, Beryllium, Cadmium, , I Chromium, and Lead .......*.........,.............................................................................. l 19 8.1.1 Antimony Data .............................................................................................. .119 ~

I r L 8.1.2 Arsenic Data ................................................................................................... 126 8.1.3 Beryliium Data ............................................................................................... 126 8.1.4 Cadmium Data ............................................................................................... 126 ??

8.1.5 Chxnium Data .............................................................................................. 122 8.1.6 Lead Data ............................... ................................................................... I22 EPA Method 29 Regression Analyses ....................................................................... 138 ??

8.2 9.0 Other Measurement Methods ................................................................................................. 167 10.0 Conclusions ........................................................................................................................ 169 BNDi References ........................................................ ......................................................... *. I72 Appendix  Statistical Analysis Procedures for the ReMap Program c _*

... 111 TABLE OF CONTENTS ( Cont.)

Figure ' Page 1 2 3 4 5 6 7 8 9 10 11 I2 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 Illustration  The Difference Between Precision and Accuracy ............................................ 7 Normal Distribution of Field Results .................................................................................... .12 Simulated Data ....................................................................................................................... 16 Regression Line and Confidence Interval for Simulated Data .............................................. 19 Schematic of Method 5 Sampling Train ................................................................................ 26 EPA Method 5 Data  Standard Deviation ( Fig. 6A) ............................................................ 37 EPA Method 5 Data  Relative Standard Deviation ( Fig. 63) ............................................. .. 3 8 thod 5 ....................................... 42 d 5 ( Front Half Only) ........... 48 .............................................. 5 1 .............................................  56 eviation ( Fig. 11B) ............................................ 57 .............................................. 60 .............................................. 62 .............................................. 68 viation ......................................... .72 Deviation .......................... .. 7 .................................................... .76 nts Using EPA Method 23 ............ 78 ion ................................................ 83 .................... 92 Method 26 ..................................... 10 1 ........................................................ 104 iv TABLE OF CONTENTS ( Cont.)

Fimre Page 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 Schematic of Method 29 Sampling Train .............................................................................. 105 Schematic of Method l 0 l b Sampling Train .......................................................................... 106 EPA Method 29/ 10 1 all 0 1 b Data  Total Mercury Standard Deviation ................. ...... ...... ... 1 1 1 EPA Method 29/ 10 1 a / l O 1 b Data  Total Mercury Relative Standard Deviation .......... ........ 1 12 Regression Line and 95% Confidence Interval

EPA Method 29 Data for Antirn ata for Antimon ata for Arsenic

..................................... 140 ......,............ 143 Regression Line and 95% C Regression Line and 95% EPA Method 29 Precision EPA Method 29 Precision d 29  Chromium ................ 146 d 29  Lead .......................... 147 Precision Estimates for Mea Precision Estimates for Mea EPA Method 29 Precision Metrics  Composite Data ......... .... . ................._...... ..................... 165 V  Table 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 TABLE OF CONTENTS ( Cont.)

Paae EPA Method 5 Data  Hamil and Camann, 1974 and 1974b ................................................ 29 EPA Method 5 Data  Dade County MWC, Hamil and Thomas, 1976 ................................ 3 1 EPA Method 5 Data  Pittsfield MWC, E g o and Chandler, 1997 ....................................... 32 SPC Factors for Identification of Data Outliers ..................................................................... 33 Consolidated Method 5 Data Set ........................................................................................... 34 Factors for Calculating Unbiased Estimates of sigma ts Based on S ..................................... 35 Method 5  Small Sample Bias Correction to Standard Deviation ....................................... 36 Method 5 Regression Analysis Results .................................................................................. 40 EPA Method 5i Data and Standard Deviation  Eli Lilly Data, ( Table 9A) .......................... 53 andard Deviation  EPA DuPont Data, ( Table 9B) .................. 55 Regression Analysis for Method 5i Data .............................................................. 58 Data as Total Mass of Tetra through Octa Dioxin pIus Furan ............................. 71 ate of Method 23 Standard Deviation ................................................................... 79 Data as ITEQ ....................................................................................................... 82 Data and Standard Deviation for HC1 ............................................... 93 uture HC1 Data .................................................................................. 102 a for Mercury ( Table 16A) ............................................................. 109 a for Mercury ( Table 16B) ............................................................. 1 10 cipated Future Mercury Data ........................................................................... 1 18 ata and Standard Deviation for Antimony ..................................... 120 ata and Standard Deviation for Arsenic ......................................... 121 ata and Standard Deviation for Beryllium ..................................... 122 ata and Standard Deviation for Cadmium ...................................... 123 ain Data and Standard Deviation for Chromium .................................... 124 ain Data m d Standard Deviation for Lead .............................................. 125 gression Analysis for Various Metals ............................................ 13 8  I f Measured PCDDRCDF Concentration Based on Range of Anticipated Future Metals Data  Antimony, Arsenic, Beryllium, Chromium, or Lead ............................................................................................................ 166 Vi This page IntentionaIly left blank. Reference Method Accuracy and Precision ( ReMAP Phase I) An Assessment of the Precision of EPA Manual Stack Emission Measurements Executive Summary This report documents results from the first phase of a study co sponsored by the American Society of Mechanical Engineers ( ASME) to assess the accuracy and precision of manual test methods adopted by the US Environmental Protection Agency ( EPA) for determining the stack concentration of selected air pollutants. The program is entitled Reference Method Accuracy and Precision and is referred to by the acronym ReMAP. The Phase 1 effort addresses the precision of the selected measurement methods. The formal Purpose Statement for the program is:

" To determine the precision of pollutant emission measurements based on analysis of available simultaneous  sample test data which were generated using EPA Manual Reference Test Methods 5 and 5i ( Particulate Matter), 23 ( Dioxin and Furan), 26 ( HCI), 29 ( multi metals), 1Ola and lOlb ( mercury) and 108 ( arsenic) at a number of stationaty air sources."

As used in the Re rogram, precision is defined as random error that inadvertently enters the measurement process. This error may enter at any stage of the measurement le recovery, or sample analysis. The impact of deviate from the true stack concentration. any manual test of the selected ibution. This distribution is xpected indicating measured Because these e Precision of a curve. One c o r n dicated by the horizontal spread of the bell & e bell curve shape is to determine the ( c) of the distribution. Alternately, the bell distributian also peated application of the the best estimate of the ators of data quality that two additional, directly measurement 99 out of 100 future single measurements. If the is repeatedly applied to a stack with a given concentration, this nes the upper and lower concentration bounds for 99% of measurement ES 1 2. The anticipated range for 99 out of 100 future triplicate measurements. Since most environmental regulations define the re able stack concentration as the average of three repeated test runs, this metric defines the anticipated range of results in triplicate ( 3 single time series) measurements due to random error in the measurement process.

For each of these precision metrics it is important to note the inherent assumption that faciiity operation.

100 single measurements will fall withi anticipated range for the average of repea to the true concentration. More precisely,

standard deviation of concentration.

lities must be assumed to of method precision mu trains are used to sim deviation of the particula measurement at the specific stack concentration.

ES 2 97 E l LJ The ReMAP program procedures required to performed a careful assessment estimate the precision of Manual of the statistical analysis Reference Methods using multi train sampling data ( see Appendix). To assure the quaiity of data used in the statistical anaiysis, an extensive effort was expended in gathering data from the original sources and carefilly evaluating them to assure that consistent data reduction procedures were used.

Conceptually, the ReMAP statistical analysis procedure is straightforward. First, data fiom a multi train test run are averaged to provide an estimate of the average C run ( Ci). The standard deviation for the test run ( Si) is also C a calculated standard deviation fiom a single test using a dual sampling probe provides a relatively poor estimate of the true standard deviation of the method ( 0) at the true concentration ( p) However, after accounting for various biases, a significant array of data from multi train tests should provide a reasonable basis for estimating the true standard deviation as a function of concentration. The ReMAP procedure is to ass at the standard deviation varies with concentration according to a power functio nship and then to fit the data to that equation using regression analysis.

regression analysis represent the best estimate av able on the standard measurement method at any given concentration ReMAP analysis procedure also provides for calculation of confidenc s define the upper and lowe lyses are summariz i for Particulate Matter  Front Half Only frdm the true ave h tacks with particulate concentrations ual train sampling and ep the simultaneous measurements. eliminate test results reening, coupled with the ReMAP anaiysis ant variation of standard deviation with concentration.

ES 3 Based on a pooled analysis, the characteristic standard deviation for Method 5i was found to be 1.43 mgdscm. Based on this best estimate of standard deviation, the ReMAP analysis indicates that 99 out of 100 Method 5i single measurements should deviate from the true concentration by no more than f 2.68 mg/ dscm. For triplicate measurements 99 out of 100 Method ata results should deviate from the true concentration by no more than f 2.12 rng rs ranging from zero a were reported in both forms. Even nts, the ReMAP results suggest that though the same data were used for both Table ES I . Anticipated Range o analysis found no statistically significant variation of standard deviation with ES 4 r I;

P L1 r: Li concentration. Pooled analysis indicates that the best estimate of standard deviation is 27 ng ITEQ/ dscm. when the emission concentration is in the range of 0.02 to 0.9 ng ITEQ/ dscm. This further indicates that 99 out of 100 future single measurements shoufd fall with 50.069 ng ITEQ/ dscm of the m e concentration and 99 out of 100 triplicate measurements should fall within 50.04 ng ITEQ/ dscm of the true concentration.

99 out of 100 Single Concentration I Measurements Truestack HCl The absolute value of anticipated range for future Method 25 measurements ( as ITEQ) are quite small in absolute t ut they are on the same order as regulatory emission limits being considered in regions. As indicated above, the best estimate of standard deviation is 0.027 ng ITEQIdscm. However, at 95% confidence, the standard deviation may be as large as 0.037 ng ITEQ/ dscm and the potential range for 99 out of 100 future measurements might deviate from the true concentration by as much as Q/ dscm. ReIying upon a single measurement has the potential to create mission limits were set at 0.095 ng ITEQ/ dscm, to be assured confidence level, measurement resuits could not exceed zero. results must be above 0.19 ng ITEQ/ dscm e true stack concentration exceeded the emissio 95% confidence Most regulations and permit limits establish compliance based on averaging results from triplicate measurem icipated range for 99 out of 100 fkture triplicate to single measurements, by 43. Thus, compliance EQ/ dscm is assured ( at the 95% codidence level) or below 0.04 ng ITEQ/ dscm. Similarly, at 95% 95 ng ITEQ/ dscm limit is assured when the three run confidence, exceeden 99 out of 100 Triplicate Measurements I Method 26 for Hydrochloric Acid ReMAP analysis of available data for Method 26 for HCI indicated that RSD is typically in the range of 5% to 10%. RSD does increase when the method is applied to stacks with very low concentration. Table ES 2 summarized the anticipated upper and lower bounds for 99 out of 100 Method 26 measurements as a hction of true stack HCI concentration.

TabIe ES 2. Anticipated Range of HC1 Measurement Results Due to Random Error in Application of Method 26.

I 20 16.1 23.9 17.7 22.3 50 41.9 58.1 45.3 54.7 100 85.8 1 14.2 91.8 108.2 I I I I I J ES 5 Methods 29, lOla and lOlb for TotaI Mercury Several measurement methods have been de concentration and for mercury speciation, multi train mercury data collected using Me data for total mercury concentration. concentration range of 50 to 78 9.6 to 12.4%. As concentration from 12.4% to for 99 out of 10 ped for measurement of rota1 emission lysis took ail availabIe The data analysis i Results Due to Random Method 29 for Multi Metals Method 29 is also used for measurement of several other metal emissions. Precision analysis was completed for six other metals including antimony, arsenic, beryllium, cadmium, chromium, and lead. With exception of cadmium, the analysis indicates that these metals behave similarly wi to measurement method precisio composite andysis was performed for the g metals and the results indicate that use of Method 29 provides an RSD that tween 13 and 18% when the individual metal concentrations are b pg/ dscm. Table ES 4 summarizes the anticipated upper and 1 9 out of 100 Sb, As, Be, Cr, and Pb measurements using Method e stack total metal concentration.

As regards cadmium measurements using M 9, the analysis indicates that standard deviation is a weaker fiction of conce at least at higher concentration ranges. The best estimate of R oncentration is 80 pg/ dscm and 18.7 % when the concentr However, at 5 pg/ dscm, predicted RSD is 38. to exceed 75%.

ES 6 h a L 100 65.7 134.3 " PLY  ,  4 80.2 119.8 I c , I L u r 1 ES 7 This page Intentionally left blank. REFERENCE METHOD ACCUR4Cl PHASE I ZIYD PRECISION ( R e m )

PRECISION OF MANUAL STACK EMISSION MEASUREMENTS 1.0 Introduction An integral part of efforts to regulate and control air pollution emissions is collection and analysis of exhaust stream samples to determine the concentration and flow rate of pollutants released to the atmosphere. The U. S. Environmental Protection Agency ( EPA) and its counterparts in other countries have developed formal methods defining the hardware and procedures for collecting and analyzing samples to quanti emissions of individual pollutants. A significant number of these methods involve manual extraction of a sample from a facility's exhaust stack, sample recovery and subsequent laboratory analysis to quantify concentration of a specific pollutant( s) in the sample. All manual processes, inc ing the various EPA measurement methods, are subject to random variations, which ultimately impact the end results. Relatively minor variations in the skill of the sampler, as well as the equipment and procedures used to extract the sample can influence the indicated quantity of sample extracted from the stack and the eficiency with which the pollutant of interest is colIected or recovered. Similarly, minor variation in laboratory hardware and procedures influence quantification of the mass or volume of pollutant in that sample. The net result of such random variation is imprecision in measurement results. The current report documents a study where available data have been gathered and analyzed to quantify the precision of key EPA manual measurement methods. The study has been conducted under the auspices of the American Society of Mechanical Engineers ( ASME) and is entitled Reference Method Accuracy and Precision1 ( ReMAP), Phase 1. S The purpose of the R e m  Phase 1 program is " to determine the precision of pollutant emission measurements based on anaiysis of available simultaneous sample2 test data which were generated 1 Precision is defined here as " Random Error" according to the new ASME PTC 19.1 1998. 2 Dual train, quad train, and simultaneous samples from different sample locations at a stationary emission source.

1 using EPA Manual Reference Test Merhods 5 and j i ( PM). 23 ( dioxin andfiran), 26 ( HCI). 29 ( multi metals, lOla and IOIb ( mercury), and 108 ( arsenic) at a number of srationary air sources."

ASME intends ReMAP to be a multi phase effort with the first phase focusing exclusively on assessment of measurement method precision. Consideration o sues associated with measurement accuracy is reserved for a later phase of ReMAP.

Three major groups have sponsored the ReMAP Phase 1 effort. First, the U. S. EPA has provided funding and personnel support to the project. Second, several industrial groups representing manufacturing companies and the waste combustion industry have provided program funding.

Finally, the ASME's Committee on Industrial and Municipal Waste has provided both financial support and overall program direction.

Although Phase I results indicate that th Methods provide differing levels of precision and that the precision typically varies utant concentratio MAP does not reach conclusions relative to p be used. Answering those questions is appro the public. The role o rifically sound d facilitate meaningful policy debate and decisi lated industries, an It is important to note from the outset that a variety of stack emission concentrations. In addition to measure skill of the stack tester), variation of process feed mat operations impact stack emission A compliance test impacted by process variation over time and will potentially indicate greater variability method itself. ntribute to variability in measured thod precision ( which includes the luding combustion fuels) and unit ngle samples will be ssion concentrations the precision of fhe measurement 2 2.0 Background The US EPA has developed and published a wide variety of methods for determining the concentration of pollutants in process effluent streams. The manual air sampling methods typically invoIve a probe for extracting a representative sample of stack effluent and means for physically capturing or chemically extracting selected pollutants from that sample. The methods further define procedures for determining the volume of sample gas extracted and for recovering the coIIected pollutant( s) from the sampling apparatus. Finally, the methods specify laboratory procedures to use for determining the quantity of pollutant collected.

In developing new measurement procedures,  PA has traditionally conducted extensive laboratory and field validation studies including tests to define the precision and biases of the method.

Procedures empioyed by EPA have evohecl over the years but generally conform to those described in a 1977 paper entitled, " How  PA Validates NSPS Methodology" ( Midgett, 1977). Mos procedures discussed by Midgett have been incorporated into EPA Method 301 which " is used whenever a source owner or operator proposes a test method to meet a U. S. Environmental Protection Agency ( EPA) requirement in the absence of a validated method."( EPA, 1992) In validation of a new method or in tests to evaluate an alternate method, EPA suggests use of four sampIing trains to simultaneously extract samples from nominally the same location in a source stack. This is commonly referred to as a quad train. For method validation, two of the four trains are configured and operated in strict accordance with the proposed method, while the other two trains are spiked with known quantities of the target analyte. Comparison of data from the two unspiked trains provides an indication of measurement precision while data from the spiked trains provides an indication of measurement bias. Data from a significant number of repeated multi train runs provide an indication of the precision and bias of the method itself. Method 301 states that " The precision of the method at the level of the standard shall not be greater than 50 percent relative standard deviation."

Several of the EPA measurement methods were developed and validated in the early days of the Agency.  PA Method 5 for measuring stack particulate concentration was published in the FederaI Register on December 23, 1971 ( 36FR 25876). Tests to validate that method were performed on 3 sources with particulate emissions ranging from 45 to 240 mddscm. In that time period, the majority of Federal particulate emission standa were established at 180 mg/ dscm ( 0.08 gr/ dscf)

corrected to 7% 02. Thus, the me ver a range that included the prevailing regulatory limits. At this emission limit, the EPA validation studies indicate that precision of Method 5 , expressed, as a relative standard deviation was on the order of 10%.

Passage of the Clean Air Act Amendments of 1990 ushered in new era in both scope and stringency of environmental regulations. Rules erning release of Hazardous Air PolIutants ( HAPS) called for regulation of I79 specific pollutants from both new and existing emission sources.

Provisions in the law, stipulate that standards must consider the " Maximum Achievable Control Technology" ( MACT). For existing sources, MACT standards shall not be less stringent than the average emission performance achieved by the subcategory. Implementation of this congressional mandate has resulted in many new emission regulations that are dramatically more stringent. F mple, in 1999 the particulate emission limit for hazardous waste incinerators was tig t performing 12% of the sources in a catego ed from 180 to 34 mg/ dscm (@ 7% 02).

Stringent emission standards raise numerous concerns a methods. The particulate standards can be used to highli method validation studies assessed acceptably precise over a rather broad applied and results used for regulatory for which it was validated. Method 5 mav be ac mgldscm) but, prior to ReMAP Phas meant all environmental stakeholders EPA measurement s. As noted, initial the Method to be for Method 30 1.

Particulate matter is not the only EPA Reference measurement Method for which there is concern.

Another example is  PA Method 23 for determining diox d furan emission concentration. The published method validation studies concentrated almo clusively on pre on and bias of 4 1 analytical procedures and largely ignored the sample collection portion of the overall method. No Agency method validation data were provided examining the precision of the entire sampling and analysis procedure. Method 23 does provide for extensive spiking of the sampling train with labeled compounds and includes tests to quantify recovery of those standards. However, data are considered acceptable if the fractional recovery of the labeled compounds falls within the range of 40% to 130%. With such broad allowable recoveries and in the absence of full system precision analysis, it is anticipated that Method 23 may provide results with exceedingly wide precision bands.

The above noted issues do not imply that Methods 5 and 23 are technically unacceptable procedures,

Instead, these issues are typical of senera1 concerns that develop when method validations are incomplete or out of date. In the absence of well documented assessments of measurement method precision, many reasonable questions are formed and nurtured. Typical questions include:

In light of the economic and public perception consequences associated with a failed compIiance test, are the current EPA measurement methods technically acceptable procedures for determining compliance with standards that have become more stringent over time?

If a method is highly imprecise, will indication of a failed compliance test withstand scrutiny of a legal challenge?

Do data indicating emission concentrations below the regulatory limit really imply Databases used to establish MACT standards are generally developed based on reports from tests using published EPA methods. A critical portion of these data  data from facilities defining the best performing 12% of the facilities  is extracted to define the MACT technology or the MACT based emission limits. Do these data characterize exceotionallv well designed and operated facilities or do the key data represent imprecision in the measurement methods? This concern applies to any analysis where the best 12% of the data are selected for examination but it is even more critical when those data indicate results below the range for which the method was validated.

5 Concerns extend beyond regulatory compliance and regulatory development. Are the EPA methods acceptable procedures for determining whether a new air pollution control device is meeting its performance guarantees? is the indicated performance representative of the control device or do the test results reflect significant imprecision in the method? I The above lists of issues and concerns are far from exhaustive. However, almost invariably, the response to such questions is that the measurement methods may not be perfect but they are the best that we have. That answer does not, however, alleviate stakeholder fears. The sponsors of ReMAP,

including the US EPA, have entered'into the program to provide tools that might be used to develop better answers. This report is, however, not intended methods addressed herein or as a substitute for Method 301.

APPROACH There is often confusion concerning the terms, precision and accuracy. Figure 1, adapted from a presentation by Dr. Greg Rigo at a meeting of ASME's Committee on Industrial and Municipal Waste clearly illustrates what the two terms imply. Imagine shooting at a target. The illustration on the left shows a wide scattering of results, almost equally distributed around the bull's eye. The illustration on the right shows a tightly grouped set of shot in these results is an indication of precision while proximity to the bull'

accuracy. The target on the left illustrates poor precision but good accuracy. The target on the right illustrates highly precise, but inaccurate shooting. Phase I of ReMAP is concerned with precision.

Accuracy is an issue for later phases of ReMAP.

Many facilities have Iarge quantities of data from repeated single train stack tests. These resu important to the facility but typicaIly, they shed little light on the precision of EPA measurement Methods. Variation in repeat measurements is in also influenced by variations in facility operation. Unfortu for separatinz these two effects. Determination of measurement method precision must be based on simultaneous determinations of stack emission concentrations, preferably with co located probes.

6  k,

p9""

' k ? a 7 Results from two or more simultaneous measurements provide information to calculate a sample standard deviation of that measurement. Two data points  a single indication of standard deviation  are not. however, a sufficient basis for defining the precision of the measurement method. Repeated simultaneous data from a given source provide an improved indication of method precision.

Repeated simultaneous measurements from a v ty of facilities further improves the data base for assessing method precision since the data co tack concentrations and a broader range of personnel applying the method.

The Re assembled a database consisting of available multi train data from a variety of selected EPA methods. A key source of these data are published and unpublished EPA reports addressing method validation. Additionally, a limited number of industry sponsored, multi

train studies have been conducted and documented. These industry reports provide significant expansion to the scope of available multi train data. Finally, the  PA has sponsored a limited number of studies where multi train tests were performed to expand the range of data for validation of previously published methods. The ReMAP program has gathered available multi train data sets for the following EPA measurement methods:

9 Methods 5 and 5i for particulate matter ( PM) emissions Method 23 for dioxin and furan emissibns Method 26 for hydrochloric acid and chlorine gas Method 29 for multi metals, and 9 9 9 c:

9 Methods IO 1 a and 10 1 b for mercury.

A search was made for validation data on the follo discovered: ods but no multi train results were Method 108 for arsenic Method 0030 and 00 10 for volatile and semi volatile organics respectively, and Method 00 1 1 for formaldehyde.

8 After the initial data collection activity, the ReMap program took two parallel paths. One path provided for detailed validation of the gathered data. Wherever possible, validation began with the original field run sheets and continued through a complete re reduction of the data. This tedious process improved the database by providing consistency in such key factors as use of consistent standard reference conditions and blank correction procedures. The parallel effort involved identifLing and refining mathematical procedures for analyzing simultaneously sampled concentration data to determine measurement precision at various appropriate concentrations.

Finally, after validating the database, the selected statistical analysis procedures ( see Appendix)

were applied to the validated database to determine the precision of the selected EPA methods at appropriate concentrations.

A final preliminary point concerns the issue of correcting data to a fixed percentage of excess oxygen. Environmental regulations almost always set a limit on the concentration of pollutants in the stack and require that the concentration be adjusted to reflect a standard stack excess oxygen ( typically 7% oxygen). For several reasons the ReMAP study does not include oxygen correction in the analysis of measurement method precision. The primary rationale is that the various chemical analyses determine the quantity of a specific analyte in the overall sample matrix. If the quantity of analyte is low, it makes little difference to the chemical analysis whether the loading is the result of effective air pollution control or if the stack has high excess air. A more pragmatic consideration comes from the available data. Several of the key EPA method validation studies failed to record the stack oxygen concentration during the tests,

The following material develops estimates of measurement method precision as a function of the average pollutant concentration. Both the concentration and the precision metrics ( when expressed in concentration terms) can be adjusted to a fixed oxygen level by applying the following 0 2 correction equation:

This, of course, requires that one have knowledge of the actual stack oxygen level as well as the desired reference oxygen level. More specifically, the precision of a measurement method,

9 referenced to a fixed percent excess iir ( say 7% 02) will vary with the 02 concentration in the stack.

This issue will be discussed in further detail in later portions of the report Report 0 rganiza tion This report is divided into two different sections front portion of the repon has been written for readers with only a passing familiarity with st a1 analysis. Included are descriptions of the measurement methods and the database of multi train results. It also includes a layman's presentation of the data analysis procedures and a presentation of the study results. The last portion of the report ( actually an appendix) provides a detailed description of the statistical analysis procedures used in ReMAP. A serious attemp ade 10 make the main body ofthe report and the appendix readable and understandable to non statisticians. 3.0 The Analysis Procedure  A Layman's Description Material presented in this section provides a brief summary of statistical analysis procedures used in ReMAP to assess precision of the selected EPA measurement methods. The presentation is written for the statistical layman and may seem overiy simpiistic ta those skilled in the statistical sciences.

Such readers are referred to the report's Appendix which includes detailed development of the .

statistical analysis procedures.

3.1 Measures of Precision As indicated earlier, imprecision in a measurement method implies that random error in the sampling ry analysis result in random variation in the indicated emission concentration.

otherical stack that emits a nearly constant concentration of some pollutant.

Imprecision from the measurement method will result in measured concentrations deviating from the true stack concentration. If the hypothetical stack is sampled many times, a plot of the results should such as the one illustrated in Figure 2. The average of a large number of ach the true concentration and most of the data points will be relatively If the average does not approach the true concentration, the sed. Individual measurements that are significantly removed from the decreasing frequency. The core objective of the ReMAP program is in anticipated results for different manual measurement methods and measurements should concentrafion.

determination of how that spread varies with the stack concentration.

There are a variety of parameters that may be used to characterize the precision of a method. The U. S. EPA has historically used standard deviation or relative standard deviation to define precision.

There are, however, several other parameters that may be equally valid precision indicators. In Figure 2, standard deviation ( denoted by the symbol 0) is indicated as a distance on either side of the mean value in the distribution. The area under the bell shaped curve bounded by f cr has special mathematical significance but for current purposes it is sufficient to note that this area covers 68.2 percent of the total area under the curve. In the example discussed 11 12 S 0 IIJ C . I .) 111 L  b. 0 I

. above relative to Figure 2, the bell shaped curve represents the expected frequency of many future measurements. Sixty eight percent of those future measurements are expected to indicate concentrations within k one standard deviation of the mean. If the value of Q is small relative to the mean, then the majority of the measurement results will occur close to the true value. Conversely, if the relative value of G is large, a larger portion of the measurement results will deviate significantly from the true concentration.

A second approach for describing measurement precision is to define a data spread expected to encompass the majority of future measurements. A convenient approach is to define concentration bounds capturing 99% of the future data. This parameter may hold special significance to facility owners and operators who must comply with emission regulations. Essentially, this parameter defines the expected concentration bounds for 99 out of 100 future measurements. Where as 68.2 %

of measurements fall within A 1.0 ( r of the mean, 99% of the measurements fall within k 2.567 G of the mean. Both the standard deviation and 99% concentration bounds represent the spread in future measurements from random variation of the measurement method. The standard deviation and 99%

concentration bounds do not include variation of the emission source.

A third precision metric is the expected spread in the average of triplicate measurements. That parameter may have special significance to facility owners, since compliance with emission regulations is typically based on the average of three runs. All three precision metrics are related by simple proportion. As no above, 99 out of 100 future single measurements will fall within the bounds of 5 2.5670. For repeated measurements, the range decreases inversely with the square root of the number of repeat measurements. Thus, ninety nine percent of the average of future triplicate measurements will fall within the bounds off: 1.482 ( 5 ( k 2.567/ 43 G). AH three metrics will be calculated in the ReMAP precision assessments for each measurement method.

3.2 Estimating Standard Deviation Having settled on metrics for describing measurement precision, the problem is reduced to determining standard deviation as a function of measurement concentration. From the outset, it is critical to understand that there is a true value of standard deviation at any given concentration but 13 we will never know its exact value. In accordance with normal statistical nomenclature, the true value of standard deviation is given the symbol Q ( si, gna). Information concerning G can be obtained from special tests using two or m ampling trains operated simultaneously in the same exhaust stream. Ideally, such tests extract sample from the same nomina osition( s) in the stack.

Each measurement will be subject to ran different results from the simultaneous measurements. Referring to the bell shaped distribution curve in Figure 2; these two measurements represent two data points, randomly selected from the total population of potential data points. If a larse number of simultaneous measurements are raken, the individual dara poinrs should generate the full distribution. Typically, however, only two, three or four sampling trains are operated simultaneously. An estimate of the standard deviation for the measurement method is obtained by calculating the standard deviation from dual o ad train tests according to equation 1 below. .

i Eq. 1 Standard deviation calculated from experimental data is referred to by the symbol S to make a distinction between this value and the true standard deviation Q. Clearly, selecting two random points from the full population of points that ma aped distribution provides a poor estimate of Q. Selecting four points provides a b ing measurement methods, the typi eated tests provide rep deviation under nearly s for assessing method e method precision is a d values of the true constant concentration conditions.

standard deviation at the characteris very strong function of concentrat standard deviation that are referred to in this report as Est. c.

There are however, several complications to the process of estimating the standard deviation from dual and quad train test results. When standard deviation is calculated from smaIl samples of a large population, the result is a biased estimate. The magnitude of the bias is dependant upon the number of data points used to estimate each value of S. A detaifed presentation on the source of this bias is 14 beyond the scope of the current discussion. However, for illustrative purposes, consider the case where a large number of data pairs are randomly selected from a known distribution. Standard deviation ( S) can be calculated for each data pair according to equation 1. Most non statisticians will anticipate that the average of the standard deviations ( S) calculated from many data pairs would closely approximate the true standard deviation ( a) for the overall distribution. As discussion in the Appendix, that anticipation would not be realized. In fact, using data pairs, the average standard deviation would be biased low by a factor of 1.253. If we repeated this example using three or four data points for each standard deviation calculation, the average standard deviation will be closer to the true value but the bias will still be present. For triplicate measurements, the bias factor is 1.128 while for quad train the bias factor is only 1.085. In the ReMAP data analysis, the standard deviation calculated from each multi train test must be multiplied by the appropriate small sample bias correction factor. This provides an unbiased estimation of standard deviation at the selected concentration. This calculated parameter is referred to as small sample, bias corrected S. .

To assess the impact of pollutant concentration on method precision, it is necessary to gather multi

train measurement data over a broad range of concentration and to fit the data to an equation relating S to average concentration ( C). The first step is to check these data for outliers and to prepare the data for analysis. Two approaches are used by ReMAP to screen data for outliers. These procedures are defined in the Appendix and illustrated in the next section of the report. The first procedure is known as the Dixon's r test. This procedure is, used to examine a group of data points collected during a single, multi train test and to determine if one or more data points in the group are outliers.

It is only applicable to tests with three or more simultaneous sampling trains. The second screening procedure is taken from Statistical Process Control ( SPC) methods and is used to identify outliers from multiple simultaneous measurements. The essence of this procedure is to compare the span between simultaneous measurements against the weighted average span for other data in a similar concentration range. Special provisions are included to account for the fact that data may exist as pair, triplicates, or as quad train results.

After outlier screening, the validated data are entered into a spreadsheet; standard deviations are calculated and then corrected for the above noted small sample bias. Figure 3 illustrates a hypothetical set of data showing small sample bias corrected standard deviation versus mean 15 . . .

c7 LL T O 0 0 0 ..

0 0 0 ?

0 0 P T P"

p" 9  1 b concentration from several multi train tests. The objective is to fit these data to an equation relating standard deviation ( Sbias corrected) to average concentration ( C). If done properly, this curve fit wilI also approximately describe the relationship between G and C. However, before fitting the data to a functional form, it is necessary to account for the differences between data sets consisting of 2,

3, or 4 simultaneous measurements. Equation 1 can be applied to dual train measurements to determine the standard deviation ( S) with only one degree of freedom. Values of S can be determined from triplicate measurements but this S value has two degrees of freedom. Thus, a triplicate measurement provides as much information about the precision of the measurement method as two dual train measurements. Similariy a quad train provides as much information a s three paired trains. In curve fin g the data, weighting factors must be applied to account for the number of degrees of freedom from each data grouping.

 r 3.3 The Relationship Between S and C i As discussed in the Appendix, a wide range of functional forms is possible for describing the relationship between standard deviation and concentration. Based on the characteristics of the availabIe data, the form selected and justified for the ReMAf program is a simple power function as 6 L = m described in Equation 2,

k  4 S = k C p Eq. 2 7

Ld where S is the estimated standard deviation for the method, C is concentration, and k and p are constants. For each measurement method, the available data are fit to this equation using a least squares regression analysis. To facilitate that regression, it is convenient to first transform equation 2 by taking logarithms to yield, r Eq. 3.

Regression analysis yields values for Ln( k) and p . The governing equation is obtained by taking inverse logs. Unfortunately, the transformation processes create yet another bias that must be accounted for.

17 In the database, an average value of the individual standard deviations can be calculated from the multi train data for a selected method. The individual values of average concentration from the database can be entered into the regression equation ( Eq. 3) to predict values of S at each value of For an unbiased model, the average value of predicted S equals the average value of S With transformation of data into the loplog plane, that value of Ln( S) { Predicted] will equal the av source of this bias is that Ln( S) f L n ( 3 This bias can be accounted for by appropriately adjusting the value of k in the model. Further explanation of this bias and the procedure for bias correct The above procedure provides a simple means of using available data to estimate the standard deviation as a function of average concentration for any given measurement method. At any selected concentration, the regression equation can be determine an estimated value of c.

Information on Est. CJ and average concentration c the anticipated range for 99 out of 100 future ind as well as the anticipated range for the average o note, however, that even after correcting for the various biases, the regression line is not a perfect indicator of true value of Q. This evaluation is a best estimate, based on the currently available data.

Addition of new data will undoubtedly cause an adjustment to the regression equa relation between Est. Q and C. stack concentration)

3.4 Confidence Intervals A critical question that must be examined is `& HOW good is the correlation?" Based on analysis of the available data, it is possible to estimate the potential for the regression line through the S versus C data. These potential bounds are referred t calculating confidence intervals are presented in the Ap he Appendix also includes several example calculations. Figure 4, taken directly from the Appendix, illustrates a hypothetical set of small sample, bias corrected S versus C data. The heavy solid line through the data represents the regression line while the arched lines above and below the regression line illustrate the upper and  P * *

4, ..

cn  C 0 m > . e . 8 O I I 0 0 0 F 0 0 F 0 0 0 0

0 a> CI,

Q) 2 k 0 0 0 0 P 0 ? rn Y a" 0 Q, Y P

I 19 lower confidence intervals. The regression line represents the best estimate of the relationship between G and average concentration. The analysis used to determine the regression line also provides information on the potential bound example, the analysis provides a best estim analysis ais0 provides information to de maximum slope of the reIation @). In Fi relation where p is at the minimum and confidence level, we know that the slope Line A and less than iliustrated by on the potential range of the leading c illustrated in Figure 4 represent the comb Confidence intervals on a linear relation will always ha the potential range for the slope term. The true be determined, but it is possible to provide t a given confidenc r the slope of th ression equation @). The , at a given confidence level, the m , the lines labeled A and B illustrate m values respectively. Specifically, at the 95%

( e. g., 95 YO), the regression interval.

The primary task for the Phase I Re deviation at given values of concentr of method precision. If 95% confidence interval 97.5% confidence that the method's Similarly, there is 97.5% confidenc bound. However, neither of the precision. The currently available method precision.

Plots such as that provided in Figure 4, frequently cause difficulty for readers. Generally, a significant portion of the individual data p? ints ( circles on Figure 4) fall outside the confidence limits. This is an expected trend since the confidence intervals represent upper and lower bounds for the regression line  not upper and lower bounds for the data. A word of caution is in order. When the statistical analysis does not require weighting, it is relatively simple to calculate confidence intervals using software routines contained in standard,

commercial spreadsheet computer programs. Recall that weighting of the data is required when the individual data points have different degrees of freedom. For example, standard software can easily be used to calculate confidence intervals in situations when all of the data consist of paired train measurements. When data weighting is required, calculation of confidence intervals becomes much more complex, requiring inversion of rather messy matrices. Advanced statistical analysis computer software generally includes routines for such analyses. Alternately, special computer software will need to be written. For the current report, detailed expianation of confidence interval calculation has been limited to those situations where calculations can be performed using software routines in standard spreadsheet computer programs such as Excel.

One additional subtle issue related to calculation of confidence intervals must be addressed before proceeding with the ReMAP analysis. The appropriate calculation process depends upon bow the confidence intervals are to be used. For the ReMAP study, the intended use of the various analyses is to determine Est. Q at discreet values of average concentration and to use Est. Q to calculate various precision metrics at those average concentrations. Method precision metrics are also calculated assuming that the true value of Q is at the upper and lower confidence intervals ( at selected concentrations). There are alternate ways of using confidence intervals that require slightly different analysis procedures.

Reh4AP statistical methodology and were deemed inappropriate for the current analysis purposes. Those procedures were carefully considered in establishing the 3.5 Summary of ReMAP Analysis Process In summary, the ReMAP analysis procedure begins with a database of available multi train data from application of an EPA measurement method. These data are screened for outiiers using procedures that purposefully try to include as much data as possible. Data should be discarded only if there is an identified problem with a measurement or if a data pair ( or a single measurement from a triplicate or quad test) is demonstrably dissimilar from the remainder of the data in the data set.

For each test run, the 2, 3, or 4 simultaneous measurements are entered into equation 1 to determine 21 the standard deviation for the run. Each of these standard deviation estimates is then multiplied by the appropriate correction factor to account for small sample bias. The array of bias corrected standard deviation data and average concentration data are weighted for the number of degrees of freedom, transformed to the Log Log plane, and subjected t linear regression analysis. This analysis determines values of k and p in the power fun used to determine a predicted value of standard d average value of S from the test data is compared to the average value of S from determine an appropriate value for the second bi correction factor. That factor is multiplied by the k parameter to provide an unbiased equation relating our best estimate of standard deviation to concentration. Next, the 95% confidence intervals are calculated over the range of available data.

mote, the confidence intervals are actually calculated in the Log Log plane. The second bias correction factor is applied to the interval when it is transformed back to the Est. a C plane.]

Data are presented in four ways. First, the data are presented in tabular and gaphicat form showing a scatter plot of calculated standard deviation and relative standard deviation concentration. The second form of data presentation is a graph of the three precision metrics elorted against concentration. Each of these metrics is normalized by the concentration and is based on the best estimate of the method standard deviation. Third, data are presented to illustrate the fact that the true value of method standard deviation co d be greater than or less than the best estimate.

There are six curves of interest. The first two curves, representing a worst case scenario, focus on the situation that would occur ; f the true method standard deviation ( Q) were best represented by the upper bound of the 95% confidence intervaI. Using that upper limit of Est. o, the upper and lower bounds of measured concentration are calculated that encompasses 99% of future measurements.

These curves, plotted against stack concentration, are denoted by the symbols C99u/ S95+ and C991/ S95+. Next, the data bands encompassing 99% of future measurements are calculated,

assuming that true standard deviation varies according to the regression equation. These two lines are given the notation C99u/ Sbest and C991lSbest. Finally, consideration is given to the case where the regression analysis has provided an over estimate of standard deviation. These curves are similar to the first two but are based on the lower 95% confidence interval. These two lines, given the symbols C99ulS95 and C991/ S95, can be considered best case scenarios. There is 97.5%

confidence that the method's precision is worse than these last two lines. i kr_ r For each of the plots describing measurement method precision, care has been taken to limit the range of the presentation to the range of the currently available data. There has been no extrapolation beyond the range for which experimental data was available.

The final data presentation is a table quantifying the anticipated range of fbture measurements at selected values of average stack concentration. These tables list C99dSbest and C99IISbest over a range of concentrations imposed or under consideration for current environmental regulations. As regards these tables as well as all other methods of describing method precision, it is important to reiterate that various parameters are not corrected to a constant excess air level.

The sections that follow examine each of the EPA Measurement Methods of interest. The first method discussed is Method 5 for determining particulate matter concentration. The precision assessment for this method is presented in great detail in hope that the reader can better understand the fuIl scope of the assessment.

23 This Page IntentionaIIy Left Blank. 4.0 EPA Particulate Matter Methods  Methods 5 and 5i Sampling hardware used for the majority of the EPA manual isokinetic measurement methods is based upon the hardware used for measuring particulate matter ( PM) concentration in stacks. The procedure for measuring stack particulate concentration has been designated EPA Method 5 and the associated hardware is referred to as a Method 5 train. ( EPA, 1987) Additional details on Method 5 .

( and other Methods discussed in this report) can be found in 40CFR Part 60  Appendix A under the heading for the Method. Figure 5 illustrates the Method5 hardware. Describing the key features of a Method 5 train serves as a convenient basis for fbrther discussion of other measurement methods addressed in this study.

In general terms, application of Method 5 involves inserting a probe into a stack and extracting a composite sample that is representative of average conditions across the stack. In a typical sampling run, sample gases are extracted from the stack for a period of approximately one hour. The stack cross section is divided into equal area segments. The probe is traversed across the stack, e stack flow from each segment for equal time periods. With a round stack, traversing typical two ports, located perpendicular to each other. The rate of sample extraction is adjusted t the velocity of gases entering the probe tip is essentially equal to the local velocity of flue gas in the stack. This is referred to as isokinetic sampling. This feature of manual method sampling is included to minimize the potential for sample bias associated with preferential capture of solid phase material according to particle size.

The extracted sample is passed through a heated line to a heated filter assembly that captures solid phase particles. The mass of particulate captured during the entire sampling period is determined gravimetrically. Additional features of the method include procedures for determining the volume of flue gas extracted from the stack during the sampling period. Using standardized protocols, the particulate concentration is determined as the ratio of the mass of particulate collected divided by the volume of flue gas extracted.

25 26 ri L P L Figure 5 illustrates the hardware components and their arrangement for Method 5 sampling. AS indicated, sample gas is extracted through a nozzle and transported in a heated glass probe to a heated filter assembly. The probe assembly consists of a glass nozzle, a heated glass probe liner, a s

type pitot probe and a thermocouple ( TK). The T/ C and pitot allow determination of the local stack gas temperature and velocity, which provides a basis for adjusting the sample extraction rate to isokinetic conditions. Particulate matter in the sample may be deposited on the nozzle and probe liner walls but the majority of the particulate matter ( typically > go%) is collected on a heated filter.

Heating of both the probe and filter assembly is required to prevent condensation of water and other condensable materials in this portion of the sampIing train ( often referred to as the front half of the train). Located downstream of the heated filter box is a series of impingers in an ice bath that remove moisture from the sample. An umbilical cord connects the impingers to the meter box. The meter box contains a dry gas meter to determine the volume of dry sample extracted, means for determining pitot probe AP, read outs for key temperatures, and a vacuum pump for adjusting sample extraction rate. The sampling rate is usually held between 0.5 and 1.0 cubic feet per minute.

After completion of a test run, the field technician thoroughly rinses the train components upstream of the filter with the appropriate solvent ( generally acetone for paniculate samples) to recover any particulate that may have been deposited on the probe walls or nozzle tip. The technician must also record a number of sampling system parameters necessary to determine the volume of gas collected and the moisture content of the flue gas. Typically an Orsat analysis is performed on the flue gas to determine the major constituents of the flue gas, particularly the oxygen concentration. Back in the laboratory, the probe rinse and filter are dried and the mass of particulate collected is determined gravimetrically. To assure that the final weig gain on the filter represents dried particulate,

repeated measurements are performed. The Sam is considered dry and results are reported when subsequent weighings agree within 0.5 mg. Particulate concentration is determined as the ratio of the particulate mass colIected divided by the volume of flue gas collected. Usually, the sample volume is determined and reported on a dry basis and adjusted to standard temperature and pressure conditions. Standard temperature and pressure conditions used by the U S  PA are 20 ° C and 760 mrn Hg. For regulatory purposes dilution effects are accounted for by correcting the measured concentration to a fixed percent oxygen ( or carbon dioxide).

27 Some states require special analysis pro ures to assess the mass of material that condenses in the impinser portion of the Method 5 sampling train. Those states o require that the mass of condensed phase material be combined with the particulate catch in probe and filter to yield a total particulate phase catch. These procedures were not used for the ReMAP study. All particulate concentration data presented and anaIyzed in the following sections represent solid phase material collected in the front half of the train only. Moreover,

the performance of particulate measurement methods sh the analysis includes back half catch from Method not be applied to measurem 4.1 Method 5 Data and Precision Analysis Multi train data included in the ReMAP database come from three main reports. The first data set includes a series of EPA sponsored studies condu search Institute in the early 1970s to validate the pa date method ( Hamil a included a coal fired PO plant and two municip units were performed using fou ltaneously ( quad trains). For this study, each train was operated b At the power plant site, testi providing a total of 16 data poi For the first MWC test, six test con concentrations ranging from runs, providing 20 individual these three test series are provided i ( and all subsequent tables listing reference. Skips in run numb difficulties and suggest that t nt operating conditions thus ge from 141 to 240 mg/ dscm.

4 individual data points with 28  w *. Table 1.  PA Method 5 Data  Hamil and Cam 1974 and I974b.

RunNumber A 8 C D I 1 205 202 204 22 1 ower Plant 207 240 155 150 141 I I 1 5 60.4 61.9 63.2 64.6 Mwc 1 ,

I 1 J 14 126 123 I34 , 144 15 153 141 161 139 16 103 106 104 103 All data expressed as mg/ dSm Data are not corrected for oxygen content 29 The next set of multi train Method 5 data is provided by a second EPA sponsored study at an MWC in Dade County, Florida. The tests were directed by Southwest Research Institute ( H a d and Thomas, 1976). These tests are unique among all data collected for validation of  PA measurement methods. The stack test location provided four sampling trains were used in each port providing a total of eight simultaneous measurements for each test condition. Moreover, a total of nine different samplin ams were used in the collaborative study.

The experiments covered a 3 week period with the test pian calling for five runs per week. Seven different laboratories analyzed the four paired sampling trains at the train was operated by a single technician maintain each week ( accounting for three of the nine parti paired trains, a separate laboratory operated each t 1 The test plan called for fifteen sampling runs, five per week for th actually completed; three the first week and five each the second and summary of the measurement results. A total of 104 data points were collected. For run 10, a probe liner was broken on one of the eight trains ( Train A operated by Laboratory 103). Accordingly, that data point was eliminated from the data analysis.

The final data source was an ASME sponsored study by Rig0 and Chandler who performed extensive muiti train experiments on a municipal w Chandler, 1997). A total of I6 Method 5 data pairs are re tor in Piasfield, Mass ( Ri . The data range for these from 14 to 74 mg/ dscm, which significantly gathered at Pittsfield used essentially every nds the overall range of the full data set. Data method of interest to the ReMAP program.

Particulate concentration results from the Rig0 and Chandler t are provided in Table 3.

The first step in the ReMAP analysis is to determine if a approach is outlined in the Appendix and includes the Statistical Process Control ( SPC) methods.

consisting of three or more simultaneous measurements and is used to identify potential outliers f the data groups are outliers. The n's r test, a procedure taken from test is applied to individual tests The Dixon's c I Pi 3 al g 31 d E e, E 3 E . e, 10 2 0 e L B 0 e, t: 3 0 l 0 c 8 CJ eJ L 4 d TabIe 3. EPA Method 5 Data  Pitt All dara expressed as mddscm. Data are not corrected for Oy~ gen content 32 F? f within the test. The SPC procedure begins by breaking the data into groups representing ranges of similar concentration. The span of data is calculated for each simultaneous measurement and then weighted according to a factor that is a function of the number of simultaneous determinations ( Le.,

pairs, quads, etc.). Next, the average weighted span is calculated for each concentration group. If the span for a given r exceeds the weighted average span, then data from that run are abnormally large, relative to afa in that concentration range. In SPC terminology, the weighting factors are referred as D4. Table 4 provides a listing of D4 parameters as a function of the number of measurements in a run. ,

Table 4: SPC Factors for Identification of Data Outliers Sample Size, n The choice of concentration ran were tested. There is a strong Table 5 combines the data prese procedures outlined above. As group representing power plant, at MWC2 and at the Dade County of these data had average PM ions above 94 mddscm. The remaining data and the Pittsfield MWC had M concentrations less than 68 mgdscm. The weighted average spread for simultaneo ments in the low concentration range was 13.1 5 mgdscm.

For measurements in the hig ion range, the weighted average data spread was 77.32 mg/ dscm. Data points are suspect if the actual measurement spread is greater than these values.

Data point number 17 in the Rigo and Chandler set marginally exceeds this limit. However, this data point has the highest concentration in the " low concentration" data group. When the spread on this data point is normalized by the mean concentration, the spread is on the same order as several other data points in the low group. The ReMAP program has a bias for retaining all data unless it is somewhat arbitrary. For the ReMAP analysis, several ranges rence for minimizing the number of data points eliminated.

rlier in Tables 1,2 and 3 and assesses the data according SPC ut, the data was separated into two range groups with the 33 Table 5. Consolidated Method 5 Data Set A11 data expressed as mg/ dscm. Dam are not corrected for 0 3 an obvious and significant outlier.

retained for subsequent anaiysis. All data points in the low concentration grouping have been The high concentration range group includes quad data and the Dade County tests using octets. Data in this range are suspect if the difference is greater than 77.32 mgldscm. Only one measurement, run number 12 from the Dade County tests, fails to meet this criteria. Data from run number I in the Dade County tests is also quite large. The data report provides no indication of measurement problems associated with either of these measurements but it is obvious from inspection that, for run number i2 the two data points collected by Laboratory 103 ( labeled A and B in Table 5) are higher than the other six determinations. For run number 1, data from Run C also appears abnormally high.

Dixon's r procedure was applied to the data from b ns and results indicate that test point C from run number 1 and test point A from run 12 are abn y high and should be considered as outliers.

All other data points in this data set pass the Dixon's r criteria. After eliminating points, the remaining data in the ion data group pass the SPA criteria.

/ the two data The next step in the analysis is to calculate the average concentration and standard deviation for each group of simultaneous measure ts and to correct the calculated sample standard deviations for the small sample bias. As noted in ier discussion, the calculated value of standard deviation from the data, S, is a biased estimate of the true standard deviation, CT. Table 6 presents the correction factors used to caIculate a data points used to calculate S.

Results of these calculations are presented in Table 7. Figures 6a and 6b present scatter plots of the data from Table 7. Figure 6a shows the scatter of bias corrected standard deviation versus the average particulate concentration. Figure 6b presents the same data, but in a sIightIy different i 35 Facility RunNo N Avg. Standard Bias Esttmared~ Conccnnation Deviation Factor Sisma All data expressed as mg/ dscm. Data 36 37 a v) 0 ua N ' 0 0 F1 0 m r 0 z 0 m 0 d u u L .. . .

L. u < format. Here the standard deviation has been normalized by the concentration and presented in units of percent. i The next portion of the data analysis is to evaluate the relationship between the estimated standard deviation and the average particulate concentration. However, before performing the regression analysis, it is necessary to weight the data according to the number of degrees of freedom for each measurement group. As discussed in the previous section, statistical assessment of data containing differing degrees of freedom involves complex matrix inversion procedures. Details of the caIculation procedure are not included here.

It is instructive to review the rationale for weighting the data. With a quad train, the four individual particulate measurements can be used in a variety of ways. For example, Train A can be grouped with Trains B, C, and D to calcuiate three different standard deviations or the four measurements can be combined in the calculation of a single value of S. Determination of S based on data from Trains A and B provides the same level of information achieved from dual train testing. Similarly, S determinations using data from Trains A and C or from Trains A and D convey the same level of information as dual train measurements. Clearly, a calculated value of S using ail four simultaneous measurements conta more information than a calculation based on two measurements. When a data set contains results from dual, triple, quad, etc. measurements it is necessary to weight the various data to account fo e relative quantity of information provided in each test.

The weighted data set is then fit to a power function relationship, as presented below.

S = kCp Eq. 1 To assist in that regression, the data is transformed into the log log plane such that the governing equation becomes:

Ln( S) = Ln( k) + pLn( C) Eq. 2 By performing the transformation, the regression analysis is linearized. Results from the analysis of data in Table 7 are summarized in Table 8 below. . .

39 A regression analysis is a mathematical procedure that yields a best estimate for the curve fit parameters. One critical question islwhether the indicated values of k and p are statistically significant. One approach to answering this question is the Student list the t statistic as a function of the confidence lev g. 95% confidence) and the nulnber of degrees of freedom. The regression analysis also produces a value of the t parameter. If the calculated t parameter is greater than the criticaI are statistically significant. Conversely, if the statistic, then the regression analysis results co Table 8, the calculated value of the t paramet parameter for 42 degrees of freedom, at the assures that there is a relationship between S regression analysis, did not occur by chance. random chance. As shown in The large relative value o f t Transformation of data from the real plane to the log log plane greatly eases the regression analysis but it introduces a potentialIy significant bias to the results. One characteristic of a linear regression analysis is that the average of the predicted values for the dependent var le should equal the average val6e from the actual data. Since the regression was performed in the log log plane, the weighted average value of Ln( S) will be the same for both the actual data However, the average value of the predicted values of S will not necessarily be equal to the average of the small sample bias corrected S values. For the current da sample bias corrected S values is 364.93 while'the sum of the ( at the observed values of concentration) is 351.013. Thus, the predicted values of S are biased low 40  , and a correction factor must be applied. For the Method 5 data, the log transformation correction factor is 364.93/ 35 1 . O 1 3 = 1.0397.

The equation describing the estimated values of standard deviation versus concentration is the best estimate available, based on available multi train experimental data, but there is uncertainty associated with this equation. The slope of the regression line @) and the value of the leading constant ( k) may be greater or smaller than predicted. Statisrical data in Table 8 can be used to quantify uncertainty in the regression equation. Specifically, the 95% confidence intervals on the regression equation will be caiculated.

The 95% confidence interval on the slope term can be expressed as I p95% = Ppredicted 5 t95%*[ SE( coeff) l Eq. 3 where Pg5% represents the upper and lower bounds of the slope coefficient, t95% is the critical t, I statistic ax the 95% confidence level and the appropriate number of desrees of freedom, and1 SE( coeff) is the standard error of the coefficient.

I , As indicated in Table 8, the predicted value of the power term in the regression equation @) is 1.3063 and the standard error of that coefficient is 0.1477. The critical t statistic for 42 degrees of freedom, at the 95% confidence level is 2.020 ( available from standard statistical tables). Thus, the best estimate for the slope of the regression line is the predicted value ( 1.3063) but with 95%

confidence it can only be concluded that the value of the p coefficient is between 1.008 and 1.605:

' Implications of the potential range of this slope term are discussed in more detail later. r.

The weighted least squares numerical analysis provided information necessary to determine confidence intervals on the regression equation. Results of those calculations ( at the 95%

confidence level) are presented in Figure 7. When plotted on log log scale, the regression equation is a straight line and the confidence intervals appear as horn shaped curves on either side of the prediction. AI1 three lines in this figure have been adjusted to include the log log transformation 41 0 0 s2 E e E rq i bias correction factor. Superimposed on Figure 7 is the small sample bias corrected data from Table 7.

The meaning of confidence intervals often confuses those with limited background in statistical analysis. The straight line through the data represents the best estimate of the relationship between standard deviation G and mean concentration, p. The confidence intervals define potential bounds for the regressi the straight line. Confidence intervals do not represent boundaries for the actual data. It is le to calculate potential bounds for data, but those bounds are referred to as tolerance intervals. Thus, it is fully anticipated that a portion of the experimental data ( individual determinations of S) will fall outside the confidence intervals.

Before proceeding with additional assessment of the Method 5 results, it is necessary to examine the implications of the regression analysis. The regression equation itself was found to be '

s = 0.021 Eq. 4 If both sides of the equation are divided by the mean concentration, the left hand term becomes S/ C,

which is the r tandard deviation ( RSD). After performing this operation, the regression equation becomes RSD = 0.02 1 lC0.306 ~ 100%. Eq. 5 This implies that the RSD increases with increasing concentration, which is a difficult result to rationalize. Typical random errors that might be attributed to the sample collection process, such as failure to adequateIy rinse particulate matter from the probe liner, should produce errors that are roughly proportionaI to the PM loading. Another, often observed error in sample collection is for a small portion of the filer to stick to the filter housing. This type of error causes an underestimation of the mass of particulate collected but the magnitude of the error will not be a function of concentration. Random error in the weighing process should also be relatively independent of PM concentration. It can even be argued that the relative magnitude of analytical error might decrease with increasing concentration. These considerations suggest that the value of the slope term ( for any 43 Method) should be expected to fall between ze slope of the regression line in Figure 7 and in d 1.0. More significa ons 4 and 5 is too high.

The forzoing statistical analysis is obvio characteristics of those data. Two facto large value for the Standard Error of th 1.008 to 1.605. This lower limit on the anticipated bounds on the regression equ by the confidence interval of the statistic A separate argument has been forwarded, suggesting that it may not be valid to group the various Method 5 data sets into a single analysis. The mathematical procedures of regression analysis predict a high value for the slo excessively high S or if the data at lower concentrations have uncharacteristically low values of S.

In the outlier analysis presented earlier, the available data were divided into two concentration groups. The high concentration group contained quad train and octet data with all tests reporting data at high concentra average PM concentration above 94 rngldscm. These data were collected in the early to mid 1970s.

The low concentration data included six quad t aired train runs from the tests at Pittsfield. The Pittsfield data was collected in the mid 1990s. The regression analysis is heavily weighted by the Dade County octet data that is also high concentration data. If the Dade County data exhibited uncharacteristically high standard deviation, the slope term from the regression analysis would be uncharacteristically high. It has been suggested that the high concentration data were collected shortly after Method 5 was first developed and that the field testing crews were still learning how to properly apply the Method was collected more than a decade later, allowing the test contrast, the Pittsfieid data sampling procedures.

A second consideration involves the stack sampling ti exception of the Rig0 and Chandler data However, for the tests at Pittsfield, stack gases we difference in sampfing time could possibly result i particulate concentration. the data include in Tabfe 5. With the es were nominally r approximately 4 hours. This es for data coIlected at lower 44 There is certainly merit to an argument that the skill level of sampling teams directly impact the standard deviation of measurement results. However, there is no direct information available to quantify the capability of testing teams or to provide relative weighting of data quality. Similarly, it is reasonable to speculate that sampling time might impact measurement precision but there is nothing within Method 5 that precludes extended sample collection times. For these reason, the analysis of the Method 5 data will continue based on the entirety of the available data but with a strong caution that slope of the true S versus C relation is probably very close to 1.0.

Not withstanding the forgoing comments, the regression line in Figure 7 provides the best estimate available for the standard deviation of data collected using Method 5. As discussed in Section 3, this estimate of standard deviation also defines the anticipated distribution of future measurements collected with that method. For example, based on currently available data, it is anticipated that repeated Method 5 measurements of a stack gas containing 100 mg/ dscm of particulate matter would exhibit a standard deviation of 8.639 mg/ dscm or 8.64 % RSD. This is a hypothetical source, where the stack PM concentration is not varying with time. Sixty eight percent of future measurements taken on this stack should 1 within 1.0 cr ( 28.639 mgldscm) and ninety nine percent of those measurements should fall w n the range of 2 2.57* 0. Thus, in the example of a stack with a PM loading of 100 mgldscm, 99 ut of 100 Method 5 measurements are expected to fall within & 22.20 mg/ dscm of the true conce ration. For the average of triplicate measurements, 99 out of 100 measurements would fall within the range o f f 2.57* 0/.\/ 3. Thus, the average of triplicate Method 5 measurements from this hypbthetical stack is ex2ected to fall within 5 12.82 mg/ dscm af the true I I concentration. ,

Figure 8 presents the predicted relative standard deviation and the 99% bounds for future single measurements as a function of stack PM concentration. Data in this figure are based on currently available data and do not include the effect of time variation in source characteristics. The X axis of Figure 8 represents the true concentration of PM in the hypothetical stack. Values indicated on the Y axis represent the precision of Method 5 at the selected values of sack concentration. The 99%

bounds are also normalized by the stack concentration and represent the anticipated range of individual measurements. Compliance with regulatory limits is typically based on the average of.

45 m . I c M I . I ; 0 0 N 0 46 c triplicate measurements. The predicted range for 99 out of 100 triplicate measurements is also included in Figure 8. When Method 5 is applied to a real stack, a wider range of experimental results can be anticipated due to time variations in source characteristics. Data presented in Figure 8 should not be extrapolated beyond the indicated limits. Further, based on physical considerations, it is anticipated that the true variation of these precision metrics with concentration are expected to be a flat line or even to decrease slightly ( whereas the curves increase) with increasing concentration.

Data in Figure 8 were generated using the predicted values of standard deviation, Recall, however,

that there is uncertainty in those estimates. It is know with 95% confidence that the relationship between standard deviation and concentration falls between the upper and lower confidence limits illustrated in Figure 7. If the actual relationship between a versus C for Method 5 conforms to the upper Confidence limits, the anticipated range of future Method 5 data will be greater than suggested by the data in Figure 8. Conversely, a tighter ranse of concentrations are anticipated if the variation in standard deviation conforms to the lower confidence limit. Figure 9 illustrates the ranges of anticipated concentrati data under three scenarios: ( 1) when standard deviation conforms to theL upper confidence limit ) when Est. CT conforms to the predicted reIationship; and ( 3) when Est. CT conforms to the lower confidence limit. The X axis in Figure 9 represents the true concentration of PM in a hypothetical stack that does not vary with time. The Y axis represents the anticipated range of measured concentra using Method 5. The upper and lower curves in the figure represent the upper and lower bo 99 out of 100 future measurements, assuming that the standard deviation1 equals the upper 95% confidence limit. There is 97.5% confidence3 that 99 out of 100 future measurements would fall below the upper curve and 97.5% confidence that the future measurements will fall above the lower curve. Similar curves are provided for the cases where Est. Q conforms to the regression curve fit and where Est. cs is equal to the lower confidence limit.

Based on the above anaIysis, and concerns over the slope of the regression equation, it is difficult to draw firm conclusion about the actual precision of Method 5. However, certain trends do appear obvious. Within the confidence bounds of the analysis and based on the available data, it appears that Method 5 standard deviation varies approximately linearly with concentration and that the 395% confidence implies that there is a 2.5% chance that the a relationship falls above the upper confidence limit and a 2.5% chance that the relationship falls below the Iower confidence limit.

47 m m n  U M L . I F1 i l C)

P t 0 P

, I h,

relative standard deviation for the method is approximately constant. For PM concentrations  3 \ between 15 and 217 mg/ dscm, the best estimate of the relative standard deviation for Method 5 is 4' between about 4.8% to 12.2%.

(

49 4.2 Method 5i Data In the middle 1990s the US.  PA began develop ate measurement method,

specifically desizned to improve measurement precision at low loadings. The method itself was published in 1999 as part of the new MACT regulation verning hazardous waste incinerators ( 62 FR 52828, Sept. 30, 1999) and has been giv designation Method 5i. The hardware configuration for Method 5i is illustrated in Figure 10. Methods 5 and 5i are similar in many respects, but there are two important hardware differences and several operational differences. The primary hardware differences are in the filter assembly for the two methods. Method 5 uses a large diameter filter that must be carefully removed from its holder as part of the sample recovery process.

Often a small quantity of the collected particulate can be lost or a small portion of the filter itself can adhere to the holder walls. This results in measurement imprecision that can potentially become critical when the total particulate catch is small. Method 5i uses a much smaller diameter filter and filter holder. The recovery and analysis procedures call for the filter to remain in its holder through the entire weishing process. This eliminates certain sources of random error but it creates another potential problem. Since the weight of the glass filter holder is much larger than the weight of the collected particulate, the analysis process must determine a small weight gain in a relatively large mass. Because of the small filter diameter, Method 5i is intended for use only under situations where the particulate concentration is expected to be below 50 mg/ dscm. The second key feature implemented with Method 5i is the reauirement that tests be conducted using dual trains. Moreover,

measurement precision requirements are defined as part of the method. ~

I Data on the precision of Method 5i comes from two studies directed primarily at evaluation of particulate matter continuous emission monitors. The first of these studies, conducted under EPA sponsorship, was executed on a hazardous waste incinerator owned by Dupont and located in WiImington, Delaware ( 62 FR 67788). The second study, sponsored by an industry consortium, was conducted at a hazardous waste incinerator owned by the Eli Lilly Company ( Eli Lilly, 1999).

Results from these two studies provide a large database for assessment of Method 5i precision. Note that there are numerous experimental programs that were recently completed ( or still underway)

using Method 5i for calibration of PM continuous emission monitoring systems. Since Method 5i 50 c I 51 requires use of dual trains, the available database ssessment of this method's precision is expected to greatly expand over time.

As noted above, Method 5i is a relatively new meas issues associated with execution of the method that si Both the  PA test report and the Eli Liliy associated with obtaining acceptab were performed but have not been used by either g program, the only data used are Tables 9a and 9b provide a sum actual data pairs, the average concentration pair, and the small sample bias corrected st there are a variety of subtle ct the precision significant learning curve er of paired train tests ysis. For the ReMAP the original study authors.

ncluded in the tables are the ted standard deviation for each Figures 1 la and 1 1 b present deviation data versus averag corrected standard deviation. Fig 11 b presents relative standard d almost 50 mg/ dscm. Also note the general char concentration range, the individual estimates of Stan discernable trend to either ethod 5i standard small sample bias

gjdscm while Figure om less than 10 to Over the entire distributed with no I ~

1 The data in Tables 9a and 9b we concentration range grouping were examined atte particulate concentrations.

runs 53,64,66 and 71 fro sts. Examination of these data points suggests that runs 64, 66, and 71 from the Eli Lilly tests are only marginally ng criteria. Several I r b f

b:

above the SPC screening criteria.

anaiysis. The spreads for data point 53 ( Eli Lilly) and 60 ( EPA D they have been deleted from the following analysis. Accordingly, those data poin re retained for subsequent were sufficiently large th

il I ' L' Table 9a. Method 5i Data and Standard Deviation  Eli LiUy Data 53 4 TabIe 9a ( Continued). Method 5i Data and Stan 54 \ RIM Number Avg Conccnmtion Standard S  Bias RSD * Bias mddscm Deliation RSD Concctcd Conected Train A Train B P 1' 9 0 =

8 m 9 0 v 8 m 8 N 8

8 1 k,

c L a ++

+

++

111 I B a 11 I t P

i L 57 i k all Method 5i data were obtained using dual trains, no weighting of the data is required. Results from the regression analysis are presented in Table I O below.

Table 10. Results of Regression Analysis for Method 5i Data Residuals T I 1.32 The regression analysis indicates that the estimated standard deviation varies with according to the relationship concentration P = 0.243 5 0.366 or from  0.123 to 0.6094 4 The predicted value ofp ( 0.243) is taken directly from the regression analysis. The 2 0.3659 term is the product of the t statistic for 1 14 degrees of freedom ( 1.980) and the s error forp ( 0.1 848). This finding is consistent with the earlier observation ( see Figure 1 la) that the standard deviation data appears to be broadiy distributed in a box covering the full range of concentration and standard deviation between zero and about 6 mgfdscm.

It is mathematically possible to construct a range of method precision metric for Method 5i as a function of concentration but the indicated relations would have little statistical significance. In such situations, the analysis approach is to calculate a pooled standard deviation for the data. The analysis procedure is described in the Appendix but essentially involves calculation of the weighted average of the variance for the available data. Variance is equal to the square of the standard deviation and the weighting factor for each data point is the number of degrees of freedom for the data point. Since all available Method 5i data are from paired train tests ( DF= I) all weighting factors are 1.0 and thus the pooled standard deviation is simply the square root of the sum of the squares for the Si values. As expIained in the Appendix, the appropriate values of Si to calculate the pooled standard deviation are taken directly from the raw data without adjustment for small sample bias.

The above described pooling procedures were applied to the data in Table 9a and b. There are 114 individual data pairs and the sum of the individual variances is 234.13. Accordingly the pooled variance is 2.054 ( 234.13/ 114) and the pooled standard deviation is 1.433 mg/ dscm. This single value is the best estimate available for the Estimated o for Method 5.

Table 1 in the Appendices provides a list of factors to calculate the confidence intervaI ( at the 95%

confidence level) for a pooIed value of Est. cr. Using linear interpolation, the values of P0.025 and P0.975 are 0.883 and 1.155 respectively. Accordingly, the upper and Iower 95% confidence intervals on Estimated c are 1.265 and 1.655 mg/ dscm respectively. The confidence intervals are afso constant values  not a function of concentration. Figure 12 presents a scatter plot of the Method Si data along with the estimated values of standard deviation and the confidence intervals.

Since Est. cr and the confidence intervals are constants, they are illustrated as straight lines in Figure 12.

59 . . I ' 1  0 8 I ! The estimate of standard deviation provides information on the anticipated range of future measurements using Method 5i. Figure 13 presents three precision. metrics for Method 5i. Included are the relative standard deviation, the expected bounds for 99 out of 100 future individual measurements as well as the anticipated bounds for the average of triplicate measurements. These precision metrics have been normalized by the concentration. Thus, even though Est. G is a constant, normalized precision metrics are strong functions of concentration.

The general presentation approach adopted for the current report is to present a figure defining the anticipated range of concentrations for 99 out of 100 future measurements, under three different scenarios for an assumed variation of standard deviation. Such a figure provides little information under conditions where the standard deviation is evaluated to be essentially a constant. It is much cleaner to simply state the anticipated variation in future measurements.

If the true standard deviation is essentiaily equal to the pooled standard deviation, then 99 out of 100 future Method Si measurements are anticipated to faif with 2 3.68 mg/ dscm of the true concentration. This assumes that there is no bias in the measurements and that the m e concentration is between about 4 and 50 mg/ dscm.

If the m e standard deviation for Method 5i is essentially equal to the lower 95%

confidence interval, then 99 out of 100 future measurements are anticipated to fall with k 3.25 mg/ dscrn of the true concentration.

If the true standard deviation for Method 5i is essentially equal to the upper 95%

confidence interval, then 99 out of 100 future measurements are anticipated to fall with t 4.25 mg/ dscm of the true concentration.

4.3 Discussion of Particulate Matter Measurement Results The forgoing discussion provides strikingly different conclusions relative to the precision of Methods 5 and 5i. Specifically, the regression analysis indicates that the standard deviation of Method 5 is a strong function of concentration. In fact, it is suggested that a reasonable interpretation of the data is that Method 5 has a constant relative standard deviation. In contrast,

61 c, u E Q) L 3 M tz 0 0 v)

0 w 0 c1 0 62 ! LA I ` Ld analysis of Method 5i data could detect no relationship between standard deviation and concentration at the 95% confidence level.

Prior discussion has suggested that random error associated with the sample collection process tends to drive the power function term @) in the regression equation toward 1.0. Similarly, random error in the analytical process tends to drive p toward zero. These are only anticipated trends but they do suggest that different types of random error have driven the assessment of these two particulate measurement methods.

w '

When Method 5 is applied at high particulate concentration, it is reasonable to anticipate that the fiIter weighing process is sufficiently precise that it contributes negligibly to the overall precision of the method. A large portion of the multi train Method 5 data was collected from high concentration stacks (> 90 mgidscm). Further, the high concentration data were collected during a time frame when many sampling teams were gaining experience with application of the method. Thus, results from the statistical analysis of Method 5 data can ea y be rationalized. 1 : / B_ u h I The Method 5i data were all collected under low particulate concentration conditions ( GO mg/ dscm with the majority of the data at much lower concentration). Under these conditions, it is anticipated Ld I that imprecision of the weighing process may . contribute significantly to the overall Method's precision. As described earlier, collected samples must be dried before recording the final particulate weight gain. Collected samples are placed in a dessicator and repeatedly weighed untii the tare weight is stabiIized. The sample is considered to have reached its final weight if the repeated weighing5 agree within k0.5 mg or 21.0% of the tare weight, whichever is greater. For a typical Method 5 or 5i particulate measurement, a sample is collected from the stack for approximately one hour during which time approximately 1 cubic meter of flue gas is extracted.

Thus the process of determining particulate loading on the filter is no more precise than the concentration measurement is no more precise than 20.5 mg/ dscm. That c measurement imprecision is insignificant when the stack concentration is on the order of 100 mg/ dscm. However, for measurements in stacks with PM concentrations on the order of 10 mg/ dscm, this represents a significant relative contribution to overall measurement precision.

5 Samples must remain in a dessicator for a minimum of 6 hours between weighings. i.

1 L L P , L . L A r Based on the above considerations it is reasonable to anticipate that the analytical portion of Method 5i measurements will contribute significantly to the overall precision of the method and that random error in the weighing p also subject to random method precision should vary with concentratio available Method 5i data have been collected measurement teams expended significant effo data from numerous tests were discarded as a result of personnel climbing the learning curve.

Finally, all data in the Method 5i data set above about 15%. All of these factors c the sampling process has a small contribution to s will not vary significantly with particulate concentration. Method 5i is associated with The above considerations are provided as one possible rationalization for the observed differences in the precision characteristics of the two methods. the characteristics of the measurement methods and the c used to assess the methods.

As regards Method 5, it appears like mpling teams collecting data in the low and high concentration ran ed to assess the method. od 5 standard deviation on concentration is less than indicated by the current data set. Further it is likely that As regards Method ji, recall that regression analy CT is a function of particulate concentrati can be confirmed at the 95% confiden performed. Consid for Methods 5 and 5, it is likely that stan with increasing stack particulate conc ncludes that no relationship It is important to place the above considerations into practical perspective. When Method Si is used to measure PM loading in low particulat data suggests that 99 O u t of 100 future true concentration. The original data set ( before outlier screening) contained 1 16 pairs of dual train  f L n t results. Assuming that the true concentration for each test is the average of the test pair, dividing the spread in each pair by 2 provides a crude way to assess this prediction. For the entire data set ( Tables 10a and 10b) the maximum spread between any data pair occurred in run number 53 from the Eli Lilly data set. The reported concentrations for that data pair were 30.3 and 37.7 with an average concentration of 34.0 mg/ dscm. Thus, for I I6 paired measurements ( 232 applications of the Method) the maximum difference between the measured concentration and the estimated true concentration was 3.7 mg/ dscm. Interestingly, that measurement was determined to be an outlier and was eliminated from the overall analysis.

It is certainly possible ( even likely) that the precision of Method 5i has some dependence on concentration. However, the forgoing analysis suggests that the constant si, oma assumption provides a reasonable and practical estimate of the Method's precision.

It is unlikely that the imprecision of Method 5 increases as rapidly with concentration as suggested by the data in Figures 7,8 and 9. It is expected that increased experience of field sampling teams,

including the lessons learned from application of Method 5, have already reduced the range of random errors impacting Method 5 results. It is anticipated that the inherent precision of Method 5 should be similar to that for Method 5i but CJ for the method almost certainly does increase with concentration. Note that the Method 5 data from the Pittsfield tests were collected in the mid 1990s and should reflect increased experience for the testing team. Data from this test series ( 16 valid data pairs ) were examined to determine the range of the data pairs relative to the pair average. For runs 15 and 17, the spread minus the mean were 4.4 and 6.8 mg/ dscm respectively while the spreads for the remainder of the data were less than 3.5 mg/ dscm. This suggests that Method 5 might provide ' slightly less precise results than Method 5i but not dramatically less. Even this observation must be tempered by the fact that the standard deviation for the method increases with increasing concentration. Based on these qualitative considerations, it is suggested that the data in Figures 7,8,

and 9 should be taken a5 an upper limit on the imprecision of Method 5. To fully assess Method 5 precision at higher concentrations (> 1 OOmg/ dscm) additional multi train data is required in that concentration range.

65 This page Intentionally Left Blank r t 5.0 EPA Method 23 For Measuring Dioxin and Furan Emissions The EPA method for measurement of dioxin and furan stack emissions is denoted as Method 23 ( 56 FR 67788 and 40 CFR Part 60  Appendix A). The hardware for the method is illustrated in Figure 14 and has several similarities to the hardware discussed previously for Method 5. The major difference is addition of a module filled with an absorbent material known as XAD. A small circulating pump maintains the temperature of the XAD module at approximately 60" F. The XAD module is spiked with known quantities of labeled compounds that are used for both system calibration and to experimentally determine the recovery of the overall sampling and analysis .

process.

The quantity of dioxin and furan collected and analyzed by the method is extremely small. Typical stack concentrations are on the order of a few ng/ dscm. To collect sufficient material for analysis by high resolution GC/ MS, each sampling run extends for at least' three hours and may last for more than 6 hours. There are eight possible homologues of both polychIorinated dibenzo( p) dioxin and dibenzofuran. Only those homolo, oues with four to eight chlorine atoms are adverse health effects. Acco ngly, some environmental regulations @ e., the rules ipal waste combustors) limit the release of all tetra through octa chlorinated dioxins nt for the fact that different dioxin and furan congeners and ikrans. Other regulations take ac have vastly different toxici Most of the world has adopted a listing of relative conge that was developed under the auspices of NATO. These relative toxicity factors are m the concentration of each congener to yield an emission concentration that is equiv toxicity that would occur if all of the indicated mass was found as the most toxi 2,3,7,8 tetrachloro dibenzo@) dioxin). Emissions expressed in this manner are International Toxic Equivalent, or ITEQ. Regardless of whether an emission stan as total mass of tetra through: octa or as ITEQ, the sampling and analysis proce Method 23 is the same. The only difference is that the ITEQ process provides significant weighting factors to a select group of individual congeners. As will be shown in material that follows,

application of these weighting factors does impact the indicated precision of the measurement method.

67 5.1 Available Mufti Train Data for Method 23 as Total PCDDPCDF Extensive effort has been expended in development of sampling and analytical methodology for determination of dioxin and furan emission concentrations. Separate procedures have evolved in Europe, Canada and the US which are similar in many respects. There are however, small differences in the analytical procedures that may have significant impact on the precision of measurement results. Accordingly, the current study focuses onIy on US EPA Method 23 since that is the method which must be used to determine compliance with US emission standards. Before presenting the available multi train data for Method 23, it is important to note that the procedures used by EPA to validate Method 23 are different from those used to validate other methods.

Specifically, in lieu of gathering multi train data from one or more source categories, the Agency I developed hardware and a procedure for dynamically spiking a sampling train with known quantities of isotopically la eled dioxin and furan congeners. Validation of the method focused on experiments determining the fractional recovery of the dynamically spiked compounds.

There are a variety of approaches that may be used to validate performance of a method.

Dynamically spiking a sampling train with a known quantity of a tracer compound is a potentially valid approach. In fact many of the EPA method validation efforts have used dynamic spiking in quad train tests to gain information on both precision and bias of an emerging method. However, use of dynamic spiking experiments as the sole approach for method validation is valid only under conditions where there is no possibility for formation of the pollutant of interest in the sampling train itself ( e. g., for methods measuring the total emission of an element such as a heavy metal). As regards dioxin and furan measurement, there is a significant potential for formation of these compounds under thermal conditions that may occur within a sampling probe or within the hot filter box of the Method. The potential for formation of the target analytes within the sampling train raises serious concern about the completeness of EPA studies validating Method 23.

The report describing the Method 23 validation effort indicates that a quad train was dynamically spiked with isotopicaily labeled dioxin and furan congeners ( MRI, 1991). The report indicates that the collected samples were analyzed for both the native congeners as well as the dynamically spiked congeners. In fact, the report includes tables listing the collected mass of native dioxin and furan in 69 5.2 Analysis of Method 23 Data for Total Dioxin and Furan. I r k' Table 1 1. Method 23 Data as Total Mass of Tetra through Octa Dioxi t L 71 vi rl c) L W a tk . I f u d 111 73 . The SPS data outlier procedures were applied to the data in Table 11. Several different range groupings were evaluated and each grouping identified Run number 7 from the Rigo, and Chandler tests as an outlier. The basis for this identification is easily seen in Figure 15. However, when presented as relative standard deviatio remainder of the data. The SPC outlie data appears to be abnormally large. , In decision was made to retain the data for the n number 7 is only slightly identify data points where the span of data point has been examined and a Data from Table 1 I were submitted to regression analysis to de weighting were required since all available results are from indicate that the estimated standard deviation varies as a concentration according to the equation S ( for Method 23 as total PCDD/ PCDF) = 0.2722 * Co. 56.

p = 0.56 k 0.48 orp lies between 0.08 and 1.04.

Figure 17 presents the regression results including the small sample bias corrected data, the regression line and the upper and lower 95% confiden precision metrics for Method 23. Data between Est. Q and C. If that relation is ng/ dscm ( excluding temporal variation), 99 out of should fall within f 30% of the true concentration. Source variation will increase those bounds. ents using Method 23 r

w k E L J i 0 s c 0 0 75 E L, r a L:

76 It is critical to recall that there is considerable uncertainty in the value of the slope term in the regression equation. Figure 19 illustrates the anticipated bounds on 99 out of 100 individual measurements for three different scenarios on the S versus C relationship. I f the regression relation is the proper description of how standard deviation varies with concentration, the anticipated spread of future data is relatively tight. For example, in that situation, samplins a stack that actually contains 20 ng/ dscm of total dioxin plus furan should result in 99 out of 100 measurements falling in the range of 16.3 to 23.7 ng/ dscm. Conversely, if the standard deviation of the method is more closely described by the upper confidence interval, a much broader range of data can be anticipated.

In this case, sampling the etical stack containing 20 ngldscm dioxin and furan using Method 23 should yield 99 out of 1 surements falling in the range of 9.13 to 30.87 ng/ dscm. LL* *

F a & h Y L r P*

L F""

b t I Table 12 presents a tabular summary of the anticipated range of measured PCDDPCDF concentration. This table i ased on the assumption that the regression equation properly describes the variation of standard deviation with concentration for the best estimate of standard deviation.

This is the best estimate available for method precision based on the available data. It is critically t 4  1 YLU"

important to reiterate that concentration information presented in this table ( as, well as the concentration data in all the tables and figure in this report) are not corrected to constant excess air level. An exam s in order. Assume that a facility operates at 11% 0 2 in the stack and must comply with a Adjusting the emission standard from 7 0 2 shows that the facility must maintain PCDDRCDF stack concentration below 24.9 ng/ dscm. If the true stack concentration was exactly 24.9 ng/ dscm, imprecision from repeated application of Method 2 ouid produce 99 out of 100 measurement results ranging from 20.7 to 29.2 ng/ dscm. When co d back to the basis of the standard, the anticipated data range is * from 29.1 to 40.9 ng/ dscm @ 7% 02. At the 95% confidence level for this hypothetical facili single measurement below 20.7ng/ dscm is below the standard while a measurement above 29.2 ngldscm is above the standard. At the 95% confidence level, resuIts between 20.7 and 29.2 could be E PCDF emission limit of 35 ng/ dscm I@

7% 02.

I & LA P m E L i L

I either in or out of actual compliance. CY c"" The above analysis clearly points to the fact that there is an insufficient body of data available to adequately assess the precision of EPA Method 23. The limited quantity of available data suggests that the precision of Method 23 conforms to the predictions presented in Table 12. However, there is t ' Table 12.

Concentration ng/ dscm &

Anticipated Range of Measured PCDDRCDF Concentration Based on Best Estimate of Method 23 Standard Deviation Measurements Measurements Lower Limit lupper Limit Lower Limit1 Upper Limj I I 0.5 0.97 0.03 0.77 0.23 2.0 4.0 6.0 8.0 0.97 2.60 I 1.40 3.03 5.52 2.4% 4.88 3.12 7.91 4.09 7.10 4.90 10.2 5.76 9.29 6.71 I 12.5 10.0 12.0 7.46 11.5 8.53 14.8 9.19 17.1 10.9 14.0 16.0 79 13.6 10.4 15.8 12.2 19.3 12.7 17.9 I 14.1 21.5 1 14.5 18.0 20.0 22.0 23.7 16.3 25.9 18.1 20.0 16.0 22.2 17.8 24.3 19.7 28.1 19.9 26.4 24.0 21.6 30.3 26.0 21.7 28.5 23.5 32.5 23.5 28.0 30.6 25.4 significant uncertainty associated with this analysis. if the actual method precision conforms to the upper confidence interval in the analysis, a can only be resolved by g particular importance wo included in the current data set. can be anticipated. The issue of concentrations. Of ions above the levels 80 5.3 Available Multi Train Data for Method 23 as ITEQ.

Data for Method 23, with results expressed as ITEQ, are the same as those for Method 23 with results expressed as total m a s of tetra through octa dioxin plus furan. Table 13 presents the ITEQ results including the results of each run, the average concentration and the calcuiated standard deviation from the run. Also included are data following application of the srnall

sample, bias correction factor. These data are illustrated in Figures 20 and 21 as scatter plots of standard deviation and relative standard deviation versus the average concentration calculated fiom the data pair. The results are generally similar to those for the Method as total PCDDPCDE except that the scales are greatly reduced. The range of the concentration data is i The above Method 23 data were submitted to regression analysis and results indicate that S is related to C according to the equation,

S = 0.4795C0.345.

Unfortunately the t statistic on the power term is only 1.02 which is well below the critical value lies that the regression equation could have occurred by chance and that, at e level, no statistically meaningful relationship between S and C was detected.

result is easily observable in the data presented in Figure 20. Note that the appear as a scattering of points at concentrations below 0.4 ng ITEQ/ dscm onlbelow 0.04 ng ITEQ/ dscm. There are four data points with standard 054 dg ITEQ/ dscm. These data might suggest that S increases with increasing t e f UT points are discounted, one can easily envision an opposite slope to a rkgression , analysis allows the potential range of the power term to be / confidence level, thep parameter could be as large as 1.051 and as small as  0.36. The magnidde of the ' uncertainty concerning the precision of Method 23 for dioxin and furan as ITEQ is further ilhstrated in Figure 22. This figure shows the small sample bias

corrected data, the regression equation and the upper and lower confidence intervals. It is I ,

?

% I u s 0 f r4 1 00 k . I +

+

r co m w m cv 8 8 8 8 8 8 83 0 I . I d 0 0 9  0 0  0 2 8 I 85 Method.

Since the above analysis failed to determine a relationship between CT and C, the alternative approach is to reevaluate the data assuming that CT is a constant. The pooled analysis procedure is to first determine the pooled variance. The 22 individual values of S from Table 13 are squared, summed, and then divided by 22 to determine the pooled variance. The pooled variance = 0.000712. The square root of this paramet s taken to determine the pooled standard deviation; pooled S = 0.0267 with 22 degrees o provide factors for determining the 95% confidence bounds on 0. Those bounds are 0.0207 and I 0.0374.

Figure 23 presents the pooled standard deviation and the 95% confidence intervals overlaid with the experimental data. Since the estimated standard deviation and the confidence intervals are constants, they are illustrated as straight lines in the Figure. The various precision metrics are presented in Figure 24. e are normalized by the average concentration and thus show The data shown in this inverse relationship If the characteristic standard deviation of Method deviation, then measurement imprecision should cause 99 out of 100 future measurements I to deviate f? om the true concentration no more than 20.068 ng ITEQ/ dscm. If the Method's characteristic standard deviation is more appropriately approximated by the upper 95%

confidence bound, then method imprecision should cause 99 out of 10 deviate from the true concentration by no more than 59.095 ng ITEQIdscm.

It is critically important that the above estimates for Method 23 imprecision be placed in perspective. Recent EPA regulations governing hazardous waste co stion SY stems and fossil fbeI fired cement plants have set dioxin and furan emission limits of 0.2 ng ITEQ/ dscm @ 7% \

86 Y L,

0 0  0 87 cu a2 m I I . . , * I . .

I .

88 c1 I L 0, The potential range for future measurements fkom measurement imprecision is large relative to the standard. At the upper 95 % confidence limit, the possible range of a single measurement ( minus 0.095 to plus 0.095 = 0.19) is essentially equal the standard itself EIC i""" i kd i i f This Page Intentionally Left Blank 90 r i 6.0 EPA Me od 26 for Hydrochloric Acid The method for measurement of hydrochloric acid and chlorine gas is designated EPA Method 26 and is fully described in 4OCFR Part 60  Appendix A under the heading for the Method. The hardware arrangement for Method 26 is illustrated in Figure 25. Impingers in the back half of the train are filled with sulfuric acid to collect HCl while the sodium hydroxide impingers collect chlorine gas.

ata for Method 26 are available from three sources, summarized in Table 14. As indicated in the table, relatively high HCl concentration data ( about 80 to 220 mgdscm) are provided by the tests of Rig0 and Chandler at an MWC facility in Pittsfield, MA Chandler, 1997). These data were collected using a quad train. Entropy Corp collec CI concentration ( 4 to 74 mg/ dscm), as part of the EPNOAQPS effort and Margeson, 1989). Two ofthe EPA/ OAQPS tests ( run Method 26 ( Steinberg and 12) were perfo midget impingers in what is now considered the standard Method 26 procedure. The other trains used fidl size impingers typically associated with Method 5 trains. A significant bias detected in the results from run number 12 and those results are not included in the ReMAP analysis. The remainder of the EPNOAQPS test was conducted using dual trains with midget impingers. Finally, very low HC1 concentration data ( 0.3 to 2.0 mg/ dscm HCI) were collected by EER as part of an effort for EPNOSW ( EER, 1997). These EPNOSW tests were executed using quad trains. Data outlier analysis was performed on these data and alI data points passed the outlier criteria set by the SPC procedures.

Figures 26 and 27 provide scatter plots of the available HCI data illustrating the standard deviation as a fimction of average HC1 concentration. Figure 26 presents all of the available data while Figure 27 includes only data from the EPNOAQPS and EPNOSW tests ( low HCl concentration data). Figure 28 presents these data as relative standard deviation. Data presented in all three of these figures have been corrected for smali sample bias. As shown, when the 91 L P I T (""

LA.%

Table 14. Method 26 Multi Train Data and Standard Deviation for HCI Fq L E l i3 = E I a M Lrc . I Q w n t; ( v 95 0 ( 0 0 w 0 m 0 w 0 F 0 . , _ I.._..  .._ .. .. ,_, "... average HCI concentration is above about 10 mg/ dscm, relative standard deviation ( RSD) for the various runs is consistently below 10%. Below IO mg/ dscm, the RSD data tend to increase sharply. Figures 26 and 27 show that the sharp increase in RSD is caused by the rapidly decreasing value of the Concentration rather than a sharp increase in standard deviation. f After applying appropriate weighting factors, the data in Table 14 were submitted to a weighted regression analysis and results indicate that the estimated standard deviation for the Method varies with HC1 concentration according to the relation:

S ( Method 26 for HCI) = 0.15259 * C0.803.

This equation includes both the small sample bias correction and the bias correction associated with the log log transfornation. The concentration term in this equation is in units of mg/ dscm.

The t statistic for the regression is 17.22, which is well above the critical t statistic for 95%

confidence and 29 degrees of freedom ( 2.042). At the 95% confidence level, the value of the power coefficient is P = 0.803 k 0.095 or between the limits of 0.707 and 0.898.

Figure 29 presents a plot of the data, along with the regression line and the upper and lower confidence limits. Note that the confidence inte Is do not deviate sigificantIy from the regression line. Using the regression equation to describe the variation of Est. G with concentration, estimates can be developed for the probable variation in measurements associated with imprecision in the Method itself Figure 30 presents resuIts of those calculations including the variation in relative standard deviation, the estimated spread for 99 out of 100 single measurements, and the estimated spread for 99 out of 100 triplicate measurements. The anticipated spread for measurements is projected to be relatively close to the true stack concentration. For example, if Method 26 is applied to a stack containing 40 mg/ dscm HCl,

results from 99 out of 100 triplicate measurements are expected fall within 10.9 ' YO of the true concentration. At Iower concentrations, the range of future measurements is predicted to 97 * 4 0 0 0 4 0 0 0 H 98 t 4 E w v1 0 . I L Y S 0 m 0 Q) .. . I L s 0 0 * m s 99 0 VI ( u 0 0 ( u 0 VI I ul I 0 G ti 0 ~ ~~ i~~ ~ ~ ~~ ~~~ ~

increase ( as a percentage of the true concentration)

conditions, the actual spread in data & om method imprecision is predicted to be relatively small.

For example, 99 out of 100 triplicate measurements in a stack containing 1.0 mgldscm HCI are expected to fa11 within the bounds of 0.80 and 1.20 mgldscm.

The data in Figure 30 assumed that the standard deviation for Method 26 varied according to the regression equation. Figure 29 presented the 95% confidence bounds for the regression equation.

Those data have been used to estimate the anticipated range for 99 out of 100 future measurements assuming confidence interval, the Results for those calcul The best estimate of the precision of Method 26 is provided by the data were presented graphically in Fi these data in tabular form. ! The first corrected for excess oxygen). The second out of 100 future single measurements ( at the fifth columns provide the i IO1 I i I i I I I 1 i 1 E Table 15. Range of Anticipated Future HC 102 PY E l b r i L ,

I  I i? t i s ,

. . . 7.0 EPA Methods 29, lOla and lOlb for Mercury Considerable effort has been expended developing methods for determination of mercury emissions from combustion sources. For details on the Methods, refer to 40 CFR Part 60  Appendix A under the headings Method 29 and Method 101. Mercury is emitted from combustion sources as either the base metal or in the + 2 vaience state. It may be in the vapor

state or it may be associated with solid phase material. At least three EPA Methods have been published or proposed and one additional method is under active development. Figure 32 illustrates the hardware used for Method 101% which is the simplest of the mercury procedures.

The sample gas is passed through a heated filter and then goes to impingers filed with a KMn04

10? 40H, S0, ( permanganate) solution. After the sampling event the filter and permanganate solutions are digested and a single combined measurement of total mercury concentration is generated. Method 29 was developed as a multi metal measurement method, including mercury.

The hardware configuration for Method 29, illustrated in Figure 33, includes a pair of nitric acid impingers foHowed by permanganate irnpingers. The various fractions are analyzed separately for a wide range of metals and results summed to develop a reported stack concentration.

Finally, Method 10 1 b, iIlustrated in Figure 34 is a simple variant on Method 29. Method 10 1 b replaces one of the nitric acid impingers in Method 29 with two water impingers. The various fractions are analyzed separately in an attempt to determine the split between Wgo and Hg' 2. The total mercury concentration is determined by adding the catch fiom the various fiactions.