4) What modifications would you suggest be made to the recommendation section? Peer Review Handbook Page C 11 CHARGE EXAMPLE 7  Economics  Valuation of Fatal Cancer Risks Background The Science Advisory Board   Environmental Economics Advisory Committee ( SABEEAC review of the draft Guidelines for Preparing Economic Analyses ( Guidelines) helped the Agency to identify valid and sound economic procedures to use when conducting benefit cost analyses of environmental policies. Both the Guidelines and the SAB EEAC review ( EPA SAB EEAC 99 020, dated September 1999) recognize that economic theory will evolve and new empirical literature will be forthcoming that may necessitate revisiting the analytic procedures contained in the Guidelines. There may also be situations where the Guidelines do not provide sufficient detail to respond to analytic questions whose answers have broad implications for the conduct of economic analysis at the EPA. Both circumstances have recently arisen, making it necessary for the Agency to submit for SAB EEAC review a document that examines the   benefit transfer  issues that arise when using the value of statistical life ( VSL) literature that is based on accidental risks to estimate the economic benefits of environmental policies that reduce fatal cancer risks.

The Guidelines provide information and guidance on the valuation of reduced mortality risks ( Chapter 7, pp. 37 43). The Agency Guidelines conclude  and we understand the SABEEAC to have concurred in their review on this subject  that one practical and well supported means to value changes in mortality risks is to use the Value of a Statistical Life ( VSL) approach. Further, in response to the SAB EEAC review, the Guidelines describe a number of important factors to consider in applying benefit transfer approaches using VSL estimates from the empirical literature on wage risk tradeoffs. Recognizing that this is an important benefit category, the Agency Guidelines stated that the EPA would   continue to conduct annual reviews of the risk valuation literature  and   reconsider and revise the recommendations in these guidelines accordingly.  Furthermore, the EPA would   seek advice from the Science Advisory Board as guidance recommendations are revised.

The Agency needs to return to the SAB EEAC and obtain additional counsel on this subject. Some economists within the government have suggested some particular approaches to dealing with the benefit transfer issues. Since the Guidelines were drafted, a few relevant articles have been published that examine benefit transfer issues surrounding the use of VSL estimates when there is a passage of time ( or latency period) between the pollution exposure and harm, or when fatal cancer risks are involved. The importance of these issues was articulated in a recently proposed regulation to reduce human health risks from radon in drinking water. The proposed rule estimated the number of reduced fatal cancers resulting from different regulatory options. The Agency presented information on the economic values for the reductions in fatal cancer risks, along with other quantified benefits. A brief discussion of some of the benefit transfer issues involved in this estimation was published in the preamble to the proposed rule for setting standards for exposure to radon from drinking water sources ( Federal Register, Page C 12 Peer Review Handbook November 2, 1999 volume 64, Number 211, pages 59245 59378). Quoting from the Federal Register notice requesting SAB review:

Latency is one of a number of adjustments or factors that are related to an evaluation of potential benefits associated with this rule, how those benefits are calculated, and when those economic benefits occur. Other factors which may influence the estimate of economic benefits associated with avoided cancer fatalities include ( 1) a possible   cancer premium  ( i. e., the additional value or sum that people may be willing to pay to avoid the experiences of dread, pain and suffering, and diminished quality of life associated with cancer related illness and ultimate fatality); ( 2) the willingness of people to pay more over time to avoid mortality risk as their income rises; ( 3) a possible premium for accepting involuntary risks as opposed to voluntary assumed risks; ( 4) the greater risk aversion of the general population compared to the workers in the wage risk valuation studies; ( 5)   altruism  or the willingness of people to pay more to reduce risk in other sectors of the population; and ( 6) a consideration of health status and life years remaining at the time of premature mortality. Use of certain of these factors may significantly increase the present value estimate. EPA therefore believes that adjustments should be considered simultaneously. The Agency also believes that there is currently neither a clear consensus among economists about how to simultaneously analyze each of these adjustments nor is there adequate empirical data to support definitive quantitative estimates for all potentially significant adjustment factors. As a result, the primary estimates of economic benefits presented in the analysis of this rule rely on the unadjusted $ 5.8 million estimate. However, EPA solicits comment on whether and how to conduct these potential adjustments to economic benefits estimates together with any rationale or supporting data commenters wish to offer. Because of the complexity of these issues, EPA will ask the Science Advisory Board ( SAB) to conduct a review of these benefits transfer issues associated with economic valuation of adjustments in mortality risks. In its analysis of the final rule, EPA will attempt to develop and present an analysis and estimate of the latency structure and associated benefits transfer issues outlined previously consistent with the recommendations of the SAB and subject to resolution of any technical limitations of the data and models.  ( page 59326)

In the process of responding to reviews prepared during deliberations on the proposed radon rule, the Agency found that the Guidelines lack sufficient detail on how to fully evaluate and characterize the different risk attributes that are central to a complete understanding of the benefit cost implications of this rule. For example, time can pass between the point of initial exposure to a carcinogen, the biological manifestation or onset of cancer in the body, the medical diagnosis of cancer, and death caused by the cancer. During development of policies affecting cancer risks, suggestions have been made to discount the VSL estimate ( i. e., $ 5.8 million recommended in the Guidelines) to account for latencies, or the delay in time between reduced exposure and when the cancer death would have occurred absent the exposure reduction.

Others argued that a suitable approach for valuing benefits from reduced cancer risks must consider simultaneously all of the benefit transfer factors related to valuing cancer risks to ensure a careful and full treatment of benefits. There is evidence in the economics literature Peer Review Handbook Page C 13 regarding many such factors ( e. g., potential premiums ascribed to cancer risk reductions due to a higher willingness to pay to avoid the dread, pain and suffering, morbidity effects, and other features of cancer endpoints) that may suggest introducing upward adjustments factors which offset any potential downward adjustments caused by accounting for cancer latency. In addition, proponents argue that adjustments for the age of population at risk, income, altruism and other risk characteristics ( e. g., controllability, voluntariness) can all have some potential influence on the value of a statistical cancer fatality ( VSCF) and therefore need to be reflected in the quantitative benefit assessment.

While developing the primary benefit estimates for reduced fatal cancer risks in the proposed radon rule, questions arose regarding the implementation of adjustments for some factors, but not others. For example, would it ever be appropriate to adjust only for latency periods, and not other factors, in the valuation of reduced cancer deaths? To help answer this and related questions regarding the valuation of cancer risks, the Agency seeks the SAB EEAC s counsel. We further ask that your guidance reflect the typical uncertainties facing EPA economists, including those surrounding the underlying risk assessments, the prediction ( or lack thereof) of latency periods for cancers, and the risk characteristics associated with the VSL approach.

Therefore, the Agency proposes to seek review of a   white paper  and list of charge questions by the SAB EEAC on the valuation and benefit transfer practices arising in the calculation of the economic benefits of reduced fatal cancer risks. The Agency seeks SABEEAC review of the treatment and presentation of quantitative and qualitative information for these types of benefits. Numeric case studies are included in the white paper, to both identify and present prospective approaches to address these issues. The results of the SAB EEAC review of this document, and responses to the specific charge questions, will be considered by the Agency during future revisions to the Guidelines, consistent with the Agency s commitment to credible and consistent economic analysis in support of the policy making process.

Charge Questions:

As the Committee considers the charge questions, it is asked to keep in mind the differing situations relating to differing degrees of data availability or uncertainty in key parameters.

1) Does the white paper accurately describe the empirical economic literature relevant to the benefit transfer issues that ensue when using the VSL literature to estimate the VSCF in a benefit cost analysis?

2) Does the white paper present the important risk and demographic factors that can affect benefit transfer approaches that use VSL estimates for VSCF? Page C 14 Peer Review Handbook 3) Does the white paper accurately describe attempts in the economic literature to measure VSCF directly?

4) There are two numeric case studies of environmental cancer risks developed for the white paper. Each presents risk assessment information that forms the basis for quantifying the number of statistical cancer fatalities that will be reduced as a consequence of a hypothetical proposed environmental policy. The case studies are then used to illustrate the outcome of using direct measures of the VSCF and benefit transfer adjustments to VSL estimates in order to calculate the VSCF.

a) Which of the valuation approaches applied to the case study designated as ALPHA are valid to use? Does this case study omit any credible alternative protocols for valuing reductions in fatal cancer risks for benefit cost analyses of environmental programs?

b) Which of the valuation approaches applied to the case study designated as OMEGA are valid to use? Does this case study omit any credible alternative protocols for valuing reductions in fatal cancer risks for benefit cost analyses of environmental programs?

5) Which economic methods illustrated with the case studies, or additional methods identified by the Committee under charge questions 4. a and 4. b, serve as credible protocols for the Agency to use in representing quantitative data, qualitative information, and sensitivity analyses for the economic value of reduced fatal cancer risks reported in benefit cost analyses? Peer Review Handbook Page C 15 CHARGE EXAMPLE 8  Hudson River PCBs Site Reassessment RI/ FS Modeling Approach Charge: Peer Review 1 Members of this peer review will be tasked to determine whether the models being used to support the decision making process for the Reassessment, and the assumptions therein, are appropriate. The peer reviewers will base their assessment on the review the Preliminary Model Calibration Report ( PMCR), an updated Technical Scope of Work for the Baseline Modeling Report ( Appendix B of the PMCR) and the responses to selected comments received from stakeholders during the public comment period on the PMCR.

In October 1996, EPA released the Preliminary Model Calibration Report ( PMCR), which described the models, datasets and assumptions being used as part of the Hudson River PCB Reassessment RI/ FS. The PMCR represents the status of the preliminary PCB modeling effort as of Fall 1995. Datasets, database corrections and other pertinent information which became available after October 1995 were not incorporated within the fate and transport modeling presented in the PMCR. The PMCR was an interim document prepared to describe work in progress and was not intended to be a conclusive report. In particular the HUDTOX model presented in the PMCR was not intended to be used as a predictive tool to assess remedial action scenarios. In addition, while time varying mechanistic models of bioaccumulation will be used along with other models to predict fish body burdens, these models are not described in the PMCR.

The PMCR was not formally peer reviewed at the time of publication, but was distributed to interested parties who were invited to submit comments and questions. Written responses were made to all of these comments and questions. In addition, the work plan contained in Appendix B of the PMCR has been revised to reflect the ongoing work being conducted as part of the Baseline Modeling effort. Results from this effort will be presented in a Baseline Modeling Report that will be formally peer reviewed.

The peer reviewers are requested to determine whether the models being used to support the decision making process for the Reassessment RI/ FS, and the assumptions therein, are appropriate. The peer reviewers are not being asked whether they would conduct the work in the same manner, only whether the work being conducted will yield scientifically credible conclusions.

It is suggested that the reviewer first read the PMCR. The Responses to Comments provides information on the context of the PMCR within the overall modeling effort and additional details beyond the PMCR results. The current work plan as revised in June 1998 reflects the ongoing Baseline Modeling effort and revisions to some of the original modeling tasks proposed in Appendix B of the PMCR. In addition, the USEPA/ TAMS Phase 2 database has been considerably revised. New datasets have been added and some earlier datasets have been extensively revised. Page C 16 Peer Review Handbook The peer reviewers are asked to comment on the following:

1) Is EPA using appropriate models, datasets and assumptions on which to base a scientifically credible decision?

2) Will the models, with the associated datasets and assumptions, be able to answer the following principal study questions as stated in the PMCR?

a) When will PCB levels in the fish population recover to levels meeting human health and ecological risk criteria under No Action?

b) Can remedies other than No Action significantly shorten the time required to achieve acceptable risk levels?

c) Are there contaminated sediments now buried and effectively sequestered from the food chain which are likely to become   reactivated  following a major flood, resulting in an increase in contamination of the fish population?

3) Specific questions:

a) Are the modeling approaches suitable for developing quantitative relationships between external forcing functions ( e. g., hydraulic flows, solids and PCB loads, sediment initial conditions, etc.) and PCB concentrations in the water column, sediments and fish? Are the models adequate for discriminating between water related and sediment related sources of PCBs?

b) Are the spatial and temporal scales of the modeling approaches adequate to answer the principal study questions? If not, what levels of spatial and temporal resolution are required to answer these questions? What supporting data are required for calibration/ validation of these spatial and temporal scales?

c) It is contemplated that PCB concentrations in fish will be estimated using several modeling approaches: an empirical probabilistic model derived from Hudson River data, a steady state model that takes into account mechanisms of bioaccumulation body burdens, and a time varying mechanistic model ( not included in the PMCR). A bi variate statistical model may also be used to provide insight into accumulations. This multi model approach is being contemplated because of the uncertainties associated with any individual model. Is this a reasonable approach or should predictions be made using a single   best  model? Peer Review Handbook Page C 17 d) Is the level of process resolution in the models adequate to answer the principal study questions? If not, what processes and what levels of resolution are required to answer these questions? What supporting data ( such as data to support specifications of a mixed depth layer, solids and scour dynamics, groundwater inflow, etc.) are required for these processes and levels of resolution?

e) The results of the modeling effort will be used, in part, to support human and ecological risk assessments. In your judgment, will the models provide estimates adequate for this purpose?

4) Are there any changes to the work effort outlined in the revised work plan that would significantly improve the outcome?

5) In terms of evaluating the overall and specific effects and behavior of PCBs in the Hudson River, are there any serious flaws in the modeling approach ( theory, structure, physical parameters, etc.) that would limit or invalidate any conclusions or further work based upon the results of these models?

Recommendations Based on your reading and analysis of the information provided, please identify and submit an explanation of your overall recommendation for the modeling effort for the Hudson River PCB Reassessment RI/ FS:

1) Acceptable as is 2) Acceptable with minor revision ( as indicated) 3) Acceptable with major revision ( as outlined) 4) Not acceptable ( under any circumstance) Page C 18 Peer Review Handbook CHARGE EXAMPLE 9  IRIS Pilot Program  Instructions to Peer Reviewers for Reviewing IRIS Summaries and Supporting Documentation The U. S. EPA is conducting a peer review of the scientific basis supporting the health hazard and dose response assessments for the subject chemical that will appear on the Agency s online database, the Integrated Risk Information System ( IRIS). Materials to be reviewed include the summary information that will appear on IRIS ( the inhalation reference concentration [ RfC], oral reference dose [ RfD], and cancer assessment) and the supporting document, the Toxicological Review, which will also be made available to the public.

A listing of Agency Guidelines and Methodologies that were used in the development of these hazard and dose response assessments included the following: The Risk Assessment Guidelines ( 1986), the ( new) Proposed Guidelines for Carcinogen Risk Assessment ( 1996), Guidelines for Developmental Toxicity Risk Assessment, ( proposed) Interim Policy for Particle Size and Limit Concentration Issues in Inhalation Toxicity, ( proposed) Guidelines for Neurotoxicity Risk Assessment, Methods for Derivation of Inhalation Reference Concentrations and Application of Inhalation Dosimetry, Recommendations for and Documentation of Biological Values for Use in Risk Assessment and Use of the Benchmark Dose Approach in Health Risk Assessment. Copies of these documents ( and/ or their relevant Sections) will be made to the reviewer upon request.

Peer review is meant to ensure that science is used credibly and appropriately in derivation of these dose response assessments. You have been chosen as an expert on the chemical under consideration, on a scientific discipline related to at least one of the assessments, or in the field of risk assessment. At least three peer reviewers per chemical are being chosen to review the scientific basis of these draft dose response assessments before they are forwarded on to the EPA's Consensus Process for final approval and adoption by the EPA. These hazard and dose response assessments will then appear on IRIS and become available as Agency consensus health effect information.

The primary function of the peer reviewer should be to judge whether the choice, use, and interpretation of data employed in the derivation of the assessments is appropriate and scientifically sound. This review is not of the recommended Agency risk assessment guidelines or methodologies used to derive cancer or RfD/ C assessments as these have been reviewed by external scientific peers, the public, and EPA Science Advisory Boards. The reviewer s comments on the application of these guidelines/ methodologies within the individual assessments is, however, welcomed and encouraged. For example, the reviewer may ascertain whether or not there is data sufficient to support use of other than default assumptions for areas such as sensitive subpopulations or linear cancer extrapolation. The reviewer may also have opinions on other areas of uncertainty such as subchronic to chronic duration ( when only a subchronic study is available) or an incomplete data base but should focus on the specific area of uncertainty rather than on the magnitude of the overall estimate. Peer Review Handbook Page C 19 Below are two groups of questions regarding this review. The first is a set of general questions that are meant to guide you through your review. It is not imperative that you specifically answer each question of this group. The second group of questions, however, are specific for the chemical assessments and deal with areas of scientific controversy or uncertainty in which the Agency may have to make a scientific judgment. Your input to this set of questions is considered vital to the review process.

Questions for IRIS Peer Reviewers  General 1) Are you aware of any other data/ studies that are relevant ( i. e., useful for the hazard identification or dose response assessment) for the assessment of the adverse health effects, both cancer and noncancer, of this chemical? Please explain fully.

2) For the RfD and RfC, has the most appropriate critical effect been chosen ( i. e., that adverse effect appearing first in a dose response continuum)? For the cancer assessment, are the tumors observed biologically significant? relevant to human health? Points relevant to this determination include whether or not the choice follows from the dose response assessment, whether the effect is considered adverse, and if the effect ( including tumors observed in the cancer assessment) and the species in which it is observed is a valid model for humans.

3) Have the noncancer and cancer assessments been based on the most appropriate studies? These studies should present the critical effect/ cancer ( tumors or appropriate precursor) in the clearest dose response relationship. If not, what other study ( or studies) should be chosen and why?

4) Studies included in the RfD and RfC under the heading " Supporting/ Additional studies" are meant to lend scientific justification for the designation of critical effect by including any relevant pathogenesis in humans, any applicable mechanistic information, any evidence corroborative of the critical effect, or to establish the comprehensiveness of the data base with respect to various endpoints ( such as reproductive/ developmental toxicity studies). Should other studies be included under the " Supporting/ Additional" category? Should some studies be removed?

5) For the noncancer assessments, are there other data that should be considered in developing the uncertainty factors or the modifying factor? Do you consider that the data support use of different ( default) values than those proposed?

6) Do the Confidence statements and weight of evidence statements present a clear rationale and accurately reflect the utility of the studies chosen, the relevancy of the effects ( cancer and noncancer) to humans, and the comprehensiveness of the Page C 20 Peer Review Handbook data base? Do these statements make sufficiently apparent all the underlying assumptions and limitations of these assessments? If not, what needs to be added?

Questions for IRIS Peer Reviewers  Chemical Specific [ example: cumene]

1) Based on the information noted in the Principal study currently designated ( Cushman et al., 1995) is the discounting of the renal effects in males justified? Is sufficient rationale given to let stand the organ weight changes in female rats as a critical effect?

2) Is the information in the Toxicological Review sufficient to consider cumene as having a low potential for causing reproductive effects? Please explain fully.

RECOMMENDATIONS Based on your reading and analysis of the information provided, please identify your overall recommendation for the IRIS materials you have reviewed as 1) Acceptable as is 2) Acceptable with minor revision ( as indicated) 3) Acceptable with major revision ( as outlined) 4) Not acceptable Peer Review Handbook Page C 21 CHARGE EXAMPLE 10  Charge to Reviewers for the WTI Draft Final Risk Assessment The draft final WTI risk assessment is divided into several volumes covering the scientific disciplines of toxicology, environmental fate and transport, combustion engineering, atmospheric modeling, exposure assessment, ecological risk assessment, and accident analysis. As a reviewer of the WTI draft final risk assessment, you should use your best technical knowledge and professional judgment to comment on the technical accuracy, completeness and scientific soundness of the assessment. Each reviewer is asked to focus on several specific issues in his or her area of expertise with comments on other areas invited but optional. Your comments will be considered in finalizing the risk assessment.

For the peer review workshop reviewers will be organized into 5 work groups: Combustion Engineering, Air Dispersion and Deposition Modeling and Accident Analysis, Toxicology, Exposure Assessment, and Ecological Risk Assessment. All reviewers should be familiar with the Executive Summary ( Volume I) and the Facility Background ( Volume II) Sections of the draft risk assessment. In addition, each work group should focus on specific Volumes as specified below:

Workgroup Risk Assessment Volumes Combustion Engineering Volume III  Facility Emissions Air Dispersion and Volume IV  Atmospheric Volume VII  Accident Deposition Modeling and Dispersion and Deposition Analysis Accident Analysis Modeling Toxicology Volume V  Human Health Risk Assessment Volume VII  Accident Analysis Exposure Assessment Volume V  Human Health Risk Assessment Volume VII  Accident Analysis Ecological Risk Assessment Volume VI  Screening Ecological Risk Assessment While reviewing these Sections of the document, please address the following general issues. Page C 22 Peer Review Handbook 1) Comment on the organization of the risk assessment document. Does the layout follow a logical format? Is the presentation of information in the document clear, concise and easy to follow?

2) Does the executive summary accurately reflect the data and methodologies used and the conclusions derived in the risk assessment?

3) Were the major recommendations of the 1993 peer review workshop for the risk assessment plan addressed?

4) As with any risk assessment, there are always additional data and method development efforts that could be undertaken to reduce the level of uncertainty. However, are there any major data or methodological gaps that would preclude the use of this risk assessment for decision making? If so, how should they be addressed?

5) What long term research would you recommend that could improve risk assessments of this type in the future?

In addition, the following workgroup specific issues should be addressed.

Emissions Characterization Emissions characterization includes identification of substances of concern and the development of emission rates for these contaminants. Emission rates were developed through a combination of site specific stack test data and models. Please comment on the following issues with respect to this aspect of the draft risk assessment.

1) To characterize the nature of the emissions, waste stream profiles were developed and entered into a database. Several refinements and adjustments ( e. g., the Subtraction Correction Factor for chlorinated compounds) were applied to the profiles before substances of concern were identified. Please comment on whether or not these adjustments are appropriate. What is the anticipated effect on the risk assessment?

2) Comment on the selected chemicals of concern. Have important chemicals been missed due to the selection technique?

3) Comment on the approaches used to estimate stack emission rates ( e. g., use of the 95% UCL of the arithmetic mean or the maximum detected value, whichever is smaller , for high end emission rates). Are the approaches appropriate? Are their effects on the risk assessment adequately characterized? Comment on the Peer Review Handbook Page C 23 adjustment made to PCDD/ PCDF emission rates to account for brominated dioxin like compounds. Also, comment on the approach to characterizing emission rates from fugitive sources ( e. g., use of the TANKS 2 model for the Carbon Adsorption Bed).

4) Comment on the identified sources of fugitive emissions. Was the approach used to select these sources appropriate? Have important sources been missed? Have emissions from process upsets been given appropriate consideration?

5) There have been a number of controlled burns at the WTI facility. Please comment on the adequacy of these data in estimating potential exposure. Please comment on the assumptions made from the tests in regard to composition of wastes received at WTI and emissions when the plant operates in the future.

6) Comment on the use of emission factors from coal burning to estimate the emission rate of fly ash from WTI. Are the factors used to adjust the coal emission rate appropriate? Are the uncertainties introduced from this approach adequately characterized?

7) Overall, is the identification of the key assumptions used in characterizing the nature and magnitude of emissions thorough? Are the magnitude and direction of effect of these assumptions on the overall risk assessment accurately characterized? Is the uncertainty and variability inherent in this analysis adequately discussed? Does the sensitivity analysis cover the major parameters expected to have an effect on the risk assessment?

Dispersion and Deposition Modeling To develop this risk assessment, computer models have been used with site specific data on emission rates and meteorological conditions to simulate the air concentrations and deposition rates for contaminants potentially emitted from the WTI facility. The models used include the Industrial Source Complex  Complex Terrain Deposition ( ISC COMPDEP), the CALPUFF, and the INPUFF models. In your review, please address the following issues.

1) Since the 1993 peer review of the risk assessment plan, a number of efforts have been completed to reduce the uncertainty associated with the air dispersion and deposition modeling. These efforts include the collection of site specific data for emission rates and meteorological conditions. Also, a wind tunnel study was conducted to evaluate the effects of the complex terrain surrounding the WTI facility. Does the risk assessment document adequately summarize these activities? Is the link between these data collection efforts, the air dispersion models, and the risk assessment clearly established? Page C 24 Peer Review Handbook 2) The results of 12 sets of sensitivity tests indicate that geophysical variables ( e. g., terrain) are more likely to affect dispersion and deposition than emission variables ( e. g., stack temperature). Were these sensitivity analyses adequate? Comment on the conclusions reached. To further examine the effect of geophysical variables, wind tunnel testing was conducted to model the terrain induced flow effects expected near WTI. It was concluded that changes in peak concentrations attributed to these effects are relatively minor and that the ISCCOMPDEP model is sufficiently conservative. Comment on this conclusion. Have these analyses helped to characterize and/ or reduce the uncertainty in the air dispersion modeling associated with the complex terrain surrounding WTI.

3) The ISC COMPDEP model does not allow for non steady state conditions such as calm winds and strong temperature inversions. Therefore, CALPUFF was used to estimate air dispersion and deposition under these conditions. However, CALPUFF gave similar peak, 24 hour, and annual average concentrations as ISCCOMPDEP Comment on the adequacy of this analysis. Comment on the conclusions reached. Has this analysis helped to characterize and/ or reduce the uncertainty in the air dispersion modeling associated with non steady state meteorological conditions?

4) Atmospheric dispersion modeling was used to estimate air concentrations of hazardous chemicals for the accident analysis. The SLAB model was used for vapor releases from spills and the mixing of incompatible wastes. ISCCOMPDEP was used for releases associated with fires. Comment on the selection of the models and inputs. Are they appropriate selections?

5) Overall, have adequate sensitivity tests been conducted to demonstrate the magnitude of variation in concentrations and deposition estimates with model inputs? Please explain fully.

Human Health Risks Human Health Risk Assessment includes hazard identification, dose response evaluation, exposure assessment, and risk characterization. To develop the risk assessment, potentially exposed populations have been identified and the magnitude, frequency, and duration of their exposure quantified. This information was then integrated with the hazard identification and dose response evaluation for the risk characterization. For this risk assessment, both carcinogenic and non carcinogenic health effects have been evaluated. In your review, please comment on the following issues. Peer Review Handbook Page C 25 Exposure 1) EPA's Exposure Assessment Guidelines identify certain exposure descriptors that should be used to characterize exposure estimates. The Guidelines define high end exposure estimates as those representing individuals above the 90th percentile on the exposure distribution but not higher than the individual in the population who has the highest exposure. Bounding exposure estimates are those that are higher than the exposure incurred by the person in the population with the highest exposure. Central tendency exposure estimates are defined as the best representation of the center of the exposure distribution ( e. g., arithmetic mean for normal distributions). Comment on whether or not the WTI exposure assessment properly characterizes each of the exposure estimates in terms of these descriptors.

2) The factors that go into estimating a central tendency or high end exposure, once the population has been defined, include the environmental media concentration, the intake rate, and the duration and/ or frequency of exposure. Comment on whether or not the WTI exposure assessment does an adequate job of describing the logical procedure of combining these factors to develop central tendency, high end, and/ or bounding estimates of exposure for each of the exposed subpopulations.

3) An important factor in an exposure assessment is identifying all of the important exposure sources. Please comment on the adequacy of the WTI assessment in identifying the important sources and pathways of exposure.

4) Have the key assumptions for estimation of chemical concentration and for estimation of exposure been identified? Are the magnitude and direction of effect correct for the assumptions that have been identified?

5) Supposedly, conservative assumptions have been applied in this assessment to account for uncertainty. Are the conservative assumptions appropriately factored into the ultimate characterization of what descriptor best applies to each exposure estimate? Please comment on whether the uncertainties were confronted in an adequate manner. If they were not, please state what should be done differently.

Hazard Identification/ Dose Response and Risk Characterization 1) To select surrogate compounds for quantitative risk assessment, a two step process was used in which chemicals were ranked on the basis of emission rate, toxicity ( both cancer and non cancer), and bioaccumulation potential. Please comment on this selection process. Are the ranking factors appropriate? Could Page C 26 Peer Review Handbook important compounds have been omitted from the analysis based on the ranking procedure?

2) For the majority of the chemicals of concern, traditional approaches to dose response evaluation were employed ( e. g., use of a slope factor for cancer and use of a RfD/ RfC for non cancer). However for certain chemicals or groups of compounds a different methodology was used. Specifically, dioxins, furans, PAHs, lead, mercury, nickel, chromium, acid gases, and particulate matter were given special consideration. Please comment on the methodology used for these compounds. Was it appropriate? Have the uncertainties associated with the methodology been adequately characterized? Comment on the assumptions used due to a lack of chemical specific data.

3) Please comment on the selection of the overall population and the various subpopulations at risk. Were site specific data, such as the informal home gardening survey, properly utilized to identify these subpopulations?

4) It is stated in the risk assessment that average risk estimates are based on average emission rates, average air dispersion/ deposition within a subarea, and typical exposure factors. Further, maximum risks are based on average emission rates, typical exposure factors, and the maximum air concentration within a subarea. Please comment on this use of the terms average and maximum risks. Are these descriptive terms appropriate given the parameters used to derive each? Please explain fully.

5) Comment on whether or not the non cancer risks of chemicals of concern have been adequately addressed by the risk assessment? For example, has an adequate discussion of endocrine disrupters been provided which either characterizes their risks or clearly explains why their risks cannot be characterized? Further, have non cancer chronic toxicities of dioxins and furans been adequately addressed in the risk assessment?

6) Please comment on whether or not the uncertainties associated with the additivity and/ or synergy of risks from pollutants emitted together from the WTI facility are adequately discussed in the risk assessment.

7) Have the key assumptions for estimation of dose and risk been identified? Are the magnitude and direction of effect correct for the assumptions that have been identified? Please comment on whether the uncertainties were confronted in an adequate manner. If they were not, please state what should be done differently. Peer Review Handbook Page C 27 8) Please comment on the overall adequacy of the risk characterization. Does the risk characterization include a statement of confidence in the risk assessment including a discussion of the major uncertainties. Are the hazard identification, dose response assessment, and exposure assessment clearly presented? Have sufficient risk descriptors which include important subgroups been presented and discussed?

Screening Ecological Risk Assessment As with the human health risk assessment, the ecological risk assessment pulls together elements of exposure analysis and dose response evaluations to develop a risk characterization. For the Screening level Ecological Risk Assessment ( SERA), Ecological Chemicals of Concern ( ECOC) and indicator species have been identified to provide conservative estimates of risk. Please address the following issues in your review.

1) Are there any components of the SERA which you feel undermine the scientific validity of the assessment? If so, what are they and can you provide suggestions to strengthen the identified components?

2) Is the organization of the document clear and does it present the material in a clear and concise manner consistent with the Framework for Ecological Risk Assessment ( EPA, 1992)? Please explain fully.

3) Uncertainties are discussed in numerous sections of the SERA and compose Section VIII of the SERA. In each case, do these discussions cover all relevant and important aspects of the uncertainties which you think should be addressed in the SERA?

4) In your opinion, what is the weakest and what is the strongest aspect of the SERA? Can you make any suggestions on how the weakest parts can be strengthened by the Agency?

5) In Section II, are the stressors, ecological effects, and both the assessment and measurement endpoints adequately characterized? Are the five emission scenarios adequate to characterize the exposures for the WTI facility? Are there other emission scenarios which you think should be included in the SERA?

6) In Section III, is the site characterization adequate to support the SERA? Why or why not?

7) In Section IV, is the tiered process used to identify the ecological chemicals of concern ( ECOC) from the initial list of potential chemicals considered Page C 28 Peer Review Handbook scientifically defensible? Does application of this tiered approach support the statement made in the SERA " by focusing on the potential risk from the selected ECOCs, the SERA provides a thorough screening level evaluation for the WTI facility?"

8) In Sections V and VI, are the exposure and ecological effects adequately characterized? Are the most appropriate estimation techniques available used? Are the assumptions clearly stated? Please explain fully.

9) In Section VIII, are there any major elements missing from the risk characterization which you think need to be included or which would strengthen the risk characterization? Does the risk characterization support the summary and conclusions presented in Section IX?

10) In Section IX, given the assumptions made and the processes used to select and evaluate chemicals, receptors, and exposure pathways, do you think the SERA adequately met its objective of not inadvertently underestimating risk?

Accident Analysis The Accident Analysis for the WTI incinerator involves evaluating the probability of an emergency incident occurring which results in the release of hazardous waste. The consequences of this release are also evaluated using exposure and human health effects information. Unlike the human health risk assessment which has a primary goal of quantifying risks, the accident analysis typically provides information that can be used to reduce the likelihood, extent and impact of possible accidents. Please comment on the following issues in your review of this aspect of the risk assessment.

1) The WTI accident assessment selected five scenarios for quantitative evaluation that were considered to be of primary concern. The scenarios are an on site spill, an on site fire, an on site mixing of incompatible waste, an off site spill, and an off site spill and fire. Please comment on the selection of these scenarios. Were any significant scenarios missed?

2) Specific chemicals were selected to evaluate each scenario. Please comment on the selections. Would other chemicals have been more appropriate?

3) Chemical specific release rates are calculated for each scenario. Please comment on the procedures used to estimate the release rates. Was an appropriate approach used? Peer Review Handbook Page C 29 4) Atmospheric dispersion modeling was used to estimate air concentrations of hazardous chemicals. Specifically, the SLAB model was used for vapor releases from spills and the mixing of incompatible wastes. ISC COMPDEP was used for releases associated with fires. Comment on the selection of the models and in puts. Are they appropriate selections? Should other models or inputs been used?

5) Please comment on the assessment's conclusions on the severity of consequences and probability of occurrence. Has the report correctly categorized the severity of the consequences of the different accident scenarios? Has the assessment adequately justified the reported probability of occurrence of each of the accident events?

6) Key assumptions were made in the identification of accident scenarios and the description of the conservative and typical events. Included were a description of the magnitude of the effect of the assumptions and direction of the effect. Please comment on the assumptions. Are they justified? Are the descriptions of the magnitude and directions of the effects correct? Has the accident assessment adequately confronted the uncertainties involved in doing this type of analysis? If not, what else should be done?

7) Comment on the appropriateness of using IDLH values for characterizing the severity of consequences in the accident analysis. Comment on the appropriateness of using 10 X LOC for chemicals for which IDLH values have not been established.

8) In the accident analysis, IDLH ( or 10 X LOC) values were used to determine the downwind distances over which adverse human health effects might occur. To evaluate the uncertainty introduced by using the IDLH, a sensitivity analysis was conducted where these distances were recalculated using the LOC ( a more stringent health criteria). Other sources of uncertainty that are identified in the accident analysis include concentration averaging times, chemical concentrations, emission rates, and meteorological conditions. For most of these parameters it is stated that conservative assumptions were used to avoid underestimating risks. Have the uncertainties inherent in the accident analysis been adequately characterized? For those parameters where sensitivity analyses were not conducted, is the conclusion that conservative assumptions have avoided underestimation valid? Peer Review Handbook Page D 1 APPENDIX D  GUIDANCE ON REQUESTING A REVIEW BY THE US EPA SCIENCE ADVISORY BOARD ( SAB) Page D 2 Peer Review Handbook Guidance on Requesting a Review By the US EPA Science Advisory Board ( SAB)

Summary The US EPA Science Advisory Board ( SAB or the   Board ) annually solicits proposals for review projects every spring for the following fiscal year. This appendix provides guidance to Programs and Regions to help them submit requests for SAB reviews. Requests should be submitted to the SAB in both hard copy and electronic versions usually by mid June for the following fiscal year. The requests may be part of the annual submissions that respond to the peer review activities of the Agency, or they may be submitted directly to the Board. Although providing requests at one defined time in the spring helps with SAB planning, we recognize that projects also come to light during other parts of the year. Please contact the SAB staff for details on making submissions during the remainder of the year ( see end of this document for contacts).

Background A key priority for the Administrator is to base Agency actions on sound scientific data, analyses, and interpretations. The Administrator issued the Agency's Peer Review Policy to increase the quality of the technical foundations upon which EPA's regulatory structures are built. The SAB is a key scientific peer review mechanism available to Programs and Regions in implementing the Peer Review Policy. However, because the Board has finite resources it cannot conduct all reviews. This document is designed to help Programs and Regions determine which projects to submit to the SAB. Note particularly that the SAB focuses on the technical underpinnings of Agency positions; i. e., risk assessment issues, in contrast to risk management issues.

The topics that are best suited for the Board's agenda are those that satisfy several of the following criteria:

1) Integrate science into Agency actions in new ways. 2) Influence long term technological developments. 3) Impact overall environmental protection. 4) Address novel scientific problems or principles. 5) Address problems that transcend federal agency or other organizational boundaries. 6) Strengthen the Agency's basic capabilities. 7) Serve Congressional or other leadership interests. 8) Deal with controversial issues.

In suggesting issues for SAB involvement, Programs and Regions should note the breadth of SAB activities: Peer Review Handbook Page D 3 Historically, most of the outputs of the Board are in the form of " full" reports. They present the findings of peer reviews of Agency document( s) and contain considerable detail about the findings and recommendations of the Board. They also address the specific questions posed by the Charge to the Board. " Letter" reports fulfill the same function as reports, but are simply shorter in length. Due to the need to be more responsive with advice, the Board has recently begun to produce more short letter reports than full reports, as they can be produced and finalized in less time.

The SAB has also introduced the " Consultation  as a means of conferring in public session with the Agency on a technical matter before the Agency has begun substantive work on that issue. The goal is to leaven EPA's thinking on an issue by brainstorming a variety of approaches to the problem very early in the development process. There is no attempt or intent to express an SAB consensus or to generate an SAB report. The Board, via a brief letter simply notifies the Administrator that a Consultation has taken place.

More recently, the Board introduced a new vehicle for communicating with its clients  the " Advisory   which provides, via a formal SAB consensus report, critical input on technical issues that arise during the Agency's issue development process. The Advisory generally involves a review of a multi year Agency project. The intent is to provide some mid course assessment to see if the Agency is heading in a scientifically credible direction. In order to maintain an objective, arms length relation with the Agency and its projects, the SAB review of the final product at some point in the future will include experts who did not participate in producing the Advisory.

The Agenda Setting Process Each Assistant Administrator and Regional Administrator is normally asked to submit a list of candidate topics for SAB action/ review. A " project sheet" ( see attached example) is used to define each topic that is nominated for SAB review. The project sheet is prepared by the requesting office and contains the following information:

1) Project title/ subject ( Descriptive short title of project)

2) Requesting Organization/ Office ( Primary office requesting review, AA/ RA level)

3) Requesting Official ( Name and position of senior official requesting review, usually office or division level  this is the person who may receive a summary briefing from the Chair following the review)

4) Program Contact ( Name/ phone number/ mail code  this is the principal contact for SAB Staff to interact with during development of the SAB review) Page D 4 Peer Review Handbook 5) Background ( brief history of the project and why it is important)

6) Tentative Charge ( what the SAB is being asked to comment on  usually a set of questions)

7) Tentative Schedule and Committee ( when the review is expected to be conducted, e. g., Winter 2001; and which SAB committee is appropriate for the review  final choice as to the review committee is at the discretion of the Board)

8) Budget Estimate ( Rough estimates of Agency funding for the subject over the past 5 years ( if applicable) and for the next 2 years ( if applicable). The SAB Executive Committee has asked for this information to help it better appreciate the level of Agency involvement in and commitment to the issue)

9) Preparer ( name, phone, office of preparer of Project Sheet and the date prepared)

The proposed topics will be examined and discussed in a number of forums:

1) The individual SAB Committees  Throughout the late Spring and Summer, the SAB Committees will be examining options for the following fiscal year, including all suggestions made by the Agency.

2) The Science Policy Council Steering Committee ( SPC SC)  The SPC SC usually meets in early summer to examine the proposals for each fiscal year. The goal is to provide cross office critique/ integration of the proposals. The SPC SC will be used as a forum for continuing discussion throughout the process and throughout the year as new topics emerge.

3) The Deputy's perspective  In the summer, the Deputy Administrator will review the requests and provide insights on priorities.

4) The SAB Executive Committee  During its summer meeting, the SAB s EC will examine the nominated topics, adding its own perspective on an appropriate agenda, using its selection criteria.

5) The Administrator  In September, the list of proposed topics will be delivered to the Administrator for information and added insights.

The completed project sheets should be submitted electronically to the SAB Deputy Staff Director ( fowle. jack@ epa. gov) and in signed hard copy ( mail code 1400A). Peer Review Handbook Page D 5 For general information, or to contact a specific SAB Staff member, please call the SAB main phone line: ( 202) 564 4533. Please ask to be connected to the Staff person handling the major review area you wish to inquire about ( e. g., human health and exposure, drinking water/ water quality, ecology, engineering, modeling, air quality, economics, research, and radiation). Page D 6 Peer Review Handbook US EPA Science Advisory Board Proposed Project Project title/ subject: Proposed Amendments to the Risk Assessment Guidelines for Carcinogens Requesting Organization/ Office: Office of Research and Development ( ORD)

Requesting Official: Name, Title, Office/ Organization Program Contact: Name, Title, Office/ Organization, 202 260 xxxx Background: EPA's Health Risk Assessment Guidelines provide generic science and science policy guidance on risk assessment issues for use in all Agency offices. EPA has currently issued or proposed nine guidelines ( or amendments) in this series, all of which have been submitted to the Science Advisory Board for review.

The current guidelines for carcinogen risk assessment were reviewed by the SAB and issued as final guidance in 1986. In 1988, the Forum initiated a public process for considering amendments to these guidelines. A Risk Assessment Technical Panel considered submissions from the public as well as information developed by experts at two public workshops in revising these guidelines.

Tentative Charge: Review the amended and expanded guidance, with special emphasis on ( a) weight of evidence issues, ( b) a new classification system, ( c) dose response modeling, and ( d) the use of pharmacokinetic and metabolic data. A more detailed charge will be negotiated with SAB at a later date.

Tentative Schedule and Committee: Winter, 2001, Environmental Health Committee Budget: FY 1997  $ xxx and yy FTE FY 1998  $ xxx and yy FTE FY 1999  $ xxx and yy FTE FY 2000  $ xxx and yy FTE FY 2001  estimated costs of $ xxx and yy FTE each year Preparer: Name, Title, Office/ Organization, 202 260 xxxx Date: June 1, 2000 Peer Review Handbook Page E 1 APPENDIX E  EXAMPLE STATEMENTS OF WORK FOR CONTRACTS Users of this Peer Review Handbook need to be aware that the examples contained in this Appendix are generalized statements of work prepared ( and in some cases modified) to emphasize certain important features ( e. g., attention to conflicts of interest, responsibilities of contractors or contracting officers, development and use of the charge to peer reviewers). Please be sure that proper and currently approved contract language is used in any EPA contract document at the time of award ( for example, see EPAAR 1552.212 71, alternate I). Page E 2 Peer Review Handbook STATEMENT OF WORK  EXAMPLE 1  Statement of Work: Technical Review Contractor for Panel Review of Assistance Agreement or Fellowship Applications 1) Purpose The purpose of this contract is to purchase peer review services of a contractor with expertise in Exploratory Research; Environmental Chemistry. The services are for peer reviewing applications received in response to the Office of Research and Development's ( ORD's) 1997 Science to Achieve Results program. These reviews shall be completed and the evaluation sheets shall be prepared prior to the reviewer's participation in a 3 day panel discussion to be held in Washington, D. C. on May 5  7, 1997 .

2) Statement of Work ORD's National Center for Environmental Research and Quality Assurance ( NCERQA) is responsible for overseeing the recently expanded research grants and fellowships programs. Each year NCERQA ( alone or in conjunction with other organizations) solicits applications in each of these programs. The applications to be reviewed under this contract were submitted in response to the solicitation for the 1997 Science to Achieve Results program. As part of the selection process, NCERQA must conduct a peer review that is designed to evaluate the scientific quality of each application; this is accomplished through the ad hoc use of technical experts.

The peer review services required by this contract necessitate the independent review of a maximum of 10 applications and the preparation of a typed evaluation summary and an overall rating for each of these applications. Each evaluation summary shall support and be consistent with the overall rating that is assigned; it also shall be completed prior to the contractor s participation in the panel discussions. After the panel discussions for the applications assigned to the contractor, the contractor shall submit all completed evaluation summaries to the designated Science Review Administrator ( SRA).

The contractor also shall serve as the panel's rapporteur for approximately 6  8 of the applications assigned. As rapporteur, the contractor shall be responsible for preparing a typed evaluation summary ( on site typing support will be provided by NCERQA) that reflects the panel's discussion of the respective application as well as the panel's overall rating ( the criteria for the panel's overall rating are the same as those for each peer reviewer's overall rating). As rapporteur, the contractor shall submit these panel evaluation summaries to the designated SRA prior to leaving the panel meeting. Peer Review Handbook Page E 3 Before the contractor shall be allowed to participate in the review process, the contractor shall have disclosed any actual or potential conflicts of interest and shall have signed and submitted to NCERQA a Conflict of Interest/ Confidentiality Form. The contractor is directed to assure that none of the conflicts disclosed are so direct and substantial as to rule out a particular reviewer. Upon receipt of an approved Purchase Order ( PO), NCERQA will send the following items to the contractor:

a) A copy of the Purchase Order or the Purchase Order number b) The applications assigned to the contractor c) For grants, a set of abstracts for all the applications being reviewed by the panel d) For persons reviewing grant applications, a copy of the pertinent section( s) of the solicitation package to provide background information; for persons reviewing fellowship applications, information on how to access the solicitation package on the Internet e) A sample evaluation form to help the contractor prepare an acceptable evaluation form for each assigned application f) A blank evaluation form for each assigned application and the criteria for completing the form and determining the overall rating g) A blank and sample invoice as well as instructions for completing and submitting the invoice to EPA h) Information on the points of contact for additional information ( e. g., NCERQA's SRA) i) Logistics information on the location and time of the panel discussions NCERQA will transmit the above items under a cover letter. In this cover letter, NCERQA will provide additional details about each item, including ( as needed) more specific instructions for the set of applications assigned to the reviewer.

Each contractor shall be responsible for making his/ her own travel reservations for hotel and transportation.

3) Reviewer Tasks a) Review the assigned applications using the guidance provided with NCERQA's evaluation form.

b) Submit completed evaluation forms to the SRA designated in the cover letter immediately following the panel discussions for the applications assigned to the contractor. THE COMPLETED FORMS MUST BE TYPED, AND THE EVALUATION SUMMARY FOR EACH APPLICATION MUST SUPPORT AND BE CONSISTENT WITH THE OVERALL RATING THAT IS ASSIGNED BY THE CONTRACTOR. IN SITUATIONS Page E 4 Peer Review Handbook WHERE THESE CONDITIONS ARE NOT MET, THE SRA WILL ASK THE CONTRACTOR TO REDO THE FORM.

c) For those applications for which the contractor is serving as the panel's rapporteur, submit a completed panel evaluation summary to the SRA designated in the cover letter prior to leaving the panel meeting. THE PANEL'S EVALUATION SUMMARY MUST BE TYPED ( ON SITE TYPING SUPPORT WILL BE PROVIDED BY NCERQA) AND BE CONSISTENT WITH THE PANEL'S OVERALL RATING. IN SITUATIONS WHERE THESE CONDITIONS ARE NOT MET, THE SRA WILL ASK THE CONTRACTOR TO REDO THE FORM.

d) Make own airline and hotel accommodations for participation in the panel review meeting. Round trip air fare must be a commercial REFUNDABLE ticket. Peer Review Handbook Page E 5 STATEMENT OF WORK  EXAMPLE 2  Peer Review of Prioritization Tool Report Work Assignment No.:

Title: Peer Review of Prioritization Tool Report Work Assignment Manager ( WAM):

Name: John Q. Government Employee Address: Office of Solid Waste Phone No.: ( 202) 260 XXXX Background:

The Waste Minimization Branch ( WMB) in the Office of Solid Waste ( OSW) is in the process of implementing the Waste Minimization National Plan, announced by the Agency on November 18, 1994. The Plan reaffirms the Agency's commitment to promote source reduction over waste management, in keeping with the policy stated in the 1984 amendments to the Resource, Conservation, and Recovery Act ( RCRA) and in the 1990 Pollution Prevention Act ( PPA). The Plan outlines major goals, objectives, and action items to achieve national reductions in the generation of hazardous wastes.

One of the objectives of the Plan is to: " develop a framework for setting national priorities; develop and distribute a flexible screening tool for identifying priorities at individual facilities; [ and] identify constituents of concern." This objective is a key building block in implementing subsequent objectives of the Plan.

In September 1995, WMB formed the Waste Minimization Prioritization Team, which includes representatives from EPA regions and states, to implement this objective. The Team has worked to assess stakeholder needs for prioritization tools and to evaluate prioritization tools that are currently available. The Team plans to summarize this work, along with its recommendations, in a report ( referred to herein as the Prioritization Tool report) that would be available in draft form in July 1996.

WMB and the Team wish to obtain independent peer review of the Prioritization Tool report prior to briefing EPA management. The report is being prepared with the support of ICF, Inc.; therefore, for the peer review to be considered independent, it must be performed by another contractor. Page E 6 Peer Review Handbook Purpose and Scope of Work:

The purpose of this work assignment is to provide support to WMB and the Team in finalizing the Prioritization Tool report by conducting an independent peer review of the report.

Work Statement:

Task 1  Management work plan and budget Within 15 days of CO approval of this work assignment, the contractor shall deliver a management work plan including a proposed level of effort, schedule, and budget for all tasks.

Task 2  Provide independent peer review of Prioritization Tool report The contractor shall provide support to WMB and the Team in preparing the Prioritization Tool report by performing an independent peer review of the report. The contractor shall establish a panel of peer reviewers including three senior level persons who collectively have extensive expertise in particular areas to be identified by the WAM upon approval of the work assignment.

Within three weeks of work assignment approval, receipt from the WAM of the necessary qualifications of peer reviewers ( in a TD), and receipt from the WAM of the peer review   charge  ( in a TD), whichever comes latest, the contractor shall identify the three peer reviewers and prepare a memo that lists the names of the peer reviewers and their affiliations and includes the peer reviewers  bio s. Within five weeks of WAM approval of the of the peer reviewers ( via a TD) and receipt of the draft Prioritization Tool report from the WAM ( via a TD), whichever comes later, the contractor shall conduct the peer review, assemble the peer review comments and recommendations in a peer review report organized by charge question, prepare an introduction to the peer review report with a clear and concise overview of the comments, and attach to the peer review report any marginal comments the peer reviewers had on the Prioritization Tool report.

It is not necessary that the peer reviewers jointly reach consensus on their findings and recommendations, since there may be limited overlap in the peer reviewers' areas of expertise and in the charge questions that they focus on. The contractor shall assume, for the purpose of estimating costs, that the draft Prioritization Tool report is roughly 100 pages in length with 200 pages of appendices, and that each peer reviewer will spend 40 hours in reviewing the report and writing comments. EPA plans to provide the report to the contractor in mid July. Peer Review Handbook Page E 7 Deliverables and Schedule:

Task Deliverable Schedule 1 Work plan and budget Within 15 days of CO ap proval of work assignment 2 Memo identifying peer reviewers Within 3 weeks of wo rk assignment approval, receipt of peer reviewer qualifications from WAM, and receipt of charge from WAM, whichever comes latest 3 Peer review report Within five weeks of WAM approval of peer reviewers and receipt of draft Prioritization Tool report from WAM, whichever comes later Other Requirements:

CONTRACTOR COMMUNICATIONS Upon approval of the Work Plan, the contractor shall maintain at least weekly communications with the Work Assignment Manager regarding the status of work on the Work Assignment.

CONFLICT OF INTEREST ( COI) [ Revised Section]

The contractor must adhere to the following requirements:

a) Upon receipt of a Work Assignment, QRT, or similar tasking document, and prior to commencement of any work, notify both the CO and PO of any actual or potential organizational or personal conflicts of interest.

b) Provide a written certification, within 20 days of receipt of a Work Assignment, QRT, or similar tasking document, that:

1) Either all conflicts of interest have been reported to the CO or that no conflicts of interest exist. The contractor is directed to assure that none of the conflicts disclosed are so direct and substantial as to rule out a particular reviewer.

2) All personnel who perform work under this Work Assignment or relating to this Work Assignment have been informed of their obligation to report personal and organizational conflicts of interest to the CO. Page E 8 Peer Review Handbook 3) The Contractor recognizes its continuing obligation to identify and report any conflicts of interest arising during performance of this Work Assignment.

c) If a conflict of interest is identified during performance under this Work Assignment, the Contractor shall immediately make a full disclosure in writing to the CO. The disclosure shall include a description of action which the Contractor has taken or proposes to take, after consultation with the CO, to avoid, mitigate, or neutralize the conflict of interest.

d) After selecting the peer reviewers but before starting the peer review, submit to EPA documentation that shows that the contractor has determined if the peer review candidates:

1) have a conflict of interest or a situation that could create the appearance of a lack of impartiality in relation to the work product 2) have had or presently have a financial relationship with EPA and summarizes for EPA its efforts to identify and propose resolution of these concerns with peer review candidates.

EXPENDITURE OF FUNDS/ HOURS In addition to the requirements of the contract, the contractor shall notify both the Project Officer and the Work Assignment Manager when 75% of funds or hours for this Work Assignment have been expended.

INFORMATION COLLECTION Any other provision of this Work Assignment notwithstanding, the contractor shall not proceed with any information collection where the same or similar information will be collected from ten or more public respondents until written approval is received from the Contracting Officer. This approval will cite an approval number from the Office of Management and Budget as required by the Paperwork Reduction Act ( PRA).

Only Federal agencies and their employees are exempt from the PRA definition of " public respondent." State agencies and their employees are classified as " public respondents."

Soliciting similar information applies to any collection method, i. e., written, oral, electronic, etc., and utilizing any approach, i. e., surveys, phone calls, focus groups, TQM, etc. The PRA applies equally to " willing participants" and participation that is mandated by law. Peer Review Handbook Page E 9 Any question of applicability of the PRA shall be resolved by submitting a complete description of the circumstances in a written request to the Contracting Officer. No collection shall be undertaken until the Contracting Officer provides written notice to the Contractor as to the applicability of the PRA. If the PRA is determined to be applicable, the Contractor shall not initiate any collection until the requisite approval is received.

The General Services Administration ( GSA), under FIRMR Bulletin B 2 administers the Interagency Reports Management Program as derived from 44 U. S. C. Chapters 29 and 31. All work performed under this Work Assignment involving federal interagency reporting must be done in full compliance with these GSA procedures.

CONFIDENTIAL BUSINESS INFORMATION If this Work Assignment requires use of RCRA Confidential Business Information ( CBI), the contract must specifically authorize the contractor to have access to RCRA CBI and the contractor shall abide by all RCRA CBI requirements and stipulations found in the RCRA CBI Security Manual and in the contract. The contractor shall identify in the Work Plan budget all estimated costs for dealing with CBI requirements. All CBI must be returned to EPA as soon as it is no longer needed under this Work Assignment or before the expiration of the Work Assignment, whichever occurs first.

PRINTING AND DUPLICATION The contractor is prohibited from performing any printing under the Government Printing and Binding Regulations. Duplication is allowed to the extent it does not exceed 5,000 impressions of a single page document or 25,000 impressions of a multiple page stand alone document, is limited to one color ( black) copies, and does not exceed the maximum image size of 10 3/ 4 by 14 1/ 4 inches. For all duplication jobs in excess of 5,000 impressions, the EPA WAM will determine in advance if the work can be performed more cost effectively and under the job or time constraints at the EPA Print Shop. If the total number of photocopies for this Work Assignment exceeds 5,000 impressions, the contractor shall identify in their Work Plan the photocopying costs by task and deliverable.

WORK ASSIGNMENT/ WORK PLAN BUDGETS The contractor shall not exceed either the dollar or PL hour budget contained in the approved Work Plan. In addition, on Quick Response Tasks ( QRTs) the contractor shall not exceed the PL hour budget of the QRT. Page E 10 Peer Review Handbook TECHNICAL DIRECTION The Designated Work Assignment Manager ( WAM) on this Work Assignment is authorized to provide technical direction to the extent allowed under EPAAR ( 1552.237 71) ( APR 1984) ( DEVIATION). Other than the Designated WAM, only the Project Officer and the Contracting Officer are authorized to provide technical direction.

Technical direction includes:

( 1) Direction to the contractor which assists the contractor in accomplishing the Statement of Work ( 2) Comments on and approval/ acceptance of reports or other deliverables Technical direction must be within the contract and the Work Assignment statement of work. The Project Officer and the WAM do not have the authority to issue technical direction which ( 1) institutes additional work outside the scope of either the contract or this Work Assignment; ( 2) constitutes a change as defined in the " Changes" clause; ( 3) causes an increase or decrease in the estimated cost of the contract or Work Assignment; ( 4) alters the period of performance or deliverable due dates; or ( 5) changes any of the other express terms or conditions of the contract or Work Assignment.

Technical direction will be issued in writing or confirmed in writing within five ( 5) calendar days after verbal issuance. The technical direction memorandum will be provided to the contractor and copies will be forwarded to the Contracting Officer and the Project Officer. If the contractor has not received written confirmation within five ( 5) calendar days of a oral issuance, the contractor must so notify the Project Officer.

INHERENTLY GOVERNMENTAL FUNCTIONS The contractor shall not perform any inherently governmental functions ( IGF) under this Work Assignment. If during the course of developing the plan of work, through receipt of technical direction, or in carrying out the assignment any portion of the effort is considered to possibly be an inherently governmental function, the contractor must immediately notify the Project Officer and the Contracting Officer.

OCCUPATIONAL HEALTH AND SAFETY Facility site visits conducted under a Work Assignment that include on site inspections or sampling must be conducted in full compliance with the Department of Labor, Occupational Safety, and Health Administration rules under 29 CFR Part 1910 and EPA Order 1440 ( Occupational Health and Safety Manual). Peer Review Handbook Page E 11 TRAVEL COSTS The contractor shall follow the requirements of Subpart 31.2 of the FAR and the Federal regulations in incurring allowable travel costs under this Work Assignment, and correspondingly must at all times seek and obtain Government rates whenever available and observe current subsistence ceilings.

QUICK RESPONSE TASKS Each Quick Response Task ( QRT) shall be confirmed in writing and approved by the Project Officer. The contractor shall respond by letter to the PO with copies to the WAM and the CO within two working days, giving a brief description of the plan of work, including best estimate of hours ( by P level) and a break out of costs to accomplish the task.

No task shall exceed a duration of 30 calendar days from start date to completion date. The level of effort for each task shall be limited to a maximum of 250 labor hours.

Quick Response Task Requests do not change the dollar or professional labor hour budgets of a Work Assignment. Page E 12 Peer Review Handbook STATEMENT OF WORK  EXAMPLE 3  External Peer Review of Protozoa Method Development Criteria Document Period of Performance: Work Plan Approval to August 1, 1997 Work Assignment Manager: Sally Q. Government Employee Office of Water U. S. Environmental Protection Agency LOE: 196 hours SOW: 2.4 BACKGROUND INFORMATION:

The United States Environmental Protection Agency ( EPA), Office of Water is charged with protecting public health and the environment from adverse exposure to chemicals and microbials in water media, such as ambient and drinking waters, wastewater/ sewage sludge and sediments. In support of this mission OW s Office of Science and Technology ( OST) develops health standards, health criteria, health advisories, and technical guidance documents for water and water related media. Under this work assignment, documents prepared by OST are to undergo peer review.

Peer review is an important component of the scientific process. It provides a focused, objective evaluation of a research proposal, publication, risk assessment, health advisory, guidance or other document submitted for review. The criticism, suggestions and new ideas provided by the peer reviewers stimulate creative thought, strengthens the reviewed document and confer credibility on the product. Comprehensive, objective peer reviews leads to good science and product acceptance within the scientific community.

Under this work assignment, the contractor will receive one document ( Protozoa Method Development Criteria Document) for peer review which is related to human health and ecological effects.

STATEMENT OF WORK:

Task 1. The contractor shall develop a work plan to address all tasks in this work assignment. The work plan shall describe the steps that will be taken by the contractor to provide for peer review, including selection of peer reviewer candidates with appropriate expertise, determining absence of conflict of interest, document and reference distribution, establishing schedules, preparing the peer review report, and submittal of the peer review package. Curriculum vitae for all persons assigned to complete this work assignment shall be provided. All P Peer Review Handbook Page E 13 levels, hours and total costs for each task will be provided and costs greater than $ 100.00 shall be itemized in detail.

Task 2. The contractor shall select a group of peer reviewers and determine their availability for the task and absence of conflict of interest, and establish a schedule for the peer review. The contractor is directed to assure that none of the conflicts disclosed are so direct and substantial as to rule out a particular reviewer. Three peer reviewers shall participate in the review. No single peer reviewer may charge more than 40 hours to this task. It is fully acceptable for peer reviewers to commit to less than 40 hours. The peer review will be conducted for the Protozoa Method Development Criteria Document. Reviewers selected by and working for the contractor shall be approved by the EPA Project Officer in writing prior to their beginning work. Minimally, all peer reviewers shall be accomplished in protozoan methods for sample recovery and analysis from water. Approval submissions shall include the reviewers  names and curriculum vitae.

Task 3. The contractor shall arrange for the selected peer reviewers to review the EPA document. Prepare the charge to the peer reviewers based on technical direction received from the EPA WAM. Provide the peer reviewers with copies of the candidate report and all relevant references and instruct the selected peer reviewers to undertake the review. The WAM will provide the contractor with the final version of the document to be reviewed.

Task 4. The contractor shall monitor peer reviewers  progress to assure timely completion. The contractor shall collate peer review comments, and organize the comments in the peer review   for comments  document. Provide the peer review document and all materials submitted by the peer reviewers to the EPA WAM.

SCHEDULE AND DELIVERABLES:

Task 1. ( Work Plan) 15 days after receipt of work assignment Task 2. 1 week after work plan approval Task 3. 1 week after selection of peer reviewers Task 4. 1 week after receiving comments from the peer reviewers TRAVEL: No travel is anticipated under this work assignment. Any travel directly chargeable to this work assignment must be submitted and approved by the project officer. Peer Review Handbook Page F 1 APPENDIX F  USEFUL FORMS Peer Review Conflict of Interest Inquiry Peer Review Checklist for Determining Whether a Work Product Needs Peer Review ( Template Provided by Region V)

Peer Review Checklist for Conducting a Peer Review ( Template Provided by Region V)

Volunteer Service Program Participation Agreement ( EPA Form 3100 14) Page F 2 Peer Review Handbook SUBJECT: PEER REVIEW CONFLICT OF INTEREST INQUIRY Dear ( Peer Reviewer  insert name):

You have been requested by EPA to serve as a Peer Reviewer for______ __________( name of project)____________. Your participation in this review will be greatly appreciated. However, it is possible that your personal affiliations and involvement in particular activities could pose a conflict of interest or create the appearance that you lack impartiality in your involvement for this peer review. Although your involvement in these activities is not necessarily grounds for exclusion from the peer review, you should consult the contact named below or other appropriate official to discuss these matters. Affiliations or activities that could potentially lead to conflicts of interest might include:

a) work or arrangements concerning future work in support of industries or other parties that could potentially be affected by regulatory developments or other actions based on material presented in the document ( or review materials) that you have been asked to review;

b) your personal benefit ( or benefit of your employer, spouse or dependent child) from the developments or other actions based on the document ( or review materials) you have been asked to review;

c) any previous involvement you have had with the development of the document ( or review materials) you have been asked to review;

d) any financial interest held by you ( or your employer, spouse or dependent child) that could be affected by your participation in this matter; and e) any financial relationship you have or have had with EPA such as research grants or cooperative agreements.

Please contact ___________________ ( name and contact info for Agency peer review official or primary contractor) to discuss any potential conflict of interest issues at your earliest convenience, but no later than _________.

[ Be sure to date and sign this inquiry] Peer Review Handbook Page F 3 PEER REVIEW CHECKLIST FOR DETERMINING WHETHER A WORK PRODUCT NEEDS PEER REVIEW Instructions: This checklist is based on the Agency s Peer Review Handbook and the October 2000 Region 5 Order   U. S. EPA Region 5 Improved Policies and Procedures: Peer Review, Records Management, and Work Product Authorization of Scientific and Technical Work Products  which constitute Region 5  s standard operating procedures for peer review. If you have any questions about peer review or need clarification when completing this checklist, please refer to the Handbook, available via the internet at http:// www. epa. gov/ ord/ spc/ 2peerrev. htm. Figure 1 on page 2 of the Handbook includes a useful flow chart and cross references to specific sections of the Handbook that are applicable to this checklist. You are also encouraged to consult with your Division or Office Peer Review Coordinator. The Division/ Office Peer Review Coordinators will periodically request information from this checklist in order to update the National Peer Review Database.

1. Title of Work Product:

2. Product Description:

3. Project Manager:

Name, Organization and Phone Number Please circle the 4. Determination if Work Product is Scientific or Technical: appropriate answer a) Is the work product a scientific, engineering, economic, social yes no science, or statistical document? ( Examples of such documents include: risk assessments, technical studies and guidance, analytical methods, scientific database designs, technical models, technical protocols, statistical surveys/ studies, technical background materials, and research plans and strategies.) b) Is the work product a scientific or technical document resulting yes no from a grant, contract or cooperative agreement? c) Will the work product be used to support a research agenda, yes no regulatory program, policy position, or other Agency position or action?

If you answered   no  to all of these questions, your work product is not subject to EPA s peer review policy for scientific or technical work products and does not need to be placed on any of the peer review lists. Please proceed to # 7 of this checklist. If you answered   yes  to any of these questions, your work product might need peer review; please continue on to # 5 of this checklist. Page F 4 Peer Review Handbook 5. Determination if Work Product is a Major Work Product: Determination of whether a work product is   major  will largely be on a case by case basis. As the continuum of work products covers the range from the obviously major to those products that clearly don t need peer review, there is no one single, easy yes/ no answer to the test of   major  ( see Handbook, Section 2.2.3). There also is no single definition of   significant.  Determination of   major  and   significant  are the responsibility of the Division or Office Director who is the official Decision Maker. Please circle the appropriate answer a) Does the work product establish a significant precedent, yes no model, or methodology? b) Does the work product address significant controversial yes no issues? c) Does the work product focus on significant emerging or yes no   cutting edge  issues? d) Does the work product have significant cross Agency or yes no inter agency implications? e) Does the work product involve a significant investment of yes no agency resources? f) Does the work product consider an innovative approach or yes no application for a previously defined problem, process or methodology? g) Is the work product required to be peer reviewed by statute yes no or other legal mandate? h) Does the work product support a regulatory decision, yes no policy or guidance of major impact? ( Major impact can mean that it will have applicability to a broad spectrum of regulated entities and other stakeholders, or that it will have narrower applicability, but with significant consequences on a smaller geographic or practical scale.) i) Is the work product an application of or modification to yes no an existing, adequately peer reviewed methodology or model that departs significantly from the situation it was originally designed to address?

If you answered   yes  to any of these questions, your work product needs peer review unless special circumstances exist; please continue on to # 6. If you answered   no  to all of these questions, your work product probably does not need peer review. However, peer review can always be done to improve the quality of the work product. Please proceed to # 7 of this checklist. Peer Review Handbook Page F 5 6. Determination Whether Circumstances Exist Where a Major Work Product Would Not Be Peer Reviewed: Please circle the appropriate answer a) Was the work product previously reviewed by recognized yes no experts or an expert body? ( Note: Peer review of an EPA work product by a recognized refereed journal strengthens the scientific credibility of the work product but does not eliminate the need to have the work product itself peer reviewed for issues and concerns to support an Agency action. See Sections 2.4.4 and 2.4.5 of the Handbook for more details.) b) Are the scientific or technical methodologies or information yes no being used commonly accepted in the field of expertise? c) Has the regulatory activity or action which the work product yes no supports been terminated or canceled? d) Is there a statutory or court ordered deadline, or a time yes no constraint which may limit or preclude peer review of the work product?

If you answered   yes  to any of these questions, your work product probably does not require peer review. This decision with the justification needs to be concurred with and signed off by the Division/ Office Director. The decision with the justification must be retained in the peer review files and noted in Peer Review Work Product List C in the National Peer Review Database. Continue on to # 7. If you answered   no  to all of these questions, proceed to # 8.

7. Next Steps For Work Products That Will Not Be Peer Reviewed: a) Division/ Office Director concurs with the decision that the work product should not be peer reviewed.

Signature of Division/ Office Director and Date Signed b) A copy of this completed checklist has been given to the Div/ Off Peer Review Coordinator and put in the official peer review files in the Division/ Office.

Signature of Div/ Off Peer Review Coordinator and Date Signed Location of Div/ Off Peer Review Files Page F 6 Peer Review Handbook c) Work product has been placed on Peer Review Work Product List C in the National Peer Review Database. ( Note: This only applies to those work products subject to the peer review policy.)

Signature of Div/ Off Peer Review Coordinator and Date Signed If all of the necessary information is complete, you are done. You don t need to proceed any further with this checklist.

8. Next Steps For Work Products That Will Be Peer Reviewed: a) Division/ Office Director has been consulted and concurs with the decision that the product should be peer reviewed.

Signature of Division/ Office Director and Date Signed b) A copy of this completed checklist has been given to the Division/ Office Peer Review Coordinator and put in the official peer review files in the Division/ Office.

Signature of Div/ Off Peer Review Coordinator and Date Signed Location of Div/ Off Peer Review Files c) Work product has been placed on Peer Review Work Product List B in the National Peer Review Database.

Signature of Div/ Off Peer Review Coordinator and Date Signed Because your work product will be peer reviewed, you need to complete a second checklist entitled   Peer Review Checklist for Conducting a Peer Review.  Peer Review Handbook Page F 7 PEER REVIEW CHECKLIST FOR CONDUCTING A PEER REVIEW Instructions: This checklist is based on the Agency s Peer Review Handbook and the October 2000 Region 5 Order   U. S. EPA Region 5 Improved Policies and Procedures: Peer Review, Records Management, and Work Product Authorization of Scientific and Technical Work Products  which constitute Region 5  s standard operating procedures for peer review. If you have any questions about peer review or need clarification when completing this checklist, please refer to the Handbook, available via the internet at http:// www. epa. gov/ ord/ spc/ 2peerrev. htm. Pages 2 4 of the Handbook contain useful flowcharts and cross references to specific sections of the Handbook that are applicable to this checklist. You are also encouraged to consult with your Division or Office Peer Review Coordinator. The Division/ Office Peer Review Coordinators will periodically request information from this checklist in order to update the National Peer Review Database.

1. Title of Work Product:

2. Product Description:

3. Project Manager: Name, Organization and Phone Number Check the box when 4. Up front Considerations for Planning the Peer Review: a) The Div/ Office Director has chosen a peer review leader for the project. ( Note: The project manager and peer review leader can be the same person.) Name of Peer Review Leader: item is completed Phone Number: Organization: b) The peer review leader has obtained appropriate peer review training before conducting the peer review. c) Key questions and issues have been identified to include in the charge to the peer reviewers. d) The Div/ Office Records Coordinator has been consulted to insure that all the files, including electronic records, will be created, maintained, retained, and disposed of appropriately and in accordance with Div/ Office and Agency procedures. e) A formal peer review record or file has been established, and provisions have been made to store any electronic records associated with the work product and peer review. Location of Record/ File: Provisions for Electronic Records: Page F 8 Peer Review Handbook Check the bo x when item is completed or circle the appropriate answer ( NA = not applicable) f) There is a source of adequate funding to pay for external peer review if external peer review is necessary and funding is needed. ( Note: Contracts NA can be used for peer review services. However, special management controls are required to ensure proper use of these contracts. See Section 3.6 of the Handbook for details.) Source of Funding: g) Resource limitations may restrict the peer review. ( If   yes  was Yes No selected, a limited peer review might be considered. However, only in very rare circumstances should resource limitations restrict peer review. Peer review must be planned for as part of a project s budget.) h) Amount of time needed for peer review( s) has been allotted given existing constraints of potential peer reviewers, deadline for the final work product, logistics for the peer review, etc. Length of Time Needed:

5. Develop the Charge to the Peer Reviewers: a) A clear, focused charge has been formulated that identifies recognized issues, asks specific questions, and invites comments or assistance. b) The charge has been included in the peer review record.

6. Select the Peer Review Mechanism: a) The work product is novel, complex, controversial, or has great Yes No cost implications. ( If the answer is   yes  to any of the above, serious thought should be given to conducting an external peer review. If the answer is   no  to all of the above, internal peer review is probably sufficient.) b) A determination has been made regarding which components or stages of the work product will be peer reviewed. ( Note: Generally, peer review is recommended for each stage of a product s development.) Components to be peer reviewed:

c) A peer review mechanism ( e. g., internal, external or a combination of both) has been chosen for the work product or stages of the work product. Mechanism: d) The work product either: 1) has been, or is being, generated as part Yes No of administrative or civil enforcement activities by U. S. EPA, or 2) likely will be used in the future to support administrative or civil enforcement activities by U. S. EPA. ( If the answer is   yes  to either item above, then the Office of Regional Council ( ORC) must be consulted if the Peer Review Leader believes an external peer review is needed or is preferable. ORC concurrence should be obtained.) Peer Review Handbook Page F 9 Check the box when item is com pleted, or circle yes or no e) The work product is going to be peer reviewed via a refereed, Yes No scientific journal. ( If the answer is   yes,  the work product still should be considered for peer review because journal peer review may not cover issues and concerns that the Agency would want peer reviewed in order to support an Agency action.) f) Logistics for conducting the peer review ( e. g., written comments will be received by mail, or will be collected at a meeting) have been included in the peer review record. g) The Div/ Off Director has concurred with the recommended method of peer review. Date of Div/ Off Director Concurrence: h) The concurrence of the Div/ Off Director has been included in the peer review record.

7. Determine the Specific Time Line for the Peer Review: a) A start date for the peer review has been selected. Start Date: b) The amount of time the peer reviewers will be given to conduct the peer review has been determined. Number of Days for Review: c) A due date for comments from the reviewers has been selected. Due Date: d) The amount of time necessary to incorporate comments from th e peer reviewers into the work product has been determined. Number of Days for Revision: e) A deadline for final completion of the work product has been determined. Due Date:

8. Select the Peer Reviewers: a) Advice was sought in developing a list of potential peer reviewer candidates who are independent of the work product and have appropriate scientific and technical expertise. b) The expertise required for the peer review has been determined. c) In reviewing the candidates, a balance and a broad enough spectrum of expertise were considered. d) In reviewing the candidates, any potential conflicts of interest were considered. Page F 10 Peer Review Handbook Check the box when item is completed e) The peer reviewers have been selected and the process for selecting the reviewers, including inquiries and resolution of potential conflicts of interest, has been documented and included in the peer review record/ file. ( Note: Conflict of Interest Inquiry Forms are available from the Regional and Div/ Off Peer Review Coordinators.)

9. Obtain and Transmit Materials for Peer Review: a) Instructions have been given to the peer reviewers which ask for written comments in a specified format by the specified deadline that are responsive to the charge. b) The peer reviewers have been provided with the essential documents, data, and information to conduct their review. Date Peer Reviewers Given Charge/ Materials: c) The peer reviewers have been instructed not to disclose draft work products to the public. d) The peer review record/ file contains all the materials given to the peer reviewers.

10. Conduct the Peer Review: a) Written comments have been received from all peer reviewers. Date all comments were received: b) All clarification or additional information necessary from the peer reviewers is received. c) The validity and objectivity of the comments have been evaluated. d) Appropriate experts/ staff/ managers have been consulted on the potential impacts of the comments on the final work product, the project schedule, and budget. e) The peer review comments have been included in the peer review record/ file.

11. Consider the Peer Review Comments: a) Decisions have been made regarding which comments are accepted and will be incorporated into the final work product, and which comments will not be incorporated. b) A memo or other written record has been prepared which responds to the peer review comments and specifies acceptance or, where thought appropriate, rebuttal and non acceptance. c) The Div/ Off Director has concurred with the decisions and written record on how to incorporate the peer reviewers comments in the work product and on which comments will not be incorporated. Date of Div/ Off Director concurrence: Peer Review Handbook Page F 11 Check the box when item is com pleted, or circle yes or no d) The concurrence of the Div/ Off Director has been included in the peer review record/ file. e) The memo or written record documenting how comments were handled and how the work product was revised has been included in the peer review record/ file. f) The work product has been revised to incorporate the acceptable comments. g) The peer review performed during the process of developing the work product has been summarized and included in the work product. h) It is necessary to send the revised work product back to the peer Yes No reviewers. ( If the answer is   yes,  proceed to item # 11i. If the answer is no, proceed to item # 12.) i) Additional comments are received, evaluated, and incorporated into the work product, and placed in the peer review record.

12. Consider Other Comments: a) Prior to finalization, the document needs additional internal Yes No and/ or external programmatic review. ( If the answer is  yes,  go to # 12b. If the answer is   no,  proceed to # 13.) b) Written comments by programmatic reviewers have been received. c) Final decisions have been made regarding which comments are accepted and will be incorporated into the final work product, and which ones will not be incorporated. d) A memo or other written record has been prepared which responds to the programmatic review comments and specifies acceptance or, where thought appropriate, rebuttal and non acceptance. e) Div/ Off Director has concurred with the decisions and written record on how to incorporate the programmatic comments. Date of Div/ Off Director concurrence: f) The memo or written record has been included in the peer review record/ file. g) The work product has been revised to incorporate the acceptable programmatic comments.

13. Finalize Work Product and Close Out Peer Review: a) The work product has been completed. b) The Div/ Off Director has approved the work product. Date of Div/ Off Director Approval: c) The Div/ Off Director approval has been included in the peer review record/ file. Page F 12 Peer Review Handbook Check the box when item is completed, or circle yes or no d) The Div/ Off Director has judged the work product to be sufficiently Yes No controversial, of significant enough interest to outside parties, or of wide enough distribution, such that it should also be authorized by the Regional Administrator ( RA), or the Deputy RA ( DRA). ( If the answer is   yes,  proceed to # 13e. If the answer is   no,  proceed to # 13f.) e) The RA or DRA has authorized the work product. Date of RA or DRA Authorization: f) The final work product has been included in the peer review record/ file.

14. Publication and Release of Reports: a) The Div/ Off Director has approved publication or release of the work product. b) The written approval by the Div/ Off Director has been included in the peer review record/ file. c) The Div/ Off Director has judged the work product to be sufficiently Yes No controversial, of significant enough interest to outside parties or of wide enough distribution, such that its distribution or release should also be authorized by the RA or DRA. If the answer is   yes,  proceed to # 14d. If the answer is   no,  proceed to # 15. ( Note: The Div/ Off Director s decision to elevate to the RA or DRA can be made concurrently with item # 13d.) d) The RA or DRA has authorized distribution or release of the work product. Date of RA or DRA Authorization:

15. Retention of Peer Review Files and Records: a) The Div/ Off official procedures for administrative records and the Agency s record retention schedules have been examined to determine how long the peer review record/ file, including electronic records, should be retained. ( Note: The required time of retention for final reports and supporting data varies depending upon the nature of the report, however, final reports which are mission related or have an EPA number and receive external distribution are generally permanent federal records.) b) The Div/ Off Records Officer or the Regional Records Officer has been consulted to help determine how long the peer review record/ file, including electronic records, should be retained. Peer Review Handbook Page F 13 Check the box when item is completed c) A location for the completed peer review record/ file has been identified, and provisions have been made to retain electronic records associated with the work product and peer review. ( Note: This can be the same location and provisions as identified in # 4e.) Location of Record/ File: Provisions for Electronic Records:

d) Someone has been assigned the responsibility for maintaining the record/ file and electronic records, and ensuring that they are either archived or destroyed appropriately. ( Note: This can be the same person as identified in # 4a.) Contact Name and Phone No: Organization:

16. Closeout of Checklist: a) Items # 1 15 of checklist have been completed.

Signature of Peer Review Leader and Date Signed b) A copy of the completed checklist has been given to the Div/ Off Peer Review Coordinator.

Signature of Div/ Off Peer Review Coordinator and Date Signed c) The completed checklist has been included in official peer review record/ file.

d) The work product has been moved from Peer Review Work Product List B to List A in the National Peer Review Database. Date Product moved to List A: Page F 14 Peer Review Handbook U. S. Environmental Protection Agency VOLUNTEER SERVICE PROGRAM PARTICIPATION AGREEMENT Name of Program Participant Citizen of U. S.

  Yes   No Project Supervisor Proposed Organizational Agreement ( Include Geographical Location)

Proposed Arrival Date Actual Arrival Date Proposed Termination Actual Termination Sponsored by Educational Institution   Yes   No Name and Address of Institution ( if   Yes  )

Describe Project ( s) on Which Par ticipant Will Work, Including Scope and Anti cipated Hour s per Week Facilities and Equipment to be Made Available by EPA Degree of Supervision to be Exercised by EPA Assistance and Degree of Cooperation Required by other Agency Personnel by Participant Project Supervisor Certifies That Services to be Produced by the Program Participant are not Services Provided for Through EPA Agency Operations ______________________________________________ Signature of Supervisor THE VOLUNTEER AGREES THAT: a. l claims for compensation from the Government of the United States are waived for any services performed; b. ernment has a non exclusive royalty free license to use or reproduce and patent or copyright material which is developed as part of and during participation in this program; and c. to the administrative instructions and requirements of the agency while on EPA premises.

______________________________________________ Participant Signature PERSONNEL OFFICE SIGNEE CONCURRENCE EPA Form 3100 14 ( Rev. b 30) Replaces the 7 72 edition and the 12 77 edition which was entitled   Unpaid Work Study Program  Any and al The Gov He/ she will adhere SIGNATURE OF PERSONNEL OFFICER OR DE DATE Peer Review Handbook Page G 1 APPENDIX G  REFERENCES CONCERNING PEER REVIEW American Chemical Society and the Conservation Foundation ( 1985) Issues in Peer Review of the Scientific Basis for Regulatory Decisions, Washington, DC, November 1985.

Browner, C. ( 1994) Peer Review Program, Washington, DC, Memorandum issued June 7, 1994. ( NOTE: Attached as Appendix A to this Handbook)

Chubin, D. ( 1994) Grants peer review in theory and practice, Evaluation Review 18: 20 30.

Chubin, D. and E. Hackett ( 1990) Peerless Science: Peer Review and Science Policy, Albany, NY: State University of New York Press.

Jasanoff, S. ( 1990) The Fifth Branch: Science Advisors as Policymakers, Cambridge, MA: Harvard University Press.

Kostoff, R. ( 1996) Peer Review in Selected Federal Agencies, presented at AAAS Annual Meeting, Baltimore, MD, February 9, 1996.

National Academy of Sciences ( 1999) Evaluating Federal Research Programs: Research and the Government Performance and Results Act, Washington, DC: National Academy Press, January 1999.

National Environmental Policy Institute ( 1996) Enhancing the Integrity and Transparency of Science in the Regulatory Process, Washington, DC: National Environmental Policy Institute, Fall 1996.

National Research Council ( 1995) Interim Report of the Committee on Research and Peer Review in EPA, Washington, DC: National Academy Press, March 1995.

National Research Council ( 2000) Strengthening Science at the U. S. Environmental Protection Agency: Research Management and Peer Review Practices, Washington, DC: National Academy Press.

Reilly, W. ( 1993) Peer review Policy, Washington, DC, Memorandum issued January 19, 1993.

Rennie, D. ( 1999) The Development and Rationale of Peer Review, In: Peer Review in Health Sciences, Fiona Godlee and Tom Jefferson, eds., BMA Books, London. Page G 2 Peer Review Handbook Science Advisory Board ( 1999) An SAB Report: Review of the Peer Review Program of the Environmental Protection Agency, EPA SAB RSAC 00 002, U. S. Environmental Protection Agency, Science Advisory Board, Washington, DC, November 1999.

Spitzer, H. ( 1995) Peer Review Practices in the Federal Government, Bethesda, MD: Environmental Network, report prepared for the American Industrial Health Council, April 26, 1995)

USEPA ( 1992) Safeguarding the Future: Credible Science, Credible Decisions, U. S. Environmental Protection Agency, Washington, DC, March 1992.

USEPA ( 1998) Science Policy Council Handbook: Peer Review, EPA 100 B 98 001, U. S. Environmental Protection Agency, Science Policy Council, Washington, DC, January 1998.

USEPA ( 1999) EPA s Selection of Peer Reviewers, EPA OIG Report No. 1999 P 217, U. S. Environmental Protection Agency, Office of the Inspector General ( OIG), Washington, DC, September 1999.

U. S. General Accounting Office ( 1994) Peer Review: EPA Needs Implementation Procedures and Additional Controls, GAO/ RCED 94 89, Washington, DC: U. S. Government Printing Office, February 1994.

U. S. General Accounting Office ( 1996) Peer Review: EPA s Implementation Remains Uneven, GAO/ RCED 96 236, Washington, DC: U. S. Government Printing Office, September 1996.

U. S. General Accounting Office ( 1999) Federal Research: Peer Review Practices at Federal Science Agencies Vary, GAO/ RCED 99 99, Washington, DC: U. S. Government Printing Office, March 1999. Peer Review Handbook Notes and Comments Peer Review Handbook Notes and Comments Peer Review Handbook Notes and Comments Peer Review Handbook Notes and Comments