The provision of peer review products or services to EPA without compensation are provided as so called " gratuitous" services. If a person or organization wishes to perform peer review services for EPA without compensation, the program must ask them to sign an agreement whereby the person or organization agrees to provide the prescribed peer review services as gratuitous services, with no expectation of receiving compensation for these services from EPA. An agreement must be executed because the Antideficiency Act ( 31 U. S. C. § 1342) prohibits the Agency's acceptance of " voluntary" services. " Voluntary" services are provided to EPA without an agreement in advance that such services are provided at no cost to EPA. Note that persons cannot waive compensation ( i. e., agree to provide gratuitous services) for which there is a statutory right to payment, unless a law permits the waiver.

3.6.2 Can I Use a Contract to Obtain Peer Review Services?

The Agency may contract for peer review services. The contract may be written solely for peer reviews or be included as one of several specifically described interrelated tasks in a contract that requires the contractor to provide more than just peer review services.

For assistance in preparing the necessary pre award documents, program officials should consult The Cookbook: How to Get Contracts Awarded in EPA and Chapter 2 of the Contracts Management Manual ( CMM). The Cookbook and Chapter 2 are available on EPA's Intranet Peer Review Handbook Page 53 ( http:// www. intranet. epa. gov/ oamintra) and on the Agency LAN Services ( Administration/ Management/ OAM Procurement Policy Information).

3.6.3 How Do I Write a Statement of Work for Contracts?

The statement of work ( SOW) must clearly specify that the contractor is responsible for preparing peer review evaluations and set forth guidelines for the peer review of scientific or technical documents. The contractor may perform the peer review with in house staff, subcontractors or consultants. Any guidelines for performing peer reviews to ensure soundness and defensibility must be developed by the program office and made part of the contract. The contractor would then ensure that the peer reviews adhered to the guidelines.

The SOW cannot simply define the role of the prime contractor as arranging for the services of others to perform peer reviews and logistics for meetings. Unless the prime contractor is clearly tasked with responsibility for performing peer reviews, individual peer reviewers' fees and associated travel expenses are not payable under the contract.

The EPA may pay for the reviewer's comments or evaluation, and also for attendance at a meeting with the Agency and other reviewers to discuss the results of the peer review. If the SOW calls for the preparation of comments or an evaluation, and specifies a meeting with the Agency and other peer reviewers to discuss the results of the peer review, payment is appropriate. The peer reviewer's attendance at the meeting would then be part of contract performance.

Example statements of work are found in Appendix D.

3.6.4 What are Advisory and Assistance Services ( AAS) or Sensitive Activities?

Contracts that provide services that support or improve Agency decision making or policy development are subject to special management controls. These services include services acquired from non Governmental sources by contract to support or improve Agency policy development, decision making, management, and administration, or research and development activities. See Federal Acquisition Regulation ( FAR) 37.201 for a more specific definition of AAS. Such services may take the form of information, advice, opinions, alternatives, conclusions, recommendations, training, and direct assistance. For additional information on advisory and assistance services and sensitive activities, program officials should review Chapter 2 of the CMM.

New contracts for these services require management approvals prior to issuance of the solicitation. For the thresholds that have been established for approval of these justifications, see Peer Review Handbook Page 54 Figure 5, Item B Management Approvals set forth after Chapter 2 of the CMM ( for current approval levels).

3.6.5 What are Some Management Controls for Contracts?

Contracting for peer review services is permitted. However, because of the potential for improper use of these contracts, special management controls are required.

a) Inherently Governmental Functions ( IGFs)  OFPP Policy Letter 92 1, dated September 23, 1992, describes ( 1) functions that are inherently governmental and must be performed only by Government employees and ( 2) functions that may be contracted, but so closely support Government employees in their performance of IGFs that the contract terms and performance require close scrutiny by Federal officials. Federal Acquisition Regulation ( FAR) coverage of inherently governmental functions is at FAR Subpart 7.5.

Peer reviews represent only a contractor's recommendations, advice or analysis of a document. Agency officials must make the official Agency decision regarding acceptability and/ or quality of the document. To ensure that Agency officials are not improperly influenced by recommendations in the peer review, management controls must be included in the contract. One possible control would be to require the peer reviewers to submit with their evaluations or comments a description of the procedures used to arrive at their recommendations; a summary of their findings; a list of sources relied upon; and make clear and substantiate the methods and considerations upon which their recommendations are based. To the extent possible, the contract should set forth any guidelines or criteria for performance of the peer review. Agency officials should document their evaluations of the quality and validity of the peer review.

b) Conflict of Interest ( COI)  Another important factor is that the objectivity of the peer review should not be improperly influenced or undermined by the contractor performing the review. To identify and avoid or mitigate actual or potential COI, the contract should include controls. Such controls might require the contractor to report on prior and current work, and prior clients that might create COI. Other controls might include Agency review and placing limits or advance approval on future work. There should also be procedures implemented to assure that the contractor does not gain an unfair advantage in future requirements as a result of their performance of peer reviews. Program officials should consult the Contracting Officer ( CO) for special contract clauses. Peer Review Handbook Page 55 FAR coverage of conflicts of interest is at FAR Subpart 9.5. The EPA Acquisition Regulations ( EPAAR) at 48 CFR Subpart 1509.5 generally mandates conflicts of interest solicitation provisions and contract clauses, but makes them optional for procurements accomplished through the use of simplified acquisition procedures as set forth in FAR Part 13. Also see 48 CFR 1509.507 1( b)( 3) and 48 CFR 1552.209 70,  71 &  72 as additional resources.

Contract for peer review services: An EPA contracting officer will include conflicts of interest solicitation provisions and contract clauses as a matter of course without involvement by the EPA project officer, if the peer review services are not obtained pursuant to the simplified acquisition procedures in FAR Part 13. If the peer review services are subcontracted pursuant to a prime contract, then the prime contractor is ordinarily required to include a conflicts of interest clause substantially similar to the conflicts of interest clause in the primary contract in its subcontract to the peer reviewer.

Although the EPA contracting officer and/ or prime contractor has the primary responsibility to include the required conflicts of interest provisions/ clauses, the EPA project officer may nevertheless wish to:

1) Highlight the conflict of interest requirements in the Scope of Work for the procurement of the peer review services 2) Develop a specific conflict of interest clause regarding the peer review at issue as a substitute or in addition to the standard conflicts of interest clause 3) Review the solicitation/ contract to make sure that the required conflicts of interest clause has been included Simplified acquisitions of peer review services: Although conflict of interests requirements are optional for simplified acquisitions, they are nevertheless a good idea. Accordingly, an EPA project officer obtaining peer review services with simplified acquisition procedures should request the purchasing agent/ contracting officer to include a conflict of interest solicitation provision and contract clause in the purchase order. Peer Review Handbook Page 56 c) Confidential Business Information ( CBI)/ Privacy Act Protected Information and Other Sensitive Information  When peer reviewers are not employees of the United States Government, it is unlikely that the EPA will have authority to give reviewers access to confidential business information in the absence of consent for such disclosure by the CBI submitter. Therefore, all documents provided to non Federal reviewers must be screened for information claimed as CBI. Even where business information has not been explicitly claimed as CBI, if it is of a kind where the submitter might be expected to object to its release, prior to release the submitter must be asked whether it wishes to assert a claim, unless the submitter has previously been informed that failure to assert a CBI claim may result in disclosure without notice. Language is included in the contract to clearly identify any required procedures or processes prior to release of any protected information, including any requirements for confidentiality agreements, as well as limits on use and disclosure of the data by contractor personnel.

d) Personal services  Under contracts, the EPA may not engage the peer reviewers in any improper personal services relationships, i. e., an arrangement under which contractor personnel are subject to relatively continuous supervision and direct control by an Agency official or employee. These relationships are characterized as one where the contractor employee interacts with the Agency in a manner similar to that of a Federal employee.

To avoid these improper relationships, program officials should write welldefined SOWs. The SOWs should set forth the requirements in detail for work to be performed independently, including the manner in which it will be evaluated. The SOW must set forth what work is to be performed not how the work is to be performed. Technical direction may be used to clarify ambiguous technical requirements to ensure efficient and effective contractor performance, and is not considered supervision or assignment of tasks. For additional information, program officials should consult EPA Order 1901.1A, Use of Contractor Services to Avoid Improper Contracting Relationships and FAR Subpart 37.1 3.6.6 Can the Agency Identify and/ or Select Peer Reviewers Through a Contract?

Program officials cannot interfere in a contractor's authority and responsibility to perform work by " selecting" who will perform the peer review ( doing so may invoke FACA  see section 2.7.3). The Federal Acquisition Regulation ( FAR) governs the Contracting Officer's ( CO's) and program officials' relationship with the contractor. Peer Review Handbook Page 57 However, the CO does have the ability to review and consent to subcontractors and consultants. The contract can also specify which individuals are key personnel and include peer reviewers. The contract will set forth the procedures governing the use and replacement of key personnel. Further, the contract can require workplans for approval by the Agency, wherein the contractor will propose the peer reviewers it is considering for selection.

Program officials should identify the qualifications required to perform the review work and the criteria for technical acceptability. The EPA may identify a pool of qualified subcontractors and consultants to the prime contractor ( listed in alphabetical order), but cannot direct the use of any particular subcontractor or consultant.

3.6.7 Can I Use Simplified Acquisition Procedures to Obtain Peer Reviewers?

In some instances, peer reviewers can be obtained via simplified acquisition procedures. The acquisition of supplies or non personal services from the open market and on a sole source basis when the aggregate amount involved in any one transaction does not exceed $ 100,000 constitutes a simplified acquisition ( FAR Part 13). The same considerations in the section 3.6.5 discussion on IGFs, COI, access to CBI, and personal services apply to simplified acquisitions. Normally, the Government issues a purchase order directly to the individual peer reviewer, instead of to a prime contractor who may subcontract for performance of the peer review.

a) Approvals  All small purchases for peer reviews are considered AAS. See Chapter 2 of the Contract Management Manual ( CMM) for the appropriate approval levels for AAS.

b) Competition  The Federal Acquisition Regulation ( FAR) requires competition for purchases in excess of $ 2,500. Purchases for more than $ 2,500 and not more than $ 100,000 are to be made only from small businesses unless the Contracting Officer is not able to obtain two or more offers from small businesses that are competitive in terms of market price, quality, and delivery. Only one source need be solicited if the Contracting Officer determines that only one source is reasonably available. Contracting Officers are encouraged to use best value.

c) Procurement Requests  Program Officers should include the following in all PRs for the purchase of peer reviews:

1) A fixed price amount at or below the simplified acquisition threshold Peer Review Handbook Page 58 2) A detailed description of the requested services, inclusive of:

( a) Total quantity per line item ( b) Estimated unit price per line item ( c) Total cost per line item ( d) Specific deliverables for each line item ( e) Total cost of the purchase request 3) Reference FAR Subpart 3.6 and Environmental Protection Agency Acquisition Regulation ( EPAAR) Subpart 1503.601 regarding sources from Government employees or organizations owned and controlled by them.

4) Provide sources from small businesses, if available.

5) If the request is a sole source purchase, justification must be provided in accordance with the EPAAR Subpart 1513.170 1.

3.6.8 How is Travel Handled with Contracts or Purchase Orders?

Funds obligated on a contract or purchase order are available to pay for the costs of producing the peer review including the travel costs and fee of the peer reviewer.

The EPA acquires peer reviews through simplified acquisitions issued directly to peer reviewers or through contracts with companies, which provide the peer review services. By issuing a purchase order or awarding a contract for peer review services, the EPA may pay not only for the peer review services/ comments, but also for attendance at a meeting with the Agency and other reviewers to discuss comments. The scope of work of the contract must require the contractor or individual peer reviewer as appropriate to provide peer review services and indicate whether the contractor or peer reviewer will be required to discuss a specific peer review work product with the Agency and/ or with other peer reviewers. Attendance at a meeting to discuss a peer review work product would then be part of the contract's performance. Thus, the contract may serve as the mechanism to pay for peer review services and associated travel expenses to provide comments to the EPA. Peer Review Handbook Page 59 3.6.9 How is Travel Handled with Special Government Employees?

The term Special Government Employee ( SGE) is defined in 18 U. S. C. 202( a) as an officer or employee of an agency who performs temporary duties, with or without compensation, for not more than 130 days in a period of 365 days, either on a full time or intermittent basis.

Travel and per diem expenses of experts hired as SGEs for peer review may only be paid through the issuance of invitational travel orders ( 5 U. S. C. § 5703). These invitational travel and per diem expenses should be charged to an appropriate EPA travel account.

Members of the SAB, SAP, and other FACA advisory committees are often brought on board as SGEs. It is not appropriate to reimburse travel or per diem expenses of advisory committee members or other SGEs through a contract. Peer Review Handbook Page 60 THIS PAGE LEFT INTENTIONALLY BLANK Peer Review Handbook Page 61 The peer review is not completed until the peer review comments are incorporated into the final work product. 4. COMPLETING A PEER REVIEW 4.1 Overview Performance of the formal peer review is not the final stage in the development of the work product. Rather, it is an important stage in developing the work product, with the final work product representing the true end of the peer review. As a result, the peer review process closes with three major activities: evaluating comments and recommendations, utilizing peer review comments for completing the final work product, and organizing and maintaining a record of the peer review.

Careful attention to all of these elements, singly and together, assures a credible peer review process. Conversely, inattention can nullify the peer review attempt. A well planned peer review applied to a reasonable quality starting work product, followed by responsible, visible utilization of peer review suggestions in the final product assures a credible product for use in Agency decision making.

The peer review is not completed until the peer review comments are incorporated into the final work product, or reasons are stated why such comments are not to be incorporated. Once the peer review is completed, the Peer Review Coordinator moves the peer reviewed scientific and/ or technical work product from List B ( or List C in some instances) to List A in their office's annual report.

4.2 Final Work Product 4.2.1 How Do I Incorporate Peer Review Comments into the Final Work Product?

The Peer Review Leader must carefully evaluate and analyze all peer review comments and recommendations. As discussed earlier, a carefully crafted charge to the peer reviewers simplifies organizing and analyzing comments. Also, any other issues that are raised need to be identified and evaluated.

The validity and objectivity of the comments need to be evaluated. Analyses may include consultation with other experts/ staff within the Office and/ or Agency. Adequate documentation is needed to show that comments are accepted or rejected  the documentation can be brief, but Peer Review Handbook Page 62 must address the legitimate, valid comments. It is the responsibility of the Peer Review Leader to obtain management approval of the approach to addressing the peer review comments.

Comments that have significant impact on time, budgetary, and/ or resource requirements are particularly important and need to be evaluated in consultation with management. These comments may lead to allocation of additional resources and a revised schedule for the completion of the work product.

4.2.2 What Actions are Potentially Forthcoming from Peer Review?

Peer review comments and recommendations may entail significant impacts on the planned project schedule, budget, or other resource requirements. Management decisions related to revisions in one or more of these areas may be appropriate.

The substantive issues or concerns expressed by peer reviewers may suggest that wider scientific and technical consultation is needed to ensure the adequacy of the work product.

The peer review comments and recommendations on a final product may provide a basis for bringing the associated project to closure.

4.2.3 Can the Identity of Peer Reviewers be Kept Anonymous?

If the matter has gone to litigation, the litigating parties can discover the names of anyone who contributed to a Federal product, including peer reviewers. Therefore, it is not possible to totally shield peer reviewers. In addition, it may be difficult to shield the names of the peer reviewers when the Agency is responding to a Freedom of Information request. However, in the ordinary course of events, you can often offer comments received without attributing the comments to a specific reviewer.

If a peer reviewer requests anonymity at the outset of the peer review, the Peer Review Leader needs to inform the peer reviewer of the above possible eventualities. The Agency will in the ordinary course of events attempt to maintain the confidentiality of the peer reviewers and their comments for public consumption, but it is recognized in many instances, for example open public meetings and the above circumstances, this can't be assured. Remember, the Agency is committed to working " as if in a fishbowl" and most of its activities are transparent to the public ( except where confidential business information is concerned). It is recognized that this may be a deterrent to possible peer reviewers, but this is a reality that has to be understood. Peer Review Handbook Page 63 4.3 Completing the Peer Review Record 4.3.1 How Do I Complete the Peer Review Record?

Once the Peer Review Leader has completed the peer review, the peer review record is brought up to date and then archived according to that organization's procedure ( see section 2.5.7). The peer review record must be indexed and maintained in an organization's archive ( repository). The location of the peer review record needs to be readily identifiable so interested parties can locate and obtain materials easily and quickly. The peer review record should be placed in any associated established public docket, if required, in addition to the organizational archive. As a courtesy, a copy of the revised work product may be sent to the peer reviewers for information.

The Peer Review Leader will collect the following materials for the peer review record and submit for archiving; including at least ( see also section 2.5.3):

a) The draft work product submitted for peer review b) Materials and information ( including the charge) given to the peer reviewers c) Written comments, information, and materials received from the peer reviewers d) Information about the peer reviewers ( such as reviewers' names, affiliations, and identified potential conflicts and their resolution)

e) Logistical information about conduct of the peer review ( such as times and locations of meetings)

f) A memorandum or other written record responding to the peer review comments specifying acceptance or, where thought appropriate, rebuttal and non acceptance g) The final work product h) Peer review summary report ( see sections 1.3.1 and 1.3.2)

4.3.2 Where Should the Peer Review Records be Kept, and for How Long?

During the active conduct of the peer review, the Peer Review Leaders maintain the peer review record with themselves until the peer review is totally completed. Minimally, the file Peer Review Handbook Page 64 should be maintained until one year after the completed peer review is reported in the next annual reporting. After that, the peer review record should be maintained for a " reasonable period of time." Establishment and maintenance of the archive where the peer review records ultimately reside are an organization's responsibility ( i. e., not that of an individual program manager or Peer Review Leader). Generally, to allow flexibility, individual offices and regions will decide the appropriate level of organizational responsibility and how they will meet the " routinely available" requirement. The peer review record may be kept with other records relating to the overall project, as long as it is easily and separately identifiable. The peer review record should be maintained in accordance with the Agency's record keeping schedule for such records. One long term archiving mechanism may be the formal archiving at the Federal Records Center in Suitland, MD. ( Note: This is the same question as section 2.5.7, but applies in this chapter as well). Peer Review Handbook Page 65 SUBJECT INDEX This is an alphabetical listing of subjects from the Handbook and the pertinent page numbers where they are found.

 A  Administrative Procedures Act ( 12) Designated Agency Ethics Official ( 48) Administrator ( 9, 17) Division Directors ( 17) Advisory and Assistance Services ( 53) Docket ( 38, 39, 63) Analytic blueprints ( 13, 32, 40)  E  Annual call letter ( 15) Economic work products ( 27) Annual reporting requirements ( 14, 18) Electronic records ( 14, 16, 38) Anonymity of peer reviewers ( 62) Environmental Impact Statement ( EIS) ( 29) Archiving ( 39, 63) Environmental regulatory models ( 29) Assistant Administrators ( 17) External peer review ( 33, 34, 46)  B   F  Balancing of peer reviewers ( 21) Federal Acquisition Regulation ( FAR) ( 53) Board of Scientific Counselors ( BOSC) ( 34) Federal Advisory Committee Act ( FACA) Budget ( 18, 32, 40) ( 41, 49)  C  Final work product ( 61) Categories of work products ( 15)  G  Charge ( 19, 37, 43) Grants ( 28) Chemical action reports ( 16) Gratuitous services ( 52) Clean Air Scientific Advisory Committee  I  ( 34) Independent peer reviewer ( 10, 12, 21, 45) Coding system ( 16) Inherently governmental functions ( IGFs) Compensation ( 52) ( 54) Competition ( 57) Interacting with peer reviewers ( 51) Completing a peer review ( 4, 61) Interagency agreement ( 28) Conducting a peer review ( 3, 43) Internal peer review ( 33) Conference proceedings ( 25) Internal peer reviewer ( 21) Confidential business information ( CBI) ( 49,  J  56) Journal article ( 16, 35) Confidential Financial Disclosure ( 47)  L  Confidentiality ( 51) Legal considerations ( 40) Conflicts of interest ( 21, 28, 47, 54) Line managers ( 17) Contracting officer ( 50, 52, 54, 56) List A ( 14) Contracts ( 28, 55) List B ( 15, 26) Contracts Management Manual ( CMM) ( 52, List C ( 15) 53) Litigation ( 15, 62) Controversy ( 27, 32)  M  Cooperative agreements ( 28) Major impact ( 27, 32) Court ordered deadlines ( 31, 36, 49) Major scientific and technical work product  D  ( 5, 9, 15, 25, 26, 30) Decision Maker ( 14, 16, 17, 26, 32) Managers planning checklist for peer review Deputy Administrator ( 14, 17, 18) ( 5) Peer Review Handbook Page 66 Materials for peer reviewers ( 49) Scientific and technical work product ( 25) Mechanisms for peer review ( 32) Selection of peer reviewers ( 45)  N  Sensitive activities ( 53) National Academy of Sciences ( 34) Signatures ( 14, 16) NEPA Products ( 29) Simplified acquisitions of peer review Non major scientific and technical work services ( 55, 57) products ( 15, 30, 31) Site specific decisions ( 28) Novelty ( 27, 32) Special Government Employee ( SGE) ( 59) NPDES permits ( 16) Stakeholder involvement ( 12)  O  Statement of work for contracts ( 53) Office Directors ( 17) Subject matter expert ( 22) Office of Research and Development ( ORD)  T  ( 14, 16, 20, 23) Tier 1 and Tier 2 ( 13, 44) Open meeting ( 41) Tier 3 ( 13)  P  Time frame for peer review ( 36, 44) Peer consultation ( 11) Travel ( 58) Peer input ( 11)  V  Peer involvement ( 10) Voluntary services ( 52) Peer review ( 9, 10) Peer Review Advisory Group ( PRAG) ( 17) Peer review comments ( 19, 38, 61) Peer Review Coordinator ( 14, 17, 19, 22) Peer Review Leader ( 18, 18, 32, 39, 48, 49, 61, 63) Peer review mechanism ( 5, 32) Peer review panel ( 22, 45, 46) Peer review record ( 5, 20, 36, 37, 63) Peer review schedule ( 18) Peer review services ( 52) Peer review summary report ( 14, 15, 18, 38) Peer reviewers ( 21, 21, 38, 49, 56) Personal services ( 56) Planning a peer review ( 2, 25) Premanufacturing notices ( PMNs) ( 16) Procurement requests ( 57) Project manager ( 13, 18, 30) Public comment ( 12)  R  RCRA permits ( 16) Record of Decision ( ROD) ( 25) Regional Administrators ( 17) Regulatory negotiations ( 14) Rulemaking ( 13, 27, 38)  S  Science Advisory Board ( SAB) ( 34, 45) Science Policy Council ( SPC) ( 14, 17, 20, 22) Scientific Advisory Panel ( SAP) ( 34, 45) Peer Review Handbook Page 67 COMMONLY USED ACRONYMS AA Assistant Administrator AAS Advisory and Assistance Services BOSC Board of Scientific Counselors CBI Confidential Business Information CFR Code of Federal Regulations CMM Contracts Management Manual CO Contract Officer COI Conflict of Interest EIS Environmental Impact Statement EPA Environmental Protection Agency EPAAR EPA Acquisition Regulations FACA Federal Advisory Committee Act FAR Federal Acquisition Regulations FOIA Freedom of Information Act FTE Full Time Equivalent GSA General Services Administration HHS Health and Human Services IGF Inherently Governmental Function IRIS Integrated Risk Information System LAN Local Area Network NEPA National Environmental Policy Act NPDES National Pollutant Discharge Elimination System OAM Office of Acquisition Management OGC Office of General Counsel OMB Office of Management and Budget ORD Office of Research and Development PMNs Premanufacture Notice PRAG Peer Review Advisory Group RA Regional Administrator RC Regional Counsel RCRA Resource Conservation and Recovery Act RIAs Regulatory Impact Analyses ROD Record of Decision SAB Science Advisory Board SAP Scientific Advisory Panel SGE Special Government Employee SOPs Standard Operating Procedures SOW Statement of Work SPC Science Policy Council WTI Waste Technologies Industries Peer Review Handbook Page 68 THIS PAGE LEFT INTENTIONALLY BLANK Peer Review Handbook Page A 1 APPENDIX A U. S. Environmental Protection Agency Peer Review Policy June 7, 1994 Peer Review Handbook Page A 2 June 7, 1994 MEMORANDUM SUBJECT: Peer Review Program TO: Assistant Administrators General Counsel Inspector General Associate Administrators Regional Administrators Staff Office Directors Today, I am reaffirming the central role of peer review in our efforts to ensure that EPA policy decisions rest on sound, credible science and data ( see attached policy statement). Toward that end, as its first major task, EPA's Science Policy Council ( SPC) is instituting a program to expand and improve peer review in all EPA offices. This memorandum gives an overview of current practices and outlines the new program.

Peer Review Practices and Policy Peer review at EPA takes several different forms, ranging from informal consultations with Agency colleagues who were not involved in developing the product to the formal, public processes of the Science Advisory Board ( SAB) and the FIFRA Scientific Advisory Panel ( SAP). In any form, peer review assists the Agency's work by bringing independent expert experience and judgment to bear on issues before the Agency to the benefit of the final product.

EPA's Peer Review Policy, which responds in part to recommendations in the " Credible Science, Credible Decisions" report, outlines general principles for peer review at EPA. Different EPA offices have undertaken various implementing activities, including an Agencywide information and planning workshop, internal guideline development, and numerous specific peer reviews. Even with these activities, however, I am concerned that EPA does not yet have a comprehensive Agency wide program for implementing its Peer Review Policy. I therefore welcome the SPC initiative toward effective, efficient implementation of the policy in all the program areas to which it applies. Peer Review Handbook Page A 3 Expanding and Improving Peer Review The Science Policy Council and its Steering Committee have outlined a dual track implementation program of planning and assistance for all Agency offices. The first track has three major milestones.

First, during the next few weeks, Steering Committee members will consult with senior management in each office to exchange information on current peer review activities, assistance needed, possible obstacles to implementation, and implementation planning.

Second, using information and materials developed during the first stage, peer review task groups in each office will develop standard operating procedures ( SOPs) for use in each office, based in part on generic guidance to be issued by the SPC and in part on peer review needs and capabilities specific to each office. The resulting SOPs will delineate as appropriate the scope of application of peer review with respect to various types of scientific and technical work products such as reports of original research, risk assessments, and analytical methods of economic analysis. OARM and OGC staff will assist each office as needed on legal, budget and administrative matters. Each AA and RA will submit draft SOPs for Steering Committee review by July 15.

Third, the SPC review group will work with each office to complete each plan by September 15.

In parallel with the above, consistent with the Peer Review Policy, the Science Policy Council will work with each AA and RA to identify " major scientific and technical work products" as peer review candidates for the coming year. This process will consider existing and new plans for internal reviews and for SAB, FIFRA SAP, and other external reviews. The two fold objective is to plan reviews for technical products covered by the Peer Review Policy and to gain experience with options and obstacles. We will use this experience to review and revise the SOPs as needed. Also, to establish a baseline for comparison, each AA and RA will identify the " major technical products" completed within his/ her program during the past 12 months.

The Science Policy Council has sent additional information to each office offering guidance on the procedures that you are asked to develop and the schedule for these activities. Please note, however, that because the policy is effective immediately, current peer review planning should continue on present schedules in parallel with developing the formal SOPs. Peer Review Handbook Page A 4 To begin this process, I have asked each Assistant Administrator and Regional Administrator to designate a Peer Review Coordinator to work with the Steering Committee on implementation activities specific to each office. I am very pleased that the Science Policy Council is taking this important step. A comprehensive peer review program is essential to maintaining and improving the quality of the analyses that underlie Agency actions. I look forward to working with you and your staff on this important activity.

/ s/

Carol M. Browner Attachment cc: Science Policy Council Science Policy Council Steering Committee Peer Review Handbook Page A 5 PEER REVIEW AND PEER INVOLVEMENT AT THE U. S. ENVIRONMENTAL PROTECTION AGENCY This document establishes the policy of the United States Environmental Protection Agency ( EPA) for peer review of scientifically and technically based work products that are intended to support Agency decisions. Peer review is presented in the context of the broader concept, peer involvement.

BACKGROUND The report " Safeguarding the Future: Credible Science, Credible Decisions" focused on 1 the state of science at EPA. The panel of experts who prepared the report emphasized the importance of peer review, especially external peer review, and the need for broader and more systematic use of it at EPA to evaluate scientific and technical work products. Their specific recommendation regarding peer review reads as follows:

" Quality assurance and peer review should be applied to the planning and results of all scientific and technical efforts to obtain data used for guidance and decisions at EPA, including such efforts in the program and regional offices. Such a requirement is essential if EPA is to be perceived as a credible, unbiased source of environmental and health information, both in the United States and throughout the world."

In response to this recommendation, then Administrator Reilly directed staff to develop an EPAwide policy statement, which he issued in January, 1993. The paragraphs below preserve the core of that earlier statement while updating it to specify the role of the Science Policy Council in guiding further implementation of the policy. Effective use of peer review is indispensable for fulfilling the EPA mission and therefore deserves high priority attention from program managers and scientists within all pertinent Headquarters and Regional Offices.

EPA/ 600/ 9 91/ 050, March 1992. 1 Peer Review Handbook Page A 6 PEER INVOLVEMENT AND PEER REVIEW EPA strives to ensure that the scientific and technical underpinnings of its decisions meet two important criteria: they should be based upon the best current knowledge from science, engineering, and other domains of technical expertise; and they should be judged credible by those who deal with the Agency. EPA staff therefore frequently rely upon peer involvement  that is, they augment their capabilities by inviting relevant subject matter experts from outside the program to become involved in one or more aspects of the development of the work products that support policies and actions.

One particularly important type of peer involvement occurs when scientifically and technically based work products undergo peer review  that is, when they are evaluated by relevant experts from outside the program who are peers of the program staff, consultants, and/ or contractor personnel who prepared the product. Properly applied, peer review not only enriches the quality of work products but also adds a degree of credibility that cannot be achieved in any other way. Further, peer review early in the development of work products in some cases may conserve future resources by steering the development along the most efficacious course.

Peer review generally takes one of two forms. The review team may consist primarily of relevant experts from within EPA, albeit individuals who have no other involvement with respect to the work product that is to be evaluated ( internal peer review). Or the review team may consist primarily of independent experts from outside EPA ( external peer review).

POLICY STATEMENT Major scientifically and technically based work products related to Agency decisions normally should be peer reviewed. Agency managers within Headquarters, Regions, laboratories, and field components determine and are accountable for the decision whether to employ peer review in particular instances and, if so, its character, scope, and timing. These decisions are made in conformance with program goals and priorities, resource constraints, and statutory or court ordered deadlines. For those work products that are intended to support the most important decisions or that have special importance in their own right, external peer review is the procedure of choice. Peer review is not restricted to the penultimate version of work products; in fact, peer review at the planning stage can often be extremely beneficial.

SCOPE Agency managers routinely make regulatory and other decisions that necessarily involve many different considerations. This policy applies to major work products that are primarily scientific and technical in nature and may contribute to the basis for policy or regulatory decisions. By contrast, this policy does not apply to non major or nontechnical matters that Peer Review Handbook Page A 7 Agency managers consider as they make decisions. Similarly, this policy does not apply to these ultimate decisions.

This policy applies where appropriate, as determined by the National and Regional Program Managers, to major scientifically and technically based work products initiated subsequent to the date of issuance. Peer review should be employed to the extent reasonable to relevant work products that currently are under development. This policy does not apply to the bases for past decisions, unless and until the relevant scientific and technical issues are considered anew in the Agency's decision making processes.

Except where it is required by law, formal peer review ( as distinguished from the Agency's normal internal review procedures) should be conducted in a manner that will not cause EPA to miss or need extension of a statutory or court ordered deadline. Agency managers still may undertake peer review if it can be conducted concurrently with necessary rulemaking steps.

LEGAL EFFECT This policy statement does not establish or affect legal rights or obligations. Rather, it confirms the importance of peer review where appropriate, outlines relevant principles, and identifies factors Agency staff should consider in implementing the policy. On a continuing basis, Agency management is expected to evaluate the policy as well as the results of its application throughout the Agency and undertake revisions as necessary. Therefore, the policy does not stand alone; nor does it establish a binding norm that is finally determinative of the issues addressed. Minor variations in its application from one instance to another are appropriate and expected; they thus are not a legitimate basis for delaying or complicating action on otherwise satisfactory scientific, technical, and regulatory products.

Except where provided otherwise by law, peer review is not a formal part of or substitute for notice and comment rulemaking or adjudicative procedures. EPA's decision whether to conduct peer review in any particular case is wholly within the Agency's discretion. Similarly, nothing in this policy creates a legal requirement that EPA respond to peer reviewers. However, to the extent that EPA decisions rely on scientific and technical work products that have been subjected to peer review, the remarks of peer reviewers should be included in the record for that decision.

IMPLEMENTATION The Science Policy Council is responsible for overseeing Agency wide implementation. Its responsibilities include promoting consistent interpretation, assessing Agency wide progress, and developing recommendations for revisions of the policy as necessary. Peer Review Handbook Page A 8 The Science Policy Council will oversee a peer review work group, which will include representatives from program units throughout EPA to effect a consistent, workable implementation of the policy. The work group will assist the programs in ( 1) formulating and, as necessary, revising standard operating procedures ( SOPs) for peer review consistent with this policy; ( 2) identifying work products that are subject to review; and ( 3) for each major work product, selecting an appropriate level and timing of peer review.

In assisting the programs, the work group will take into account statutory and court deadlines, resource implications, and availability of disinterested peer reviewers. The group will work closely with Headquarters offices and the Regional Offices toward ensuring effective, efficient uses of peer review in supporting their mission objectives. However, the Assistant Administrators and Regional Administrators remain ultimately responsible for developing SOPs, identifying work products subject to peer review, determining the type and timing of such review, documenting the process and outcome of each peer review, and otherwise implementing the policy within their organizational units.

Because peer review can be time consuming and expensive, Agency managers within Headquarters, Regions, laboratories, and field components are expected to plan carefully with respect to its use  taking account of program priorities, resource considerations, and any other relevant constraints as well as the policy goal of achieving high quality, credible underpinnings for decisions. External peer reviewers should be chosen carefully to ensure an independent and objective evaluation. The affiliations of peer reviewers should be identified on the public record, so as to avoid undercutting the credibility of the peer review process by conflicts of interest.

The policy is effective immediately. The peer review work group mentioned above will identify the focal point to whom comments and questions should be addressed and, from time to time, will provide further information about implementation activities.

/ s/ APPROVED: DATE: JUN 7 1994 CAROL M. BROWNER, ADMINISTRATOR Peer Review Handbook Page B 1 APPENDIX B Examples of Charges Please note  certain questions that are posed in charges can be responded to with a yes or no answer. Clearly, this is not the type of response we generally want, therefore, it is important to phrase charge questions carefully to ensure that you receive a fully satisfactory and thoughtful response. Where a yes or no answer might be expected, be sure to ask for a full explanation supporting the yes or no answer. Peer Review Handbook Page B 2 CHARGE EXAMPLE 1  CHARGE TO REVIEWERS FOR THE WTI DRAFT FINAL RISK ASSESSMENT The draft final WTI risk assessment is divided into several volumes covering the scientific disciplines of toxicology, environmental fate and transport, combustion engineering, atmospheric modeling, exposure assessment, ecological risk assessment, and accident analysis. As a reviewer of the WTI draft final risk assessment, you should use your best technical knowledge and professional judgment to comment on the technical accuracy, completeness and scientific soundness of the assessment. Each reviewer is asked to focus on several specific issues in his or her area of expertise with comments on other areas invited but optional. Your comments will be considered in finalizing the risk assessment.

For the peer review workshop reviewers will be organized into 5 work groups: Combustion Engineering, Air Dispersion and Deposition Modeling and Accident Analysis, Toxicology, Exposure Assessment, and Ecological Risk Assessment. All reviewers should be familiar with the Executive Summary ( Volume I) and the Facility Background ( Volume II) sections of the draft risk assessment. In addition, each work group should focus on specific Volumes as specified below:

Workgroup Risk Assessment Volumes Combustion Engineering Volume III  Facility Emissions Air Dispersion and Volume IV  Atmospheric Volume VII  Accident Deposition Modeling and Dispersion and Deposition Analysis Accident Analysis Modeling Toxicology Volume V  Human Health Volume VII  Accident Risk Assessment Analysis Exposure Assessment Volume V  Human Health Volume VII  Accident Risk Assessment Analysis Ecological Risk Assessment Volume VI  Screening Ecological Risk Assessment Peer Review Handbook Page B 3 While reviewing these sections of the document, please address the following general issues.

1) Comment on the organization of the risk assessment document. Does the layout follow a logical format? Is the presentation of information in the document clear, concise and easy to follow?

2) Does the executive summary accurately reflect the data and methodologies used and the conclusions derived in the risk assessment?

3) Were the major recommendations of the 1993 peer review workshop for the risk assessment plan addressed?

4) As with any risk assessment, there are always additional data and method development efforts that could be undertaken to reduce the level of uncertainty. However, are there any major data or methodological gaps that would preclude the use of this risk assessment for decision making? If so, how should they be addressed?

5) What long term research would you recommend that could improve risk assessments of this type in the future?

In addition, the following workgroup specific issues should be addressed.

Emissions Characterization Emissions characterization includes identification of substances of concern and the development of emission rates for these contaminants. Emission rates were developed through a combination of site specific stack test data and models. Please comment on the following issues with respect to this aspect of the draft risk assessment.

1) To characterize the nature of the emissions, waste stream profiles were developed and entered into a database. Several refinements and adjustments ( e. g., the Subtraction Correction Factor for chlorinated compounds) were applied to the profiles before substances of concern were identified. Please comment on whether or not these adjustments are appropriate. What is the anticipated effect on the risk assessment?

2) Comment on the selected chemicals of concern. Have important chemicals been missed due to the selection technique? Peer Review Handbook Page B 4 3) Comment on the approaches used to estimate stack emission rates ( e. g., use of the 95% UCL of the arithmetic mean or the maximum detected value, whichever is smaller, for high end emission rates). Are the approaches appropriate? Are their effects on the risk assessment adequately characterized? Comment on the adjustment made to PCDD/ PCDF emission rates to account for brominated dioxin like compounds. Also, comment on the approach to characterizing emission rates from fugitive sources ( e. g., use of the TANKS 2 model for the Carbon Adsorption Bed).

4) Comment on the identified sources of fugitive emissions. Was the approach used to select these sources appropriate? Have important sources been missed? Have emissions from process upsets been given appropriate consideration?

5) There have been a number of controlled burns at the WTI facility. Please comment on the adequacy of these data in estimating potential exposure. Please comment on the assumptions made from the tests in regard to composition of wastes received at WTI and emissions when the plant operates in the future.

6) Comment on the use of emission factors from coal burning to estimate the emission rate of fly ash from WTI. Are the factors used to adjust the coal emission rate appropriate? Are the uncertainties introduced from this approach adequately characterized?

7) Overall, is the identification of the key assumptions used in characterizing the nature and magnitude of emissions thorough? Are the magnitude and direction of effect of these assumptions on the overall risk assessment accurately characterized? Is the uncertainty and variability inherent in this analysis adequately discussed? Does the sensitivity analysis cover the major parameters expected to have an effect on the risk assessment?

Dispersion and Deposition Modeling To develop this risk assessment, computer models have been used with site specific data on emission rates and meteorological conditions to simulate the air concentrations and deposition rates for contaminants potentially emitted from the WTI facility. The models used include the Industrial Source Complex  Complex Terrain Deposition ( ISC COMPDEP), the CALPUFF, and the INPUFF models. In your review, please address the following issues.

1) Since the 1993 peer review of the risk assessment plan, a number of efforts have been completed to reduce the uncertainty associated with the air dispersion and deposition modeling. These efforts include the collection of site specific data for Peer Review Handbook Page B 5 emission rates and meteorological conditions. Also, a wind tunnel study was conducted to evaluate the effects of the complex terrain surrounding the WTI facility. Does the risk assessment document adequately summarize these activities? Is the link between these data collection efforts, the air dispersion models, and the risk assessment clearly established?

2) The results of 12 sets of sensitivity tests indicate that geophysical variables ( e. g., terrain) are more likely to affect dispersion and deposition than emission variables ( e. g., stack temperature). Were these sensitivity analyses adequate? Comment on the conclusions reached. To further examine the effect of geophysical variables, wind tunnel testing was conducted to model the terrain induced flow effects expected near WTI. It was concluded that changes in peak concentrations attributed to these effects are relatively minor and that the ISC COMPDEP model is sufficiently conservative. Comment on this conclusion. Have these analyses helped to characterize and/ or reduce the uncertainty in the air dispersion modeling associated with the complex terrain surrounding WTI.

3) The ISC COMPDEP model does not allow for non steady state conditions such as calm winds and strong temperature inversions. Therefore, CALPUFF was used to estimate air dispersion and deposition under these conditions. However, CALPUFF gave similar peak, 24 hour, and annual average concentrations as ISCCOMPDEP Comment on the adequacy of this analysis. Comment on the conclusions reached. Has this analysis helped to characterize and/ or reduce the uncertainty in the air dispersion modeling associated with non steady state meteorological conditions?

4) Atmospheric dispersion modeling was used to estimate air concentrations of hazardous chemicals for the accident analysis. The SLAB model was used for vapor releases from spills and the mixing of incompatible wastes. ISCCOMPDEP was used for releases associated with fires. Comment on the selection of the models and inputs. Are they appropriate selections?

5) Overall, have adequate sensitivity tests been conducted to demonstrate the magnitude of variation in concentrations and deposition estimates with model inputs? Please explain fully.

Human Health Risks Human Health Risk Assessment includes hazard identification, dose response evaluation, exposure assessment, and risk characterization. To develop the risk assessment, potentially exposed populations have been identified and the magnitude, frequency, and duration of their Peer Review Handbook Page B 6 exposure quantified. This information was then integrated with the hazard identification and dose response evaluation for the risk characterization. For this risk assessment, both carcinogenic and non carcinogenic health effects have been evaluated. In your review, please comment on the following issues.

Exposure 1) EPA's Exposure Assessment Guidelines identify certain exposure descriptors that should be used to characterize exposure estimates. The Guidelines define high end exposure estimates as those representing individuals above the 90th percentile on the exposure distribution but not higher than the individual in the population who has the highest exposure. Bounding exposure estimates are those that are higher than the exposure incurred by the person in the population with the highest exposure. Central tendency exposure estimates are defined as the best representation of the center of the exposure distribution ( e. g., arithmetic mean for normal distributions). Comment on whether or not the WTI exposure assessment properly characterizes each of the exposure estimates in terms of these descriptors.

2) The factors that go into estimating a central tendency or high end exposure, once the population has been defined, include the environmental media concentration, the intake rate, and the duration and/ or frequency of exposure. Comment on whether or not the WTI exposure assessment does an adequate job of describing the logical procedure of combining these factors to develop central tendency, high end, and/ or bounding estimates of exposure for each of the exposed subpopulations.

3) An important factor in an exposure assessment is identifying all of the important exposure sources. Please comment on the adequacy of the WTI assessment in identifying the important sources and pathways of exposure.

4) Have the key assumptions for estimation of chemical concentration and for estimation of exposure been identified? Are the magnitude and direction of effect correct for the assumptions that have been identified?

5) Supposedly, conservative assumptions have been applied in this assessment to account for uncertainty. Are the conservative assumptions appropriately factored into the ultimate characterization of what descriptor best applies to each exposure estimate? Please comment on whether the uncertainties were confronted in an adequate manner. If they were not, please state what should be done differently. Peer Review Handbook Page B 7 Hazard Identification/ Dose Response and Risk Characterization 1) To select surrogate compounds for quantitative risk assessment, a two step process was used in which chemicals were ranked on the basis of emission rate, toxicity ( both cancer and non cancer), and bioaccumulation potential. Please comment on this selection process. Are the ranking factors appropriate? Could important compounds have been omitted from the analysis based on the ranking procedure?

2) For the majority of the chemicals of concern, traditional approaches to dose response evaluation were employed ( e. g., use of a slope factor for cancer and use of a RfD/ RfC for non cancer). However for certain chemicals or groups of compounds a different methodology was used. Specifically, dioxins, furans, PAHs, lead, mercury, nickel, chromium, acid gases, and particulate matter were given special consideration. Please comment on the methodology used for these compounds. Was it appropriate? Have the uncertainties associated with the methodology been adequately characterized? Comment on the assumptions used due to a lack of chemical specific data.

3) Please comment on the selection of the overall population and the various subpopulations at risk. Were site specific data, such as the informal home gardening survey, properly utilized to identify these subpopulations?

4) It is stated in the risk assessment that average risk estimates are based on average emission rates, average air dispersion/ deposition within a subarea, and typical exposure factors. Further, maximum risks are based on average emission rates, typical exposure factors, and the maximum air concentration within a subarea. Please comment on this use of the terms average and maximum risks. Are these descriptive terms appropriate given the parameters used to derive each? Please explain fully.

5) Comment on whether or not the non cancer risks of chemicals of concern have been adequately addressed by the risk assessment? For example, has an adequate discussion of endocrine disrupters been provided which either characterizes their risks or clearly explains why their risks cannot be characterized? Further, have non cancer chronic toxicities of dioxins and furans been adequately addressed in the risk assessment?

6) Please comment on whether or not the uncertainties associated with the additivity and/ or synergy of risks from pollutants emitted together from the WTI facility are adequately discussed in the risk assessment. Peer Review Handbook Page B 8 7) Have the key assumptions for estimation of dose and risk been identified? Are the magnitude and direction of effect correct for the assumptions that have been identified? Please comment on whether the uncertainties were confronted in an adequate manner. If they were not, please state what should be done differently.

8) Please comment on the overall adequacy of the risk characterization. Does the risk characterization include a statement of confidence in the risk assessment including a discussion of the major uncertainties. Are the hazard identification, dose response assessment, and exposure assessment clearly presented? Have sufficient risk descriptors which include important subgroups been presented and discussed?

Screening Ecological Risk Assessment As with the human health risk assessment, the ecological risk assessment pulls together elements of exposure analysis and dose response evaluations to develop a risk characterization. For the Screening level Ecological Risk Assessment ( SERA), Ecological Chemicals of Concern ( ECOC) and indicator species have been identified to provide conservative estimates of risk. Please address the following issues in your review.

1) Are there any components of the SERA which you feel undermine the scientific validity of the assessment? If so, what are they and can you provide suggestions to strengthen the identified components?

2) Is the organization of the document clear and does it present the material in a clear and concise manner consistent with the Framework for Ecological Risk Assessment ( EPA, 1992)? Please explain fully.

3) Uncertainties are discussed in numerous sections of the SERA and compose Section VIII of the SERA. In each case, do these discussions cover all relevant and important aspects of the uncertainties which you think should be addressed in the SERA?

4) In your opinion, what is the weakest and what is the strongest aspect of the SERA? Can you make any suggestions on how the weakest parts can be strengthened by the Agency?

5) In Section II, are the stressors, ecological effects, and both the assessment and measurement endpoints adequately characterized? Are the five emission scenarios adequate to characterize the exposures for the WTI facility? Are there other emission scenarios which you think should be included in the SERA? Peer Review Handbook Page B 9 6) In Section III, is the site characterization adequate to support the SERA? Why or why not?

7) In Section IV, is the tiered process used to identify the ecological chemicals of concern ( ECOC) from the initial list of potential chemicals considered scientifically defensible? Does application of this tiered approach support the statement made in the SERA " by focusing on the potential risk from the selected ECOCs, the SERA provides a thorough screening level evaluation for the WTI facility?"

8) In Sections V and VI, are the exposure and ecological effects adequately characterized? Are the most appropriate estimation techniques available used? Are the assumptions clearly stated? Please explain fully.

9) In Section VIII, are there any major elements missing from the risk characterization which you think need to be included or which would strengthen the risk characterization? Does the risk characterization support the summary and conclusions presented in Section IX?

10) In Section IX, given the assumptions made and the processes used to select and evaluate chemicals, receptors, and exposure pathways, do you think the SERA adequately met its objective of not inadvertently underestimating risk?

Accident Analysis The Accident Analysis for the WTI incinerator involves evaluating the probability of an emergency incident occurring which results in the release of hazardous waste. The consequences of this release are also evaluated using exposure and human health effects information. Unlike the human health risk assessment which has a primary goal of quantifying risks, the accident analysis typically provides information that can be used to reduce the likelihood, extent and impact of possible accidents. Please comment on the following issues in your review of this aspect of the risk assessment.

1) The WTI accident assessment selected five scenarios for quantitative evaluation that were considered to be of primary concern. The scenarios are an on site spill, an on site fire, an on site mixing of incompatible waste, an off site spill, and an off site spill and fire. Please comment on the selection of these scenarios. Were any significant scenarios missed?

2) Specific chemicals were selected to evaluate each scenario. Please comment on the selections. Would other chemicals have been more appropriate? Peer Review Handbook Page B 10 3) Chemical specific release rates are calculated for each scenario. Please comment on the procedures used to estimate the release rates. Was an appropriate approach used?

4) Atmospheric dispersion modeling was used to estimate air concentrations of hazardous chemicals. Specifically, the SLAB model was used for vapor releases from spills and the mixing of incompatible wastes. ISC COMPDEP was used for releases associated with fires. Comment on the selection of the models and inputs Are they appropriate selections? Should other models or inputs been used?

5) Please comment on the assessment's conclusions on the severity of consequences and probability of occurrence. Has the report correctly categorized the severity of the consequences of the different accident scenarios? Has the assessment adequately justified the reported probability of occurrence of each of the accident events?

6) Key assumptions were made in the identification of accident scenarios and the description of the conservative and typical events. Included were a description of the magnitude of the effect of the assumptions and direction of the effect. Please comment on the assumptions. Are they justified? Are the descriptions of the magnitude and directions of the effects correct? Has the accident assessment adequately confronted the uncertainties involved in doing this type of analysis? If not, what else should be done?

7) Comment on the appropriateness of using IDLH values for characterizing the severity of consequences in the accident analysis. Comment on the appropriateness of using 10 X LOC for chemicals for which IDLH values have not been established.

8) In the accident analysis, IDLH ( or 10 X LOC) values were used to determine the downwind distances over which adverse human health effects might occur. To evaluate the uncertainty introduced by using the IDLH, a sensitivity analysis was conducted where these distances were recalculated using the LOC ( a more stringent health criteria). Other sources of uncertainty that are identified in the accident analysis include concentration averaging times, chemical concentrations, emission rates, and meteorological conditions. For most of these parameters it is stated that conservative assumptions were used to avoid underestimating risks. Have the uncertainties inherent in the accident analysis been adequately characterized? For those parameters where sensitivity analyses were not conducted, is the conclusion that conservative assumptions have avoided underestimation valid? Peer Review Handbook Page B 11 CHARGE EXAMPLE 2  IRIS Pilot Program Instructions to Peer Reviewers for Reviewing IRIS Summaries and Supporting Documentation The U. S. EPA is conducting a peer review of the scientific basis supporting the health hazard and dose response assessments for the subject chemical that will appear on the Agency's online database, the Integrated Risk Information System ( IRIS). Materials to be reviewed include the summary information that will appear on IRIS ( the inhalation reference concentration [ RfC], oral reference dose [ RfD], and cancer assessment) and the supporting document, the Toxicological Review, which will also be made available to the public.

A listing of Agency Guidelines and Methodologies that were used in the development of these hazard and dose response assessments included the following: The Risk Assessment Guidelines ( 1986), the ( new) Proposed Guidelines for Carcinogen Risk Assessment ( 1996), Guidelines for Developmental Toxicity Risk Assessment, ( proposed) Interim Policy for Particle Size and Limit Concentration Issues in Inhalation Toxicity, ( proposed) Guidelines for Neurotoxicity Risk Assessment, Methods for Derivation of Inhalation Reference Concentrations and Application of Inhalation Dosimetry, Recommendations for and Documentation of Biological Values for Use in Risk Assessment and Use of the Benchmark Dose Approach in Health Risk Assessment. Copies of these documents ( and/ or their relevant sections) will be made to the reviewer upon request.

Peer review is meant to ensure that science is used credibly and appropriately in derivation of these dose response assessments. You have been chosen as an expert on the chemical under consideration, on a scientific discipline related to at least one of the assessments, or in the field of risk assessment. At least three peer reviewers per chemical are being chosen to review the scientific basis of these draft dose response assessments before they are forwarded on to the EPA's Consensus Process for final approval and adoption by the EPA. These hazard and dose response assessments will then appear on IRIS and become available as Agency consensus health effect information.

The primary function of the peer reviewer should be to judge whether the choice, use, and interpretation of data employed in the derivation of the assessments is appropriate and scientifically sound. This review is not of the recommended Agency risk assessment guidelines or methodologies used to derive cancer or RfD/ C assessments as these have been reviewed by external scientific peers, the public, and EPA Science Advisory Boards. The reviewer's comments on the application of these guidelines/ methodologies within the individual assessments is, however, welcomed and encouraged. For example, the reviewer may ascertain whether or not there is data sufficient to support use of other than default assumptions for areas such as sensitive subpopulations or linear cancer extrapolation. The reviewer may also have Peer Review Handbook Page B 12 opinions on other areas of uncertainty such as subchronic to chronic duration ( when only a subchronic study is available) or an incomplete data base but should focus on the specific area of uncertainty rather than on the magnitude of the overall estimate.

Below are two groups of questions regarding this review. The first is a set of general questions that are meant to guide you through your review. It is not imperative that you specifically answer each question of this group. The second group of questions, however, are specific for the chemical assessments and deal with areas of scientific controversy or uncertainty in which the Agency may have to make a scientific judgment. Your input to this set of questions is considered vital to the review process.

Questions for IRIS Peer Reviewers  General 1) Are you aware of any other data/ studies that are relevant ( i. e., useful for the hazard identification or dose response assessment) for the assessment of the adverse health effects, both cancer and noncancer, of this chemical? Please explain fully.

2) For the RfD and RfC, has the most appropriate critical effect been chosen ( i. e., that adverse effect appearing first in a dose response continuum)? For the cancer assessment, are the tumors observed biologically significant? relevant to human health? Points relevant to this determination include whether or not the choice follows from the dose response assessment, whether the effect is considered adverse, and if the effect ( including tumors observed in the cancer assessment) and the species in which it is observed is a valid model for humans.

3) Have the noncancer and cancer assessments been based on the most appropriate studies? These studies should present the critical effect/ cancer ( tumors or appropriate precursor) in the clearest dose response relationship. If not, what other study ( or studies) should be chosen and why?

4) Studies included in the RfD and RfC under the heading " Supporting/ Additional studies" are meant to lend scientific justification for the designation of critical effect by including any relevant pathogenesis in humans, any applicable mechanistic information, any evidence corroborative of the critical effect, or to establish the comprehensiveness of the data base with respect to various endpoints ( such as reproductive/ developmental toxicity studies). Should other studies be included under the " Supporting/ Additional" category? Should some studies be removed? Peer Review Handbook Page B 13 5) For the noncancer assessments, are there other data that should be considered in developing the uncertainty factors or the modifying factor? Do you consider that the data support use of different ( default) values than those proposed?

6) Do the Confidence statements and weight of evidence statements present a clear rationale and accurately reflect the utility of the studies chosen, the relevancy of the effects ( cancer and noncancer) to humans, and the comprehensiveness of the data base? Do these statements make sufficiently apparent all the underlying assumptions and limitations of these assessments? If not, what needs to be added?

Questions for IRIS Peer Reviewers  Chemical Specific [ example: cumene]

1) Based on the information noted in the Principal study currently designated ( Cushman et al., 1995) is the discounting of the renal effects in males justified? Is sufficient rationale given to let stand the organ weight changes in female rats as a critical effect?

2) Is the information in the Toxicological Review sufficient to consider cumene as having a low potential for causing reproductive effects? Please explain fully.

RECOMMENDATIONS Based on your reading and analysis of the information provided, please identify your overall recommendation for the IRIS materials you have reviewed as 1) Acceptable as is 2) Acceptable with minor revision ( as indicated) 3) Acceptable with major revision ( as outlined) 4) Not acceptable Peer Review Handbook Page B 14 CHARGE EXAMPLE 3  Science Advisory Board Review of the Agency's National Risk Management Research Laboratory's ( NRMRL) Program The Office of Research and Development ( ORD) requests that the Science Advisory Board review the Agency's National Risk Management Research Laboratory's ( NRMRL) program.

In the " Strategic Plan for the Office of Research and Development" ( EPA, 1996a), ORD described the relationship of risk assessment to the risk management process, and emphasized the need for scientific and engineering research to enable sound risk management decisions and actions. Within the framework of that strategic plan, NRMRL's mission is to conduct research to reduce uncertainties and costs associated with making and implementing environmental risk management decisions. NRMRL has therefore developed a research agenda to reduce risk uncertainty that also focuses on those important, relevant issues where it can make a difference.

The charge to the SAB is to:

1) Examine and critique the research programmatic directions such as whether NRMRL is pursuing the most appropriate research problem areas;

2) Comment on strategic directions, e. g. use of its core technical competencies, transition from primarily extramural to an intramural R& D organization, leveraging with other agencies and organization;

3) Review and comment on the effectiveness of NRMRL's approach to science management, e. g. measures of success and science quality, soundness of peer review process;

4) Examine and critique the relationship of NRMRL's risk management research and its intended role in the risk assessment/ risk management paradigm; and 5) Review and comment on the strategic balance for the next decade among pollution prevention, technology development, remediation, and risk management assessment activities. Peer Review Handbook Page B 15 CHARGE EXAMPLE 4  Science Advisory Board ( SAB) Review of the Technical Aspects of the Multi Agency Radiation Survey and Site Investigation Manual ( MARSSIM)

The EPA Science Advisory Board ( SAB) is asked to review the technical aspects of the Multi Agency Radiation Survey and Site Investigation Manual ( MARSSIM). The review document was developed collaboratively by four Federal agencies, departments and commissions having authority for control of radioactive materials: Department of Defense, Department of Energy, Environmental Protection Agency, and Nuclear Regulatory Commission. MARSSIM addresses the need for a nationally consistent approach to conducting radiation surveys of potentially radioactively contaminated sites that are being considered for release to the public. A condition of release is a demonstration that residual radioactivity levels do not exceed a specified risk or dose level, also known as a release criterion. MARSSIM provides guidance to users performing and assessing the results of such a demonstration for surface soils and building surfaces.

The SAB is asked by the Agency's Office of Radiation and Indoor Air ( ORIA) to respond to the following charge in its review:

1) Is the overall approach to the planning, data acquisition, data assessment, and data interpretation as described in the MARSSIM technically acceptable? Please explain fully.

2) Are the methods and assumptions for demonstrating compliance with a dose or risk based regulation technically acceptable? Please explain fully.

3) Are the hypotheses and statistical tests and their method of application appropriate? Please explain fully. Peer Review Handbook Page B 16 CHARGE EXAMPLE 5  Science Advisory Board ( SAB) Review of the Statistical Performance of the Agency's Protozoan Oocyst Monitoring Methods The Science Advisory Board ( SAB) is asked to review a report describing the statistical performance of the Agency's protozoan oocyst monitoring methods. Agency staff recognized that the protozoan analysis methodology that had been formally adopted for the Information Collection Rule ( ICR) was crude and had very low and highly variable recoveries of added oocysts. The statistical methodology was considered by Agency staff to be necessary to determine whether the Agency can take advantage of the large monitoring program agreed to under the negotiated rulemaking process.

The Agency charge asks that the SAB evaluate the report and address the following concerns:

1) Evaluate the factual and conceptual soundness of the approach and methods used, and the soundness of the results and conclusions of the report.

2) Evaluate the viability of the assumptions and conditions tested in the report.

3) Evaluate the suitability of the report as a basis for making a decision on the use of protozoan monitoring data for a national impact assessment.

4) Evaluate whether the degree of accuracy and precision of the protozoan method is acceptable for an impact analysis. Peer Review Handbook Page B 17 CHARGE EXAMPLE 6  Science Advisory Board ( SAB) Review of the Environment Monitoring and Assessment Program ( EMAP) Research Strategy and Research Plan The Science Advisory Board ( SAB) is asked to review the Environment Monitoring and Assessment Program ( EMAP) Research Strategy and Research Plan. The review is requested by the Office of Research and Development ( ORD) with the following specific charge issues:

1) Previous peer reviews recommended that EMAP develop a close working relation with EPA Program Offices and other federal monitoring efforts. Does the EMAP strategy support the [ Office of Science and Technology Policy's Committee on Environment and Natural Resources] CENR National Monitoring Framework and EPA Program Offices?

2) Previous peer reviews recommended that EMAP initiate a focused research program on indicator development. Does the intramural EMAP program on ecological indicator development, coordinated with the [ ORD Science To Achieve Results] STAR solicitations, respond to this research need?

3) Previous peer reviews recommended that the EMAP design be modified to include a set of nonrandomly selected sentinel sites with intensive data collection. Does the development of Index Sites as outdoor laboratories in the national parks ([ National Park Service] NPS and [ US Geological Survey] USGS) and selected estuaries ([ National Oceanic and Atmospheric Administration] NOAA) add this dimension to the EMAP?

4) Previous peer reviews recommended that EMAP combine effects oriented and stressor oriented monitoring approaches. Do the focused geographic demonstration pilot studies ( initially in the Mid Atlantic region) combine these elements? Peer Review Handbook Page B 18 THIS PAGE LEFT INTENTIONALLY BLANK Peer Review Handbook Page C 1 APPENDIX C GUIDANCE ON REQUESTING A REVIEW BY THE SCIENCE ADVISORY BOARD ( SAB) Peer Review Handbook Page C 2 Guidance on Requesting a Review By the Science Advisory Board ( SAB)

Summary The Science Advisory Board ( SAB) annually solicits proposals for review projects every spring for the following fiscal year. This appendix provides guidance to Programs and Regions to help them submit requests for SAB reviews. Requests should be submitted to the Science Advisory Board in both hard copy and electronic versions usually by mid June for the following fiscal year. The requests may be part of the annual submissions that respond to the peer review activities of the Agency, or they may be submitted directly to the Board. Although providing requests at one defined time in the spring helps with SAB planning, we recognize that projects also come to light during other parts of the year. Please contact the SAB staff for details on making submissions during the remainder of the year ( see end of this document for contacts).

Background A key priority for Administrator Browner is to base Agency actions on sound scientific data, analyses, and interpretations. She issued the Agency's Peer Review Policy to increase the quality of the technical foundations upon which EPA's regulatory structures are built. The SAB is a key scientific peer review mechanism available to Programs and Regions in implementing the Peer Review Policy. However, because the Board has finite resources it cannot conduct all reviews. This document is designed to help Programs and Regions determine which projects to submit to the SAB. Note particularly that the SAB focuses on the technical underpinnings of Agency positions; i. e., risk assessment issues, in contrast to risk management issues.

The topics that are best suited for the Board's agenda are those that satisfy several of the following criteria:

1) Integrate science into Agency actions in new ways. 2) Influence long term technological developments. 3) Impact overall environmental protection. 4) Address novel scientific problems or principles. 5) Address problems that transcend federal agency or other organizational boundaries. 6) Strengthen the Agency's basic capabilities. 7) Serve Congressional or other leadership interests. 8) Deal with controversial issues.

In suggesting issues for SAB involvement, Programs and Regions should note the breadth of SAB activities: Peer Review Handbook Page C 3 Historically, most of the outputs of the Board are in the form of " full" reports. They present the findings of peer reviews of Agency document( s) and contain considerable detail about the findings and recommendations of the Board. They also address the specific questions posed by the Charge to the Board. " Letter" reports fulfill the same function as reports, but are simply shorter in length. Due to the need to be more responsive with advice, the Board has recently begun to produce more short letter reports than full reports, as they can be produced and finalized in less time.

The SAB has also introduced the " Consultation" as a means of conferring in public session with the Agency on a technical matter before the Agency has begun substantive work on that issue. The goal is to leaven EPA's thinking on an issue by brainstorming a variety of approaches to the problem very early in the development process. There is no attempt or intent to express an SAB consensus or to generate an SAB report. The Board, via a brief letter simply notifies the Administrator that a Consultation has taken place.

More recently, the Board introduced a new vehicle for communicating with its clients  the " Advisory"  which provides, via a formal SAB consensus report, critical input on technical issues that arise during the Agency's issue development process. The Advisory generally involves a review of a multi year Agency project. The intent is to provide some mid course assessment to see if the Agency is heading in a scientifically credible direction. In order to maintain an objective, arms length relation with the Agency and its projects, the SAB review of the final product at some point in the future will include experts who did not participate in producing the Advisory.

The Agenda Setting Process Each Assistant Administrator and Regional Administrator is normally asked to submit a list of candidate topics for SAB action/ review. A " project sheet" ( see attached example) is used to define each topic that is nominated for SAB review. The project sheet is prepared by the requesting office and contains the following information:

1) Project title/ subject ( Descriptive short title of project)

2) Requesting Organization/ Office ( Primary office requesting review, AA/ RA level)

3) Requesting Official ( Name and position of senior official requesting review, usually office or division level  this is the person who may receive a summary briefing from the Chair following the review) Peer Review Handbook Page C 4 4) Program Contact ( Name/ phone number/ mail code  this is the principal contact for SAB Staff to interact with during development of the SAB review)

5) Background ( brief history of the project and why it is important)

6) Tentative Charge ( what the SAB is being asked to comment on  usually a set of questions)

7) Tentative Schedule and Committee ( when the review is expected to be conducted, e. g., Winter 1998; and which SAB committee is appropriate for the review  final choice as to the review committee is at the discretion of the Board)

8) Budget Estimate ( Rough estimates of Agency funding for the subject over the past 5 years ( if applicable) and for the next 2 years ( if applicable). The SAB Executive Committee has asked for this information to help it better appreciate the level of Agency involvement in and commitment to the issue)

9) Preparer ( name, phone, office of preparer of Project Sheet and the date prepared)

The proposed topics will be examined and discussed in a number of forums:

1) The individual SAB Committees  Throughout the late Spring and Summer, the SAB Committees will be examining options for the following fiscal year, including all suggestions made by the Agency.

2) The Science Policy Council Steering Committee ( SPC SC)  The SPC SC usually meets in early summer to examine the proposals for each fiscal year. The goal is to provide cross office critique/ integration of the proposals. The SPC SC will be used as a forum for continuing discussion throughout the process and throughout the year as new topics emerge.

3) The Deputy's perspective  In the summer, the Deputy Administrator will review the requests and provide insights on priorities.

4) The SAB Executive Committee  During its summer meeting, the SAB's EC will examine the nominated topics, adding its own perspective on an appropriate agenda, using its selection criteria.

5) The Administrator  In September, the list of proposed topics will be delivered to Administrator Browner for information and added insights. Peer Review Handbook Page C 5 The completed project sheets should be submitted electronically to the SAB Deputy Staff Director ( fowle. jack@ epamail. epa. gov) and in signed hard copy ( mail code 1400).

SAB Staff Contacts Dr. Donald Barnes ( 202 260 4126)  Staff Director; Designated Federal Officer ( DFO) for the Executive Committee:

Dr. Jack Fowle ( 202 260 8325)  Deputy Staff Director.

Mr. Robert Flaak ( 202 260 5133)  Team Leader, Committee Operations Staff; DFO for the Clean Air Scientific Advisory Committee ( CASAC); DFO for the Research Strategies Advisory Committee ( RSAC).

Ms. Kathleen Conway ( 202 260 2558)  DFO for the Environmental Engineering Committee ( EEC).

Dr. Jack Kooyoomjian ( 202 260 2560)  DFO for the Radiation Advisory Committee ( RAC); DFO for the Advisory Council on Clean Air Compliance Analysis ( Council).

Mr. Tom Miller ( 202 260 5886)  DFO for the Drinking Water Committee ( DWC); DFO for the Environmental Economics Advisory Committee ( EEAC).

Mr. Sam Rondberg ( 202 260 2559) and LCDR Roslyn Edson ( PHS)  DFOs for the Environmental Health Committee ( EHC); DFOs for the Integrated Human Exposure Committee ( IHEC)

Ms. Stephanie Sanzone ( 202 260 6557)  DFO for the Ecological Processes and Effects Committee ( EPEC).

All SAB staff can be contacted via Agency Email. Peer Review Handbook Page C 6 Science Advisory Board Proposed Project Project title/ subject: Proposed Amendments to the Risk Assessment Guidelines for Carcinogens Requesting Organization/ Office: Office of Research and Development ( ORD)

Requesting Official: Name, Title, Office/ Organization Program Contact: Name, Title, Office/ Organization, 202 260 xxxx Background: EPA's Health Risk Assessment Guidelines provide generic science and science policy guidance on risk assessment issues for use in all Agency offices. EPA has currently issued or proposed nine guidelines ( or amendments) in this series, all of which have been submitted to the Science Advisory Board for review.

The current guidelines for carcinogen risk assessment were reviewed by the SAB and issued as final guidance in 1986. In 1988, the Forum initiated a public process for considering amendments to these guidelines. A Risk Assessment Technical Panel considered submissions from the public as well as information developed by experts at two public workshops in revising these guidelines.

Tentative Charge: Review the amended and expanded guidance, with special emphasis on ( a) weight of evidence issues, ( b) a new classification system, ( c) dose response modeling, and ( d) the use of pharmacokinetic and metabolic data. A more detailed charge will be negotiated with SAB at a later date.

Tentative Schedule and Committee: Winter, 1998, Environmental Health Committee Budget: FY 1995  $ xxx and yy FTE FY 1996  $ xxx and yy FTE FY 1997  $ xxx and yy FTE FY 1998  $ xxx and yy FTE FY 1999  FY2000  estimated costs of $ xxx and yy FTE each year Preparer: Name, Title, Office/ Organization, 202 260 xxxx Date: June 1, 1997 Peer Review Handbook Page D 1 APPENDIX D Example Statements of Work for Contracts Peer Review Handbook Page D 2 STATEMENT OF WORK  EXAMPLE 1  Statement of Work: Technical Review Contractor for Panel Review of Assistance Agreement or Fellowship Applications 1) Purpose The purpose of this contract is to purchase peer review services of a contractor with expertise in Exploratory Research; Environmental Chemistry. The services are for peer reviewing applications received in response to the Office of Research and Development's ( ORD's) 1997 Science to Achieve Results program. These reviews shall be completed and the evaluation sheets shall be prepared prior to the reviewer's participation in a 3 day panel discussion to be held in Washington, D. C. on May 5  7, 1997.

2) Statement of Work ORD's National Center for Environmental Research and Quality Assurance ( NCERQA) is responsible for overseeing the recently expanded research grants and fellowships programs. Each year NCERQA ( alone or in conjunction with other organizations) solicits applications in each of these programs. The applications to be reviewed under this contract were submitted in response to the solicitation for the 1997 Science to Achieve Results program. As part of the selection process, NCERQA must conduct a peer review that is designed to evaluate the scientific quality of each application; this is accomplished through the ad hoc use of technical experts.

The peer review services required by this contract necessitate the independent review of a maximum of 10 applications and the preparation of a typed evaluation summary and an overall rating for each of these applications. Each evaluation summary shall support and be consistent with the overall rating that is assigned; it also shall be completed prior to the contractor's participation in the panel discussions. After the panel discussions for the applications assigned to the contractor, the contractor shall submit all completed evaluation summaries to the designated Science Review Administrator ( SRA).

The contractor also shall serve as the panel's rapporteur for approximately 6  8 of the applications assigned. As rapporteur, the contractor shall be responsible for preparing a typed evaluation summary ( on site typing support will be provided by NCERQA) that reflects the panel's discussion of the respective application as well as the panel's overall rating ( the criteria for the panel's overall rating are the same as those for each peer reviewer's overall rating). As rapporteur, the contractor shall submit these panel evaluation summaries to the designated SRA prior to leaving the panel meeting. Peer Review Handbook Page D 3 Before the contractor shall be allowed to participate in the review process, the contractor shall have disclosed any actual or potential conflicts of interest and shall have signed and submitted to NCERQA a Conflict of Interest/ Confidentiality Form. The contractor is directed to assure that none of the conflicts disclosed are so direct and substantial as to rule out a particular reviewer. Upon receipt of an approved Purchase Order ( PO), NCERQA will send the following items to the contractor:

a) A copy of the Purchase Order or the Purchase Order number b) The applications assigned to the contractor c) For grants, a set of abstracts for all the applications being reviewed by the panel d) For persons reviewing grant applications, a copy of the pertinent section( s) of the solicitation package to provide background information; for persons reviewing fellowship applications, information on how to access the solicitation package on the Internet e) A sample evaluation form to help the contractor prepare an acceptable evaluation form for each assigned application f) A blank evaluation form for each assigned application and the criteria for completing the form and determining the overall rating g) A blank and sample invoice as well as instructions for completing and submitting the invoice to EPA h) Information on the points of contact for additional information ( e. g., NCERQA's SRA) i) Logistics information on the location and time of the panel discussions NCERQA will transmit the above items under a cover letter. In this cover letter, NCERQA will provide additional details about each item, including ( as needed) more specific instructions for the set of applications assigned to the reviewer.

Each contractor shall be responsible for making his/ her own travel reservations for hotel and transportation.

3) Reviewer Tasks a) Review the assigned applications using the guidance provided with NCERQA's evaluation form.

b) Submit completed evaluation forms to the SRA designated in the cover letter immediately following the panel discussions for the applications assigned to the contractor. THE COMPLETED FORMS MUST BE TYPED, AND THE EVALUATION SUMMARY FOR EACH APPLICATION MUST SUPPORT AND BE CONSISTENT WITH THE OVERALL RATING THAT IS ASSIGNED BY THE CONTRACTOR. IN SITUATIONS Peer Review Handbook Page D 4 WHERE THESE CONDITIONS ARE NOT MET, THE SRA WILL ASK THE CONTRACTOR TO REDO THE FORM.

c) For those applications for which the contractor is serving as the panel's rapporteur, submit a completed panel evaluation summary to the SRA designated in the cover letter prior to leaving the panel meeting. THE PANEL'S EVALUATION SUMMARY MUST BE TYPED ( ON SITE TYPING SUPPORT WILL BE PROVIDED BY NCERQA) AND BE CONSISTENT WITH THE PANEL'S OVERALL RATING. IN SITUATIONS WHERE THESE CONDITIONS ARE NOT MET, THE SRA WILL ASK THE CONTRACTOR TO REDO THE FORM.

d) Make own airline and hotel accommodations for participation in the panel review meeting. Round trip air fare must be a commercial REFUNDABLE ticket. Peer Review Handbook Page D 5 STATEMENT OF WORK  EXAMPLE 2  Peer Review of Prioritization Tool Report Work Assignment No.:

Title: Peer Review of Prioritization Tool Report Work Assignment Manager ( WAM):

Name: John Q. Government Employee Address: Office of Solid Waste Phone No.: ( 202) 260 XXXX Background:

The Waste Minimization Branch ( WMB) in the Office of Solid Waste ( OSW) is in the process of implementing the Waste Minimization National Plan, announced by the Agency on November 18, 1994. The Plan reaffirms the Agency's commitment to promote source reduction over waste management, in keeping with the policy stated in the 1984 amendments to the Resource, Conservation, and Recovery Act ( RCRA) and in the 1990 Pollution Prevention Act ( PPA). The Plan outlines major goals, objectives, and action items to achieve national reductions in the generation of hazardous wastes.

One of the objectives of the Plan is to: " develop a framework for setting national priorities; develop and distribute a flexible screening tool for identifying priorities at individual facilities; [ and] identify constituents of concern." This objective is a key building block in implementing subsequent objectives of the Plan.

In September 1995, WMB formed the Waste Minimization Prioritization Team, which includes representatives from EPA regions and states, to implement this objective. The Team has worked to assess stakeholder needs for prioritization tools and to evaluate prioritization tools that are currently available. The Team plans to summarize this work, along with its recommendations, in a report ( referred to herein as the Prioritization Tool report) that would be available in draft form in July 1996.

WMB and the Team wish to obtain independent peer review of the Prioritization Tool report prior to briefing EPA management. The report is being prepared with the support of ICF, Inc.; therefore, for the peer review to be considered independent, it must be performed by another contractor. Peer Review Handbook Page D 6 Purpose and Scope of Work:

The purpose of this work assignment is to provide support to WMB and the Team in finalizing the Prioritization Tool report by conducting an independent peer review of the report.

Work Statement:

Task 1  Management work plan and budget Within 15 days of CO approval of this work assignment, the contractor shall deliver a management work plan including a proposed level of effort, schedule, and budget for all tasks.

Task 2  Provide independent peer review of Prioritization Tool report The contractor shall provide support to WMB and the Team in preparing the Prioritization Tool report by performing an independent peer review of the report. The contractor shall establish a panel of peer reviewers including three senior level persons who collectively have extensive expertise in particular areas to be identified by the WAM upon approval of the work assignment.

Within three weeks of work assignment approval, receipt from the WAM of the necessary qualifications of peer reviewers ( in a TD), and receipt from the WAM of the peer review " charge" ( in a TD), whichever comes latest, the contractor shall identify the three peer reviewers and prepare a memo that lists the names of the peer reviewers and their affiliations and includes the peer reviewers' bio's. Within five weeks of WAM approval of the of the peer reviewers ( via a TD) and receipt of the draft Prioritization Tool report from the WAM ( via a TD), whichever comes later, the contractor shall conduct the peer review, assemble the peer review comments and recommendations in a peer review report organized by charge question, prepare an introduction to the peer review report with a clear and concise overview of the comments, and attach to the peer review report any marginal comments the peer reviewers had on the Prioritization Tool report.

It is not necessary that the peer reviewers jointly reach consensus on their findings and recommendations, since there may be limited overlap in the peer reviewers' areas of expertise and in the charge questions that they focus on. The contractor shall assume, for the purpose of estimating costs, that the draft Prioritization Tool report is roughly 100 pages in length with 200 pages of appendices, and that each peer reviewer will spend 40 hours in reviewing the report and writing comments. EPA plans to provide the report to the contractor in mid July. Peer Review Handbook Page D 7 Deliverables and Schedule:

Task Deliverable Schedule 1 Work plan and budget Within 15 days of CO approval of work assignment 2 Memo identifying peer reviewers Within 3 weeks of work assignment approval, receipt of peer reviewer qualifications from WAM, and receipt of charge from WAM, whichever comes latest 3 Peer review report Within five weeks of WAM approval of peer reviewers and receipt of draft Prioritization Tool report from WAM, whichever comes later Other Requirements:

CONTRACTOR COMMUNICATIONS Upon approval of the Work Plan, the contractor shall maintain at least weekly communications with the Work Assignment Manager regarding the status of work on the Work Assignment.

CONFLICT OF INTEREST The contractor must adhere to the following requirements:

1) Upon receipt of a Work Assignment, QRT, or similar tasking document, and prior to commencement of any work, notify both the CO and PO of any actual or potential organizational or personal conflicts of interest.

2) Provide a written certification, within 20 days of receipt of a Work Assignment, QRT, or similar tasking document, that:

a) Either all actual or potential organizational conflicts of interest have been reported to the CO or that no actual or potential organizational conflicts of interest exist. The contractor is directed to assure that none of the conflicts disclosed are so direct and substantial as to rule out a particular reviewer.

b) All personnel who perform work under this Work Assignment or relating to this Work Assignment have been informed of their obligation to report personal and organizational conflicts of interest to the CO. Peer Review Handbook Page D 8 c) The Contractor recognizes its continuing obligation to identify and report any actual or potential conflicts of interest arising during performance of this Work Assignment.

3) If an actual or potential organizational conflict of interest is identified during performance under this Work Assignment, the Contractor shall immediately make a full disclosure in writing to the CO. The disclosure shall include a description of action which the Contractor has taken or proposes to take, after consultation with the CO, to avoid, mitigate, or neutralize the actual or potential conflict of interest.

EXPENDITURE OF FUNDS/ HOURS In addition to the requirements of the contract, the contractor shall notify both the Project Officer and the Work Assignment Manager when 75% of funds or hours for this Work Assignment have been expended.

INFORMATION COLLECTION Any other provision of this Work Assignment notwithstanding, the contractor shall not proceed with any information collection where the same or similar information will be collected from ten or more public respondents until written approval is received from the Contracting Officer. This approval will cite an approval number from the Office of Management and Budget as required by the Paperwork Reduction Act ( PRA).

Only Federal agencies and their employees are exempt from the PRA definition of " public respondent." State agencies and their employees are classified as " public respondents."

Soliciting similar information applies to any collection method, i. e., written, oral, electronic, etc., and utilizing any approach, i. e., surveys, phone calls, focus groups, TQM, etc. The PRA applies equally to " willing participants" and participation that is mandated by law.

Any question of applicability of the PRA shall be resolved by submitting a complete description of the circumstances in a written request to the Contracting Officer. No collection shall be undertaken until the Contracting Officer provides written notice to the Contractor as to the applicability of the PRA. If the PRA is determined to be applicable, the Contractor shall not initiate any collection until the requisite approval is received.

The General Services Administration ( GSA), under FIRMR Bulletin B 2 administers the Interagency Reports Management Program as derived from 44 U. S. C. Chapters 29 and 31. All work performed under this Work Assignment involving federal interagency reporting must be done in full compliance with these GSA procedures. Peer Review Handbook Page D 9 CONFIDENTIAL BUSINESS INFORMATION If this Work Assignment requires use of RCRA Confidential Business Information ( CBI), the contract must specifically authorize the contractor to have access to RCRA CBI and the contractor shall abide by all RCRA CBI requirements and stipulations found in the RCRA CBI Security Manual and in the contract. The contractor shall identify in the Work Plan budget all estimated costs for dealing with CBI requirements. All CBI must be returned to EPA as soon as it is no longer needed under this Work Assignment or before the expiration of the Work Assignment, whichever occurs first.

PRINTING AND DUPLICATION The contractor is prohibited from performing any printing under the Government Printing and Binding Regulations. Duplication is allowed to the extent it does not exceed 5,000 impressions of a single page document or 25,000 impressions of a multiple page stand alone document, is limited to one color ( black) copies, and does not exceed the maximum image size of 10 3/ 4 by 14 1/ 4 inches. For all duplication jobs in excess of 5,000 impressions, the EPA WAM will determine in advance if the work can be performed more cost effectively and under the job or time constraints at the EPA Print Shop. If the total number of photocopies for this Work Assignment exceeds 5,000 impressions, the contractor shall identify in their Work Plan the photocopying costs by task and deliverable.

WORK ASSIGNMENT/ WORK PLAN BUDGETS The contractor shall not exceed either the dollar or PL hour budget contained in the approved Work Plan. In addition, on Quick Response Tasks ( QRTs) the contractor shall not exceed the PL hour budget of the QRT.

TECHNICAL DIRECTION The Designated Work Assignment Manager ( WAM) on this Work Assignment is authorized to provide technical direction to the extent allowed under EPAAR ( 1552.237 71) ( APR 1984) ( DEVIATION). Other than the Designated WAM, only the Project Officer and the Contracting Officer are authorized to provide technical direction.

Technical direction includes:

( 1) Direction to the contractor which assists the contractor in accomplishing the Statement of Work ( 2) Comments on and approval/ acceptance of reports or other deliverables Peer Review Handbook Page D 10 Technical direction must be within the contract and the Work Assignment statement of work. The Project Officer and the WAM do not have the authority to issue technical direction which ( 1) institutes additional work outside the scope of either the contract or this Work Assignment; ( 2) constitutes a change as defined in the " Changes" clause; ( 3) causes an increase or decrease in the estimated cost of the contract or Work Assignment; ( 4) alters the period of performance or deliverable due dates; or ( 5) changes any of the other express terms or conditions of the contract or Work Assignment.

Technical direction will be issued in writing or confirmed in writing within five ( 5) calendar days after verbal issuance. The technical direction memorandum will be provided to the contractor and copies will be forwarded to the Contracting Officer and the Project Officer. If the contractor has not received written confirmation within five ( 5) calendar days of a oral issuance, the contractor must so notify the Project Officer.

INHERENTLY GOVERNMENTAL FUNCTIONS The contractor shall not perform any inherently governmental functions ( IGF) under this Work Assignment. If during the course of developing the plan of work, through receipt of technical direction, or in carrying out the assignment any portion of the effort is considered to possibly be an inherently governmental function, the contractor must immediately notify the Project Officer and the Contracting Officer.

OCCUPATIONAL HEALTH AND SAFETY Facility site visits conducted under a Work Assignment that include on site inspections or sampling must be conducted in full compliance with the Department of Labor, Occupational Safety, and Health Administration rules under 29 CFR Part 1910 and EPA Order 1440 ( Occupational Health and Safety Manual).

TRAVEL COSTS The contractor shall follow the requirements of Subpart 31.2 of the FAR and the Federal regulations in incurring allowable travel costs under this Work Assignment, and correspondingly must at all times seek and obtain Government rates whenever available and observe current subsistence ceilings.

QUICK RESPONSE TASKS Each Quick Response Task ( QRT) shall be confirmed in writing and approved by the Project Officer. The contractor shall respond by letter to the PO with copies to the WAM and the Peer Review Handbook Page D 11 CO within two working days, giving a brief description of the plan of work, including best estimate of hours ( by P level) and a break out of costs to accomplish the task.

No task shall exceed a duration of 30 calendar days from start date to completion date. The level of effort for each task shall be limited to a maximum of 250 labor hours.

Quick Response Task Requests do not change the dollar or professional labor hour budgets of a Work Assignment. Peer Review Handbook Page D 12 STATEMENT OF WORK  EXAMPLE 3  External Peer Review of Protozoa Method Development Criteria Document Period of Performance: Work Plan Approval to August 1, 1997 Work Assignment Manager: Sally Q. Government Employee Office of Water U. S. Environmental Protection Agency LOE: 196 hours SOW: 2.4 BACKGROUND INFORMATION:

The United States Environmental Protection Agency ( EPA), Office of Water is charged with protecting public health and the environment from adverse exposure to chemicals and microbials in water media, such as ambient and drinking waters, wastewater/ sewage sludge and sediments. In support of this mission OW's Office of Science and Technology ( OST) develops health standards, health criteria, health advisories, and technical guidance documents for water and water related media. Under this work assignment, documents prepared by OST are to undergo peer review.

Peer review is an important component of the scientific process. It provides a focused, objective evaluation of a research proposal, publication, risk assessment, health advisory, guidance or other document submitted for review. The criticism, suggestions and new ideas provided by the peer reviewers stimulate creative thought, strengthens the reviewed document and confer credibility on the product. Comprehensive, objective peer reviews leads to good science and product acceptance within the scientific community.

Under this work assignment, the contractor will receive one document ( Protozoa Method Development Criteria Document) for peer review which is related to human health and ecological effects.

STATEMENT OF WORK:

Task 1. The contractor shall develop a work plan to address all tasks in this work assignment. The work plan shall describe the steps that will be taken by the contractor to provide for peer review, including selection of peer reviewer candidates with appropriate expertise, determining absence of conflict of interest, document and reference distribution, establishing schedules, preparing the peer review report, and submittal of the peer review package. Curriculum vitae for all Peer Review Handbook Page D 13 persons assigned to complete this work assignment shall be provided. All P levels, hours and total costs for each task will be provided and costs greater than $ 100.00 shall be itemized in detail.

Task 2. The contractor shall select a group of peer reviewers and determine their availability for the task and absence of conflict of interest, and establish a schedule for the peer review. The contractor is directed to assure that none of the conflicts disclosed are so direct and substantial as to rule out a particular reviewer. Three peer reviewers shall participate in the review. No single peer reviewer may charge more than 40 hours to this task. It is fully acceptable for peer reviewers to commit to less than 40 hours. The peer review will be conducted for the Protozoa Method Development Criteria Document. Reviewers selected by and working for the contractor shall be approved by the EPA Project Officer in writing prior to their beginning work. Minimally, all peer reviewers shall be accomplished in protozoan methods for sample recovery and analysis from water. Approval submissions shall include the reviewers' names and curriculum vitae.

Task 3. The contractor shall arrange for the selected peer reviewers to review the EPA document. Prepare the charge to the peer reviewers based on technical direction received from the EPA WAM. Provide the peer reviewers with copies of the candidate report and all relevant references and instruct the selected peer reviewers to undertake the review. The WAM will provide the contractor with the final version of the document to be reviewed.

Task 4. The contractor shall monitor peer reviewers' progress to assure timely completion. The contractor shall collate peer review comments, and organize the comments in the peer review " for comments" document. Provide the peer review document and all materials submitted by the peer reviewers to the EPA WAM.

SCHEDULE AND DELIVERABLES:

Task 1. ( Work Plan) 15 days after receipt of work assignment Task 2. 1 week after work plan approval Task 3. 1 week after selection of peer reviewers Task 4. 1 week after receiving comments from the peer reviewers TRAVEL: No travel is anticipated under this work assignment. Any travel directly chargeable to this work assignment must be submitted and approved by the project officer. Peer Review Handbook Page D 14 THIS PAGE LEFT INTENTIONALLY BLANK Peer Review Handbook Page E 1 APPENDIX E References Concerning Peer Review American Chemical Society and the Conservation Foundation ( 1985) Issues in Peer Review of the Scientific Basis for Regulatory Decisions, Washington, DC, November 1985.

Browner, C. ( 1994) Peer Review Program, Washington, DC, Memorandum issued June 7, 1994. ( NOTE: Attached as Appendix A to this Handbook)

Chubin, D. ( 1994) Grants peer review in theory and practice, Evaluation Review 18: 20 30.

Chubin, D. and E. Hackett ( 1990) Peerless Science: Peer Review and Science Policy, Albany, NY: State University of New York Press.

Jasanoff, S. ( 1990) The Fifth Branch: Science Advisors as Policymakers, Cambridge, MA: Harvard University Press.

Kostoff, R. ( 1996) Peer Review in Selected Federal Agencies, presented at AAAS Annual Meeting, Baltimore, MD, February 9, 1996.

National Environmental Policy Institute ( 1996) Enhancing the Integrity and Transparency of Science in the Regulatory Process, Washington, DC: National Environmental Policy Institute, Fall 1996.

National Research Council ( 1995) Interim Report of the Committee on Research and Peer Review in EPA, Washington, DC: National Academy Press, March 1995.

Reilly, W. ( 1993) Peer review Policy, Washington, DC, Memorandum issued January 19, 1993.

Science Advisory Board ( 1992) Safeguarding the Future: Credible Science, Credible Decisions, Washington, DC, SAB Report issued March 1992.

Spitzer, H. ( 1995) Peer Review Practices in the Federal Government, Bethesda, MD: Environmental Network, report prepared for the American Industrial Health Council, April 26, 1995) Peer Review Handbook Page E 2 U. S. General Accounting Office ( 1994) Peer Review: EPA Needs Implementation Procedures and Additional Controls, GAO/ RCED 94 89, Washington, DC: U. S. Government Printing Office, February 1994.

U. S. General Accounting Office ( 1996) Peer Review: EPA's Implementation Remains Uneven, GAO/ RCED 96 236, Washington, DC: U. S. Government Printing Office, September 1996. Peer Review Handbook Notes and Comments Peer Review Handbook Notes and Comments Peer Review Handbook Notes and Comments Peer Review Handbook Notes and Comments