Science Policy Council HANDBOOK United States Office of Science Policy EPA 100 B 98 001 Environmental Protection Office of Research and Development January 1998 Agency Washington, DC 20460 www. epa. gov THIS PAGE LEFT INTENTIONALLY BLANK EPA 100 B 98 001 January 1998 U. S. Environmental Protection Agency PEER REVIEW HANDBOOK Prepared for the U. S. Environmental Protection Agency by members of the Peer Review Advisory Group, a group of EPA's Science Policy Council Principal Authors Kerry L. Dearfield, Ph. D. A. Robert Flaak Office of Science Policy Science Advisory Board Office of Research and Development Office of the Administrator Major Contributors Jean C. Schumann Nancy W. Wentworth Office of Solid Waste Office of Research and and Emergency Response Development Roland B. Hemmett, Ph. D. Arnold M. Kuzmack, Ph. D. Region 2 Office of Water Science Policy Council U. S. Environmental Protection Agency Washington, DC 20460 Peer Review Handbook Page ii DISCLAIMER This document has been reviewed in accordance with U. S. Environmental Protection Agency policy and approved for publication and distribution to the Agency. Mention of trade names or commercial products does not constitute endorsement or recommendation for use. Peer Review Handbook Page iii TABLE OF CONTENTS FOREWORD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page ix SUMMARY OF THE PEER REVIEW PROCESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 1 Figure 1  Flowchart for Planning a Peer Review . . . . . . . . . . . . . . . . . . . . . . . . . . Page 2 Figure 2  Flowchart for Conducting a Peer Review . . . . . . . . . . . . . . . . . . . . . . . . Page 3 Figure 3  Flowchart for Completing a Peer Review . . . . . . . . . . . . . . . . . . . . . . . . Page 4 Managers Planning Checklist for Peer Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 5 PEER REVIEW GUIDANCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 7 1. THE NEED FOR PEER REVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 9 1.1 Overview Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 9 1.2 Understanding Peer Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 9 1.2.1 Why use Peer Review? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 9 1.2.2 What is Peer Involvement? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 10 1.2.3 What is Peer Review? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 10 1.2.4 What is Peer Input? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 11 1.2.5 How is Peer Review Different from Peer Input? . . . . . . . . . . . . . . Page 11 1.2.6 Can Someone Who Provided Peer Input Become an Independent Peer Reviewer for the Same Work Product Later in the Process? Page 12 1.2.7 How is Peer Review Different from Public Comment? . . . . . . . . Page 12 1.2.8 How is Peer Review Different from Stakeholder Involvement? . . Page 12 1.2.9 What Role does Peer Review have in the Regulatory Development Process? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 13 1.2.10 What Role does Peer Review have in Regulatory Negotiations? . Page 14 1.3 Annual Agency Reporting Requirements . . . . . . . . . . . . . . . . . . . . . . . . . Page 14 1.3.1 What are the Annual Reporting Requirements? . . . . . . . . . . . . . . Page 14 1.3.2 What Listings are Required for the Annual Reporting? . . . . . . . . Page 14 1.3.3 Is There a Coding System for Work Products on the Annual Reporting Lists? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 16 1.3.4 When will the Handbook Itself be Revised? . . . . . . . . . . . . . . . . . Page 17 1.4 The Roles of People and Organizations in Peer Review . . . . . . . . . . . . . . Page 17 1.4.1 Who is Ultimately Accountable for Peer Review? . . . . . . . . . . . . Page 17 1.4.2 Who are the Agency Staff involved in Peer Review? . . . . . . . . . . Page 17 Peer Review Handbook Page iv 1.4.3 Who are the Decision Makers & What are Their Responsibilities? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 17 1.4.4 Who are the Peer Review Leaders & What are Their Responsibilities? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 18 1.4.5 Who are the Peer Review Coordinators & What are Their Responsibilities? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 19 1.4.6 Who are the Peer Reviewers? . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 21 1.4.7 What are the Responsibilities of Peer Reviewers? . . . . . . . . . . . . Page 21 1.4.8 What is an Independent Peer Reviewer? . . . . . . . . . . . . . . . . . . . . Page 21 1.4.9 When does an Agency Internal Peer Reviewer Qualify as Independent? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 21 1.4.10 What is a Peer Review Panel? . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 22 1.4.11 What is a Subject Matter Expert? . . . . . . . . . . . . . . . . . . . . . . . . . Page 22 1.4.12 What is the Role of the Science Policy Council ( SPC)? . . . . . . . Page 22 1.4.13 What is the Role of the Peer Review Advisory Group ( PRAG)? . Page 22 1.4.14 What is the Role of the Office of Research and Development ( ORD)? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 23 2. PLANNING A PEER REVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 25 2.1 Overview Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 25 2.2 Determining Which Work Products to Peer Review . . . . . . . . . . . . . . . . . Page 25 2.2.1 What are Scientific and Technical Work Products? . . . . . . . . . . . Page 25 2.2.2 What Scientific and Technical Work Products Need Peer Review? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 25 2.2.3 How Does One Determine Whether a Scientific and/ or Technical Work Product is " Major"? . . . . . . . . . . . . . . . . . . . . . . Page 26 2.2.4 What Economic Work Products Need Peer Review? . . . . . . . . . . Page 27 2.2.5 How Should Peer Review be Handled for Products Developed under an Interagency Agreement? . . . . . . . . . . . . . . . . Page 28 2.2.6 Should Products from Grants, Contracts and Cooperative Agreements Receive Peer Review? . . . . . . . . . . . . . . . . . . . . . . . . Page 28 2.2.7 Should Site Specific Decisions be Subject to Peer Review? . . . . . Page 28 2.2.8 Should NEPA Products ( e. g., EISs) be Subject to Peer Review? . Page 29 2.2.9 Should Environmental Regulatory Models be Peer Reviewed? . . Page 29 2.2.10 Is Peer Review Needed for Other Organization's Work Products that Have been Submitted to EPA for Use in Decision Making? . Page 29 2.2.11 Can Work Products That are Not Determined to be Major Still be Peer Reviewed? . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 30 Peer Review Handbook Page v 2.3 Determining Which Work Products Do Not Receive Peer Review . . . . . Page 30 2.3.1 Are There Circumstances When a Major Work Product is Not Peer Reviewed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 30 2.3.2 What Products Normally Do Not Need Peer Review? . . . . . . . . . Page 31 2.4 Choosing a Peer Review Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 32 2.4.1 How Do You Determine the Appropriate Peer Review Mechanism? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 32 2.4.2 What are Examples of Internal Peer Review? . . . . . . . . . . . . . . . . Page 33 2.4.3 What are Examples of External Peer Review? . . . . . . . . . . . . . . . Page 34 2.4.4 What is the Role of Peer Review by a Refereed Scientific Journal? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 35 2.4.5 Is Peer Review Necessary when Journal Articles are Used in an Agency Work Product? . . . . . . . . . . . . . . . . . . . . . . . . Page 35 2.4.6 When and How Often Should Peer Review Occur? . . . . . . . . . . . Page 36 2.4.7 What Factors are Considered in Setting the Time Frame for Peer Review? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 36 2.4.8 Which Office/ Region or Other Agency is Responsible for Conducting the Peer Review? . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 37 2.5 Creating the Peer Review Record . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 37 2.5.1 What is the Peer Review Record? . . . . . . . . . . . . . . . . . . . . . . . . . Page 37 2.5.2 How Can the Peer Review Record Improve the Peer Review Process? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 37 2.5.3 What Should Be in the Peer Review Record? . . . . . . . . . . . . . . . . Page 37 2.5.4 What Should I Do with a Peer Review Record That Pertains to a Rulemaking Action? . . . . . . . . . . . . . . . . . . . . . . . . . Page 38 2.5.5 When Should the Peer Review Record Building Process Begin? . Page 38 2.5.6 What are the Differences in Record Keeping for a Review by an Individual Compared to a Panel? . . . . . . . . . . . . . . . . . . . . . Page 39 2.5.7 Where Should the Peer Review Record be Kept and For How Long? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 39 2.6 Budget Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 40 2.6.1 What Budgetary Factors Should I Consider in a Peer Review? . . Page 40 2.6.2 What Input is Needed for the Annual Budget Formulation and Budget Execution Process? . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 40 2.7 Legal Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 40 2.7.1 Are There Legal Ramifications From the Peer Review Policy? . . Page 40 2.7.2 Is Legal Advice Needed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 41 2.7.3 Is Peer Review Subject to the Federal Advisory Committee Act ( FACA)? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 41 Peer Review Handbook Page vi 3. CONDUCTING A PEER REVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 43 3.1 Overview Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 43 3.2 Charge to the Peer Reviewers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 43 3.2.1 What is a Charge? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 43 3.2.2 What are the Essential Elements of a Charge? . . . . . . . . . . . . . . . Page 44 3.2.3 Where Can I Get an Example of a Charge? . . . . . . . . . . . . . . . . . . Page 44 3.3 Time Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 44 3.3.1 What are the Factors in Scheduling a Peer Review? . . . . . . . . . . . Page 44 3.4 Selection of Peer Reviewers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 45 3.4.1 What are Considerations for Selecting Peer Reviewers? . . . . . . . Page 45 3.4.2 Where Do I Find Peer Reviewers? . . . . . . . . . . . . . . . . . . . . . . . . Page 45 3.4.3 Are External or Internal Peer Reviewers Preferred? . . . . . . . . . . . Page 46 3.4.4 What is Important in the Mix of a Peer Review Panel? . . . . . . . . Page 46 3.4.5 What is a Conflict of Interest? . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 47 3.4.6 What Techniques Help Ensure Disclosure and Appropriate Resolution of Conflicts of Interest? . . . . . . . . . . . . . . . . . . . . . . . . Page 47 3.4.7 Can Parties External to EPA Pay for Their Own Peer Reviews? . Page 48 3.4.8 Are There Constraints to Selecting Peer Reviewers? . . . . . . . . . . Page 49 3.5 Materials for Peer Reviewers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 49 3.5.1 What Instructions Do You Give Peer Reviewers? . . . . . . . . . . . . Page 49 3.5.2 What Materials Should be Sent to Peer Reviewers? . . . . . . . . . . . Page 50 3.5.3 How Closely can EPA Interact with Peer Reviewers During the Review? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 51 3.6 Peer Review Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 52 3.6.1 What are Gratuitous Services for Peer Review? . . . . . . . . . . . . . . Page 52 3.6.2 Can I Use a Contract to Obtain Peer Review Services? . . . . . . . . Page 52 3.6.3 How Do I Write a Statement of Work for Contracts? . . . . . . . . . . Page 53 3.6.4 What are Advisory and Assistance Services ( AAS) or Sensitive Activities? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 53 3.6.5 What are Some Management Controls for Contracts? . . . . . . . . . Page 54 3.6.6 Can the Agency Identify and/ or Select Peer Reviewers Through a Contract? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 56 3.6.7 Can I Use Simplified Acquisition Procedures to Obtain Peer Reviewers? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 57 3.6.8 How is Travel Handled with Contracts or Purchase Orders? . . . . Page 58 3.6.9 How is Travel Handled with Special Government Employees? . . Page 59 Peer Review Handbook Page vii 4. COMPLETING A PEER REVIEW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 61 4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 61 4.2 Final Work Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 61 4.2.1 How Do I Incorporate Peer Review Comments into the Final Work Product? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 61 4.2.2 What Actions are Potentially Forthcoming from Peer Review? . . Page 62 4.2.3 Can the Identity of Peer Reviewers be Kept Anonymous? . . . . . . Page 62 4.3 Completing the Peer Review Record . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 63 4.3.1 How Do I Complete the Peer Review Record? . . . . . . . . . . . . . . . Page 63 4.3.2 Where Should the Peer Review Records be Kept, and for How Long? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 63 SUBJECT INDEX . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 65 COMMONLY USED ACRONYMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page 67 APPENDIX A  Peer Review Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page A 1 Peer Review Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page A 2 Peer Review and Peer Involvement at the U. S. Environmental Protection Agency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page A 5 APPENDIX B  Examples of Charges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page B 1 CHARGE EXAMPLE 1  Charge to Reviewers for the WTI Draft Final Risk Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page B 2 CHARGE EXAMPLE 2  IRIS Pilot Program . . . . . . . . . . . . . . . . . . . . . . . . . Page B 11 CHARGE EXAMPLE 3  Science Advisory Board Review of the Agency's National Risk Management Research Laboratory's ( NRMRL) Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page B 14 CHARGE EXAMPLE 4  Science Advisory Board ( SAB) Review of the Technical Aspects of the Multi Agency Radiation Survey and Site Investigation Manual ( MARSSIM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page B 15 CHARGE EXAMPLE 5  Science Advisory Board ( SAB) Review of the Statistical Performance of the Agency's Protozoan Oocyst Monitoring Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page B 16 CHARGE EXAMPLE 6  Science Advisory Board ( SAB) Review of the Environment Monitoring and Assessment Program ( EMAP) Research Strategy and Research Plan . . . . . . . . . . . . . . . . . . . . . . . . . . Page B 17 Peer Review Handbook Page viii APPENDIX C  Guidance on Requesting a Review by the Science Advisory Board ( SAB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page C 1 APPENDIX D  Example Statements of Work for Contracts . . . . . . . . . . . . . . . . . . . . . Page D 1 STATEMENT OF WORK  EXAMPLE 1  Statement of Work: Technical Review Contractor for Panel Review of Assistance Agreement or Fellowship Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page D 2 STATEMENT OF WORK  EXAMPLE 2  Peer Review of Prioritization Tool Report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Page D 5 STATEMENT OF WORK  EXAMPLE 3  External Peer Review of Protozoa Method Development Criteria Document . . . . . . . . . . . . . . . Page D 12 APPENDIX E  References Concerning Peer Review . . . . . . . . . . . . . . . . . . . . . . . . . . Page E 1 Peer Review Handbook Page ix The Policy requires peer review of the basis of the decision ( i. e., the underlying major scientific and/ or technical work products), not the decision itself. FOREWORD EPA's Science Policy Council ( SPC) has organized this Peer Review Handbook as guidance to EPA staff and managers on the organization and conduct of peer review pursuant to the Administrator's June 7, 1994 Peer Review Policy statement. The Handbook is based in part on the central themes set forth in the Policy statement ( see Appendix A for the full policy):

Major scientifically and technically based work products related to Agency decisions normally should be peer reviewed. Agency managers within Headquarters, Regions, laboratories, and field components determine and are accountable for the decision whether to employ peer review in particular instances and, if so, its character, scope, and timing. These decisions are made in conformance with program goals and priorities, resource constraints, and statutory or court ordered deadlines. For those work products that are intended to support the most important decisions or that have special importance in their own right, external peer review is the procedure of choice. Peer review is not restricted to the penultimate version of work products; in fact, peer review at the planning stage can often be extremely beneficial.

In addition, the Handbook augments these themes by stating and explaining widelyaccepted principles and practices that have long guided peer review in the universities, in private research organizations, and at the EPA and other government agencies.

The goal of the Peer Review Policy and this Handbook is to enhance the quality and credibility of Agency decisions by ensuring that the scientific and technical work products underlying these decisions receive appropriate levels of peer review by independent scientific and technical experts. To serve this goal, the Handbook provides information and outlines procedures in several different areas:

 basic principles and definitions, including distinctions between peer review and peer input, public comment, and stakeholder involvement;

 preparing for peer review, including identifying work products, identifying appropriate peer review mechanisms, and identifying qualified experts; and,

 conducting and completing peer reviews, including materials required for peer review, creating a peer review record, and utilizing peer review comments. Peer Review Handbook Page x This Handbook has three parts. The first contains flow charts that outline the key steps in conducting a peer review, along with a managers' checklist for planning peer reviews. The second part contains peer review guidance detailing the procedures outlined in the flow charts in a question and answer format. The third part contains Appendices including the 1994 Peer Review Policy and examples to help perform quality peer reviews. Some procedures outlined in the 1994 Policy have been completed while others are continuing to change in line with Agency experience. These changes have been incorporated into the current Handbook.

Dorothy E. Patton, Ph. D. Executive Director, Science Policy Council Science Policy Council Fred Hansen, OA, Chair Lynn Goldman, OPPTS, Vice Chair Joseph Alexander, ORD Donald Barnes, OA Tudor Davies, OW Jeanne Fox, Region 2 Sylvia Lowrance, OECA Ramona Trovato, OA Michael Shapiro, OSWER Lawrence Weinstock, OAR Robert Wolcott, OPPE Science Policy Council Steering Committee Dorothy Patton, ORD, Chair Donald Barnes, OA Patricia Cirone, Region 10 William Farland, ORD Michael Feldman, OCFO Penelope Fenner Crisp, OPPTS Michael Firestone, OPPTS Jerri Anne Garl, Region 5 Roland Hemmett, Region 2 Carl Mazza, OAR James Nelson, OGC Jennifer Orme Zavaleta, ORD Peter Preuss, ORD Larry Reed, OSWER Joseph Reinert, OPPE Rosemarie Russo, Region 4 Vanessa Vu, OPPTS Mary Ellen Weber, OPPTS Jeanette Wiltse, OW William Wood, ORD Science Policy Council Staff Edward Bender Kerry Dearfield James Rowe Mary McCarthy O'Reilly Dorothy Patton, Executive Director, Science Policy Council Peer Review Handbook Page 1 U. S. Environmental Protection Agency SUMMARY OF THE PEER REVIEW PROCESS This section of the Peer Review Handbook contains flowcharts and descriptions of the major steps in conducting a peer review. Cross references to the appropriate section on Peer Review Guidance are shown in parenthesis and bolded.

The Managers Planning Checklist for Peer Review ( on page 5) is designed to give Managers/ Decision Makers a simple tool to help plan for a successful peer review. It asks questions that should be considered by a manager or Decision Maker during the peer review process to insure that necessary actions are taking place. This checklist is also intended to be used by staff ( especially Peer Review Leaders and Coordinators) to inform managers and Decision Makers on some of the key steps and considerations that are necessary in carrying out a successful peer review. Prepare Work Product Work Product is Not Subject to Peer Review Policy Work Product is Not Placed on any List Is it a Scientific and Technical Work Product?

Is the Work Product Major? Would the Work Product Still Benefit from Peer Review?

Work Product is Not a Candidate for Peer Review Place Work Product on List C Work Product is a Candidate for Peer Review  Identify Basis for Charge  Identify Key Staff  Create Peer Review Record  Ensure Availability of Funds  Determine Overall Time Frame Place Work Product on List B NO YES YES NO NO YES Continue with Conducting a Peer Review Does Work Product Consist only of Science Previously Peer Reviewed? YES NO Peer Review Handbook Page 2 Figure 1  Flowchart for Planning a Peer Review 1. Determination of " major scientific and technical work product":  It is a scientific, engineering, economic, or statistical document ( § 2.2.1)  Determine if the work product is major ( § 2.2.3)  Major products meet certain criteria ( § 2.2.3)

2. Work product is a candidate for peer review :  Major work products are subject to peer review ( § 2.2.2)  Some non major work products have to be evaluated to determine if peer review would still benefit the product ( § 2.2.11)

3. Work product is not a candidate for peer review :  Most non major work products are typically not candidates for peer review ( § 2.3.2)  Major work product consists only of science previously peer reviewed and adequate under the Agency's Policy ( § 2.3.1)  Place work product on List C ( § 1.3.2c))

4. If a work product is subject to peer review:  Identify basis for charge ( § 3.2.1)  Identify key staff ( § 1.4)  Create a peer review record ( § 2.5)  Ensure source of funding for the peer review ( § 2.6)  Estimate completion date for peer review ( § 3.3.1)  Place the work product on List B ( § 1.3.2b)) Work Product ( Ready for Peer Review)

Develop the Charge Select the Peer Review Mechanism Select Peer Reviewers Set the Timelines, including Deadlines Send Materials to Peer Reviewers & Conduct Review Complete Peer Review and obtain Comments of Peer Reviewers Add to Peer Review Record Continue with Completing a Peer Review Peer Review Handbook Page 3 Figure 2  Flowchart for Conducting a Peer Review 1. Develop the charge ( § 3.2.2):  Determine which key issues to address  Include in peer review record ( § 2.5.3)

2. Select a peer review mechanism ( § 2.4)  Internal ( § 2.4.2)  External ( § 2.4.3)  Mail review ( § 2.4.3)  Face to face meeting ( § 2.4.3)  One time or multiple meetings ( § 2.4.6)  Include logistical information in peer review record ( § 2.5.3)

3. Determine the specific time line ( § 3.3):  When will the review be started  What are the intermediate check points  What is the deadline for completion 4. Select peer reviewers ( § 3.4):  Determine sources of peer reviewers ( § 3.4.2)  Determine expertise required ( § 3.4.4)  Consider balance/ address ( § 3.4.4)  Consider conflicts of interest ( § 3.4.5 & 3.4.6)  Include documentation in peer review record ( § 2.5.3)

5. Materials for the peer review ( § 3.5):  Obtain materials from Program for review  Prepare instructions for peer reviews ( § 3.5.1)  Forward materials to peer reviewers ( § 3.5.2)  Include copy of materials in peer review record ( § 2.5.3)

6. Conduct the peer review  Obtain written comments from reviewers  Include in peer review record ( § 2.5.3)

Note: Some of these steps may occur concurrently. Receive Comments from Peer Reviewers Are Peer Reviewers Comments used in Work Product?

Document Why Not Used and Place in Peer Review Record Document How Used and Place in Peer Review Record Incorporate Comments into Work Product Finalize the Work Product Update the Peer Review Record Move the Work Product from List B to List A Done NO YES Peer Review Handbook Page 4 Figure 3  Flowchart for Completing a Peer Review 1. Evaluate comments from peer reviewers ( § 4.2.1)  Responding and reacting to comments  Obtaining clarification, if needed 2. Comments that are considered, but not used ( § 4.3.1)  Determine why not used and document  Include comments in peer review record ( § 2.5.3)

3. Comments that are useful ( § 4.3.1)  Revise the work product by incorporating comments  Send revised work product back to peer reviewers, if necessary  Include comments in peer review record ( § 2.5.3)

4. Finalize work product ( § 4.3.1)  Include in peer review record ( § 2.5.3)  Move work product from List B to List A Peer Review Handbook Page 5 Managers Planning Checklist for Peer Review 1) Title of Work Product: ___________________________________________________

2) What Decision/ Rule/ Regulation/ Action Does this Work Product Support: ________ _______________________________________________________________________

3) Determination of Major Scientific and Technical Work Products  Is the work product scientific or technical __ yes __ no?

 Is the work product __ major or __ non major?

4) Determining What Peer Review is Needed  If major, peer review is needed  If not major, is peer review still needed  When does the review need to be done?

 How much time will be needed to conduct/ complete the review?

 Are there court ordered deadlines or other constraints?

 Has senior management ( AA/ RA/ others) been informed of progress/ problems?

 What would constitute success for this review?

5) Determining the Resources for Peer Review  What is the priority of this project relative to other projects in the same office?

 What resources are needed to conduct the review?

 What are the impacts of the review on personnel?

 Who will lead the peer review?

 Who will conduct the peer review?

 Who will maintain the peer review record?

 Where will the peer review record be kept?

 What mechanism will be used for the peer review?

 Has the charge been developed?

 Has internal and external coordination been initiated/ completed?

 Have arrangements for interim/ final sign offs ( e. g., for the charge, the panel, on any changes to the final work product) been made?

 How will results of the review be presented and addressed in the final work product ( e. g., in a preamble, in an accompanying appendix  as well as changes in the work product itself)?

 Has the work product been entered onto List B or C, as appropriate?

6) Comments: _____________________________________________________________ Peer Review Handbook Page 6 THIS PAGE LEFT INTENTIONALLY BLANK Peer Review Handbook Page 7 U. S. Environmental Protection Agency PEER REVIEW GUIDANCE Peer Review Handbook Page 8 THIS PAGE LEFT INTENTIONALLY BLANK Peer Review Handbook Page 9 Peer review is not free; however, not doing peer review can be costly. 1. THE NEED FOR PEER REVIEW 1.1 Overview Statement Peer review at the U. S. Environmental Protection Agency ( EPA) takes many different forms depending on the nature of the work product, relevant statutory requirements, and officespecific policies and practices. In January 1993, responding to recommendations in the report Safeguarding the Future: Credible Science, Credible Decisions, former Administrator William Reilly issued an Agency wide policy for peer review. Administrator Carol Browner reaffirmed the central role of peer review in the Agency on June 7, 1994 and instituted an Agency wide implementation program ( see Appendix A). Following Agency wide implementation, office and region specific standard operating procedures ( SOPs) were written and have been used since 1994. Based on the experiences of those managing peer review throughout the Agency since then, as reported to the Science Policy Council ( SPC) and its Peer Review Advisory Group ( PRAG), we found that a single Agency wide document on peer review was needed. Therefore, this Peer Review Handbook is created as a single, centralized form of implementation guidance for Agency staff and managers. While the Handbook supersedes the original SOPs, Agency offices and regions can still prepare brief, tailored guidance that meets their individual needs to supplement the information in this Handbook.

1.2 Understanding Peer Review 1.2.1 Why use Peer Review?

Peer review is intended to uncover any technical problems or unresolved issues in a preliminary ( or draft) work product through the use of independent experts. This information is then used to revise that draft product so that the final work product will reflect sound technical information and analyses. Peer review is a process for enhancing a scientific or technical work product so that the decision or position taken by the Agency, based on that product, has a sound, credible basis. To be most effective, peer review of a major scientific and/ or technical work product needs to be incorporated into the up front planning of any action based on the work product  this includes obtaining the proper resource commitments ( people and money) and establishing realistic schedules.

Peer review of major scientific and technical work products should not be looked upon as another " hurdle" in the Agency decision making processes. While peer review requires that time Peer Review Handbook Page 10 and resources be planned into the decision making process, the benefits justifies the added cost. Peer review enhances the credibility and acceptance of the decision based on the work product. By ensuring a sound basis for decisions, greater cost savings are realized since decisions will not be challenged as often and extra effort will not be required to go back and redo the work product. So while peer review is not free, the cost of not doing peer review is usually much more expensive. Furthermore, not conducting a peer review can potentially place the Agency in the position of attempting to defend a scientifically invalid position  which can be very costly in terms of both resources, and more importantly, credibility.

1.2.2 What is Peer Involvement?

As defined in the Peer Review Policy, peer involvement is the process whereby Agency staff involve subject matter experts from outside their program in one or more aspects of the development of work products. Peer involvement, therefore, constitutes active outreach to and participation by the broad scientific, engineering, and economics communities beyond the Agency ( external) as well as within the Agency ( internal). Typically, peer involvement takes two general forms: peer input ( ongoing discussions during the development of the work product) and peer review ( an evaluation of a workplan, preliminary draft or the like, or most often, the critical, final objective expert evaluation of the work product).

1.2.3 What is Peer Review?

Peer review is a documented critical review of a specific Agency major scientific and/ or technical work product. The peer review is conducted by qualified individuals ( or organizations) who are independent of those who performed the work, but who are collectively equivalent in technical expertise ( i. e., peers) to those who performed the original work. The peer review is conducted to ensure that activities are technically adequate, competently performed, properly documented, and satisfy established quality requirements. The peer review is an in depth assessment of the assumptions, calculations, extrapolations, alternate interpretations, methodology, acceptance criteria, and conclusions pertaining to the specific major scientific and/ or technical work product and of the documentation that supports them. Peer review may provide an evaluation of a subject where quantitative methods of analysis or measures of success are unavailable or undefined; such as research and development. Peer review is usually characterized by a one time interaction or a limited number of interactions by independent peer reviewers. Peer review can occur during the early stages of the project or methods selection, or as typically used, as part of the culmination of the work product, ensuring that the final product is technically sound. Peer Review Handbook Page 11 Peer Input is not a substitute for Peer Review 1.2.4 What is Peer Input?

Many Agency work products are developed with the input of various scientific and technical experts inside and outside the Agency. Like the contribution made by peer reviewers, peer input is valuable and enhances the scientific or technical basis of the products. Peer input, sometimes referred to as peer consultation, generally connotes an interaction during the development of an evolving Agency work product, providing an open exchange of data, insights, and ideas. Peer input may be characterized by a continued and iterative interaction with scientific experts during work product development. A common example of peer input is the input received from workgroup members during development of a product. Many Agency products are developed through the efforts of a workgroup, which may include external experts, such as State and Tribal representatives. These workgroup members have an active, ongoing participation in developing the work product. Another example of obtaining peer input is of an Agency office sending a draft work product to a list of stakeholder representatives for general comments ( stakeholder representatives often include experts who could be considered " peers").

1.2.5 How is Peer Review Different from Peer Input?

The key distinctions between peer input as described above and formal peer review are the independence of the peer reviewers and their level of involvement. The goal of peer review is to obtain an independent, third party review of the product from experts who haven't substantially contributed to its development. When experts have a material stake in the outcome of the peer review ( such as a regulated party) or have participated substantially in the development of the product ( such as a workgroup member), those experts' reviews may not qualify as unbiased, independent peer review and may be better characterized as peer input.

It is clear that peer input provides valuable contributions to the development of the work product. However, peer input does not substitute for peer review. Once a work product is considered major, it is a candidate for peer review and entered on List B ( Candidate Products for Future Peer Review  see section 1.3.2b))  even though the work product may already have a substantial amount of peer input. In other words, one cannot argue that a peer review is not necessary if a major work product has received " enough" peer input. If the work product is not considered major and has had peer input, it is entered on List C ( Products for Which a Decision has been made not to Peer Review  see section 1.3.2c)) with comments about the extent of the peer input. Peer Review Handbook Page 12 Public comment does not substitute for peer review.

Stakeholder involvement is not a peer review mechanism. 1.2.6 Can Someone Who Provided Peer Input Become an Independent Peer Reviewer for the Same Work Product Later in the Process?

Generally, the answer is no as that expert is no longer independent, but rather a contributor to the work product. There may be special circumstances where the expertise is so narrow that another peer reviewer isn't available. The Peer Review Leader ( see section 1.4.4) will normally be responsible for making this determination.

1.2.7 How is Peer Review Different from Public Comment?

Peer review and public comment are mutually exclusive. Public comment solicited from the general public through the Federal Register or by other means is often required by the Administrative Procedures Act, relevant statutes or both. Public comment can also be solicited for policy purposes. The Agency takes public comment on some strictly scientific products and almost all regulatory decisions. Public commenters usually include a broad array of people with an interest in the technical analysis or the regulatory decision; some are scientific experts ( which may provide some peer input), some are experts in other areas, and some are interested non experts. The critical distinction is that public comment doesn't necessarily draw the kind of independent, expert information and in depth analyses expected from the peer review process. Public comment is open to all issues, whereas the peer review process is limited to consideration of technical issues. While it may be an important component of the review process, public comment does not substitute for peer review.

1.2.8 How is Peer Review Different from Stakeholder Involvement?

Stakeholder involvement occurs when the Agency works with external interest groups that have some stake in or concerns over the outcome of the technical work product or regulatory position. This is an interactive process, working with other agencies, industry groups, regulated community experts, environmental groups, other interest groups that represent a broad spectrum of the regulated community, etc., and usually strives for a consensus approach. The goal of peer review, on the other hand, is to obtain an independent, third party review. Stakeholder involvement is not a peer review mechanism  as with peer input, once a decision is made that peer review is needed, stakeholder involvement does not substitute for peer review even though it adds value to the work product. Peer Review Handbook Page 13 1.2.9 What Role does Peer Review have in the Regulatory Development Process?

The peer review of scientific and technical work products that support rulemaking actions is an important, fundamental step in the policy setting process and which affirms the credibility of the Agency. Because new rules, and the work products supporting them, must often withstand intense scrutiny by the general public and the stakeholders involved in the action, the peer review process selected for such work products needs to be well planned and documented. The rule or regulation itself is not subject to the Peer Review Policy. However, if the rule or regulation is supported by a major scientific and/ or technical work product, that work product should be peer reviewed prior to its use in the rule ( see section 2.2 for determination of major work products). The decision to peer review or not peer review any scientific and/ or technical work product will be documented through the Agency's annual peer review reporting process ( see section 1.3). Remember, public comment and stakeholder involvement do not constitute peer review.

Tier 1 and Tier 2 rulemakings, are by definition important, major Agency rulemakings within the Agency. Therefore, work products supporting Tier 1 and Tier 2 rules in particular ( including rules that are determined to be " significant" by OMB under Executive Order 12866 because they have an economic impact of $ 100 million or more) should be closely scrutinized to determine whether they meet the criteria for major ( see section 2.2.3). Work products supporting Tier 3 rulemakings may also be considered major and thus candidates for peer review. External peer review is the procedure of first choice for a work product that is intended to support a Tier 1 or Tier 2 rulemaking. Although acceptable in certain circumstances, any decision to use an internal peer review mechanism for such work products would be the exception rather than the rule. For work products supporting a Tier 3 rule, internal or external peer review may be appropriate depending on the nature of the product and other factors ( see section 2.4.1). For Tier 1 and Tier 2 rulemakings, the Final Agency Review/ closure memo needs to indicate that the Peer Review Policy was followed. For Tier 3 rulemakings, the action memo needs to indicate that the Peer Review Policy was followed.

Analytic blueprints are required for Tier 1 and Tier 2 rulemakings, and are encouraged for Tier 3 rulemakings; some individual EPA offices require it for Tier 3. For peer review purposes, the analytic blueprint is the process whereby the project manager identifies the supporting scientific and technical work products and identifies needed peer review. It also shows the schedule of the peer review in the context of the schedule for the overall rulemaking. In general, peer review should be completed as early in the process as practicable. Where possible, peer review of work products should be completed prior to issuance of the draft regulation. In some cases, support work products for final regulations may require an additional peer review if those scientific and technical work products change significantly after the public comment period. Peer Review Handbook Page 14 1.2.10 What Role does Peer Review have in Regulatory Negotiations?

Regulatory negotiations are not candidates for peer review; however, to ensure final decisions are based on sound and credible science, the major scientific and technical work products that support the negotiation need peer review before the negotiation takes place.

1.3 Annual Agency Reporting Requirements 1.3.1 What are the Annual Reporting Requirements?

The Peer Review Coordinator ( for each AA/ RA; see section 1.4.5) will organize an annual review of all peer review activities and submit this information to the Office of Research and Development ( ORD). ORD will staff this function at the direction of the Deputy Administrator. In the Deputy Administrator's annual call for submissions, guidance on format and submission of this information will be provided. ORD will review the submissions for completeness, i. e. all information is provided and products are accounted for each year. ORD will then provide a review of the completeness of the information in the submissions through consultation with the appropriate persons in each organization ( see section 1.4). ORD will then consolidate the information and findings for the SPC and the Deputy Administrator. Any conflicts arising from the review will be resolved by the Deputy Administrator. The due date for the annual reporting will be announced each year in the annual call letter; however, for planning purposes, it is normally due in the early summer.

1.3.2 What Listings are Required for the Annual Reporting?

Three listings of products are required for the annual reporting:

a) List A  Products Peer Reviewed Since 1991 1) List A is a cumulative list of peer reviewed products from 1991 to the present.

2) For each new work product entered onto List A, a short report summarizing the peer review must also be provided and attached to List A. This summary report is signed by the appropriate Decision Maker ( see section 1.4.3). Signatures by Decision Makers should be affixed as originals in the peer review record and copies submitted with List A. However, for reports submitted by electronic means, the submitted report should identify the name of the signer and the date signed. Peer Review Handbook Page 15 3) The format and content of the summary report will be provided in the annual call letter ( see section 1.3.1).

b) List B  Candidate Products for Future Peer Review 1) List B is a listing of products that are expected to be peer reviewed in the near future.

2) This List contains major scientific and technical work products and any non major scientific and technical work products for which peer review has been deemed necessary or appropriate.

3) Work products placed on List B remain on List B until they are either peer reviewed ( after which they are moved to List A) or a decision is made not to peer review that work product ( at which point it is moved to List C).

4) The peer review summary report that eventually accompanies each List A peer review product should be initiated at the time a work product is decided to be peer reviewed and filled in as the peer review proceeds. This is easier than waiting until the peer review of the work product is completed while the details are still fresh.

c) List C  Products for Which a Decision has Been Made Not to Peer Review 1) List C is a cumulative listing of all scientific and technical work products that do not receive peer review ( see section 2.3).

2) List C includes: ( a) any major scientific and technical work products for which a decision was made not to peer review; ( b) those work products which were originally placed on List B, but for which it was decided that peer review was not necessary ( e. g., the product was not used in decision making; the project was canceled); and ( c) all non major scientific and technical work products.

3) List C also includes several categories of work products that would not normally receive peer review. It is not necessary to list these work products individually on List C, however, the total number prepared by the organization must be included on List C. The organization may need to identify the individual products if requested ( e. g., due to litigation, FOIA, Peer Review Handbook Page 16 etc.). Such categories can include, but may not be limited to: chemical action reports, RCRA permits, scientific analyses for Premanufacturing Notices ( PMNs) that are conducted on a routine basis and that do not deviate from established practice, and NPDES permits.

4) List C also includes any scientific papers ( articles) that were published in a credible peer reviewed journal ( i. e. for purposes of listing journal articles, only published articles, rather than projected articles, is intended). It is not necessary to list these papers individually on List C, however, the total number prepared by the organization and peer reviewed by journals must be included on List C. The organization may need to identify the individual papers and where published, if requested ( e. g., due to litigation, FOIA, etc.). ( See sections 2.4.4 and 2.4.5).

5) Each work product on List C needs a brief description in the " comments" column of the reason( s) it is not being peer reviewed. In addition, each Decision Maker must provide signed documentation that identifies the work products that they decided not to peer review ( other than work products that are included in categories in 3) and 4), above). ORD will provide a standardized format for this documentation. Signatures by Decision Makers should be affixed as originals in the peer review record and copies submitted with List C. However, for reports submitted by electronic means, the submitted document should identify the name of the signer and the date signed.

1.3.3 Is There a Coding System for Work Products on the Annual Reporting Lists?

The work products that are placed on these lists should be assigned a discrete code number. A coding system will uniquely identify each work product when it is first listed on any of these lists. This will assist in tracking each work product from year to year and when it moves from one list to another. The code will be: YEAR ( 2 digits)  OFFICE/ REGION ( usual initials)  RUNNING NUMBER ( 4 digits and in sequential numbering for that year the work product was first placed on any list); for example, 97 OAR 0001, 98 OW 0010, 99 R05 0012, 00 OPPTS 0054, or 01 R10 0002.

The code number needs to be assigned at the time the annual report is developed ( although the code number can be assigned anytime during the year that the peer review is Peer Review Handbook Page 17 planned). The code number is assigned by the Peer Review Coordinator to provide a single, consistent numbering sequence.

1.3.4 When will the Handbook Itself be Revised?

During each annual reporting cycle, suggestions for revisions to the Handbook should be submitted. A decision will be made by the Science Policy Council ( SPC) on whether to revise the Handbook, in part or total, or not based on the suggestions. The SPC will then direct the Peer Review Advisory Group ( PRAG) to draft the revision( s) for Agency comment and SPC approval.

1.4 The Roles of People and Organizations in Peer Review 1.4.1 Who is Ultimately Accountable for Peer Review?

Under the June 7, 1994 Peer Review Policy, the Administrator has designated the Assistant Administrators and Regional Administrators ( AAs and RAs) to be accountable for implementing the Policy in their respective organizations. The Deputy Administrator is ultimately responsible for peer review across the Agency and is the final arbitrator of conflicts and concerns about peer review.

1.4.2 Who are the Agency Staff involved in Peer Review?

The principal Agency staff involved are Decision Makers ( and their line managers), Peer Review Leaders and Peer Review Coordinators. In addition, staff in ORD have been designated by the Deputy Administrator for ensuring the Agency's Peer Review Policy requirements are met.

1.4.3 Who are the Decision Makers & What are Their Responsibilities?

The AA/ RA is the ultimate Decision Maker for their organization and is accountable for the decisions regarding the identification of major scientific and technical work products and the mechanism( s) of peer review utilized for each of the products. The AA/ RA may designate the Office Directors and Division Directors ( or other appropriate level line managers) as the frontline Decision Makers.

Generally, the Decision Makers [ usually line managers] decide whether a work product is major and needs peer review or not, and what peer review mechanism to use. Furthermore, the Decision Makers commit the resources needed to ensure a proper peer review. Decision Makers are responsible for ensuring that the peer reviews are properly performed and documented. Peer Review Handbook Page 18 Specific responsibilities of the Decision Maker( s) are the following:

a) Determine which work products in their organization require peer review b) Designate ( in conjunction with the Project Manager) a Peer Review Leader to organize the peer review c) Provide advice, guidance, and support to the Peer Review Leader in the preparation, conduct, and completion of the peer review d) Ensure that sufficient funds are designated in the office's budget request to conduct the peer review; also ensure that adequate resources and/ or extramural management support are available for the peer review e) Establish a realistic peer review schedule f) Designate the stage( s) of product development where peer review is appropriate g) Ensure that the results of peer review are carried forward in the work product h) By signature, document the decisions made that are reported in the annual reporting to the SPC and Deputy Administrator; this includes signing the peer review summary report for each completed peer review i) Certify any decision NOT to peer review a product by signature on a List C submission 1.4.4 Who are the Peer Review Leaders & What are Their Responsibilities?

The Peer Review Leader organizes and oversees the peer review for a specific individual work product. This person( s) can be the Decision Maker( s), but will usually be someone who is authorized by the Decision Maker to organize, conduct, and complete the peer review. This individual may also be the Project Manager for the work product. The Peer Review Leader will obtain the assistance and support of the Peer Review Coordinator ( see below) as well as any others within the Agency to help perform the peer review. The Peer Review Leader will be chosen on a case by case basis depending on the work product needing peer review. Peer Review Handbook Page 19 Specific responsibilities of the Peer Review Leader are these:

a) Keep the Decision Maker informed of the status of a given project; provide Peer Review Coordinator with data for the annual report b) Organize, conduct, and complete the peer review following Agency procedures c) Establish and maintain the peer review record for the specific individual peer review currently being performed ( see section 2.5); this includes initiating the peer review summary report for the Decision Maker to sign when the peer review is completed d) Select the peer reviewers in consultation with others involved with the peer review ( e. g., Decision Maker or responsible line manager)

e) Advise peer reviewers of their responsibilities f) Brief management and obtain management approval on the approach to responding to peer reviewer comments g) Provide information to Decision Maker on the charge, profile of peer reviewers, the peer review comments, and how those comments are to be used h) Notify the Peer Review Coordinator that the peer review is completed for the annual report i) Archive the peer review record in a manner consistent with their organization's archiving procedures 1.4.5 Who are the Peer Review Coordinators & What are Their Responsibilities?

The Peer Review Coordinator is designated by the AA/ RA to coordinate and monitor peer review activities in their respective organization or organizational unit. This person must be of sufficient stature and judgment to have the access to and confidence of all levels of office or regional management when needed. The Peer Review Coordinator is the main contact for their organization; they can also direct interested parties to other persons/ contacts in the office on specific work products ( e. g., Peer Review Leader). Peer Review Handbook Page 20 Specific responsibilities of the Peer Review Coordinator are these:

a) General oversight responsibility for the Office's or Region's peer review process b) Report peer review activities to the AA/ RA c) Help mediate difficult issues between their organization and others; if they can't resolve issue, then bring the issue to the attention of the appropriate level Decision Makers in each organization for resolution.

d) Function as the liaison with ORD and the Science Policy Council ( SPC):

1) Represent office/ region before the SPC 2) Advise ORD of any changes in the list of work products and peer review mechanisms during the annual reporting, and when necessary, at other times 3) Participate in Agency peer review training, workshops, etc., as requested and disseminate this information to the organization; coordinate and/ or present training within their organization e) Submit information on organization's peer review candidates for each year as requested ( this is the annual reporting, see section 1.3)

1) Generate and update Lists A, B, and C 2) Assure the proper approval signature on the completed submission with the accompanying explanation for any departures from the Policy f) Establish procedures to assure that the required work product peer review documentation ( i. e., peer review record) is filed and maintained in an appropriate manner ( see section 2.5)

g) Provide advice, guidance, and support to the various Peer Review Leaders for the performance of the peer reviews h) Distribute Agency wide peer review guidance and materials to appropriate office/ region personnel, as requested Peer Review Handbook Page 21 The quality of the peer review is dependent on the competence and independence of the reviewers. 1.4.6 Who are the Peer Reviewers?

Peer reviewers are individuals who have technical expertise in the subject matter of the work product undergoing peer review. Peer reviewers can come from EPA, another Federal agency, or from outside of the Federal government.

1.4.7 What are the Responsibilities of Peer Reviewers?

Peer reviewers should maintain the confidentiality of the product, perform the review in a timely manner, and be unbiased and objective. Peer reviewers need to be willing participants in the peer review process  they should agree to read all materials, attend all sessions, and protect confidential information that arises.

1.4.8 What is an Independent Peer Reviewer?

An independent peer reviewer is an expert who wasn't associated with the generation of the specific work product either directly by substantial contribution to its development or indirectly by significant consultation during the development of the specific product. The independent peer reviewer, thus, is expected to be objective.

Independence is freedom from institutional, ideological, or technical bias regarding the issues under review and is necessary for objective, fair, and responsible evaluation of the work product. If a selected reviewer has a particular scientific or technical perspective, it may be desirable to balance the review with peer reviewers of other perspectives. Ideally, peer reviewers should be free of real or perceived conflicts of interest or there should be a balancing of interests among peer reviewers. If there are potential conflicts of interest ( real or perceived), they should be fully identified to ensure a credible peer review. ( See section 3.4 for further information).

1.4.9 When does an Agency Internal Peer Reviewer Qualify as Independent?

An Agency independent peer reviewer is one who comes from a different organizational unit than the one where the review question or document originates. A different organizational unit usually denotes, at minimum, a different office ( i. e., above a division level) within the organization. In particular, a reviewer shouldn't come from within the chain of command, either upward or downward. Peer Review Handbook Page 22 1.4.10 What is a Peer Review Panel?

A peer review panel can range from a few individuals to ten or more, depending on the issue being investigated, the time available and any limitations on resources. Individuals who serve as peer reviewers must have appropriate scientific and technical expertise that covers the broad spectrum of expertise required to treat the issues/ questions presented in the charge.

1.4.11 What is a Subject Matter Expert?

A subject matter expert is one who has specific scientific and technical expertise in the matter under review. The importance of scientific and technical expertise in the subject matter is obvious, however, knowledge or just " knowing" about the subject area isn't equivalent to expertise in the subject matter. For Agency decisions, a multi disciplinary group of experts corresponding to the disciplines that contribute to complex Agency decisions is often necessary for a full and complete peer review. For example, a risk assessment that relies on both animal and human data usually requires experts in both areas for a complete review. For economic analyses, experts from the corresponding economic disciplines are necessary.

1.4.12 What is the Role of the Science Policy Council ( SPC)?

According to the 1994 Peer Review Policy statement: " The Science Policy Council is responsible for overseeing Agency wide implementation. Its responsibilities include promoting consistent interpretation, assessing Agency wide progress, and developing recommendations for revisions of the policy as necessary." The SPC meets its responsibilities through coordination with the Peer Review Coordinators, the Peer Review Advisory Group ( PRAG) and the Office of Research and Development ( ORD).

The SPC, PRAG and ORD are not responsible for identifying specific products for peer review or determining the level of review or mechanisim for that review; those functions are the responsibility of management within each Office or Region.

1.4.13 What is the Role of the Peer Review Advisory Group ( PRAG)?

The Science Policy Council has created the Peer Review Advisory Group ( PRAG) to assist in the implementation of the Agency's Peer Review Policy. The primary role of the PRAG is to provide interpretation of the policy and to assist the SPC and Agency Offices and Regions in the annual update of the Peer Review Handbook. Peer Review Handbook Page 23 1.4.14 What is the Role of the Office of Research and Development ( ORD)?

The Deputy Administrator has designated the Office of Research and Development ( ORD) to assist the Program Offices and Regions in the collection and review of information that is contained in the annual submission of Lists A, B and C ( for detailed information, see sections 1.3.1, 1.3.2 and 1.3.3). Peer Review Handbook Page 24 THIS PAGE LEFT INTENTIONALLY BLANK Peer Review Handbook Page 25 When in doubt about whether to peer review a work product or not, always decide to make it a candidate for peer review. 2. PLANNING A PEER REVIEW 2.1 Overview Statement Planning a peer review is a critical first step to ensure a successful peer review of a work product. The initial step is to determine whether your work product requires peer review. Once you have determined that a peer review will be conducted, the Decision Makers and Peer Review Leaders need to plan an appropriate review. This includes identification of resources ( budget and personnel), the schedule for the completion of the peer review, the mechanism for peer review, the choice of peer reviewers, and the development of the peer review record.

2.2 Determining Which Work Products to Peer Review 2.2.1 What are Scientific and Technical Work Products?

The first step in determining which work products require peer review, is to identify products that are scientific and/ or technical in nature. Scientific and technical work products are documents or positions that are used to support a research agenda, regulatory program, policy position or other Agency position or action. Categories include the following: risk assessments, technical studies and guidance, analytical methods, scientific database designs, technical models, technical protocols, statistical survey/ studies, technical background materials, technical guidance, and research plans and strategies.

Products that wouldn't be considered scientific and technical work products can include those: that address procedural matters ( e. g., planning, reporting, coordination, notification); that are primarily policy statements ( e. g., relocation policy); that are conference proceedings ( unless the proceedings are used as the scientific basis for an Agency action or decision); and that are decision documents ( e. g., Record of Decision ( ROD)  the decision document itself is not subject to the Peer Review Policy, but the underlying scientific and/ or technical support work product is a candidate for peer review).

2.2.2 What Scientific and Technical Work Products Need Peer Review?

The principle underlying the Peer Review Policy is that all major scientific and technical work products used in decision making will be peer reviewed. The process for identifying which of these products is Peer Review Handbook Page 26 " major" ( and thus a candidate for peer review) and then determining the mechanism of review will take into account various criteria and the circumstances surrounding the use of that work product. To maintain flexibility, the Decision Maker( s) for peer review should consider the full field of possible work products that could benefit from peer review and the full spectrum of peer review mechanisms for each product. Once a decision is made to perform peer review, the product is listed in the annual submission of Candidate Products for Future Peer Review ( List B  see section 1.3.2b)).

2.2.3 How Does One Determine Whether a Scientific and/ or Technical Work Product is " Major"?

Determinations of a scientific and/ or technical work product as " Major" will largely be case by case. The continuum of work products covers the range from the obviously major, which clearly need peer review, to those products which are not major and clearly don't need peer review. The rest of the work products fall in between those two distinctions. This " middleground probably represents the majority of work products, each of which needs to be evaluated closely and be compared to certain criteria ( see below). The Decision Maker needs to make a judgment as to whether a work product meets the criteria for major or not. There is no easy single yes/ no test of major covering the whole continuum of work products. A rule of thumb to remember  if there is any doubt about whether a work product needs peer review, then go ahead and consider it a candidate for peer review ( and place it on List B  Candidate Products for Future Peer Review).

Scientific and technical work products that are used to support a regulatory program or policy position and that meet one or more of the following criteria are candidates for peer review:

a) Establishes a significant precedent, model, or methodology b) Addresses significant controversial issues c) Focuses on significant emerging issues d) Has significant cross Agency/ inter agency implications e) Involves a significant investment of Agency resources f) Considers an innovative approach for a previously defined problem/ process/ methodology Peer Review Handbook Page 27 g) Satisfies a statutory or other legal mandate for peer review Usually, a major scientific and/ or technical work product supports a regulatory decision or policy/ guidance of major impact. Major impact can mean that it will have applicability to a broad spectrum of regulated entities and other stakeholders, or that it will have narrower applicability, but with significant consequences on a smaller geographic or practical scale. The scientific and/ or technical work that underlies many of the Agency's major rulemakings and policy and guidance documents of general applicability would be designated " major" under this scope of impact criterion because of their far reaching or significant impacts.

The novelty or controversy associated with the work product helps determine whether it is major or not. A major work product may be novel or innovative, precedential, controversial, or emerging (" cutting edge"). An application of an existing, adequately peer reviewed methodology or model to a situation that departs significantly from the situation it was originally designed to address is a candidate for peer review. Similarly, a modification of an existing, adequately peer reviewed methodology or model that departs significantly from its original approach is a candidate for peer review. Determination of " significant departure" as used in this section is the responsibility of the Decision Maker.

In summary, a major scientific or technical work product has a major impact, involves precedential, novel, and/ or controversial issues, or the Agency has a legal and/ or statutory obligation to conduct a peer review.

2.2.4 What Economic Work Products Need Peer Review?

The following three types of major economic work products require peer review:

a) internal Agency guidance for conducting economic analysis b) new economic methodologies or unique or novel applications of existing economic methodologies, particularly those that are pathbreaking c) broad scale economic assessments of regulatory programs, such as the Congressionally mandated study of the costs and benefits of the Clean Air Act For all these types of economic work products, we will pursue an independent and expert external peer review, such as by the recently reconstituted Environmental Economics Advisory Committee ( a subcommittee of the SAB), or other appropriate experts. The straight forward application of accepted, previously peer reviewed economic methods or analyses in regulatory Peer Review Handbook Page 28 impact analyses supporting rulemakings or policy development, however, will typically not be subject to formal peer review. Of course, if the particular facts and circumstances of any piece of economic analysis warrant peer review beyond that described above, the Agency will accommodate those on a case by case basis.

2.2.5 How Should Peer Review be Handled for Products Developed under an Interagency Agreement?

When funds are passed to another agency, it will likely be placed in a contract, cooperative agreement, or grant. In many instances, the receiving agency's guidance for peer review will not be the same as ours. ( See sections 2.2.6, 2.2.8, 2.2.10 and 3.6 for further details).

2.2.6 Should Products from Grants, Contracts and Cooperative Agreements Receive Peer Review?

If there is a scientific and/ or technical work product resulting from a grant, contract, or cooperative agreement and it is considered major and will likely be used in Agency decisionmaking the work product needs peer review. Since it would probably result in a perceived, if not real, conflict of interest, a group that is generating the work product usually cannot conduct or perform the peer review of its own work product. Exceptions may be made in certain instances for organizations that have adequate and well established recognized procedures for peer review, such as the National Academy of Sciences. In practice, the Agency may need to peer review the product on its own, or arrange with an independent third group ( e. g., via another extramural vehicle) to arrange for the peer review. The Agency should not use the major scientific and technical work products from grants, contracts, or cooperative agreements to support decision making unless the work products are peer reviewed for both scientific and technical rigor and applicability to the specific use to be made of the product.

2.2.7 Should Site Specific Decisions be Subject to Peer Review?

The site specific decision itself is not subject to peer review and doesn't need peer review based solely on the Peer Review Policy. However, if a site specific decision is supported by a major scientific and/ or technical work product, that work product needs peer review. While the same considerations for major apply here, several of the criteria above ( see section 2.2.3; specifically criteria b, c, d, and g) are considered more useful for regional consideration than other criteria. So generally speaking, a close examination of how the underlying major scientific and/ or technical work product is adapted to the site specific circumstances is required. Peer Review Handbook Page 29 2.2.8 Should NEPA Products ( e. g., EISs) be Subject to Peer Review?

Not everything requires peer review, and in the case of an Environmental Impact Statement ( EIS) prepared under the requirements of the National Environmental Policy Act ( NEPA), the document already has received extensive review through the " scoping" and interagency review processes that are part of NEPA.

The rule of thumb is that if the underlying scientific and/ or technical work product is major, then the work product needs peer review. In general, the Agency's role in the NEPA document would suggest what sort of review the document gets. If EPA is developing the document as part of an EPA action/ decision ( EPA is the Lead agency under NEPA), and it meets the definition of " major," then it needs independent peer review. If it is not a major work product ( little impact, non controversial, etc.), then peer input/ continuing involvement might well be appropriate.

On the other hand, if EPA is reviewing an EIS from another Agency ( EPA is not the Lead agency under NEPA), it is likely that we are reviewing for conflicts with EPA policy and general environmental concerns. However, EPA must ask if the underlying major scientific and/ or technical work product that supports the EIS has been peer reviewed. If not, this would raise concern about the full credibility and soundness of the EIS based on the science and technical support. EPA should work with the other organization/ agency to ensure that the major scientific and/ or technical work product receives adequate peer review.

2.2.9 Should Environmental Regulatory Models be Peer Reviewed?

Generally, yes. Specific guidelines for the peer review of environmental regulatory models have been published by the Agency. These can be found on the EPA web site under the Science Policy Council home page ( http:// www. epa. gov/ ORD/ spc).

2.2.10 Is Peer Review Needed for Other Organization's Work Products that Have been Submitted to EPA for Use in Decision Making?

Yes. Any scientific and/ or technical work product that is used in Agency decision making and is considered major needs peer review regardless if the work product is produced by the Agency or another organization. It is hoped that the other organization outside the Agency has had the work product independently peer reviewed and the peer review meets the intent of the Agency's Peer Review Policy and EPA's proposed use of the product. Agency staff should examine closely the particulars of the peer review to ensure independence and a conscious effort to incorporate the peer reviewers' comments into the final work product. If there are perceived, Peer Review Handbook Page 30 or real, conflicts of interest, this may preclude the use of that peer review and, in those instances, another peer review would be needed. See section 2.4 for further details on peer review mechanisms.

2.2.11 Can Work Products That are Not Determined to be Major Still be Peer Reviewed?

Yes, they could be. Scientific and technical work products that do not come under the " major" distinction discussed above may nonetheless be candidates for peer review. For example, a project manager may decide to use peer review because of particular program needs and goals. Peer review may also be warranted because it adds substantial value to the work product.

2.3 Determining Which Work Products Do Not Receive Peer Review 2.3.1 Are There Circumstances When a Major Work Product is Not Peer Reviewed?

There may be circumstances where a work product is considered major, but a decision for no peer review can then be justified. For example:

a) Additional peer review is not required with work that has been previously reviewed by recognized experts or an expert body. For example, a cancer risk assessment methodology or an exposure modeling technique that was the subject of earlier peer review would not require additional peer review, even if the product supported a significant Agency decision.

b) Additional peer review is not required if an application of an adequately peer reviewed work product does not depart significantly from its scientific or technical approach ( see section 2.2.3).

c) Additional peer review is not required when the scientific and/ or technical methodologies or information being used are commonly accepted in the field of expertise ( this would need the appropriate documentation to support the commonly held view).

d) Most often, a major work product would not receive peer review when the regulatory activity or action which the work product supports is terminated or canceled  no further action, including peer review, is necessary. Peer Review Handbook Page 31 e) In a few instances, statutory and court ordered deadlines and other time constraints may limit or preclude peer review of products that would otherwise be considered major. However, it is up to the Decision Maker( s) to make every attempt possible to assure that peer review of major work products occurs taking into account these deadlines.

f) Very rarely, resource limitations may also restrict peer review. Programs or Regions will evaluate these circumstances on a case by case basis; decisions will be based on consultations involving line management, the Project Manager, the Peer Review Leader, and the Peer Review Coordinator.

If peer review of a major scientific and/ or technical work product is not conducted, a written justification must be placed in the " comments" section of List C ( Products for Which a Decision has Been Made Not to Peer Review). The justification is signed off by the appropriate Decision Maker ( see section 1.3.2c)).

2.3.2 What Products Normally Do Not Need Peer Review?

Products that are not major scientific and technical work products normally do not require peer review under the intent of the Peer Review Policy. Most of these scientific and technical work products are then placed on List C with a written comment of why it was determined to be not major ( see section 1.3.2c)). This justification is signed off by the appropriate Decision Maker to assure that all scientific and technical work products received consideration for peer review.

Some scientific and technical work products are not considered major and generally do not need to be placed on List C ( Products for Which a Decision has Been Made Not to Peer Review). These types of work products typically include: derivative products ( i. e., a product that only summarizes an already peer reviewed product or products), compendiums of existing models, methods and/ or technologies; or preliminary or incidental analyses prepared separately from the work product ultimately used to support an Agency action or decision ( e. g., during the course of developing a rule, managers may direct staff to prepare various " what if" analyses; those that aren't used in the work product do not need to be listed). Peer Review Handbook Page 32 The mechanism of the peer review matches the importance and complexity of the major work product. 2.4 Choosing a Peer Review Mechanism 2.4.1 How Do You Determine the Appropriate Peer Review Mechanism?

During the planning of a peer review, the Decision Maker and the Peer Review Leader may consider several mechanisms for the peer review of major scientific and technical work products. These options range from consultations with EPA colleagues not involved in developing the product to a large and formal panel of outside subject matter experts. The peer review effort might be a focused one time evaluation, or could encompass several examinations over the course of a project. In principle, peer review provides the greatest credibility for major work products when it involves wellqualified external reviewers, is intensive in its examination, and operates through a more or less formal process. As a practical matter, however, time and resource considerations in many cases impose limitations on what can be reasonably achieved. Arranging for the most appropriate and feasible peer review will involve good judgment and a willingness to consider substance, time, and resource tradeoffs. Developing a peer review plan that provides for appropriate depth, timing, and content is an important matter for early consideration by the Decision Maker and Peer Review Leader. Note that use of peer input, public or stakeholder involvement does not qualify as peer review.

The approach best suited to a specific work product will depend on the nature of the topic and the intended final product. Generally, the more novel or complex the science or technology, the greater the cost implications of the impending decision, and the more controversial the issue, then the stronger the indication for a more extensive and involved peer review and for external peer review in particular. Certain work products will clearly lend themselves to extensive external peer review; generally these will be products with large impacts ( e. g., those that support Tier 1 and Tier 2 rulemakings). Other major work products may not need a large scale external peer review and may utilize a less involved, less resource intensive review. The peer review of some products may be better served with some form of internal peer review or a combination of internal and external peer review.

The choice of peer review mechanism will depend upon the experience and assessment of the Decision Maker( s) dealing with peer review issues. It is important to make this choice at the time that the work is planned ( for products supporting rule makings, at the analytic blueprint stage) so that peer review costs and time can be budgeted into the work plan. Essentially, the level of peer review matches the impact and complexity of the major work product. For Peer Review Handbook Page 33 example, a rule under development carries considerable weight and deserves careful handling and attention; therefore, the supporting work product deserves similar care and attention for its peer review. Both internal and external peer review mechanisms are available, have been used in the past, and have served to address the needs and challenges of a particular peer review situation. Nevertheless, no single peer review mechanism is likely to work best in all situations. Some useful rules of thumb include:

a) Major work products intended to support the most important decisions, or that have special importance in their own right, ordinarily should be the subject of external peer review. Generally, the more complex, novel and/ or controversial the product, the more the Decision Maker should consider implementing a large scale peer review involving external experts.

b) Major work products that are less complex, novel, or controversial may not need such a large scale and external peer review. These products might be subject to one of the less extensive, less resource intensive review processes.

c) Group discussion with peer reviewers can be very helpful at some point in the peer review process. On the other hand, simply soliciting individual comments is easier, faster, and less expensive. Individual review is probably more appropriate for peer review at the early stages of a product's development or for products with less impact and complexity.

d) Strict time constraints, such as a court ordered deadline, can make a less involved or formal peer review mechanism imperative. But Decision Makers and Peer Review Leaders must make maximum efforts to assure that such a process is perceived as systematic and objective.

2.4.2 What are Examples of Internal Peer Review?

a) Independent experts from within the Agency ( e. g., ORD experts on non cancer effects of lead reviews a draft article on benchmark dose)

b) An ad hoc panel of independent experts from within the Agency ( e. g., an independent internal workgroup convened to examine the case for the classification of a chemical as a carcinogen)

c) Technical merit review by scientists in an Agency laboratory ( e. g., an initial review of the risk assessment for a regional incinerator by Agency scientists) Peer Review Handbook Page 34 2.4.3 What are Examples of External Peer Review?

a) Independent experts from outside the Agency ( e. g., a letter review by outside scientists)

b) An ad hoc panel of independent experts outside the Agency ( e. g., a group is convened to develop a consensus on the carcinogenicity of a particular industrial chemical)

c) Agency sponsored peer review workshops ( e. g., a review of potential indicators of ecosystem damage)

d) Review by an established Federal advisory committee such as the Science Advisory Board ( SAB), FIFRA Scientific Advisory Panel ( SAP), ORD's Board of Scientific Counselors, or the Clean Air Scientific Advisory Committee ( e. g., a review of a criteria document for a particular chemical risk)

e) Agency based federal advisory committee ( other than those established and discussed in d above)

f) Agency appointed special board or commission ( e. g., a review of the risk assessment methodology prepared by the Clean Air Act Commission on Risk Assessment) Note: The Office of General Counsel should be consulted regarding EPA's authority to establish and finance the activities of a commission or board.

g) Interagency committee ( e. g., a review of prospective research plans by the Committee on the Environment and Natural Resources coordinated by the White House)

h) A committee convened by another federal agency or government organization ( e. g., a review of the Dioxin Reassessment by the HHS Committee to Coordinate Environmentally Related Programs)

i) Review by non governmental groups ( e. g., a Society of Risk Analysis review of cancer guidelines)

j) Review by the National Academy of Sciences ( e. g., a review of the state of current knowledge about children's health risks from pesticide exposures) Peer Review Handbook Page 35 2.4.4 What is the Role of Peer Review by a Refereed Scientific Journal?

Peer review performed under the auspices of a peer reviewed, refereed, published journal contributes to the scientific and technical credibility of the reviewed product. Peer review of an EPA produced scientific and technical article by a recognized refereed journal is a satisfactory form of peer review for the purposes of publication in that journal. Peer review by that journal is also considered adequate for reviewing the scientific credibility and validity of the findings ( or data) in that article. Products so reviewed are included on List C ( Products for Which a Decision has Been Made Not to Peer Review) ( see section 1.3.2c)). No other action is needed under the Peer Review Policy unless the conditions outlined in section 2.4.5 apply.

Prior to submitting an article to a journal for peer review, EPA employees are encouraged to have the article internally peer reviewed ( see section 1.4.9); such internal peer review is already common procedure in certain parts of EPA. Articles may also need examination in accordance with any organizational clearance procedures, especially when the author is presenting him or herself as an EPA employee. For EPA employees, conflict of interest regulations will also apply.

2.4.5 Is Peer Review Necessary when Journal Articles are Used in an Agency Work Product?

Peer review for publication in a journal is usually performed for specific reasons for that journal. The use of articles that have been peer reviewed by a credible journal strengthens the scientific and technical credibility of any work product in which the article appears, but does not eliminate the need to have the work product itself peer reviewed. For instance, journal peer review may not cover issues and concerns that the Agency would want peer reviewed to support an Agency action. Under these circumstances, the major scientific and/ or technical work product in which the article appears becomes a candidate for peer review. A journal article authored by EPA employees would be used in the same manner as an article published by anyone else in a credible, well recognized journal.

If an Agency work product is based solely on a single article that has received peer review by a credible journal ( e. g., where a model is suggested for a singular use that fits a specific Agency need), journal peer review of the article is not a substitute for peer review of the Agency work product. Generally, when a major work product is supported by more than one journal article, that work product becomes a candidate for peer review. These decisions need to be documented in the peer review record. Peer Review Handbook Page 36 One important factor to remember with regard to the use of articles that have received journal peer review deals with the availability of documentation from that peer review. Ideally, EPA needs to maintain a clear, easily accessible record of the peer review to assure the credibility and validity of the peer review ( see section 2.5 for details on peer review record). Much of the information required for the Agency peer review record may not be readily accessible from a journal publisher or editorial board ( e. g., names of the peer reviewers, the charge to them, and their specific comments). However, in most cases this documentation from the journal peer review is not needed to complete the Agency's peer review record for a given action  in extreme cases, such as where litigation is involved, such documentation may be needed.

2.4.6 When and How Often Should Peer Review Occur?

The Decision Maker and Peer Review Leader have significant discretion in deciding the timing and the frequency of peer review. Options abound, each with merits depending on the context and specified peer review objectives.

A single peer review event, beginning when the final draft work product becomes available, is the approach usually taken in many situations. On the other hand, a peer review far earlier in a project cycle could be a superior approach in some circumstances. For example, early review might be beneficial at the stage of research design or data collection planning where the product involves extensive primary data collection. Or, there may be substantial incremental benefit to conducting several peer reviews along the way, particularly where a project involves complex tasks, has decision branching points, or could be expected to produce controversial findings. ( See also section 3.4 on the selection of peer reviewers).

2.4.7 What Factors are Considered in Setting the Time Frame for Peer Review?

Several factors impact how quickly a peer review may be needed. These include deadlines for completion of a project, research program, or rulemaking, funding availability, availability of quality peer reviewers, and statutory and/ or court ordered deadlines.

Peer review sometimes leads to new information and analyses. Reviewers may make recommendations for new research that would alter the work product and thus modify the scientific/ technical basis for the action or rule it supports. For this reason, a completed peer review is desirable before issuing any proposal for public comment. If that is not logistically possible because of court or statutory deadlines, or other appropriate reasons, the Decision Maker should make every effort to complete the peer review before the close of the comment period. Because peer review comments on such work products could be of sufficient magnitude to Peer Review Handbook Page 37 warrant a revision to the proposed action or rule, Decision Makers should exercise diligence in completing the peer review prior to the proposal stage whenever possible.

2.4.8 Which Office/ Region or Other Agency is Responsible for Conducting the Peer Review?

The organization of the Decision Maker is normally responsible for conducting the peer review. Responsibility for conducting a peer review can be negotiable when more than one Agency office or region or other agencies are involved. Usually, the degree of involvement by any of the organizations and agencies and their ability to fund peer review will often determine who has the lead for the peer review.

2.5 Creating the Peer Review Record 2.5.1 What is the Peer Review Record?

It is the formal record ( file) of decision on the conduct of the peer review, the type of peer review performed, and a summary of the outcome of that peer review. It includes sufficient documentation for an uninvolved person to understand what actually happened and why. The Peer Review Leader ( with the program manager if there is one) creates a separate, clearly marked peer review file section within the overall file for development of the work. Once the peer review is completed, it is the responsibility of the Peer Review Leader to ensure that the peer review record is filed and maintained in accordance with the organization's procedures.

2.5.2 How Can the Peer Review Record Improve the Peer Review Process?

A good peer review record allows future reference to what happened during the peer review, and helps Decision Makers make appropriate use of peer reviewer input. In addition, a good record helps ensure that EPA's Peer Review Policy is followed. The Peer Review Leader is responsible for ensuring that the peer review record for individual work products is collected and maintained until completion of the peer review effort.

2.5.3 What Should Be in the Peer Review Record?

The peer review record should include all materials considered by the individual peer reviewers of the peer review panel, as well as their written comments and other input. Such materials include, at a minimum ( see also section 4.3.1):

a) The draft work product submitted for peer review Peer Review Handbook Page 38 b) Materials and information ( including the charge) given to the peer reviewers c) Written comments, information, and materials received from the peer reviewers d) Information about the peer reviewers ( such as reviewers' names, affiliations, and identified potential conflicts and their resolution)

e) Logistical information about conduct of the peer review ( such as times and locations of meetings)

f) A memorandum or other written record responding to the peer review comments specifying acceptance or, where thought appropriate, rebuttal and non acceptance g) The final work product h) Peer review summary report ( see sections 1.3.1 and 1.3.2)

When deciding if particular materials should be included in the record, the Peer Review Leader should consider whether the materials would help reconstruct the peer review process and results at a later time. If the materials may be helpful, they should be part of the peer review record.

In addition to hard copies of materials, Peer Review Leaders need to maintain electronic copies of the materials ( e. g., charge) that are necessary for the annual reports compiled by the Peer Review Coordinators. Peer Review Leaders should discuss what are those materials and requirements for the annual report with their coordinators.

2.5.4 What Should I Do with a Peer Review Record That Pertains to a Rulemaking Action?

The Peer Review Leader should coordinate with their program's docket office to see that proper docketing requirements are satisfied for a peer review of a work product supporting a new rule. The Peer Review Leader is also responsible for notifying the workgroup chair as well as the Peer Review Coordinator ( for the annual report) that a peer review is completed.

2.5.5 When Should the Peer Review Record Building Process Begin?

An early start at developing and maintaining a peer review record will help ensure the record is complete and helpful. Ideally, the record begins when the decision to peer review a Peer Review Handbook Page 39 work product is made ( and includes the written decision). The Peer Review Leader needs to construct the peer review record from this point on  this will avoid potentially time consuming reconstruction at a later point. Note that the peer review record is not complete until it contains a copy of the final work product which addresses the peer review comments.

2.5.6 What are the Differences in Record Keeping for a Review by an Individual Compared to a Panel?

Strictly speaking, a true peer review requires more than a single reviewer. A review conducted by one individual will rarely provide the depth of commentary required to improve the work product. In addition, you will not receive the range of views and richness necessary to ensure improvement.

In the case of a review panel, there will often be conflicting comments that must be resolved. This resolution should be in the record.

2.5.7 Where Should the Peer Review Record be Kept and For How Long?

During the active conduct of the peer review, the Peer Review Leaders maintain the peer review record with themselves until the peer review is totally completed. Minimally, the file should be maintained until one year after the completed peer review is reported in the next annual reporting. After that, the peer review record should be maintained for a " reasonable period of time." Establishment and maintenance of the archive where the peer review records ultimately reside are an organization's responsibility ( i. e., not that of an individual program manager or Peer Review Leader). Generally, to allow flexibility, individual offices and regions will decide the appropriate level of organizational responsibility and how they will meet any " routinely available" requirements. The peer review record may be kept with other records relating to the overall project, as long as it is easily and separately identifiable.

There are also specific requirements regarding the use of dockets for record keeping; however, these are not covered in this Peer Review Handbook. The documents contained in the peer review record should be maintained in accordance with the Agency's record keeping retention schedule for such records. One long term archiving mechanism may be the formal archiving at the Federal Records Center in Suitland, MD. Peer Review Handbook Page 40 Peer review is part of the normal cost of doing business. 2.6 Budget Planning 2.6.1 What Budgetary Factors Should I Consider in a Peer Review?

Resources needed to implement the Peer Review Policy need to be requested through the usual Agency budgetary processes. The budget formulation process within the Executive Branch is followed, after appropriation bills are passed by Congress, by budget execution. These two processes provide opportunities to secure resources for activities carried out by Headquarters and Regional offices, including peer review. The major work products for which decisions for peer review have been made ( List B candidates) need to have adequate funding for peer review in budget requests for the coming fiscal year. Similarly, adequate funding needs to appear in the actual approved operating budget to ensure their conduct. For purposes of budget planning, the costs of peer review would include the FTE cost of staff, the contract or other costs associated with the use of outside peer reviewers, and the administrative costs of conducting a review ( copying, travel expenses, etc).

2.6.2 What Input is Needed for the Annual Budget Formulation and Budget Execution Process?

Senior Management in Office and Regions ( including Decision Makers and budget officers) need to be sure that budget requests include anticipated resources for peer review. Peer review needs to be considered as a normal part of doing business. Peer review resource considerations should also be addressed in the analytic blueprint for Agency rule making actions.

2.7 Legal Considerations 2.7.1 Are There Legal Ramifications From the Peer Review Policy?

The Peer Review Policy does not establish or affect legal rights or obligations. Rather, it confirms the importance of peer review where appropriate, outlines relevant principles, and identifies factors Agency staff should consider in implementing the policy. Except where provided otherwise by law, peer review is not a formal part of or substitute for notice and comment rulemaking or adjudicative procedures. EPA's decision to conduct peer review in any particular case is wholly within the Agency's discretion. Similarly, nothing in the Policy creates a legal requirement that EPA respond to peer reviewers. However, to the extent that EPA Peer Review Handbook Page 41 decisions rely on scientific and technical work products that have been subjected to peer review, the remarks of peer reviewers should be included in the record for that decision.

2.7.2 Is Legal Advice Needed?

AA/ RA staff and management should work regularly with individual OGC/ Regional Counsel ( RC) staff assigned to Agency activities. Peer Review Leaders should initially consult with their customary OGC/ RC advisors for legal advice or referral. Headquarters attorneys have specialties in specific areas and can be consulted as needed ( e. g., FACA considerations ( see below); contractual responsibilities; ethics and potential conflicts of interest).

2.7.3 Is Peer Review Subject to the Federal Advisory Committee Act ( FACA)?

It depends on how the peer review is carried out. The Federal Advisory Committee Act, 5 U. S. C. App. 2, imposes certain open meeting, balanced membership, and chartering requirements ( with OMB and GSA approval) before the Agency " establishes or utilizes" an " advisory committee" for advice or recommendations.

Peer review carried out by formal and established ( chartered) Federal advisory committees ( such as the Science Advisory Board or the FIFRA Science Advisory Panel) is always subject to FACA requirements. However, not all peer review is carried out by established Federal advisory committees. For example, if EPA conducts a peer review meeting for the purpose of obtaining advice from the individual attendees and not for the purpose of obtaining a peer review product from the group as a whole ( consensus), the meeting would not be subject to FACA.

Committees established by an outside organization ( such as by a Federal contractor) to provide the outside party with advice and recommendations are probably exempt from FACA. You should be aware, however, that the outside party's committee could be subject to FACA if EPA manages or controls the committee ( e. g., EPA selects the members of the panel, runs the meeting, etc.). Also remember, a contractor cannot arrange to have peer review performed on a work product the contractor itself prepared.

EPA officials conducting peer review may wish to mitigate the potential for a FACA challenge by seeking balanced participation at peer review meetings and allowing interested members of the public to attend. Questions concerning the applicability of FACA to peer review meetings should be addressed to the FACA experts in the Cross Cutting Issues Division of OGC ( Mail Code 2322 at Headquarters). Peer Review Handbook Page 42 THIS PAGE LEFT INTENTIONALLY BLANK Peer Review Handbook Page 43 The time spent preparing a good charge is well spent, and is crucial for an effective peer review. 3. CONDUCTING A PEER REVIEW 3.1 Overview Statement The success and usefulness of any peer review depends on the quality of the draft work product submitted for peer review, the care given to the statement of the issues or " charge," the match between the peer review draft product and the form of peer review, the match between the peer review draft product and the scientific/ technical expertise of the reviewers, and Agency use of peer review comments in the final product. It is not simply enough to conduct a peer review; each of the foregoing elements requires serious attention.

3.2 Charge to the Peer Reviewers 3.2.1 What is a Charge?

As part of each peer review, the Peer Review Leader must formulate a clear, focused charge that identifies recognized issues and invites comments or assistance. This request signals the Agency's awareness of potential issues and its receptivity to expert recommendations. The charge to peer reviewers usually makes two general requests. First, it focuses the review by presenting specific questions and concerns that the Agency expects the reviewers to address. Secondly, it invites general comments on the entire work product. The specific and general comments should focus mostly on the scientific and technical merits of the work product and, where germane, whether the scientific/ technical studies have been applied in a sound manner. Remember, the peer review is not for the decision or action itself, but for the underlying scientific and/ or technical work product. Focused questions greatly simplify the task of collating, analyzing and synthesizing peer review comments on a topical basis. The questions should be specific enough to get helpful comments, but not so specific ( unless very specific points are needed to be addressed) that they preclude creative responses. Moreover, the written responses to these questions by peer reviewers help the Agency create a peer review record. As a general rule, the time drafting a good charge letter is well spent and is necessary for an effective peer review. Peer Review Handbook Page 44 3.2.2 What are the Essential Elements of a Charge?

a) Brief overview or introduction ( describe what the work product is, how it was developed, how it will be used),

b) As needed, a brief description or listing of any background materials provided to the peer reviewers, and c) The issues or questions to be addressed by the peer reviewer( s).

In addition, the following should also be included as important administrative components of a charge: a) the due date of reviewer comments and format of reviewer responses, and b) the point of contact in case peer reviewers have questions.

3.2.3 Where Can I Get an Example of a Charge?

Appendix B contains examples of successful charges that cover a variety of issues. These include: WTI Risk Assessment, Region 10 review of the Eastern Columbia Plateau sole source aquifer, letter reviews for IRIS ( Integrated Risk Information System) entries, and several examples of charges for SAB reviews. Appendix C provides guidance for obtaining Science Advisory Board ( SAB) services.

3.3 Time Line 3.3.1 What are the Factors in Scheduling a Peer Review?

The peer review schedule is a critical feature of the process. The schedule must take into account the availability of a quality draft work product, availability of appropriate experts, time available for using peer review comments, deadlines for the final work product, and logistical aspects of the peer review ( e. g., contracting procedures).

The schedule for peer review should take into account the overall rulemaking ( or other decision making) schedule. For rules, in particular those in Tier 1 and Tier 2, the scheduling of the peer review should be included in the development of the analytic blueprint. Peer review sometimes leads to new information and analyses, or recommendations for new research that would alter the work product and thus modify the scientific/ technical basis for the action. For this reason, it is usually advisable to complete the peer review before taking public comment, or at least before the close of the public comment period. Peer Review Handbook Page 45 3.4 Selection of Peer Reviewers 3.4.1 What are Considerations for Selecting Peer Reviewers?

Selection of independent peer reviewers is not a trivial task, and it is crucial to an effective peer review. It is important that peer reviewers be selected for independence and scientific/ technical expertise. However, the very need to have experienced individuals on a peer review, along with the desire to have appropriate technical balance and representation, can mean that the selection of potential peer reviewers often comes from those who are considered as having a potential bias. To reduce the concern that a potential panel may have unnecessary bias, it may be useful to obtain an informal review of the expertise and balance of potential peer reviewers from others in your organization, from OGC or even from outside groups. Sometimes selecting individuals who have served in a variety of organizations rather than a single one for an extended period, provides expertise with diverse perspective. The emphasis on independence and expertise applies equally to government experts and experts from the larger scientific community.

Some peer reviews can be conducted with two or three reviewers; others involve panels of peer reviewers. In either case, each peer reviewer should have recognized technical expertise that bears on the subject matter under discussion. In instances where there is more than one peer reviewer, the peer reviewers of a work product should represent a balanced range of technically legitimate points of view. In addition, cultural diversity and " address" ( e. g., industrial, academic, or environmental community) are other factors that can play a role in selecting peer reviewers.

3.4.2 Where Do I Find Peer Reviewers?

Recommendations for potential peer reviewers can be identified from a number of organizations. These include external groups such as the affected party( ies), special interest groups, public interest groups, environmental groups, professional societies, trade or business associations, state organizations or agencies, Native American Tribes, colleges and universities, the National Research Council, and other Federal agencies with an involvement in or familiarity with the issue. Agency associated groups include the staff of the Science Advisory Board ( SAB) or the Scientific Advisory Panel ( SAP), and relevant scientific and technical experts from Program or Regional offices.

In certain circumstances, existing peer review organizations such as the SAB or SAP may be used to conduct a peer review. These groups establish their own criteria for accepting work and coordination must be made directly with them ( see Appendix C for SAB procedures). Both SAB and SAP conduct formal, public, external peer reviews. Peer Review Handbook Page 46 Occasionally, a member of the scientific community will offer his/ her services for peer review during an ongoing peer review. These offers may be at no cost or based on an expectation that reimbursement will be made. Disposition of these unsolicited offers will be handled on a case by case basis by the Peer Review Leader, and as necessary, in consultation with the Peer Review Coordinator, the Office of General Counsel ( OGC), and appropriate Decision Makers.

If you use a contract mechanism to conduct a peer review, the contractor may have its own pool of scientific and technical experts for peer review. With contractors, EPA can provide information on potential sources of peer reviewers for conducting a peer review if such a listing were prepared in alphabetical order, and contained an Agency disclaimer that EPA doesn't recommend any particular individual or firm on the list. Furthermore, when utilizing a contract mechanism to conduct peer review, EPA is not permitted to direct the prime contractor to a specific sub contractor or peer reviewer nor is EPA permitted to directly interact with the peer reviewer ( sub contractor). All interaction must be with the prime contractor's designated representative and not the contractor's staff.

3.4.3 Are External or Internal Peer Reviewers Preferred?

External peer reviewers are generally preferred, particularly for most final work products. For some work products, like those reviewed at interim steps, either external or internal peer review may be appropriate. Selection of internal peer reviewers should be based upon technical expertise, available time and " address"  that is, they should not come from the immediate office or group producing the product or have any other connection with the product or document being peer reviewed. External peer reviewers should be selected based upon technical expertise as well, however, care must be taken not to use individuals who have been involved in the development of the work product. ( See also sections 1.4.6 to 1.4.9).

3.4.4 What is Important in the Mix of a Peer Review Panel?

A peer review panel or group can number from just a few individuals to ten or more, depending on the issue, the time and resources available, and the broad spectrum of expertise required to treat the range of issues/ questions in the charge. Naturally, experts whose understanding of the specific technical area( s) being evaluated are critical; nevertheless, it is also important to include a broad enough spectrum of other related experts to completely evaluate the relevant impacts on other less obvious concerns ( i. e., to comment not only if the job is being done right, but also whether the right job is being done). For example, for health related peer reviews, experts in such fields as ecology and economics may provide very useful insights. Although persons who are familiar and have a substantial reputation in the field are often called Peer Review Handbook Page 47 upon repeatedly to be reviewers, it is important to keep a balance with new people who bring fresh perspectives to the review of a work product.

There is usually a continuum of views on any issue. To the extent possible or practicable, selected experts should have technically legitimate points of views that fall to either side of the central position along the continuum, but not too far to either extreme. This will help maintain a balanced viewpoint, while allowing all views to be expressed and discussed. A balanced panel will allow consensus building ( if consensus is the object of a particular peer review; if not, it allows a spectrum of reviews for the Agency to evaluate). As a general rule, experts who have made public pronouncements on an issue ( e. g., those who have clearly " taken sides") may have difficulty in reaching consensus positions and should be avoided.

3.4.5 What is a Conflict of Interest?

Conflict of interest is a situation in which a person is unable or potentially unable to render impartial assistance or advice to the Agency, or the person's objectivity in performing the work is or might be otherwise impaired, or a person has an unfair competitive advantage. Generally, a conflict of interest or a perceived conflict of interest arises when the person is affected by his/ her private interests, or when he/ she or his/ her associates would derive benefit from incorporation of their point of view in an Agency product. Whenever there are questions about conflicts of interest, you should contact the appropriate official in OGC for clarification.

3.4.6 What Techniques Help Ensure Disclosure and Appropriate Resolution of Conflicts of Interest?

Before finalizing the selection of reviewers, the Peer Review Leader should ask potential reviewers if they have any real or perceived conflicts of interest. One way of identifying conflicts is to ask potential reviewers about current and prior work, and prior clients that might create conflicts or the appearance thereof. This information becomes part of the peer review record.

Care must be taken to reduce the possibility for real or apparent conflicts of interest between the reviewers and the work product under review. Various tools are available to identify and limit conflicts of interest ( e. g., attention to the employment, financial, and professional affiliations of the participants; filing Confidential Financial Disclosure Forms ( SF 450) in the case of members of Federal Advisory Committees; exploring directly the issue with each of the participants before the review process takes place; and disclosing publicly at the beginning of meetings any previous involvement with the issue). Established peer review groups such as the Science Advisory Board provide useful models for addressing balance and conflict of interest Peer Review Handbook Page 48 issues. Assistance in determining legal conflicts of interest and in providing an appropriate response can be obtained through the Office of the General Counsel. In addition, additional advice can be obtained from the Designated Agency Ethics Officials ( DAEO).

Of course, conflicts do not necessarily arise merely because a peer reviewer knows something about the subject matter. In fact, experts with a stake in the outcome  and therefore a potential conflict  may be some of the most knowledgeable and up to date experts because they have concrete reasons to maintain their expertise. Such experts could be used provided the potential conflicts of interest are disclosed and the peer review panel or group being used as a whole is balanced. In some cases, however, the conflict may be so direct and substantial as to rule out a particular expert, for instance, a potential peer reviewer who may have a client or employer with a direct financial stake in the matter under review ( e. g., a manufacturer of a chemical under review). However, review of a general methodology that applies to numerous chemicals would not necessarily raise such a concern. ( Note: language that reflects these concerns should be made part of contracts/ statements of work ( SOW) or purchase orders ( PO)  see section 3.6 and Appendix D).

A Peer Review Leader may want to adopt measures that will prevent creation of conflicts as the peer review is underway. Such measures might include clauses in a contract or purchase order that requires reviewers to receive advance approval on future work, or places limits on such work, while they are performing the current peer review. The intent behind such measures is to avoid having the peer reviewer gain an unfair future advantage as a result of their participation on a peer review process. Note that at some level these types of measures will discourage experts from serving as peer reviewers. ( See section 3.6.5 for further information dealing with contracts and suggestions for appropriate management controls).

3.4.7 Can Parties External to EPA Pay for Their Own Peer Reviews?

There may be instances where parties external to EPA will want to conduct and/ or pay for a peer review on a particular work product ( presumably their own work product or one they are closely interested in, or they wouldn't be interested in expending resources). This may look benign at first blush, but is a very complex and sensitive situation that can raise significant concerns for perceived and/ or actual conflicts of interest for interested parties " paying" for a peer review of their own work product. While the Agency cannot prevent external parties from conducting and paying for a peer review, it is desireable that any such peer review meets the intent of the Agency's Peer Review Policy and adheres to the principles and guidance in this Handbook. If the external party submits their work product and accompanying peer review, the materials will be treated by the Agency as anything else submitted for the Agency's evaluation. Peer Review Handbook Page 49 We will evaluate the work product and the peer review for scientific credibility and validity before making any decisions based on the materials.

3.4.8 Are There Constraints to Selecting Peer Reviewers?

Sometimes the need for a peer review is accelerated due to a court ordered deadline or other time sensitive requirements. In such cases, it is difficult, if not impossible to obtain external peer reviewers in time to conduct a full external peer review. It may even be impossible to conduct a small scale internal peer review using just a few individuals. Mechanisms for identifying and using a small number of peer reviewers should be developed so that quick, effective peer review can be included for even the most rapidly moving products.

Another possible constraint involves confidential business information ( CBI). To evaluate certain Agency generated studies properly, some peer reviewers may need access to CBI. However, unless the reviewers are Federal employees with CBI clearance, the Agency doesn't have the independent authority to disclose CBI to them. Therefore, whenever contemplating the use of outside peer reviewers, Agency staff should determine whether the reviewers will need access to CBI. If they don't have CBI clearance, the Office of the General Counsel should be consulted on whether it is practical to obtain the consent of CBI submitters to disclose the information to peer reviewers.

Offices need to be aware of the requirements of the Federal Advisory Committee Act ( FACA) when establishing peer review mechanisms ( see section 2.7.3). Federal advisory committees that are subject to chartering by the General Services Administration must hold meetings that are open to the public, and have balanced membership requirements. The Office of the General Counsel should be consulted regarding the applicability of FACA to peer review panels.

3.5 Materials for Peer Reviewers 3.5.1 What Instructions Do You Give Peer Reviewers?

The Peer Review Leader is responsible for ensuring that peer reviewers understand and comply with these responsibilities ( see section 3.6 if a contract is involved):

a) Advise the Agency of any real or perceived conflicts of interest b) Provide written comments in specified format by the specified deadline that are responsive to the charge Peer Review Handbook Page 50 c) Comply with the request for not disclosing draft work products to the public 3.5.2 What Materials Should be Sent to Peer Reviewers?

For a peer review to be successful, peer reviewers should receive several documents at the beginning of the process. Typically, the most important among these documents are the charge letter and the current work product. The charge letter describes what the peer reviewers are being asked to do, and should serve to focus and structure the review. The work product is, of course, the material being subject to peer review.

Remember, no documents should be provided directly to a potential peer reviewer when that reviewer is going to be working under a contract or purchase order. In the case of a contract, the Agency provides the work product with associated background material to be peer reviewed to the prime contractor who in turn distributes these documents to the peer reviewers. In the case of a purchase order, the " charge or statement of work" must be part of the PO ( purchase order) and the provision of any documents needs to be coordinated with the contracting officer handling the order.

a) Essential documentation for each peer reviewer includes:

1) A current copy of the work product to be peer reviewed with associated background material. The work product needs to be of the best possible scientific/ technical quality to ensure an adequate and useful peer review.

2) A clear charge or statement of work seeking informed comment on identified issues to properly focus the efforts of the peer reviewers and ensure that their individual efforts can be compared or contrasted.

3) Some information concerning the process that you use for the peer review, including the due date of reviewer comments, the format of those responses, and a point of contact in case the peer reviewer has questions. Responses should be written and submitted to the Peer Review Leader by an agreed upon deadline. In certain rare cases, oral commentary may be sufficient. However, in such cases, a follow up written response for the record is required.

4) In some cases, Agency materials being peer reviewed will be available to the public, even if they are marked as drafts. For example, all materials reviewed by the SAB are available. Agency managers may also decide Peer Review Handbook Page 51 that a broad accessibility has benefits for the Agency. In other cases, confidentiality needs to be maintained. In these cases, each peer reviewer must be informed of the need for confidentiality with regard to the release of Agency products that are stamped as " DRAFT" or " DRAFT  Do Not Cite, Quote, or Release." Premature release of draft Agency products, views, or positions is inappropriate and can be damaging to the credibility of the Agency or the peer reviewer. While not necessarily having legal consequences, such language will be included in the charge to the peer reviewers. Other mechanisms to use in discouraging premature release include a disclaimer that appears in a separate section at the front of the document and creating the document with watermarks clearly delineating DRAFT status ( or a header or footer that states DRAFT status) on every page. In addition, in any solicitation for peer reviewers, the necessity for confidentiality and the non release of materials shall be emphasized.

b) Useful, but not critical materials that may be sent to peer reviewers include:

1) The name, address, and phone and fax numbers, and/ or Internet address of each peer reviewer working on the specific review 2) A bibliography and/ or any particularly relevant scientific articles from the literature 3) A work product that has line numbering added in the margin for ease in providing and referencing comments c) Peer Reviewers should be given what is needed to complete their task  they should not be overburdened with excess material.

3.5.3 How Closely can EPA Interact with Peer Reviewers During the Review?

The Peer Review Leader normally has administrative contacts with the reviewers during the development and conduct of the peer review. In some cases ( e. g., SAB peer review), peer reviewers may also receive a briefing on the product to be peer reviewed. Otherwise, the Peer Review Leader and other EPA staff should not contact the reviewers during the course of the review. Such contact can lead to perceived conflicts or inappropriate direction that could compromise the independence of the review. Peer Review Handbook Page 52 If peer review is conducted under a contract mechanism, EPA must limit direct contact to the prime contractor's designated representative and not have general contact and direction to the contractor's staff or peer reviewers ( sub contractors). Note, when a peer review is conducted under a contract, there are constraints where EPA staff are prohibited from contacting peer reviewers to avoid personal services arrangements. Personal services contracts exist when the nature of the relationship between the contractor and the EPA can be characterized as an employer  employee relationship. Any communications with peer reviewers must be only from the Peer Review Leader or contracting officer.

3.6 Peer Review Services A range of peer review services are available to the Agency including internal, external ( gratuitous services, contracts, purchase order), and Special Government Employee ( SGE) mechanisms. The mechanism selected is generally based on the nature of the scientific or technical work product.

3.6.1 What are Gratuitous Services for Peer Review?