During the active conduct of the peer review, the Peer Review Leaders maintain the peer review record with themselves until the peer review is totally completed. Minimally, the file should be maintained until one year after the completed peer review is reported in the next annual reporting. After that, the peer review record should be maintained for a   reasonable period of time.  Establishment and maintenance of the archive where the peer review records ultimately reside are an organization s responsibility ( i. e., not that of an individual program manager or Peer Review Leader). Generally, to allow flexibility, individual offices and regions will decide the appropriate level of organizational responsibility and how they will meet any   routinely available  requirements. The peer review record may be kept with other records relating to the overall project, as long as it is easily and separately identifiable.

There are also specific requirements regarding the use of dockets for record keeping; however, these are not covered in this Peer Review Handbook. The documents contained in the peer review record should be maintained in accordance with the Agency s record keeping retention schedule for such records ( for details, see EPA s National Records Management Program; http:// www. epa. gov/ records/ ). One long term archiving mechanism may be the formal archiving at the Federal Records Center in Suitland, MD.

2.5.8 Are Internal Peer Review Comments Included in the Peer Review Record?

An internal EPA peer review must be a formal process to be considered a legitimate peer review. This process adheres to the guidance found in this Handbook for planning, conducting, and completing a peer review. When you follow this formal process to obtain peer review from EPA peers ( see Section 1.4.9), then the whole record of that internal peer review is included in the peer review record. This includes all the materials detailed in Section 2.5.3 ( also see Section Page 48 Peer Review Handbook 4.3.1). Conducting a formal internal peer review is not the same thing as informal input from your EPA colleagues ( i. e.,   colleagues down the hall ), nor peer input from Agency personnel helping to develop the work product, nor organizational review and clearance processes. Such inputs from these informal processes should not be placed in the peer review record. The peer review record should contain only the information obtained when you conduct a formal internal peer review.

In some cases, an internal EPA peer review may be followed by a separate external peer review. In this case, the formal record of the internal peer review should be included in the peer review record, however, the external peer review will generally stand as the peer review of greater significance since it is viewed more independent in nature, has broader fields of available expertise which can be brought to bear on the issues and often includes greater depth for specific disciplines. Peer Review Leaders should ensure that any applicable EPA record keeping regulations are followed ( for details, see EPA s National Records Management Program; http:// www. epa. gov/ records/).

2.6 Budget Planning 2.6.1 What Budgetary Factors Should I Consider in a Peer Review?

Resources needed to implement the Peer Review Policy need to be requested through the usual Agency budgetary processes. The budget formulation process within the Executive Branch is followed, after appropriation bills are passed by Congress, by budget execution. These two processes provide opportunities to secure resources for activities Peer review is part of the carried out by Headquarters and Regional offices, normal cost of doing including peer review. The major work products for which business.

decisions for peer review have been made ( List B ( Candidate Products for Future Peer Review) candidates) need to have adequate funding for peer review in budget requests for the coming fiscal year. Similarly, adequate funding needs to appear in the actual approved operating budget to ensure their conduct. For purposes of budget planning, the costs of peer review would include the FTE cost of staff, the contract or other costs associated with the use of outside peer reviewers, and the administrative costs of conducting a review ( copying, travel expenses, etc).

2.6.2 What Input is Needed for the Annual Budget Formulation and Budget Execution Process?

Senior Management in Office and Regions ( including Decision Makers and budget officers) need to be sure that budget requests include anticipated resources for peer review. Peer review needs to be considered as a normal part of doing business. Peer review resource Peer Review Handbook Page 49 considerations should also be addressed in the analytic blueprint for Agency rule making actions.

2.7 Legal Considerations 2.7.1 Are There Legal Ramifications From the Peer Review Policy?

The Peer Review Policy does not establish or affect legal rights or obligations. Rather, it confirms the importance of peer review where appropriate, outlines relevant principles, and identifies factors Agency staff should consider in implementing the policy. Except where provided otherwise by law, peer review is not a formal part of or substitute for notice and comment rulemaking or adjudicative procedures. EPA s decision to conduct peer review in any particular case is wholly within the Agency s discretion. Similarly, nothing in the Policy creates a legal requirement that EPA respond to peer reviewers. However, to the extent that EPA decisions rely on scientific and technical work products that have been subjected to peer review, the remarks of peer reviewers should be included in the record for that decision.

2.7.2 Is Legal Advice Needed?

AA/ RA staff and management should work regularly with individual OGC/ Regional Counsel ( RC) staff assigned to Agency activities. Peer Review Leaders should initially consult with their customary OGC/ RC advisors for legal advice or referral. Headquarters attorneys have specialties in specific areas and can be consulted as needed ( e. g., FACA considerations ( see below); contractual responsibilities; ethics and potential conflicts of interest).

2.8 Federal Advisory Committee Act ( FACA) Considerations The Federal Advisory Committee Act, 5 U. S. C. App. 2, imposes certain open meeting ( public announcement in the Federal Register), balanced membership, and chartering requirements ( with the approval of the General Services Administration ( GSA)) before the Agency establishes, controls or manages an   advisory committee  for advice or recommendations. Peer review carried out by formal and established ( chartered) Federal advisory committees, such as the Science Advisory Board ( SAB) or the FIFRA Scientific Advisory Panel ( SAP), is always subject to FACA requirements. However, FACA does not apply to every EPA and contractor run peer reviews.

In this section you will find information on the applicability of FACA to EPA or contractor run peer reviews. Page 50 Peer Review Handbook 2.8.1 When Do FACA Requirements Apply to EPA Run Peer Reviews?

In most cases, Federal Advisory Committee Act ( FACA) requirements apply to EPA run peer reviews that are conducted by formal and established ( chartered) Federal advisory committees, such as the Science Advisory Board ( SAB) or the FIFRA Scientific Advisory Panel ( SAP). These groups have the following characteristics:

a) the group is established, controlled, or managed by EPA;

b) the group has a fixed membership, established purpose, and a set agenda; and c) the group strives to produce collegial, rather than, individual advice to EPA EPA run peer reviews that are not originally intended to be subject to FACA, but which exhibit the above characteristics, may unintentionally become subject to FACA. Questions concerning the applicability of FACA to peer review meetings should be addressed to the FACA experts in the Cross Cutting Issues Law Office of OGC ( Mail Code 2322A at Headquarters), or the appropriate Office of Regional Counsel.

2.8.2 When Are EPA Run Peer Reviews Not Subject to FACA?

If EPA conducts a peer review with the purpose of obtaining advice from the individual peer reviewers and not for the purpose of obtaining a peer review product from the group ( as a collective or consensus body), the peer review would, in most cases, not be subject to FACA. Peer review participants provide only their own views or recommendations and do not vote nor do they provide collective or consensus recommendations to EPA. When referring to the recommendations of the individual reviewers, EPA should not characterize these recommendations using terms such as   collective  or   consensus.  As a general matter, workshops and   letter reviews  that seek individual views or comments are usually not subject to the requirements of FACA.

In addition to ensuring that peer reviewers only provide comments as individuals, EPA officials may wish to lessen the potential for a challenge under the Federal Advisory Committee Act ( FACA) by seeking balanced participation at peer review meetings, and allowing interested members of the public to attend, and ensuring that they have access to appropriate materials.

Non FACA meetings may be announced in the Federal Register ( providing that it is clear in the notice that such meetings are not subject to FACA) as it provides the public with useful information and a point of contact concerning the peer review. In addition, non FACA ( as well as FACA) meetings should also be advertised via other avenues ( e. g., the Web, local newspapers, and mailing lists). Peer Review Handbook Page 51 2.8.3 How Do I Ensure that a Contractor Run Peer Review Does Not Become Subject to FACA?

Committees ( or other peer review groups) established, controlled or managed by an outside organization ( such as by an EPA contractor) to provide that outside organization with advice and recommendations ( that will be submitted eventually to EPA as a contractor report) are not subject to FACA. Although FACA should not apply to contractor run peer reviews, there are things that you ( i. e., EPA) can inadvertently do that may invoke FACA.

The following are considerations that you should be aware of when a contractor runs a peer review ( e. g., letter review, panel, workshop, etc.) for EPA:

a) the outside party s peer review could be subject to FACA if EPA establishes, manages or controls the peer review group ( e. g., EPA selects the members of the panel, runs the meeting, etc.).

b) EPA should not provide contractors with a draft agenda or suggested format for peer review meetings. EPA contractors should manage and control the process, including running any meetings.

c) At the request of the EPA contractor, EPA can provide a briefing to the peer reviewers ( e. g., in a conference call with the contractor on the line) on the history or background of the development of the peer review document. EPA only should provide technical or background information and not use the call to take over the contractor s peer review group. Not only should the contractor be   on the line  but it should be very clear to all participants that the contractor is in charge of the call. The contractor, not EPA, should invite people to participate, make all administrative arrangements, conduct the meeting, and control the agenda.

d) EPA employees may attend the peer review panel meetings or workshops. However, they may not take over the control of the meeting. The contractor should call on them to speak when appropriate, but they should limit their participation to providing technical and/ or background information, and not attempt to, or appear to, take over the contractor s meeting.

e) Since FACA does not apply when a contractor establishes, controls, or manages a peer review, the contractor need not avoid terms such as   collective  or   consensus  when reporting agreement among its peer reviewers ( subcontractors). Page 52 Peer Review Handbook f) EPA may provide comments to the contractor on the contractor s peer review only to the extent that the Agency is verifying that the contractor has completed satisfactorily the report in accordance with the work assignment. EPA should not attempt to make changes in the contractor s conclusions; this would compromise the independence of the peer review conducted by the contractor. Peer Review Handbook Page 53 3. CONDUCTING A PEER REVIEW 3.1 Overview Statement The success and usefulness of any peer review depends on the quality of the draft work product submitted for peer review, the care given to the statement of the issues or " charge," the match between the peer review draft product and the form of peer review, the match between the peer review draft product and the scientific/ technical expertise of the reviewers, and Agency use of peer review comments in the final product. It is not simply enough to conduct a peer review; each of the foregoing elements requires serious attention.

3.2 Charge to the Peer Reviewers 3.2.1 What is a Charge?

As part of each peer review, the Peer Review Leader must formulate a clear, focused charge that identifies recognized issues and invites comments or assistance. This request signals the Agency's awareness of potential issues and its receptivity to expert recommendations. The charge to peer reviewers usually makes two general requests. First, it focuses the review by presenting specific questions and concerns that the Agency expects the reviewers to address. Secondly, it invites general comments on the entire work The time spent preparing a good product. The specific and general comments should charge is well spent, and is crucial focus mostly on the scientific and technical merits of for an effective peer review. the work product and, where germane, whether the scientific/ technical studies have been applied in a sound manner. Remember, the peer review is not for the decision or action itself, but for the underlying scientific and/ or technical work product. Focused questions greatly simplify the task of collating, analyzing and synthesizing peer review comments on a topical basis. The questions should be specific enough to get helpful comments, but not so specific ( unless very specific points are needed to be addressed) that they preclude creative responses. Moreover, the written responses to these questions by peer reviewers help the Agency create a peer review record. As a general rule, the time drafting a good charge letter is well spent and is necessary for an effective peer review.

3.2.2 What are the Essential Elements of a Charge?

a) Brief overview or introduction ( describe what the work product is, how it was developed, how it will be used) Page 54 Peer Review Handbook b) As needed, a brief description or listing of any background materials provided to the peer reviewers c) The issues or questions to be addressed by the peer reviewer( s)

1) The due date of reviewers  comments 2) The format of reviewer responses 3) The point of contact in case peer reviewers have questions 3.2.3 Where Can I Get an Example of a Charge?

Appendix C ( Examples of Charges) contains examples of successful charges that cover a variety of issues. Appendix D ( Guidance on Requesting a Review by the Science Advisory Board) provides guidance for obtaining Science Advisory Board ( SAB) services.

3.2.4 Can a Stakeholder Provide Input to the Charge to the Peer Reviewers?

Yes. EPA may decide to obtain stakeholder input on the charge to the peer reviewers, but EPA must make the final determination on what elements to include in the charge to ensure that it meets the needs of the EPA peer review.

a) If you obtain stakeholder input, you must include any and all interested parties to the extent feasible based upon statutory, regulatory, budgetary and/ or time constraints. Do not limit input to one stakeholder only ( e. g., a responsible party or environmental group).

b) If EPA has hired a contractor to perform the peer review, it should still be EPA personnel who obtain stakeholder input and provide the list of charge questions to the contractor.

c) If you form a committee of stakeholders to help develop the charge, be aware that your committee may become subject to the requirements of FACA ( see Section 2.8).

3.2.5 Who Writes the Charge When I Hire a Contractor to Conduct the Peer Review?

In general, if EPA hires a contractor to perform the peer review, EPA must allow the contractor independence in conducting the review. However, with regard to the charge, EPA can Peer Review Handbook Page 55 and should provide the contractor with the substantive list of questions that EPA wants included in the charge letter to ensure that the peer review meets EPA s needs. The list of charge questions can be incorporated into the Statement of Work if the EPA project manager has the list ready at that time. Based on this list, the contractor would then prepare and submit the actual charge letter to the peer reviewers. Prior to submitting the charge letter to the peer reviewers, the contractor should give EPA an opportunity to review the charge letter to ensure that it meets EPA s needs. EPA cannot submit the charge directly to the peer reviewers.

3.2.6 Is it Okay For Me to Ask a Contractor to Develop the Charge to the Peer Reviewers?

No. EPA will provide the charge questions to the contractor as discussed above. The contractor, though, may provide assistance and advice in the development of the charge.

3.3 Time Line 3.3.1 What are the Factors in Scheduling a Peer Review?

The peer review schedule is a critical feature of the process. The schedule must take into account the availability of a quality draft work product, availability of appropriate experts, time available for using peer review comments, deadlines for the final work product, and logistical aspects of the peer review ( e. g., contracting procedures).

The schedule for peer review should take into account the overall rulemaking ( or other decision making) schedule. For rules, in particular those in Tier 1 and Tier 2, the scheduling of the peer review should be included in the development of the analytic blueprint. Peer review sometimes leads to new information and analyses, or recommendations for new research that would alter the work product and thus modify the scientific/ technical basis for the action. For this reason, it is usually advisable to complete the peer review before taking public comment, or at least before the close of the public comment period.

3.4 Selection of Peer Reviewers 3.4.1 What are Considerations for Selecting Peer Reviewers?

Selection of independent peer reviewers is not a trivial task, and it is crucial to an effective peer review. It is important that peer reviewers be selected for independence and scientific/ technical expertise. Therefore, EPA should always make every effort to use peer reviewers who do not have any real or perceived bias or conflict of interest and who are completely independent. However, the very need to have experienced individuals on a peer review, along with the desire to have appropriate technical balance and representation, can mean Page 56 Peer Review Handbook that the selection of potential peer reviewers often comes from those who are considered as having a potential bias. To reduce the concern that a potential panel may have unnecessary bias, it may be useful to obtain an informal review of the expertise and balance of potential peer reviewers from others in your organization, from OGC or even from outside groups. Sometimes selecting individuals who have served in a variety of organizations rather than a single one for an extended period, provides expertise with diverse perspective. The emphasis on independence and expertise applies equally to government experts and experts from the larger scientific community.

Some peer reviews can be conducted with two or three reviewers; others involve panels of peer reviewers. In either case, each peer reviewer should have recognized technical expertise that bears on the subject matter under discussion. The peer reviewers of a work product should represent a balanced range of technically legitimate points of view. In addition, cultural diversity and " address" ( e. g., industrial, academic, or environmental community) are other factors that can play a role in selecting peer reviewers.

3.4.2 Where Do I Find Peer Reviewers?

Recommendations for potential peer reviewers can be identified from a number of organizations. These include external groups such as the affected party( ies), special interest groups, public interest groups, environmental groups, professional societies, trade or business associations, state organizations or agencies, Native American Tribes, colleges and universities, the National Research Council, and other Federal agencies with an involvement in or familiarity with the issue. Agency associated groups include the staff of the Science Advisory Board ( SAB) or the Scientific Advisory Panel ( SAP), and relevant scientific and technical experts from Program or Regional offices.

In certain circumstances, existing peer review organizations such as the SAB or SAP may be used to conduct a peer review. These groups establish their own criteria for accepting work and coordination must be made directly with them ( see Appendix D  Guidance on Requesting a Review by the Science Advisory Board) for SAB procedures. Both SAB and SAP conduct formal, public, external peer reviews.

Occasionally, a member of the scientific community will offer his/ her services for peer review during an ongoing peer review. These offers may be at no cost or based on an expectation that reimbursement will be made. Disposition of these unsolicited offers will be handled on a case by case basis by the Peer Review Leader, and as necessary, in consultation with the Peer Review Coordinator, the Office of General Counsel ( OGC), and appropriate Decision Makers. Peer Review Handbook Page 57 If you use a contract mechanism to conduct a peer review, the contractor will have its own pool of scientific and technical experts for peer review. With contractors, EPA can provide information on potential sources of peer reviewers for conducting a peer review if such a listing is prepared in alphabetical order. In no way should EPA insist that the contractor select from EPA s list, or require that the contractor receive EPA approval before selecting any peer reviewer ( sometimes known as a   subcontractor ). However, EPA should review the list of reviewers for conformance to work assignment specifications and adherence to conflict of interest concerns. Furthermore, when utilizing a contract mechanism to conduct peer review, EPA is not permitted to direct the prime contractor to a specific subcontractor ( or peer reviewer) nor is EPA permitted to direct the peer reviewer ( subcontractor). All interactions with the peer reviewers must be coordinated through the prime contractor. ( See Section 3.6 for further information)

Keep in mind that for contracting purposes, contractors are required to obtain Contracting Officer ( CO) approval of subcontractors and the CO generally seeks the input of the work assignment manager ( WAM  the WAM may be the Peer Review Leader in many cases) before approving the use of subcontractors. In this case, as noted above, the Peer Review Leader should review the list of potential reviewers for conformance to the work assignment specifications and adherence to conflict of interest concerns.

3.4.3 Are External or Internal Peer Reviewers Preferred?

External peer reviewers are generally preferred, particularly for most final work products. For some work products, like those reviewed at interim steps, either external or internal peer review may be appropriate. Selection of internal peer reviewers should be based upon technical expertise, available time and " address"  that is, they should not come from the immediate office or group producing the product or have any other connection with the product or document being peer reviewed. External peer reviewers should be selected based upon technical expertise as well, however, care must be taken not to use individuals who have been involved in the development of the work product. ( See Section 1.2.6; see also Sections 1.4.6 to 1.4.9).

3.4.4 What is Important in the Mix of a Peer Review Panel?

A peer review panel or group can number from just a few individuals to ten or more, depending on the issue, the time and resources available, and the broad spectrum of expertise required to treat the range of issues/ questions in the charge. Objective technical expertise and lack of a conflict of interest are critical in selecting peer reviewers. Naturally, experts whose understanding of the specific technical area( s) being evaluated are necessary; nevertheless, it is also important to include a broad enough spectrum of other related experts to completely evaluate the relevant impacts on other less obvious concerns ( i. e., to comment not only if the job is being done right, but also whether the right job is being done). For example, for health related Page 58 Peer Review Handbook peer reviews, experts in such fields as ecology and economics may provide very useful insights. Although persons who are familiar and have a substantial reputation in the field are often called upon repeatedly to be reviewers, it is important to keep a balance with new people who bring fresh perspectives to the review of a work product.

There is usually a continuum of views on any issue. To the extent possible or practicable, selected experts should have technically legitimate points of views that fall along the continuum. A review panel should include experts that are considered   mainstream  ( nearer the center of the continuum) as well as those further along the continuum ( while generally avoiding the extremes). This will help maintain a balanced review panel, while allowing all views to be expressed and discussed. A balanced panel will allow consensus building ( if consensus is the object of a particular peer review; if not, it allows a spectrum of ( re) views for the Agency to evaluate). As a general rule, experts who have made public pronouncements on an issue ( e. g., those who have clearly " taken sides") may have difficulty in being objective and should be avoided.

3.4.5 What is a Conflict of Interest?

The matter of obtaining a fair and credible peer review, as well as maintaining the credibility of the Agency and the Agency s scientific products, is of paramount importance. Peer review leaders are strongly encouraged to obtain peer reviewers who do not have a legal or perceived conflict of interest ( i. e., creates the appearance that the peer reviewer lacks impartiality or objectivity). In reality, we recognize that a totally independent peer reviewer is rare and very difficult to identify ( see Section 3.4.6 for steps to take to ensure a credible peer review).

Conflict of interest is a situation in which, because of other activities or relationships with other persons, an individual is unable or potentially unable to render impartial assistance or advice to the Agency, or the person s objectivity in performing the work is or might be otherwise impaired, or a person has an unfair competitive advantage. Generally, a conflict of interest of interest arises when the person is affected by his/ her private interests, when he/ she or his/ her associates would derive benefit from incorporation of their point of view in an Agency product, or when their professional standing and status or the significance of their principal area of work might be affected by the outcome of the peer review. Clearly, peer reviewers should not be placed in the position of reviewing their own research and analyses that form the basis of the work product under review as this might impair their objectivity. Whenever there are questions about conflicts of interest, you should contact the appropriate official in OGC for clarification.

Be aware that COI can be used in a generic fashion in speaking of common, everyday experience that also involves elements of ethics. In most cases, that is how it is treated in the Peer Review Handbook. However, COI is a very specific issue when used in a governmental Peer Review Handbook Page 59 contracting sense. In the contracting sense, the Federal Acquisition Regulations ( FAR), the Environmental Protection Agency Acquisition Regulations ( EPAAR), and other internal Agency documents define COI and describe the Agency s COI policies and procedures. The Handbook provides guidance in some of these aspects related to peer review ( see Section 3.6 and Appendix E  Example Statements of Work for Contracts), however, specific questions should be addressed to the Office of Acquisition Management.

3.4.6 What Techniques Help Ensure Disclosure and Appropriate Resolution of Conflicts of Interest?

Before finalizing the selection of reviewers, the Peer Review Leader should ascertain whether each potential peer reviewer s involvement in certain activities could pose a conflict of interest ( COI) or create the appearance that the peer reviewer lacks impartiality. One way of identifying conflicts is to ask potential reviewers about current and prior work, and prior clients that might create conflicts or the appearance of a lack of impartiality in carrying out peer review activities. This information obtained by the Peer Review Leader becomes part of the peer review record. When the peer review process is being conducted by a contractor, the requirement for addressing peer reviewers  possible conflicts of interest should be highlighted in the Statement of Work of the work ordering instrument ( e. g., Work Assignment, Delivery Order, Task Order, etc.) and is a matter that is bound by contractual clauses with the Contracting Officer as the final Decision Maker in contracting matters.

Care must be taken to reduce actual or potential organizational or personal conflicts of interest between the reviewers and the work product under review. Remember, each potential conflict situation is unique and must be treated on a case by case basis. The following are considerations that should be addressed in evaluating COI:

a) attention to the employment, financial, and professional affiliations of the participants;

b) exploring directly the issue with each of the participants before the review process takes place;

c) disclosing publicly at the beginning of meetings any previous involvement with the issue;

d) in the cases of regular government employees and Special Government Employees ( SGEs), filing annual ( or updated, as appropriate) Confidential Financial Disclosure Forms ( OGE Form 450), discussing any potential conflicts of interest with their designated ethics official ( DEO), and advising the Peer Review Leader of any relevant concerns; Page 60 Peer Review Handbook e) in the case of non Federal peer reviewers, provide them a copy of the peer review COI inquiry ( see Appendix F  Useful Forms). This form is sent to each prospective peer reviewer by the Peer Review Leader ( or contractor, in the case of contractor run reviews) to advise them of the need to address COI issues prior to the actual review taking place. A follow up contact with the Peer Review Leader ( or contractor, in the case of contractor run reviews) is then made to discuss any relevant issues. The Peer Review Leader then documents this effort in the peer review record; this includes a summary provided by the contractor documenting their inquiries and efforts.

The Peer Review Leader needs to ensure that the peer review COI inquiry ( see Appendix F  Useful Forms) took place and this appears in the peer review record. You should ensure that any documentation in the peer review record does not contain information subject to privacy laws ( for example, see The Privacy Act of 1974 , 5 U. S. C. § 552a  Records maintained on individuals; a copy can be found at: http:// www. usdoj. gov/ 04foia/ privstat. htm). Peer Review Leaders should ensure that any applicable EPA record keeping regulations are followed ( for details, see EPA s National Records Management Program; http:// www. epa. gov/ records/). For information on Freedom of Information Act ( FOIA) requests, please see EPA s Ethics Advisory 88 12 ( http:// intranet. epa. gov/ ogcrmo01/ ethics/ 88 12. htm). If you have any questions, be sure to ask your appropriate Office of General Counsel and/ or office contract official( s).

Established peer review groups such as the Science Advisory Board provide useful models for addressing balance and COI issues. Assistance in identifying conflicts of interest and in providing an appropriate response can be obtained from the Office of the General Counsel. Additional advice can be obtained from one of the Agency s designated ethics officials ( DEOs). Assistance in evaluating conflicts of interest in a contractual situation require the involvement of the Contracting Officer and the resources available within the Office of Acquisition Management.

Of course, conflicts do not necessarily arise merely because a peer reviewer knows something about the subject matter. In fact, experts with a stake in the outcome  and therefore a potential conflict  may be some of the most knowledgeable and up to date experts because they have concrete reasons to maintain their expertise. Such experts could be used provided the potential conflicts of interest are disclosed and the peer review panel or group being used as a whole is balanced. In some cases, however, the conflict may be so direct and substantial as to rule out a particular expert, for instance, a potential peer reviewer who may have a client or employer with a direct financial stake in the particular specific party matter under review ( e. g., i) a Federal grant or contract to the potential peer reviewer or his/ her employer that relates to the matter under review; ii) the potential peer reviewer s or their company s work on a specific chemical under review). However, review of a general methodology that applies to numerous Peer Review Handbook Page 61 chemicals would not necessarily raise such a concern. ( Note : COI language should be made part of contracts/ statements of work ( SOW) or purchase orders ( PO)  see Section 3.6.)

A Peer Review Leader may also want to adopt measures that will prevent creation of conflicts as the peer review is underway. Any measures contemplated that involve a contractual action must be coordinated with the cognizant Contracting Officer. Some measures might include clauses in a contract or purchase order that require reviewers to receive advance approval on future work, or place limits on such work, while they are performing the current peer review. Note that at some level these types of measures will discourage experts from serving as peer reviewers. ( See Section 3.6.5 for further information dealing with contracts and suggestions for appropriate management controls.)

3.4.7 Can Parties External to EPA Pay for Their Own Peer Reviews?

There may be instances where parties external to EPA will want to conduct and/ or pay for a peer review on a particular work product ( presumably their own work product or one they are closely interested in, or they wouldn t be interested in expending resources). This may look benign at first blush, but is a very complex and sensitive situation that can raise significant concerns for perceived and/ or actual conflicts of interest for interested parties   paying  for a peer review of their own work product. While the Agency cannot prevent external parties from conducting and paying for a peer review, it is desirable that any such peer review meets the intent of the Agency s Peer Review Policy and adheres to the principles and guidance in this Handbook. If the external party submits their work product and accompanying peer review, the materials will be treated by the Agency as anything else submitted for the Agency s evaluation.

We will evaluate the work product and the peer review for scientific credibility and validity before making any decisions based on the materials.

[ This issue is under active Agency consideration at this time; therefore, users of this Handbook should be aware that the language in Section 3.4.7 will likely change in the future.]

3.4.8 Are There Constraints to Selecting Peer Reviewers?

Sometimes the need for a peer review is accelerated due to a court ordered deadline or other time sensitive requirements. In such cases, it is difficult, if not impossible to obtain external peer reviewers in time to conduct a full external peer review. It may even be impossible to conduct a small scale internal peer review using just a few individuals. Mechanisms for identifying and using a small number of peer reviewers should be developed so that quick, effective peer review can be included for even the most rapidly moving products. Page 62 Peer Review Handbook Another possible constraint involves confidential business information ( CBI). To evaluate certain Agency generated studies properly, some peer reviewers may need access to CBI. However, unless the reviewers are Federal employees with CBI clearance, the Agency doesn t have the independent authority to disclose CBI to them. Therefore, whenever contemplating the use of outside peer reviewers, Agency staff should determine whether the reviewers will need access to CBI. If they don t have CBI clearance, the Office of the General Counsel should be consulted on whether it is practical to obtain the consent of CBI submitters to disclose the information to peer reviewers.

Offices need to be aware of the requirements of the Federal Advisory Committee Act ( FACA) when establishing peer review mechanisms ( see Section 2.8). Federal advisory committees that are subject to chartering by the General Services Administration must hold meetings that are open to the public, and have balanced membership requirements. The Office of the General Counsel should be consulted regarding the applicability of FACA to peer review panels.

3.4.9 If State Employees are Used as Peer Reviewers, Can EPA Pay Them for this Service?

Maybe. First, the Peer Review Leader needs to determine: a) if the State agency has a policy on whether their employees can perform this type of work for EPA, and b) if the person will be acting as an individual or as an employee of the State. If the State allows the person to be a peer reviewer for EPA and the person is acting as an individual, we can pay them for their peer review services. If the State allows the person to be a peer reviewer and the person will be acting as a State employee, we can only pay them if State policy allows us to. The most efficient vehicle for paying the State employee will likely be a contract or purchase order ( see Sections 3.6.6 & 3.6.7), but you should consult with OGC and the procurement office for advice. In most instances, EPA can also pay travel expenses ( consult with your administrative staff for details).

If we don t pay a State person for their peer review services, they must sign an agreement stating that they don t expect payment ( See Section 3.6.1  gratuitous services).

3.5 Materials for Peer Reviewers 3.5.1 What Instructions Do You Give Peer Reviewers?

The Peer Review Leader is responsible for ensuring that peer reviewers understand their responsibilities ( see Section 3.6 if a contract is involved): Peer Review Handbook Page 63 a) Advise the Agency of actual or potential organizational or personal conflicts of interest or other matters that would create the appearance of a lack of impartiality ( e. g., see Section 3.4.6)

b) Provide written comments in specified format by the specified deadline that are responsive to the charge c) Comply with any requests for not disclosing draft work products to the public 3.5.2 What Materials Should be Sent to Peer Reviewers?

For a peer review to be successful, peer reviewers should receive several documents at the beginning of the process. Typically, the most important among these documents are the charge letter and the current work product. The charge letter describes what the peer reviewers are being asked to do, and should serve to focus and structure the review. The work product is, of course, the material being subject to peer review.

Remember, no documents should be provided directly to a potential peer reviewer when that reviewer is going to be working under a contract or purchase order. In the case of a contract, the Agency provides the work product with associated background material to be peer reviewed to the prime contractor who in turn distributes these documents to the peer reviewers. In the case of a purchase order, the   charge or statement of work  must be part of the PO ( purchase order) and the provision of any documents needs to be coordinated with the purchasing agent handling the order.

a) Essential documentation for each peer reviewer includes:

1) A current copy of the work product to be peer reviewed with associated background material. The work product needs to be of the best possible scientific/ technical quality to ensure an adequate and useful peer review.

2) A clear charge or statement of work seeking informed comment on identified issues to properly focus the efforts of the peer reviewers and ensure that their individual efforts can be compared or contrasted.

3) Some information concerning the process that you use for the peer review, including the due date of reviewer comments, the format of those responses, and a point of contact in case the peer reviewer has questions. Responses should be written and submitted to the Peer Review Leader by an agreed upon deadline. In certain rare cases, oral commentary may be Page 64 Peer Review Handbook sufficient. However, in such cases, a follow up written response for the record is required.

4) In some cases, Agency materials being peer reviewed will be available to the public, even if they are marked as drafts. For example, all materials reviewed by the SAB are available. Agency managers may also decide that a broad accessibility has benefits for the Agency. In other cases, confidentiality needs to be maintained. In these cases, each peer reviewer must be informed of the need for confidentiality with regard to the release of Agency products that are stamped as " DRAFT" or " DRAFT  Do Not Cite, Quote, or Release." Premature release of draft Agency products, views, or positions is inappropriate and can be damaging to the credibility of the Agency or the peer reviewer. While not necessarily having legal consequences, such language will be included in the charge to the peer reviewers. Other mechanisms to use in discouraging premature release include a disclaimer that appears in a separate section at the front of the document and creating the document with watermarks clearly delineating DRAFT status ( or a header or footer that states DRAFT status) on every page. In addition, in any solicitation for peer reviewers, the necessity for confidentiality and the non release of materials shall be emphasized.

b) Useful, but not critical materials that may be sent to peer reviewers include:

1) The name, address, and phone and fax numbers, and/ or Internet address of each peer reviewer working on the specific review 2) A bibliography and/ or any particularly relevant scientific articles from the literature 3) A work product that has line numbering added in the margin for ease in providing and referencing comments c) Peer Reviewers should be given what is needed to complete their task  they should not be overburdened with excess material.

3.5.3 How Closely can EPA Interact with Peer Reviewers During the Review?

a) When EPA Conducts the Peer Review  The Peer Review Leader normally has administrative contacts with the reviewers during the development and conduct of the peer review. In some cases ( e. g., SAB peer review), peer reviewers may also receive a briefing on the product to be peer reviewed. Otherwise, the Peer Peer Review Handbook Page 65 Review Leader and other EPA staff should not contact the reviewers during the course of the review. Such contact can lead to perceived conflicts or inappropriate direction that could compromise the independence of the review.

b) When the Contractor Conducts the Peer Review  If peer review is conducted under a contract mechanism, EPA must limit direct contact to the prime contractor s designated representative and not have general contact and direction to the contractor s staff or peer reviewers ( sub contractors). Note, when a peer review is conducted under a contract, there are constraints where EPA staff are prohibited from contacting peer reviewers to avoid personal services arrangements. Personal services contracts exist when the nature of the relationship between the contractor and the EPA can be characterized as an employer  employee relationship. Any communications with peer reviewers must be coordinated through the prime contractor.

3.6 Peer Review Services A range of peer review services are available to the Agency including internal, external ( gratuitous services, contracts, purchase order), and Special Government Employee ( SGE) mechanisms. The mechanism selected is generally based on the nature of the scientific or technical work product.

3.6.1 What are Gratuitous Services for Peer Review?

The provision of peer review products or services to EPA without compensation are provided as so called   gratuitous  services. If a person wishes to perform peer review services for EPA without compensation, the EPA must ask them to sign an agreement whereby the person agrees to provide the prescribed peer review services as gratuitous services, with no expectation of receiving compensation for these services from EPA ( see Appendix F for EPA Form 3100 14 which is used in situations where a gratuitous services contract is not used). An agreement ( such as Form 3100 14) must be executed because the Antideficiency Act ( 31 U. S. C. § 1342) prohibits the Agency s acceptance of   voluntary  services (  voluntary services  are services provided to EPA without an agreement in advance that such services are provided at no cost to EPA). Note that persons cannot waive compensation ( i. e., agree to provide gratuitous services) for which there is a statutory right to payment, unless a law permits the waiver. ( For situations concerning State employees, see Section 3.4.10. Laws that permit services without compensation are 5 U. S. C. § 3109 for experts or consultants and 5 U. S. C. § 3111 for student volunteers.) Page 66 Peer Review Handbook 3.6.2 Can I Use a Contract to Obtain Peer Review Services?

The Agency may contract for peer review services. The contract may be written solely for peer reviews or be included as one of several specifically described interrelated tasks in a contract that requires the contractor to provide more than just peer review services.

For assistance in preparing the necessary pre award documents, program officials should consult Chapter 2 of the Contracts Management Manual ( CMM). Chapter 2 is available on EPA s Intranet ( http:// epawww. epa. gov/ oamintra/ policy/ cmm. pdf ) and on the Agency LAN Services ( Administration/ Management/ OAM Procurement Policy Information).

3.6.3 How Do I Write a Statement of Work for Contracts?

The Statement of Work ( SOW) must clearly specify that the contractor is responsible for preparing peer review evaluations and set forth guidelines for the peer review of scientific or technical documents. The contractor may perform the peer review with in house staff, subcontractors or consultants. Any guidelines for performing peer reviews to ensure soundness and defensibility must be developed by the program office and made part of the contract. The contractor would then ensure that the peer reviews adhere to the guidelines.

The SOW should include the list of questions that EPA wants the contractor to include in the charge to the peer reviewers. While the contractor will be the one that will prepare and send the formal charge to the peer reviewers, EPA should provide the list of questions to the contractor ( see Section 3.2 for general discussion on charge to peer reviewers).

Within the SOW, the contractor should be directed to inquire whether prospective peer reviewers have any actual or potential organizational or personal conflicts of interest or other matters that would create the appearance of a lack of impartiality, including whether they have had or presently have a financial relationship with EPA. Further, the SOW needs to specify that contractors are required to provide EPA with a summary of the procedures and efforts made to identify, disclose, and ensure that no independence or conflict of interest concerns arise during the performance of the contract.

The SOW cannot simply define the role of the prime contractor as arranging for the services of others to perform peer reviews and logistics for meetings. Unless the prime contractor is clearly tasked with responsibility for performing peer reviews, individual peer reviewers' fees and associated travel expenses are not payable under the contract.

The EPA may pay for the reviewer's comments or evaluation, and also for attendance at a meeting with the Agency and other reviewers to discuss the results of the peer review. If the SOW calls for the preparation of comments or an evaluation, and specifies a meeting with the Peer Review Handbook Page 67 Agency and other peer reviewers to discuss the results of the peer review, payment is appropriate. The peer reviewer's attendance at the meeting would then be part of contract performance.

Example statements of work are found in Appendix E ( Example Statements of Work for Contracts).

3.6.4 What are Advisory and Assistance Services ( AAS) or Sensitive Activities?

Contracts that provide services that support or improve Agency decision making or policy development are subject to special management controls. These services include services acquired from non Governmental sources by contract to support or improve Agency policy development, decision making, management, and administration, or research and development activities. See Federal Acquisition Regulation ( FAR) 37.201 for a more specific definition of AAS. Such services may take the form of information, advice, opinions, alternatives, conclusions, recommendations, training, and direct assistance. For additional information on advisory and assistance services and sensitive activities, program officials should review Chapter 2 of the CMM.

New contracts for these services require management approvals prior to issuance of the solicitation. For the thresholds that have been established for approval of these justifications, see Figure 5, Item B Management Approvals set forth after Chapter 2 of the CMM ( for current approval levels).

3.6.5 What are Some Management Controls for Contracts?

Contracting for peer review services is permitted. However, because of the potential for improper use of these contracts, special management controls are required.

a) Inherently Governmental Functions ( IGFs)   OFPP Policy Letter 92 1, dated September 23, 1992, describes ( 1) functions that are inherently governmental and must be performed only by Government employees and ( 2) functions that may be contracted, but so closely support Government employees in their performance of IGFs that the contract terms and performance require close scrutiny by Federal officials. Federal Acquisition Regulation ( FAR) coverage of inherently governmental functions is at FAR Subpart 7.5.

Peer reviews represent only a contractor's recommendations, advice or analysis of a document. Agency officials must make the official Agency decision regarding acceptability and/ or quality of the document. To ensure that Agency officials are not improperly influenced by recommendations in the peer review, management Page 68 Peer Review Handbook controls must be included in the contract. One possible control would be to require the peer reviewers to submit with their evaluations or comments a description of the procedures used to arrive at their recommendations; a summary of their findings; a list of sources relied upon; and make clear and substantiate the methods and considerations upon which their recommendations are based. To the extent possible, the contract should set forth any guidelines or criteria for performance of the peer review. Agency officials should document their evaluations of the quality and validity of the peer review.

b) Conflict of Interest ( COI)  Another important factor is that the objectivity of the peer review should not be improperly influenced or undermined by the contractor performing the review. To identify and avoid or mitigate actual or potential COI, the contract should include controls. Such controls might require the contractor to report on prior and current work, and prior clients that might create COI. Other controls might include Agency review and placing limits or advance approval on future work. There should also be procedures implemented to assure that the contractor does not gain an unfair advantage in future requirements as a result of their performance of peer reviews. Program officials should consult the Contracting Officer ( CO) for special contract clauses.

FAR coverage of conflicts of interest is at FAR Subpart 9.5. The EPA Acquisition Regulations ( EPAAR) at 48 CFR Subpart 1509.5 generally mandates conflicts of interest solicitation provisions and contract clauses, but makes them optional for procurements accomplished through the use of simplified acquisition procedures as set forth in FAR Part 13. Also see 48 CFR 1509.507 1( b)( 3) and 48 CFR 1552.209 70,  71 &  72 as additional resources.

Contract for peer review services: An EPA contracting officer will include conflicts of interest solicitation provisions and contract clauses as a matter of course without involvement by the EPA project officer, if the peer review services are not obtained pursuant to the simplified acquisition procedures in FAR Part 13. If the peer review services are subcontracted pursuant to a prime contract, then the prime contractor is ordinarily required to include a conflicts of interest clause substantially similar to the conflicts of interest clause in the primary contract in its subcontract to the peer reviewer.

Although the EPA contracting officer and/ or prime contractor has the primary responsibility to include the required conflicts of interest provisions/ clauses, the EPA project officer may nevertheless wish to: Peer Review Handbook Page 69 1) Highlight the conflict of interest requirements in the Statement of Work ( SOW) for the procurement of the peer review services 2) Develop a specific conflict of interest clause regarding the peer review at issue as a substitute or in addition to the standard conflicts of interest clause 3) Review the solicitation/ contract to make sure that the required conflicts of interest clause has been included 4) Ensure that the SOW includes a requirement for the contractor to provide EPA a summary of the work performed to identify and resolve conflict of interest/ independence concerns with peer review candidates Simplified acquisitions of peer review services: Although conflict of interests requirements are optional for simplified acquisitions, they are nevertheless a good idea. Accordingly, an EPA project officer obtaining peer review services with simplified acquisition procedures ( see Section 3.6.7) should request the purchasing agent/ contracting officer to include a conflict of interest solicitation provision and contract clause in the purchase order.

c) Confidential Business Information ( CBI)/ Privacy Act Protected Information and Other Sensitive Information  When peer reviewers are not employees of the United States Government, it is unlikely that the EPA will have authority to give reviewers access to confidential business information in the absence of consent for such disclosure by the CBI submitter. Therefore, all documents provided to non Federal reviewers must be screened for information claimed as CBI. Even where business information has not been explicitly claimed as CBI, if it is of a kind where the submitter might be expected to object to its release, prior to release the submitter must be asked whether it wishes to assert a claim, unless the submitter has previously been informed that failure to assert a CBI claim may result in disclosure without notice. Language is included in the contract to clearly identify any required procedures or processes prior to release of any protected information, including any requirements for confidentiality agreements, as well as limits on use and disclosure of the data by contractor personnel.

d) Personal services  Under contracts, the EPA may not engage the peer reviewers in any improper personal services relationships, i. e., an arrangement under which contractor personnel are subject to relatively continuous supervision and direct control by an Agency official or employee. These relationships are characterized Page 70 Peer Review Handbook as one where the contractor employee interacts with the Agency in a manner similar to that of a Federal employee.

To avoid these improper relationships, program officials should write well defined SOWs. The SOWs should set forth the requirements in detail for work to be performed independently, including the manner in which it will be evaluated. The SOW must set forth what work is to be performed not how the work is to be performed. Technical direction may be used to clarify ambiguous technical requirements to ensure efficient and effective contractor performance, and is not considered supervision or assignment of tasks. For additional information, program officials should consult EPA Order 1901.1A, Use of Contractor Services to Avoid Improper Contracting Relationships and FAR Subpart 37.1 3.6.6 Can I Identify and/ or Select Peer Reviewers When Using a Contract?

No. Program officials cannot interfere in a contractor's authority and responsibility to perform work by " selecting" who will perform the peer review ( doing so may invoke FACA  see Section 2.8). The Federal Acquisition Regulation ( FAR) governs the Contracting Officer s ( CO's) and program officials' relationship with the contractor. Keep in mind that for contracting purposes, contractors are required to obtain Contracting Officer ( CO) approval of subcontractors and the CO generally seeks the input of the work assignment manager ( WAM  the WAM may be the Peer Review Leader in many cases) before approving the use of subcontractors.

EPA can establish criteria for the sort of individuals that might participate on a peer review panel. However, the Agency must not be involved in the selection of individual peer reviewers, and should avoid commenting on the contractor s selection of peer reviewers other than to determine whether the panel, once selected, meets the criteria established. EPA may identify a pool of qualified subcontractors and consultants to the prime contractor ( listed in alphabetical order), but cannot direct the use of any particular subcontractor or consultant.

3.6.7 Can I Use Simplified Acquisition Procedures to Obtain Peer Reviewers?

In some instances, peer reviewers can be obtained via simplified acquisition procedures. The acquisition of supplies or non personal services from the open market and on a sole source basis when the aggregate amount involved in any one transaction does not exceed $ 100,000 constitutes a simplified acquisition ( FAR Part 13). The same considerations in the Section 3.6.5 discussion on IGFs, COI, access to CBI, and personal services apply to simplified acquisitions. Normally, the Government issues a purchase order directly to the individual peer reviewer, instead of to a prime contractor who may subcontract for performance of the peer review. Peer Review Handbook Page 71 a) Approvals  All small purchases for peer reviews are considered Advisory and Assistance Services ( AAS). See Chapter 2 of the Contract Management Manual ( CMM) for the appropriate approval levels for AAS.

b) Competition  The Federal Acquisition Regulation ( FAR) requires competition for purchases in excess of $ 2,500. Purchases for more than $ 2,500 and not more than $ 100,000 are to be made only from small businesses unless the Contracting Officer is not able to obtain two or more offers from small businesses that are competitive in terms of market price, quality, and delivery. Only one source need be solicited if the Contracting Officer determines that only one source is reasonably available. Contracting Officers are encouraged to use best value.

c) Procurement Requests  Program Officers should include the following in all PRs for the purchase of peer reviews:

1) A fixed price amount at or below the simplified acquisition threshold 2) A detailed description of the requested services, inclusive of:

( a) Total quantity per line item ( b) Estimated unit price per line item ( c) Total cost per line item ( d) Specific deliverables for each line item ( e) Total cost of the purchase request 3) Reference FAR Subpart 3.6 and Environmental Protection Agency Acquisition Regulation ( EPAAR) Subpart 1503.601 regarding sources from Government employees or organizations owned and controlled by them.

4) Provide sources from small businesses, if available.

5) If the request is a sole source purchase, justification must be provided in accordance with the EPAAR Subpart 1513.170 1. Page 72 Peer Review Handbook 3.6.8 How is Travel Handled with Contracts or Purchase Orders?

Funds obligated on a contract or purchase order are available to pay for the costs of producing the peer review including the travel costs and fee of the peer reviewer.

The EPA acquires peer reviews through simplified acquisitions issued directly to peer reviewers or through contracts with companies, which provide the peer review services. By issuing a purchase order or awarding a contract for peer review services, the EPA may pay not only for the peer review services/ comments, but also for participation in a meeting with the Agency and other reviewers to discuss comments. The scope of work of the contract must require the contractor or individual peer reviewer as appropriate to provide peer review services and indicate whether the contractor or peer reviewer will be required to discuss a specific peer review work product with the Agency and/ or with other peer reviewers. Participation in a meeting to discuss a peer review work product would then be part of the contract's performance. Thus, the contract may serve as the mechanism to pay for peer review services and associated travel expenses to provide comments to the EPA.

3.6.9 How is Travel Handled with Special Government Employees?

The term Special Government Employee ( SGE) is defined in 18 U. S. C. 202( a) as an officer or employee of an agency who performs temporary duties, with or without compensation, for not more than 130 days in a period of 365 days, either on a full time or intermittent basis.

Travel and per diem expenses of experts hired as SGEs for peer review may only be paid through the issuance of invitational travel orders ( 5 U. S. C. § 5703). These invitational travel and per diem expenses should be charged to an appropriate EPA travel account. The Federal Travel Regulations govern the invited travelers reimbursement.

Members of the SAB, SAP, and other FACA advisory committees are often brought on board as SGEs. It is not appropriate to reimburse travel or per diem expenses of advisory committee members ( SGEs) through a contract. Peer Review Handbook Page 73 4. COMPLETING A PEER REVIEW 4.1 Overview Performance of the formal peer review is not the final stage in the development of the work product. Rather, it is an important stage in developing the work product, with the final work product representing the true end of the peer review. As a result, the peer review process closes with three major activities: evaluating comments and recommendations, utilizing peer review comments for completing the final work product, and organizing and maintaining a record of the peer review.

Careful attention to all of these elements, singly and together, assures a credible peer review process. Conversely, inattention can nullify the peer review attempt. A well planned peer review applied to a reasonable quality starting work product, followed by responsible, visible utilization of peer review suggestions in the final product assures a credible product for use in Agency decision making. The peer review is not complete untilThe peer review is not complete until the the peer review comments are peer review comments are incorporated into the incorporated into the final work final work product, or reasons are stated why product. such comments are not to be incorporated. However, for the purposes of the annual report to the Agency s Peer Review Product Tracking Database only, the work product can be moved from List B ( Candidate Products for Future Peer Review) ( or List C  Products for Which a Decision has been made not to Peer Review) in some instances) to List A ( Products Peer Reviewed Since 1991) when the Decision Maker decides on how the peer review comments will be addressed and this decision is documented in the peer review record.

The peer review record is completed only when it contains a copy of the final work product ( when there is one) that addresses the peer review comments.

4.2 Final Work Product 4.2.1 How Do I Incorporate Peer Review Comments into the Final Work Product?

The Peer Review Leader must carefully evaluate and analyze all peer review comments and recommendations. As discussed earlier, a carefully crafted charge to the peer reviewers simplifies organizing and analyzing comments. Also, any other issues that are raised need to be identified and evaluated. Page 74 Peer Review Handbook The validity and objectivity of the comments need to be evaluated. Analyses may include consultation with other experts/ staff within the Office and/ or Agency. Adequate documentation is needed to show that comments are accepted or rejected   the documentation can be brief, but must address the legitimate, valid comments. The peer review record must contain a document describing the Agency s response to the peer review comments.

The Peer Review Leader should brief the Decision Maker ( including all appropriate managers in the Peer Review Leader s chain of command) on the charge, profile of peer reviewers, the peer review comments, and provide a proposal on how to address the peer review comments. It is the responsibility of the Peer Review Leader to obtain Decision Maker approval of the approach to addressing the peer review comments. The Peer Review Leader should clearly identify for the Decision Maker any major peer review comments that will not be accepted and why.

Comments that have significant impact on time, budgetary, and/ or resource requirements are particularly important and need to be evaluated in consultation with management. These comments may lead to allocation of additional resources and a revised schedule for the completion of the work product.

4.2.2 What Actions are Potentially Forthcoming from Peer Review?

Peer review comments and recommendations may entail significant impacts on the planned project schedule, budget, or other resource requirements. Management decisions related to revisions in one or more of these areas may be appropriate.

The substantive issues or concerns expressed by peer reviewers may suggest that wider scientific and technical consultation is needed to ensure the adequacy of the work product.

The peer review comments and recommendations on a final product may provide a basis for bringing the associated project to closure.

4.2.3 What Should the Final Work Product Say About the Peer Review Process?

If the product has been peer reviewed, you should describe the peer review in the document. Frequently, this will be part of a description of the process of developing the product. It can be brief and does not need to describe the process or discuss the peer review comments in great detail. The description can be included in an introduction, preamble, or appendix.

When there were significant peer review comments, and particularly if they are not being accepted, the document will generally discuss the issue and describe the reasons for the Agency s choices in the appropriate sections of the document. The level of detail that is needed Peer Review Handbook Page 75 is a matter of judgment and will depend on the importance and degree of controversy of the issue.

If a major scientific or technical product has not been peer reviewed, this fact should be noted in the document, perhaps in an introduction or description of its scope. This section should briefly indicate the reasons that peer review was not conducted.

Derivative products of major scientific or technical products ( such as fact sheets, press releases, and brochures) do not need to discuss whether the underlying products were peer reviewed.

4.2.4 Can the Identity of Peer Reviewers be Kept Anonymous?

Yes, up to a point. In the ordinary course of events, you can often discuss comments received without attributing the comments to a specific reviewer. However, if the matter has gone to litigation, the litigating parties can discover the names of anyone who contributed to a Federal product, including peer reviewers. Therefore, it is not possible to totally shield peer reviewers. In addition, it may be difficult to shield the names of the peer reviewers when the Agency is responding to a Freedom of Information Act ( FOIA) request.

If a peer reviewer requests anonymity at the outset of the peer review, the Peer Review Leader needs to inform the peer reviewer of the above possible eventualities. The Agency will in the ordinary course of events attempt to maintain the confidentiality of the peer reviewers and their comments from public disclosure, but it is recognized in many instances, for example open public meetings and the above circumstances, this can t be assured. Remember, the Agency is committed to working   as if in a fishbowl  and most of its activities are transparent to the public ( except where confidential business information is concerned). It is recognized that this may be a deterrent to possible peer reviewers, but this is a reality that has to be understood.

4.3 Completing the Peer Review Record 4.3.1 How Do I Complete the Peer Review Record?

Once the Peer Review Leader has completed the peer review and the final work product ( where one is prepared), the peer review record is brought up to date and then archived according to that organization s procedure ( see Section 4.3.2). The peer review record must be indexed and maintained in an organization s archive ( repository). The location of the peer review record needs to be readily identifiable so interested parties can locate and obtain materials easily and quickly. The peer review record should be placed in any associated established public docket, if required, in addition to the organizational archive. As a courtesy, a copy of the revised work product may be sent to the peer reviewers for information. Page 76 Peer Review Handbook The Peer Review Leader will collect the following materials for the peer review record and submit for archiving; including at least ( see also Section 2.5.3):

a) The draft work product submitted for peer review b) Materials and information ( including the charge) given to the peer reviewers c) Written comments, information, and materials received from the peer reviewers d) Information about the peer reviewers ( such as reviewers  names, affiliations, and a summary of efforts to identify potential conflicts and their resolution)

e) Logistical information about conduct of the peer review ( such as times and locations of meetings)

f) A memorandum, or other written record, approved by the Decision Maker, responding to the peer review comments specifying acceptance or, where thought appropriate, rebuttal and non acceptance g) The final work product 4.3.2 Where Should the Peer Review Records be Kept, and for How Long?

During the active conduct of the peer review, the Peer Review Leaders maintain the peer review record with themselves until the peer review is totally completed. Minimally, the file should be maintained until one year after the completed peer review is reported in the next annual reporting. After that, the peer review record should be maintained for a   reasonable period of time.  Establishment and maintenance of the archive where the peer review records ultimately reside are an organization s responsibility ( i. e., not that of an individual program manager or Peer Review Leader). Generally, to allow flexibility, individual offices and regions will decide the appropriate level of organizational responsibility and how they will meet the   routinely available  requirement. The peer review record may be kept with other records relating to the overall project, as long as it is easily and separately identifiable. The peer review record should be maintained in accordance with the Agency s record keeping schedule for such records. One long term archiving mechanism may be the formal archiving at the Federal Records Center in Suitland, MD. ( Note: This is the same question as Section 2.5.7, but applies in this chapter as well). Peer Review Handbook Page 77 4.3.3 Is Information Regarding a Peer Review Subject to Release under FOIA?

Yes, it is subject to release if EPA receives a Freedom of Information Act ( FOIA) request, unless the peer review information meets the criteria for an exemption under the FOIA. ( http:// www. epic. org/ open_ gov/ foia/ us_ foia_ act. html). Peer Review Handbook Page 79 SUBJECT INDEX This is an alphabetical listing of subjects from the Handbook and the pertinent page numbers where they are found.

 A  Administrative Procedures Act ( 12) Administrator ( 9, 17) Advisory and Assistance Services ( 66) Analytic blueprints ( 13, 25, 40, 49) Annual reporting requirements ( 14, 18) Anonymity of peer reviewers ( 75) Archiving ( 47, 75) Assistant Administrators ( 17)  B  Balancing of peer reviewers ( 22) Board of Scientific Counselors ( BOSC) ( 41) Budget ( 18, 40, 48)  C  Candidate for peer review ( 26) Categories of work products ( 16, 25) Charge ( 46, 53, 76) Chemical action reports ( 16) Clean Air Scientific Advisory Committee ( 41) Compensation ( 65) Competition ( 70) Completing a peer review ( 4, 73) Conducting a peer review ( 3, 53) Conference proceedings ( 25) Confidential business information ( CBI) ( 61, 69) Confidential Financial Disclosure ( 59) Confidentiality ( 63) Conflicts of interest ( 22, 31, 58, 67) Contracting officer ( 57, 63, 68, 69) Contracts ( 31, 64, 65, 68) Contracts Management Manual ( CMM) ( 65, 67) Controversy ( 27, 40) Cooperative agreements ( 31) Court ordered deadlines ( 44, 61)  D  Decision Maker ( 17, 26, 39) Deputy Administrator ( 14, 17, 18) Designated Ethics Official ( DEO) ( 59) Division Directors ( 17) Docket ( 46, 47, 75)  E  Economic work products ( 27) Electronic records ( 46) Environmental Impact Statement ( EIS) ( 35) Environmental regulatory models ( 36) Executive Order 12866 ( 13, 29) External peer review ( 40, 41, 57)  F  Federal Acquisition Regulation ( FAR) ( 67) Federal Advisory Committee Act ( FACA) ( 49, 62) Federal Register ( 12, 50) Final work product ( 73) Freedom of Information Act ( FOIA) ( 16, 42, 60, 77) Frequency of peer review ( 43)  G  Grants ( 31) Gratuitous services ( 65)  I  Independent peer reviewer ( 10, 11, 21, 55) Inherently governmental functions ( IG Fs) ( 67) Interacting with peer reviewers ( 64) Interagency agreement ( 31) Internal peer review ( 41) Internal peer reviewer ( 22)  J  Journal article ( 16, 42)  L  Legal considerations ( 49) Letter reviews ( 41, 50) List A ( 15) List B ( 15) List C ( 15, 16) List D ( 16, 42) Litigation ( 16, 75)  M  Major impact ( 27, 40) Major scientific and technical work product ( 5, 9, 15, 26, 37) Page 80 Peer Review Handbook Managers planning checklist for peer review ( 5) Materials for peer reviewers ( 62) Mechanisms for peer review ( 39)  N  National Academy of Sciences ( 31, 42) NEPA Products ( 35) Non major scientific and technical work products ( 16, 37, 38) Novelty ( 27, 40) NPDES permits ( 16)  O  Office Directors ( 17) Office of Research and Development ( ORD) ( 14, 20, 23) Open meeting ( 49) Other organization s work products ( 36)  P  Peer consultation ( 11) Peer input ( 10) Peer involvement ( 10) Peer review ( 9, 10) Peer Review Advisory Group ( PRAG) ( 17, 23) Peer review comments ( 46, 47, 73, 76) Peer Review Coordinator ( 14, 17, 20, 23) Peer Review Leader ( 17 19, 39, 47, 61, 62, 73, 75) Peer review materials ( 62) Peer review mechanism ( 5, 39) Peer review panel ( 22, 56, 57) Peer Review Product Tracking ( PRPT) database ( 14, 19, 34, 45) Peer review record ( 5, 21, 45, 75) Peer review schedule ( 18) Peer review services ( 65) Peer reviewers ( 21, 46, 55, 56, 61, 70, 76) Personal services ( 69) Planning a peer review ( 2, 25) Premanufacturing notices ( PMNs) ( 16) Privacy Act ( 60) Procurement requests ( 71) Project manager ( 18, 19, 37) Public comment ( 12)  R  RCRA permits ( 16) Record of Decision ( ROD) ( 25) Records management ( 60) Regional Administrators ( 17) Regulatory development process ( 13) Regulatory negotiations ( 14) Rulemaking ( 13, 27, 46)  S  Science Advisory Board ( SAB) ( 28, 41, 50, 56) Science Policy Council ( SPC) ( 14, 17, 20, 23) Scientific Advisory Panel ( SAP) ( 41, 50, Scientific and technical work product ( 41, 50, 56) Selection of peer reviewers ( 25) Sensitive activities ( 55) Simplified acquisitions of peer review services ( 66) Site specific decisions ( 69, 70) Social Science work product ( 35) Special Government Employee ( SGE) ( 30) Stakeholder involvement ( 59, 72) State employees ( 12, 13, 40, 54) Statement of work for contracts ( 62) Strategic plans ( 59, 66) Subcontractor ( 25) Subject matter expert ( 51, 57)  T  Tier 1 and Tier 2 ( 22) Tier 3 ( 13, 55) Time frame for peer review ( 13) Training ( 44, 55) Travel ( 18, 19)  V  Voluntary consensus standards ( 71) Voluntary services ( 39)  W  Work Assignment Manager ( WAM) ( 57) Peer Review Handbook Page 81 COMMONLY USED ACRONYMS AA Assistant Administrator AAS Advisory and Assistance Services ASTM American Society of Testing and Materials BOSC Board of Scientific Counselors CBI Confidential Business Information CFR Code of Federal Regulations CMM Contracts Management Manual CO Contract( ing) Officer COI Conflict of Interest DEO Designated Ethics Official EIS Environmental Impact Statement EPA Environmental Protection Agency EPAAR EPA Acquisition Regulations FACA Federal Advisory Committee Act FAR Federal Acquisition Regulations FIFRA Federal Insecticide, Fungicide and Rodenticide Act FOIA Freedom of Information Act FTE Full Time Equivalent GSA General Services Administration HHS Health and Human Services IAG Interagency Agreement IGF Inherently Governmental Function IRIS Integrated Risk Information System ISO International Organization for Standardization LAN Local Area Network NAS National Academy of Sciences NEPA National Environmental Policy Act NPDES National Pollutant Discharge Elimination System NTTAA National Technology Transfer and Advancement Act of 1995 OAM Office of Acquisition Management OEI Office of Environmental Information OGC Office of General Counsel OMB Office of Management and Budget ORD Office of Research and Development PMNs Premanufacture Notice PO Purchase Order PRAG Peer Review Advisory Group PRPT Peer Review Product Tracking Database RA Regional Administrator RC Regional Counsel RCRA Resource Conservation and Recovery Act RIAs Regulatory Impact Analyses Page 82 Peer Review Handbook ROD Record of Decision SAB Science Advisory Board SAP Scientific Advisory Panel SGE Special Government Employee SOPs Standard Operating Procedures SOW Statement of Work SPC Science Policy Council USC United States Code WTI Waste Technologies Industries WAM Work Assignment Manager Peer Review Handbook Page A 1 APPENDIX A  U. S. ENVIRONMENTAL PROTECTION AGENCY PEER REVIEW POLICY June 7, 1994 Page A 2 Peer Review Handbook June 7, 1994 MEMORANDUM SUBJECT: Peer Review Program TO: Assistant Administrators General Counsel Inspector General Associate Administrators Regional Administrators Staff Office Directors Today, I am reaffirming the central role of peer review in our efforts to ensure that EPA policy decisions rest on sound, credible science and data ( see attached policy statement). Toward that end, as its first major task, EPA s Science Policy Council ( SPC) is instituting a program to expand and improve peer review in all EPA offices. This memorandum gives an overview of current practices and outlines the new program.

Peer Review Practices and Policy Peer review at EPA takes several different forms, ranging from informal consultations with Agency colleagues who were not involved in developing the product to the formal, public processes of the Science Advisory Board ( SAB) and the FIFRA Scientific Advisory Panel ( SAP). In any form, peer review assists the Agency s work by bringing independent expert experience and judgment to bear on issues before the Agency to the benefit of the final product.

EPA s Peer Review Policy, which responds in part to recommendations in the   Credible Science, Credible Decisions  report, outlines general principles for peer review at EPA. Different EPA offices have undertaken various implementing activities, including an Agency wide information and planning workshop, internal guideline development, and numerous specific peer reviews. Even with these activities, however, I am concerned that EPA does not yet have a comprehensive Agency wide program for implementing its Peer Review Policy. I therefore welcome the SPC initiative toward effective, efficient implementation of the policy in all the program areas to which it applies. Peer Review Handbook Page A 3 Expanding and Improving Peer Review The Science Policy Council and its Steering Committee have outlined a dual track implementation program of planning and assistance for all Agency offices. The first track has three major milestones.

First, during the next few weeks, Steering Committee members will consult with senior management in each office to exchange information on current peer review activities, assistance needed, possible obstacles to implementation, and implementation planning.

Second, using information and materials developed during the first stage, peer review task groups in each office will develop standard operating procedures ( SOPs) for use in each office, based in part on generic guidance to be issued by the SPC and in part on peer review needs and capabilities specific to each office. The resulting SOPs will delineate as appropriate the scope of application of peer review with respect to various types of scientific and technical work products such as reports of original research, risk assessments, and analytical methods of economic analysis. OARM and OGC staff will assist each office as needed on legal, budget and administrative matters. Each AA and RA will submit draft SOPs for Steering Committee review by July 15.

Third, the SPC review group will work with each office to complete each plan by September 15.

In parallel with the above, consistent with the Peer Review Policy, the Science Policy Council will work with each AA and RA to identify   major scientific and technical work products  as peer review candidates for the coming year. This process will consider existing and new plans for internal reviews and for Science Advisory Board ( SAB), FIFRA Scientific Advisory Panel ( SAP), and other external reviews. The two fold objective is to plan reviews for technical products covered by the Peer Review Policy and to gain experience with options and obstacles. We will use this experience to review and revise the SOPs as needed. Also, to establish a baseline for comparison, each AA and RA will identify the   major technical products  completed within his/ her program during the past 12 months.

The Science Policy Council has sent additional information to each office offering guidance on the procedures that you are asked to develop and the schedule for these activities. Please note, however, that because the policy is effective immediately, current peer review planning should continue on present schedules in parallel with developing the formal SOPs. Page A 4 Peer Review Handbook To begin this process, I have asked each Assistant Administrator and Regional Administrator to designate a Peer Review Coordinator to work with the Steering Committee on implementation activities specific to each office. I am very pleased that the Science Policy Council is taking this important step. A comprehensive peer review program is essential to maintaining and improving the quality of the analyses that underlie Agency actions. I look forward to working with you and your staff on this important activity.

/ s/

Carol M. Browner Attachment cc: Science Policy Council Science Policy Council Steering Committee Peer Review Handbook Page A 5 PEER REVIEW AND PEER INVOLVEMENT AT THE U. S. ENVIRONMENTAL PROTECTION AGENCY This document establishes the policy of the United States Environmental Protection Agency ( EPA) for peer review of scientifically and technically based work products that are intended to support Agency decisions. Peer review is presented in the context of the broader concept, peer involvement.

BACKGROUND The report   Safeguarding the Future: Credible Science, Credible Decisions  1 focused on the state of science at EPA. The panel of experts who prepared the report emphasized the importance of peer review, especially external peer review, and the need for broader and more systematic use of it at EPA to evaluate scientific and technical work products. Their specific recommendation regarding peer review reads as follows:

  Quality assurance and peer review should be applied to the planning and results of all scientific and technical efforts to obtain data used for guidance and decisions at EPA, including such efforts in the program and regional offices. Such a requirement is essential if EPA is to be perceived as a credible, unbiased source of environmental and health information, both in the United States and throughout the world.

In response to this recommendation, then Administrator Reilly directed staff to develop an EPA wide policy statement, which he issued in January, 1993. The paragraphs below preserve the core of that earlier statement while updating it to specify the role of the Science Policy Council in guiding further implementation of the policy. Effective use of peer review is indispensable for fulfilling the EPA mission and therefore deserves high priority attention from program managers and scientists within all pertinent Headquarters and Regional Offices.

1 EPA/ 600/ 9 91/ 050, March 1992. Page A 6 Peer Review Handbook PEER INVOLVEMENT AND PEER REVIEW EPA strives to ensure that the scientific and technical underpinnings of its decisions meet two important criteria: they should be based upon the best current knowledge from science, engineering, and other domains of technical expertise; and they should be judged credible by those who deal with the Agency. EPA staff therefore frequently rely upon peer involvement  that is, they augment their capabilities by inviting relevant subject matter experts from outside the program to become involved in one or more aspects of the development of the work products that support policies and actions.

One particularly important type of peer involvement occurs when scientifically and technically based work products undergo peer review  that is, when they are evaluated by relevant experts from outside the program who are peers of the program staff, consultants, and/ or contractor personnel who prepared the product. Properly applied, peer review not only enriches the quality of work products but also adds a degree of credibility that cannot be achieved in any other way. Further, peer review early in the development of work products in some cases may conserve future resources by steering the development along the most efficacious course.

Peer review generally takes one of two forms. The review team may consist primarily of relevant experts from within EPA, albeit individuals who have no other involvement with respect to the work product that is to be evaluated ( internal peer review). Or the review team may consist primarily of independent experts from outside EPA ( external peer review ).

POLICY STATEMENT Major scientifically and technically based work products related to Agency decisions normally should be peer reviewed. Agency managers within Headquarters, Regions, laboratories, and field components determine and are accountable for the decision whether to employ peer review in particular instances and, if so, its character, scope, and timing. These decisions are made in conformance with program goals and priorities, resource constraints, and statutory or court ordered deadlines. For those work products that are intended to support the most important decisions or that have special importance in their own right, external peer review is the procedure of choice. Peer review is not restricted to the penultimate version of work products; in fact, peer review at the planning stage can often be extremely beneficial.

SCOPE Agency managers routinely make regulatory and other decisions that necessarily involve many different considerations. This policy applies to major work products that are primarily scientific and technical in nature and may contribute to the basis for policy or regulatory decisions. By contrast, this policy does not apply to non major or nontechnical matters that Agency managers consider as they make decisions. Similarly, this policy does not apply to these ultimate decisions. Peer Review Handbook Page A 7 This policy applies where appropriate, as determined by the National and Regional Program Managers, to major scientifically and technically based work products initiated subsequent to the date of issuance. Peer review should be employed to the extent reasonable to relevant work products that currently are under development. This policy does not apply to the bases for past decisions, unless and until the relevant scientific and technical issues are considered anew in the Agency s decision making processes.

Except where it is required by law, formal peer review ( as distinguished from the Agency s normal internal review procedures) should be conducted in a manner that will not cause EPA to miss or need extension of a statutory or court ordered deadline. Agency managers still may undertake peer review if it can be conducted concurrently with necessary rulemaking steps.

LEGAL EFFECT This policy statement does not establish or affect legal rights or obligations. Rather, it confirms the importance of peer review where appropriate, outlines relevant principles, and identifies factors Agency staff should consider in implementing the policy. On a continuing basis, Agency management is expected to evaluate the policy as well as the results of its application throughout the Agency and undertake revisions as necessary. Therefore, the policy does not stand alone; nor does it establish a binding norm that is finally determinative of the issues addressed. Minor variations in its application from one instance to another are appropriate and expected; they thus are not a legitimate basis for delaying or complicating action on otherwise satisfactory scientific, technical, and regulatory products.

Except where provided otherwise by law, peer review is not a formal part of or substitute for notice and comment rulemaking or adjudicative procedures. EPA s decision whether to conduct peer review in any particular case is wholly within the Agency s discretion. Similarly, nothing in this policy creates a legal requirement that EPA respond to peer reviewers. However, to the extent that EPA decisions rely on scientific and technical work products that have been subjected to peer review, the remarks of peer reviewers should be included in the record for that decision.

IMPLEMENTATION The Science Policy Council is responsible for overseeing Agency wide implementation. Its responsibilities include promoting consistent interpretation, assessing Agency wide progress, and developing recommendations for revisions of the policy as necessary.

The Science Policy Council will oversee a peer review work group, which will include representatives from program units throughout EPA to effect a consistent, workable implementation of the policy. The work group will assist the programs in ( 1) formulating and, as necessary, revising standard operating procedures ( SOPs) for peer review consistent with this Page A 8 Peer Review Handbook policy; ( 2) identifying work products that are subject to review; and ( 3) for each major work product, selecting an appropriate level and timing of peer review.

In assisting the programs, the work group will take into account statutory and court deadlines, resource implications, and availability of disinterested peer reviewers. The group will work closely with Headquarters offices and the Regional Offices toward ensuring effective, efficient uses of peer review in supporting their mission objectives. However, the Assistant Administrators and Regional Administrators remain ultimately responsible for developing SOPs, identifying work products subject to peer review, determining the type and timing of such review, documenting the process and outcome of each peer review, and otherwise implementing the policy within their organizational units.

Because peer review can be time consuming and expensive, Agency managers within Headquarters, Regions, laboratories, and field components are expected to plan carefully with respect to its use  taking account of program priorities, resource considerations, and any other relevant constraints as well as the policy goal of achieving high quality, credible underpinnings for decisions. External peer reviewers should be chosen carefully to ensure an independent and objective evaluation. The affiliations of peer reviewers should be identified on the public record, so as to avoid undercutting the credibility of the peer review process by conflicts of interest.

The policy is effective immediately. The peer review work group mentioned above will identify the focal point to whom comments and questions should be addressed and, from time to time, will provide further information about implementation activities.

/ s/ DATE: JUN 7 1994 CAROL M. BROWNER , ADMINISTRATOR APPROVED: Peer Review Handbook Page B 1 APPENDIX B  SOUND SCIENCE AND PEER REVIEW IN RULEMAKING Using EPA s regulatory agenda as the cornerstone of the Agency s mission, the Deputy Administrator s memo implements an additional requirement as part of the ongoing effort to use peer review to enhance the quality and credibility of Agency decisions. Page B 2 Peer Review Handbook June 15, 1999 MEMORANDUM SUBJECT: Sound Science and Peer Review in Rulemaking TO: The Regulatory Policy Council Over a year ago, the Agency enhanced its peer review policy and procedures with the finalization and release of the Peer Review Handbook. The Handbook was developed to serve as a guide for Agency scientists and managers on the organization and conduct of peer review. With our regulatory agenda as the cornerstone of the Agency s mission, I would now like to implement an additional requirement as part of the ongoing effort to use peer review to enhance the quality and credibility of Agency decisions.

Beginning July 12, 1999, the action memorandum accompanying each rule submitted for signature must explicitly address the use of peer review in that rulemaking. The Administrator and I want to ensure that the Agency takes complete advantage of the benefits that thorough peer reviews can provide in our deliberations on the many challenging decisions that must be made. A key step in producing credible regulatory decisions supported by well documented scientific data is ensuring that our work is based on broadly accepted scientific studies. Peer review is an essential tool for accomplishing this goal.

In addition, I ask that, beginning today, planning for peer review be specifically addressed in each new Analytic Blueprint. The general guidance on integrating peer reviews into the rulemaking process is included as Attachment A. Detailed guidance on action memoranda, updated to address peer review, is provided in Attachment B. Guidance regarding which scientific or technical studies must be peer reviewed is contained in the fact sheet on   Peer Review in the Rulemaking Process  in Attachment C and in the Peer Review Handbook available at http:// www. epa. gov/ ordntrnt/ ORD/ spc/ sopmenu. htm. I am also taking this opportunity to bring to your attention a memorandum from the Office of General Counsel that discusses the implications of a recent decision in a challenge to a peer review brought against the Agency under the Federal Advisory Committee Act ( Attachment D) [ note added: this attachment is not attached to the Peer Review Handbook].

I appreciate your support in this effort. For additional information about peer review in the rulemaking process, please contact your Steering Committee representative, your peer review coordinator ( Attachment E) [ note added: this attachment is not attached to the Peer Review Handbook], or Phil Schwartz in the Office of Policy at xxx xxxx.

[ signed 6/ 15/ 99, AX 9807793] Peter D. Robertson Acting Deputy Administrator Peer Review Handbook Page B 3 ATTACHMENT A Regulatory Management Guidance PEER REVIEW REQUIREMENTS Office of Policy FOR EPA RULEMAKING June 1999 Reference: RMD 5 BACKGROUND On February 25, 1998, the Agency reaffirmed its commitment to quality decision making by issuing the comprehensive Peer Review Handbook. The Peer Review Handbook is available at: http:// www. epa. gov/ ordntrnt/ ORD/ spc/ sopmenu. htm, which amplified and clarified key Agency peer review policy issues. The Handbook states that, in general, all major scientific and technical work products related to Agency decisions should normally be peer reviewed. Peer review is a documented, critical review of a work product, performed by experts who are independent of those who developed the product. Work products that are developed to support regulations, such as risk assessments, should be evaluated to determine whether they meet the criteria for " major scientific and/ or technical work products" and thus require peer review ( See Attachment C).

Regulations themselves are not peer reviewed. The straight forward application of accepted, previously peer reviewed economic methods or analyses in Regulatory Impact Analyses ( RIA's), are not typically subject to formal peer review. If, however, the particular facts and circumstances of any piece of economic analysis in an RIA warrant peer review, the Agency will accommodate those needs on a case by case basis.

PLANNING FOR PEER REVIEW Remember that:

  Analytical Blueprints should discuss the plan for peer review of major scientific and technical products. This should include a schedule for the review and should identify the resources that will be needed.

  When negotiating rule deadlines with courts or with litigants, be sure that your proposals allow adequate time for any needed peer reviews.

PEER REVIEW REQUIREMENTS Program offices have the primary responsibility for complying with the Agency peer review requirements. Senior managers must confirm in their Action Memoranda that their offices have complied with the peer review requirements; detailed explanations are not necessary. If no major scientific and technical product as defined by the Peer Review Handbook was used to support the action, include the following statement in the action memorandum: Page B 4 Peer Review Handbook Peer Review There were no major scientific or technical products supporting this action as defined by the Agency's Peer Review Handbook. We did not, therefore, submit any support documents for peer review.

If a major scientific or technical product was used to support the action, include the following statement in the Action Memorandum:

Peer Review [ Insert Name of AAship] has complied with the Agency's Peer Review requirements with respect to the underlying major scientific and technical products supporting this action.

You may add any details you think important, but you may not modify this compliance statement. If you utilized peer reviewed products, but you could not fully comply with the peer review requirements, explain in the Action Memorandum why the deviation was necessary.

FINAL AGENCY REVIEW ( FAR)

For rules that go through Final Agency Review, OP will document compliance in the FAR memorandum. This documentation will be based on the peer review information included in the draft Action Memorandum.

These requirements will be effective on July 12, 1999. After that date, OP's Office of Regulatory Management and Information ( ORMI) will only forward for signature those actions that include one of the following:

  a statement that no major scientific or technical documents were utilized to support the rulemaking;

  a statement of compliance with the peer review requirements for major scientific and technical documents;

  an explanation for deviation from the peer review requirements.

Questions? Contact your Regulatory Steering Committee representative or Phil Schwartz in the Regulatory Management Division at ( 202) 260 5493. Peer Review Handbook Page B 5 ATTACHMENT B Regulatory Management Guidance Office of Policy FRAMEWORK FOR ACTION MEMORANDA Revised June 1999 Reference RMD 6 OVERVIEW The overview provides background information which briefly describes the rule, characterizes the environmental issue( s) or public health problem( s) being addressed, and summarizes the history of the regulatory action. It should explain why EPA is taking this action, and where appropriate, cover the following points:

  Define the court or statutory deadline;

  Identify whether the action amends the CFR, and if so, explain what kind of amendment ( technical, procedural, etc.);

  Identify other regulatory actions underway that will affect this particular program or sector;

  Describe the specific environmental issue( s) or public health problem( s) being addressed, and the goal intended by taking this action;

  Describe what the action does, and specifically, how the regulated community is affected ( performance standards, specific requirements); and   Describe how flexible the implementation of the action will be for states and regulated entities.

REGULATORY IMPACTS Summarize the costs and benefits of the regulatory action ( include costs to State and local governments, and tribal communities) and the results of the economic and risk analyses. Discuss the economic impacts on all stakeholders including, where appropriate, the results of the regulatory flexibility analysis and the specific effects of the action on small entities. Explain the roles of cost benefit and cost effectiveness analyses in shaping the regulatory approach.

Describe the reporting and Record keeping burden and what the Agency has done to reduce it. Indicate possible impacts on other Federal agencies. Page B 6 Peer Review Handbook ADMINISTRATOR'S PRIORITIES If appropriate, discuss how the regulatory action relates to the Administrator's priorities. Explain how the regulatory action relates to a specific priority only where the relationship is significant, clear and distinct.

STAKEHOLDER INVOLVEMENT Briefly discuss the role of both government entities and private sector stakeholders in the development of the action. Summarize the concerns they have raised and what the Agency has done to address them.

INTERNAL REVIEW Identify whether the regulatory action was developed under Tier 1,2, or 3. Describe any outstanding issues from workgroup closure or other internal review. Identify program offices or regions with outstanding issues and indicate why they cannot be resolved or accommodated. Also, provide the basis for any decision made to not address an identified cross media impact. Where there is not a formal workgroup, identify those offices that have reviewed the package.

PEER REVIEW If you did not utilize major scientific and technical products as defined by the Peer Review Handbook to support the action, include the following statement:

There were no major scientific or technical products supporting this action as defined by the Agency's Peer Review Handbook. We did not, therefore, submit any support documents for peer review.

If a major scientific or technical product has been used to support the action, include the following statement:

[ Insert Name of AAship] has followed the Agency's Peer Review requirements with respect to the underlying, major scientific and technical products supporting this action.

You may add any details you think important, but you may not modify this compliance statement. If you utilized peer reviewed products, but you could not fully comply with the peer review requirements, explain why the deviation was necessary. Peer Review Handbook Page B 7 PLAIN LANGUAGE Either in the action memorandum or in a separate note briefly describe which elements of plain language have been addressed.

OMB TRANSACTION Highlight significant issues resulting from OMB's review under Executive Order 12866. Explain any changes made to the regulatory action as a result of that review. If OMB review was waived, please indicate this.

ANTICIPATED PUBLIC REACTION Describe the type of public response anticipated and identify both the involved stakeholders and the nature of their expected response. Characterize the likely reaction to the action by all interested parties including industry; environmental groups; Congress; state, local, and tribal governments; and OMB. Explain why the Agency should take the action despite any controversial response anticipated from the public.

If the regulatory action will not be issued for public comment ( i. e., direct final rule, administrative stay, etc.), explain the basis for the Agency's decision not to solicit comment.

RECOMMENDATION Identify the action the administrator is expected to take.

NOTE: An Action Memo should generally not exceed 5 pages. Page B 8 Peer Review Handbook ATTACHMENT C Fact Sheet Peer Review in the Rulemaking Process What Role does Peer Review have in the Regulatory Development Process? Peer review of scientific and technical work products affirms the scientific credibility of the Agency's rulemakings. Work products need to be well planned and documented because new rules, and the work products supporting them, may have substantial cost impacts and must often withstand intense public scrutiny.

What Work Products Need Peer Review? While rules themselves are not subject to peer review, any major scientific and/ or technical work products that support rules should be peer reviewed. ( See Section 2.2 of the Peer Review Handbook http:// www. epa. gov/ ordntrnt/ ORD/ spc/ sopmenu. htm to determine which work products are considered major.) Designations of a scientific and/ or technical work product as " major" will largely be case by case. Those that are used to support a regulatory program or policy position and that meet one or more of the following criteria are candidates for peer review:

a) Establishes a significant precedent, model, or methodology b) Addresses significant controversial issues c) Focuses on significant emerging issues d) Has significant cross Agency/ inter agency implications e) Involves a significant investment of Agency resources f) Considers an innovative approach for a previously defined problem/ process/ methodology The straight forward application of accepted, previously peer reviewed economic methods or analyses in Regulatory Impact Analyses ( RIA's), are not typically subject to formal peer review. If, however, the particular facts and circumstances of any piece of economic analysis in an RIA warrant peer review, the Agency will accommodate those needs on a case by case basis. The decision whether to peer review any work product will be documented through the Agency's annual peer review reporting process ( see Section 1.3 of the Handbook).

When should determinations about the need for peer review be made? The need for, and scheduling of, peer reviews should be addressed in the Analytic Blueprint that is required for all Tier 1 and 2 regulatory actions. Work products supporting Tier 3 rules may also be considered major and Analytic Blueprints are encouraged for Tier 3 rules. In any event, the need for, and scheduling of, peer reviews should be addressed early in the development of Tier 3 actions. Peer Review Handbook Page C 1 APPENDIX C  EXAMPLES OF CHARGES Please note  certain questions that are posed in charges can be responded to with a yes or no answer. Clearly, this is not the type of response we generally want, therefore, it is important to phrase charge questions carefully to ensure that you receive a fully satisfactory and thoughtful response. Where a yes or no answer might be expected, be sure to ask for a full explanation supporting the yes or no answer.

Charges can run the gamut from rather simplistic to highly complex. The examples shown here cover a variety of types. Examples 1 through 6 have less complex questions and are looking for the overall quality of the reports. Examples 7 through 11 have numerous technical questions that need to be addressed and are therefore more complex in their nature. Page C 2 Peer Review Handbook CHARGE EXAMPLE 1  Framework for Developing a Living Resources Research and Monitoring Plan for the Peconic Estuary Background The Peconic Estuary is located on the eastern end of Long Island, New York. Under the Federal Clean Water Act, the Peconic Estuary was named an " Estuary of National Significance" in 1992. Because of its high concentration of rare, threatened and endangered species and habitats, The Nature Conservancy named the Peconic Ecosystem as one of the " Last Great Places in the Western Hemisphere."

The Peconic Estuary Program ( PEP) released its draft Comprehensive Conservation and Management Plan ( CCMP) in September 2, 1999. The draft Plan addresses the environmental management issues facing the estuary and its watershed, including brown tide ( a nuisance algal bloom), nutrients, habitats and living resources, pathogens, and toxics, as well post CCMP management structure, public education and outreach, and financing. A final Plan is now under preparation. The preparation of the Framework document is called for in the draft CCMP.

With population increasing in the watershed, the Peconic Estuary is being threatened by over development and misuse of its resources. To fully realize the impacts of people and their activities on this system, there must be a better understanding of how the Peconic Estuary functions ecologically. This knowledge can only be achieved through comprehensive research and monitoring of the entire ecosystem.

The long term goal of this Framework for Developing a Living Resources Research and Monitoring Plan is to develop a strategy of coordinated research and monitoring to fill significant information gaps and assist in the planning, conservation, and management of the Peconic Estuary. A key component of this strategy is to develop an applied, multi scale, integrated approach to gain a better understanding of the estuary. To achieve, this goal, the Framework for Developing a Living Resources Research and Monitoring Plan sets broad priorities, provides the context for specific studies, guidelines for a detailed living resources research and monitoring plan, and stimulus for funding agencies and organizations and researchers. The short term goal of this Framework is to provide a basis for securing funding for the priority initiatives described in the Framework document.

In order of importance, the objectives of this Framework document are to:

1) understand threats so as to improve resource protection through management and conservation;

2) examine the biology and ecology of particular organisms identified in the CCMP as important either due to their commercial or recreational value or their role in the food web and ecosystem; and, Peer Review Handbook Page C 3 3) undertake system wide studies of the Peconic Estuary to understand the significance of key habitats and ecosystem productivity.

Recommendations included in this framework are based on information gaps identified in the Characterization Report of the Living Resources of the Peconic Estuary ( Bortman and Niedowski 1998), the Habitat Module of the draft CCMP, PEP Natural Resource Subcommittee meetings, and recommendations made at a PEP sponsored Living Resources Research and Monitoring workshop held in 1998. Copies of the draft CCMP and Characterization Report are included with this package.

Tasks The peer review charge is to assess the adequacy of this document to provide a framework for integrated, system wide ecological research and monitoring to understand the dynamic, multi scale ecological patterns and processes that sustain biota and their supporting natural systems in the Peconic Estuary.

Specific Questions:

Peer Reviewers shall answer/ comment on the following:

1) Is the organization of the document appropriate and does it present the material in a clear and concise manner? Please explain fully.

2) In your opinion, what are the weakest and the strongest aspects of the framework? Please make suggestions on how the weakest parts can be strengthened.

3) Are there any elements missing from the framework which you think need to be included or which would strengthen the document? Please explain fully.

4) Are you aware of any other significant data/ studies that are relevant and should be included or referenced in this document? Please explain fully.

5) Is the stated goal realistic? Are the stated objectives adequately met? Please explain fully.

6) Examine and critique the interrelationship among the specified assessment, research, and monitoring priorities.

7) Does this framework present a holistic approach to preserve, protect and restore the estuary? Does the approach cover all the relevant and important areas that warrant investigation? If not, please state what should be done differently. Page C 4 Peer Review Handbook 8) Are the initial set of assessment, research, and monitoring priorities adequate to provide the information needed to successfully protect and manage this system? Please comment on the selection of priorities. Were any significant priorities missed? What other priorities should be specified and why? Peer Review Handbook Page C 5 CHARGE EXAMPLE 2  EPA Science Advisory Board ( SAB) Review of the Environment Monitoring and Assessment Program ( EMAP) Research Strategy and Research Plan The Science Advisory Board ( SAB) is asked to review the Environment Monitoring and Assessment Program ( EMAP) Research Strategy and Research Plan. The review is requested by the Office of Research and Development ( ORD) with the following specific charge issues:

1) Previous peer reviews recommended that EMAP develop a close working relation with EPA Program Offices and other federal monitoring efforts. Does the EMAP strategy support the [ Office of Science and Technology Policy's Committee on Environment and Natural Resources] CENR National Monitoring Framework and EPA Program Offices?

2) Previous peer reviews recommended that EMAP initiate a focused research program on indicator development. Does the intramural EMAP program on ecological indicator development, coordinated with the [ ORD Science To Achieve Results] STAR solicitations, respond to this research need?

3) Previous peer reviews recommended that the EMAP design be modified to include a set of nonrandomly selected sentinel sites with intensive data collection. Does the development of Index Sites as outdoor laboratories in the national parks ([ National Park Service] NPS and [ US Geological Survey] USGS) and selected estuaries ([ National Oceanic and Atmospheric Administration] NOAA) add this dimension to the EMAP?

4) Previous peer reviews recommended that EMAP combine effects oriented and stressor oriented monitoring approaches. Do the focused geographic demonstration pilot studies ( initially in the Mid Atlantic region) combine these elements? Page C 6 Peer Review Handbook CHARGE EXAMPLE 3  EPA Science Advisory Board Review of the Agency's National Risk Management Research Laboratory s ( NRMRL) Program The Office of Research and Development ( ORD) requests that the Science Advisory Board review the Agency's National Risk Management Research Laboratory s ( NRMRL) program.

In the   Strategic Plan for the Office of Research and Development  ( EPA, 1996a), ORD described the relationship of risk assessment to the risk management process, and emphasized the need for scientific and engineering research to enable sound risk management decisions and actions. Within the framework of that strategic plan, NRMRL s mission is to conduct research to reduce uncertainties and costs associated with making and implementing environmental risk management decisions. NRMRL has therefore developed a research agenda to reduce risk uncertainty that also focuses on those important, relevant issues where it can make a difference.

The charge to the SAB is to:

1) Examine and critique the research programmatic directions such as whether NRMRL is pursuing the most appropriate research problem areas;

2) Comment on strategic directions, e. g., use of its core technical competencies, transition from primarily extramural to an intramural R& D organization, leveraging with other agencies and organization;

3) Review and comment on the effectiveness of NRMRL s approach to science management, e. g., measures of success and science quality, soundness of peer review process;

4) Examine and critique the relationship of NRMRL s risk management research and its intended role in the risk assessment/ risk management paradigm; and 5) Review and comment on the strategic balance for the next decade among pollution prevention, technology development, remediation, and risk management assessment activities. Peer Review Handbook Page C 7 CHARGE EXAMPLE 4  EPA Science Advisory Board ( SAB) Review of the Technical Aspects of the Multi Agency Radiation Survey and Site Investigation Manual ( MARSSIM)

The EPA Science Advisory Board ( SAB) is asked to review the technical aspects of the Multi Agency Radiation Survey and Site Investigation Manual ( MARSSIM). The review document was developed collaboratively by four Federal agencies, departments and commissions having authority for control of radioactive materials: Department of Defense, Department of Energy, Environmental Protection Agency, and Nuclear Regulatory Commission. MARSSIM addresses the need for a nationally consistent approach to conducting radiation surveys of potentially radioactively contaminated sites that are being considered for release to the public. A condition of release is a demonstration that residual radioactivity levels do not exceed a specified risk or dose level, also known as a release criterion. MARSSIM provides guidance to users performing and assessing the results of such a demonstration for surface soils and building surfaces.

The SAB is asked by the Agency s Office of Radiation and Indoor Air ( ORIA) to respond to the following charge in its review:

1) Is the overall approach to the planning, data acquisition, data assessment, and data interpretation as described in the MARSSIM technically acceptable? Please explain fully.

2) Are the methods and assumptions for demonstrating compliance with a dose or risk based regulation technically acceptable? Please explain fully.

3) Are the hypotheses and statistical tests and their method of application appropriate? Please explain fully. Page C 8 Peer Review Handbook CHARGE EXAMPLE 5  Economics  Benefits Transfer from Adults to Children Background EPA established the Office of Children s Health Protection ( OCHP) in 1997 to support its efforts to increase the protection of children s health throughout its programs. One of the many difficult issues the Office is addressing is the appropriate treatment of children s health effects in the economic analyses performed by the Agency. Policy analysis efforts at the Agency often rely on the benefits transfer technique, and very few of the Agency s benefit transfers have explicitly addressed children s health issues. In addition, no accepted systematic process for conducting benefits transfer currently exists. To assist the Agency in its efforts, this paper discusses the benefits transfer technique as it applies to estimating values for children s health. The first section provides some general background on the technique, and its application to estimate health related values. The second section raises important general issues to consider when conducting a benefits transfer for children s health values. The last section discusses the implications of using the benefits transfer method to estimate values for children s health. The scarcity and state of existing child oriented health valuation literature suggests that it may be necessary to transfer adult oriented values to estimate child related values ( Neumann and Greenwood 1999). However, the results of this paper suggest that transfer of these value estimates to children at best provides estimates for a scoping analysis. In cases where these scoping exercises indicate that children's health values may be a crucial component in the policy analysis, primary research should be undertaken to estimate child related values.

1) Assess the appropriateness of transferring health benefit values estimated for adult populations to children. Describe the specific issues that arise in these transfers. Which variables or situations improve or decrease the appropriateness of transferring benefit values from adults to children?

2) Does the analysis support the proposition that the value of children s health effects should be estimated differently than adult health effects? Why or why not?

3) Identify issues for further research that would improve our ability to estimate values for children s health effects. Peer Review Handbook Page C 9 CHARGE EXAMPLE 6  Economics  Study of Municipal Government Costs and Financial Impacts from Environmental Regulations Background Municipalities play a major role in supplying environmental services. Local governments have taken responsibility for providing drinking water, sewage treatment, and waste disposal in a majority of communities. Over the past fifteen to twenty years, most of the mandates found in the federal environmental legislation enacted in the early 1970s have been met. The increase in the number of people served and improvements in the quality of local environmental services have been considerable, as has the investment in public infrastructure to meet these laws.

Recent revisions to the environmental legislation have established a broader and more stringent set of standards to be met by suppliers of environmental services. As a result, many local governments are now faced with having to maintain all or some part of their public services at a higher level of performance. To meet these new standards will require additional investments in capital, and increases in rates charged to customers for environmental services.

Improvements in environmental services are but one of several demands being made of local public infrastructure. Studies prepared on public infrastructure needs and the availability of funds to meet these needs indicate that there will be an excess demand for money to rebuild and improve upon the existing stock of public infrastructure. Therefore, it is important to recognize that additional environmental requirements will have to compete with other infrastructure needs ( e. g., highways, bridges), as well as other public services ( e. g., police, education, health and welfare programs) provided at the local level.

Given the increasing demand for public services, this study examines what additional investments the new environmental legislation will require local governments to undertake, and the likelihood that they will face difficulties raising the necessary funds through capital markets and revenues from customers. The economic impacts of individual EPA actions are considered during the regulatory process in those situations permitted by environmental statutes. The unique feature of this study is its attempt to estimate the cumulative costs and impacts of meeting a combined set of EPA requirements, and to determine whether they will place a significant burden on the fiscal conditions of local governments, and require them to significantly increase existing charges for improved environmental services.

Please find attached a copy of the draft study and appendices for your review. This version of the report reflects Agency comments received on an earlier draft. We expect that this version of the report, with some additional minor modifications, will be the final version. Your comments will be useful in preparing the final version of the report and discussing the findings of the reports with the public. Page C 10 Peer Review Handbook To assist in your review of the report, we ask that you pay particular attention to the following questions:

1) Do state and local governments and financial markets consider household costs ( measured as a percent of household income) and selected municipal financial information ( debt service to general revenues or taxable property values) when evaluating the ability of enterprise systems and municipalities to issue bonds or obtain loans?

2) Having selected a series of financial indicators, are the criteria used in the analysis acceptable?

3) Do the results support our conclusions? What additional conclusions can be reached from the analysis?