Comment: Commenter (2475) expresses concern that the Administrator could claim there is no budget to support independent peer review, making it infeasible, and justify an exemption.
Response: The EPA is not finalizing the criterion for an Administrator's exemption related to the infeasibility of conducting independent peer review on all pivotal regulatory science used to justify regulatory decisions for reasons outlined in OMB Final Information Quality Bulletin for Peer Review. The EPA is also clarifying that the Agency is not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review does not typically include reanalysis of the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information) and, thus, peer review is not considered a replacement for the data availability requirements of this rule.
The EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the OMB Bulletin for Peer Review and the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review. The EPA Peer Review Handbook, 4th Edition, describes the process for peer reviews conducted by a Federal Advisory Committee and the National Academy of Sciences. If the Agency conducts peer review on pivotal science, the EPA shall ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied in analyzing dose-response data and the implications of those assumptions for the results. The considerations in 40 CFR 30.5, the Administrator's exemption in 40 CFR 30.7, and the EPA's incremental approach to maximizing transparency by first focusing on dose-response data are intended to minimize potential unintended consequences, such as the potential skew toward newer, domestic, and/or industry-funded studies suggested by the commenters. In addition, the EPA is maintaining language from the 2020 SNPRM in 40 CFR 30.3 stating that statutory requirements and corresponding implementing regulations will control in the event of any conflict, and clarifying in this preamble that the requirements in this final rule set the overarching structure and principles for transparency in significant regulatory actions and influential scientific information. The EPA plans to develop internal implementing guidance and promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers. The EPA believes the rule requirements provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical and privacy considerations in order to ensure the Agency maintains a strong scientific basis for its decision-making.
<IP_ADDRESS>.3 Process of Independent Peer Review
 Recommendations for a Process to Conduct Independent Peer Review
Comment: Commenter (4420) provides the following detailed recommendations and rationale for a process to conduct the independent peer review.
      A Call for the Standardization of the Review of an Epidemiology Studies  -  Establishment of an Agency wide Epidemiology Study Peer Review Council and their product Record of Epidemiological Study Review (ROESR).
      Each epidemiology study occurring in the open literature or otherwise needs to have a supporting formal Record of Epidemiological Study Review (ROESR) that clearly delineates the justification for a decision to include or not include the study in the regulatory decisions for the chemical or environmental situation. The production of the ROESR will consist of the separate and independent reports of several Sub-Committees that will be integrated by the Epidemiological Study Peer Review Council and signed by the Council Chairperson and each member of the Council. It is important to have independent sub-discipline committees review the study so that the biases of other sub-disciplines are minimized. The committee members should be drawn from the relevant staff throughout the Agency as well as other government agencies as needed.
      The roles of the five suggested Sub-Committees are as follows:
      Endpoint Evaluation Sub-Committee: Depending upon the lesion/ environmental condition claimed by the epidemiological study, the endpoint evaluation committee would be experts in cancer, behavioral response, or other appropriate discipline for interpretation of the significance of the lesion. This committee need not actually see the supporting data since they will be judging the endpoint itself relative to the plausibility that the lesion is actually in excess of the norms and that all confounds are addressed as well as consistency reported for the subjects in the cohort. Commentary on any known chemicals or environmental factors that also promote the occurrence of the lesion/condition will also be provided. This evaluation would limit its conclusions with regard to how well the character of the endpoint was assessed for and whether or not the increase in the lesion reported is plausibly related to treatment or within normal variation. This Sub-Committee may require additional input from the study authors regarding how the endpoint was characterized in all subjects.
      Statistical Evaluation Sub-Committee: The sole role of the statistical evaluation sub-committee is limited to determining if the statistical methods were/were not appropriate and adequately conducted to support the conclusion of the report. A committee of statisticians with expertise in epidemiological studies would need to be established for this evaluation. This Sub-Committee would be very critical in determining if the original data are needed to do an independent analysis.
      Exposure Assessment Sub-Committee: The role of the exposure assessment committee is limited to determining that the methodology and reliability used to determine the exposure of the cohort is adequate and appropriate. The committee would provide commentary on whether or not exposure was supported by analytical chemical data or by oral history. Also, if actual persons exposed provided the oral history by direct conversation, survey or by telephone. The committee will determine if second-hand exposure information by a relative (or friend or coworker) provided the exposure information is reliable. The exposure assessment is always a problematic area and may also be critical in a decision to provide supporting data.
      Analytical Chemistry Assessment Sub-Committee: The role of the analytical chemistry committee is limited to determining if the analytical techniques/ methodology were appropriate and adequate for the study. In some cases where no analytical chemistry data were generated, the committee will comment on the need to have included such data and/or if quantitative analysis of exposure data was even possible. Since the analytical chemistry data such as sample preparation, method of detection, and lower limit of detection are generic, there should be no issue for the study authors to provide this information. However, this Sub-Committee may still determine that analytical data from the subjects in the cohort be provided.
      Animal Toxicity Assessor or Sub-Committee: An actual Sub-Committee may not be necessary for this evaluation since other committees should already have characterized the toxicity profile based on animal studies. For example, for a pesticide, this assessor would normally be the person most responsible for reviewing the guideline studies or other required studies. If not then a committee that reviews the animal study data may need to be convened. This assessment will also include in addition to the standard set of required studies, reports from the open literature suggesting the association of the lesion with the chemical in animals as well as structure activity relationships (SAR). However, any publication supporting the association or mechanism of action will need its own Record of Review (ROR) signed by two qualified toxicologists. If the lesion occurs in animals or has SAR correlates, such information can support the epidemiological findings. If not, it does not automatically dismiss the epidemiological findings since the induction of the lesion may be unique to humans. There should be no need for this Sub-Committee to require additional data from the epidemiological report.
      Council Chairperson and resulting product: ROESR: Once the five Sub-Committees finish their respective reports, they meet together with the Council members to discuss their findings and conclusions. Each Sub-Committee will assert if additional or original data supporting the study are required or not. The Council Chairperson, with the approval of the Council, makes the final determination in deciding if the epidemiological study provides quality data that can or should not be incorporated into the regulatory decisions. The Council Chairperson with the Council's approval can overrule a request for additional data made by any of the Sub-Committees. However, clear justification for overruling a Sub-Committee's decisions must be provided. For example, the Council may determine that the association between the chemical or environmental condition is otherwise strong enough to support regulatory action to protect the public health. The Council Chairperson (or secretary of the Council) will prepare the ROESR product that contains the decision with supporting justification. The ROESR product would include the report of the Council with the Chairperson and Council members signatures. Separate attachments for the signed Sub-Committee reports and the original epidemiological report (i.e. a publication) will also be attached. The Council Chairperson should not be in the same program that has responsibility for regulating the chemical or environmental condition. That is, if the chemical is a pesticide, the Chairperson cannot be from the Office of Pesticide Programs (OPP). Representatives of OPP may still be on the Council if an epidemiological study with a pesticide is being reviewed.
      The purpose of this procedure is to assure that responsible persons for each of the critical sub-disciplines independently reviewed the study in terms of their expertise. The resulting ROESR that either determines that the study can be used in the absence of additional original data or the need for critical additional data is determined will be clearly indicated. The Council Chairperson and Council and the sub-discipline committees own their decisions and are responsible for defending them. If industry or any concerned public interest groups object to the conclusions, they can address the conclusions presented in the ROESR and/or the sub-disciplines. Therefore, the decisions in the resulting ROESR should be transparent. The current system renders too much power to individuals that are sometimes not clearly identified who can profit by making career projects for themselves based on the face value of the publication.
      Preparing a ROESR for such studies with the several sub-disciplines all independently contributing to the Agency's determination of the conclusions in the report should greatly help to minimize controversies. The ROESR is not a final conclusion (no documents in EPA should be considered final). It is very likely that a Science Advisory Panel will ultimately review the ROESR. Also, as new information is generated, the ROESR can be updated and the recommendations for use (or otherwise) can be adjusted as appropriate.
Response: EPA appreciates this recommendation. However, EPA does not agree that it needs to establish this type of process. The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. All Agency peer review follows the recommended procedures and approaches consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook, 4[th] Edition. The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect.
Comment: Commenter (4594) recommends that EPA partner with scientific organizations and professional communities to administer and manage these reviews. The commenter states that such outsourcing and partnerships will help to ensure that EPA gains access to independent and highly qualified experts; and will promote greater public confidence in the independence of these peer-reviews. The commenter notes that this kind of process for managing peer-review will also allow EPA to more cost-effectively, nimbly, and rapidly conduct reviews as it will not require EPA to substantially increase staffing for convening reviews. According to the commenter, such a process will also provide EPA with greater capacity to conduct reviews on timescales that do not needlessly delay regulatory and rulemaking schedules.
Response: Thank you for the recommendation. However, EPA does not agree that it should partner with scientific organizations and professional communities to administer and manage the reviews under this rule. The purpose of this action is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. This rulemaking is not intended to modify the Agency's interpretations of "best available science." The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. All Agency peer review follows the recommended procedures and approaches consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook, 4th Edition. The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect. Further, the EPA plans to promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers.
Comment: Commenter (6180) supports formal peer reviews, but also alternatively supports rigorous, detailed, and transparent development of pivotal regulatory science which takes into account public comments and input from a wide range of experts and stakeholders. As an example, the commenter refers to EPA's 1991 publication of the Technical Support Document (TSD) for Water Quality-based Toxics Control. The commenter provides that, while no formal peer review took place, the TSD underwent a rigorous and detailed multi-year review by "a diverse group of people from EPA Regions, states, environmental groups, trade associations, academia, private industry, and municipalities."
Response: Thank you for the recommendation. All Agency technical work products go through rigorous internal review and, when appropriate, peer review. The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. In the final rule, the EPA shall conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein. All Agency peer review follows the recommended procedures and approaches found in the EPA's Peer Review Handbook [4[th] Edition]." The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect. Further, the EPA plans to promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers.
Comment: Commenters (6376, 6891) suggest that there should and can be a balance between transparency and the ability to draw from peer-reviewed scientific studies from outside sources.
Commenter (6891) advises EPA to recognize, as it has in the past, that a variety of quality assurance and quality control measures can result in properly validated data that is appropriate for use in justifying regulatory decisions, not just that which has been independently peer reviewed. The commenter recommends that EPA adopt a broader view of what is necessary to ensure validity, rather than require independent peer review in all instances.
Commenter (6376) states that, while EPA needs to evaluate and judge the merits of different data sets, it is impractical to expect the Agency to supplant the role of peer-review. The commenter notes that the Agency is explicit in their intention to rely on internal review of the scientific studies used to make substantial regulatory decisions. The commenter expresses that, theoretically, this practice makes sense for the EPA, but it could be problematic in application. The commenter provides that there is an ongoing downsizing process across the federal government and the EPA, in particular, is currently executing its mission with fewer staff members and is planning on decreasing those numbers even more in the coming years. According to the commenter, practically speaking, this means substantive internal review of the scientific studies used to inform "significant regulatory decisions" could be unrealistic based on the Agency's workforce. The commenter asserts that the ability to utilize peer-reviewed studies needs to be maintained as EPA moves towards creating more transparency.
Response: All Agency technical work products go through rigorous internal review and, when appropriate, peer review. The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. In the final rule, the EPA shall conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein. All Agency peer review follows the recommended procedures and approaches found in the EPA's Peer Review Handbook [4[th] Edition]." The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect. Further, the EPA plans to promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers.
The EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the OMB Bulletin for Peer Review and the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review. Although not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounts for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
Clarification of the Independent Peer Review Process is Needed
Comment: Commenters (4594, 5165, 5419, 6125, 6142, 6185, 6376, 6868, 6915, 6935) express concern that the proposed language of §30.7 is accompanied by inadequate explanation, leaving commenters to speculate about how it would apply, or what problems its implementation might present. Commenter (6142) states that there is no value in this second peer review without guidelines ensuring the review is rigorous and unbiased in nature and conducted by subject matter experts. Commenter (6935) suggests that the regulatory language in the proposed language of §30.7 should be expanded to address these issues. Commenter (6376) believes the proposal would benefit from a clearly outlined process for utilizing peer-reviewed science to ensure regulatory decisions can be based on the best available science. The commenters request clarification and additional detail related to the following questions:
 Who will conduct and manage the peer-review process? (4594, 6142)
 Will these reviews be managed by the Office of Research and Development or by the various regulatory offices within EPA? (4594)
 Does EPA have appropriate staffing, expertise, and resources to manage these peer-reviews? (4594)
 Would EPA need to hire new experts, how many, in what fields and how much would this cost? (5165)
 What are the selection criteria for peer reviewers? (5419)
 What are the decision criteria? (5419)
 How the proposed rule compares to existing processes through the SAB and third‐party contractor‐managed peer reviews? (5419)
 What scientific standards will be applied? (6142)
 How transparent will the evaluation will be to the study's authors, the scientific community, and the public? (6142)
 Would EPA's staff scientists conduct these validations and reviews, or would EPA contract this out to third parties? (6185)
 How would EPA determine that third parties have the necessary qualifications and resources to perform these validations and reviews? (6185)
 Would peer reviewers be anonymous, or will their names and reviews be made public? (6185)
 How will an EPA agency "independent peer review" process be funded? (6868)
 How will EPA vet reviewers to identify persons who are purportedly more competent than those already used in past or current peer review processes? (6915)
 What is the relationship between EPA's existing science advisors and the new requirement for independent peer review? (6935)
 If the new independent peer review process will be supported by new external advisors, how will such advisors be identified and selected, and will such a process will be used for each regulatory action on an ad hoc basis? (6935)
Commenter (6915) notes that despite the existing peer review process, EPA apparently proposes to require that EPA itself conduct an additional "independent" review. See 83 Fed. Reg. at 18,774. Yet the commenter states that the proposal nowhere discusses how EPA would vet reviewers to identify persons who are purportedly more competent than those already used in past or current peer review processes, or the level of EPA staffing and associated costs that would be needed for additional review -- only stating that EPA will implement the proposed rule in a manner "that minimizes costs." Id. at 18,774.
Commenter (6185) suggests that EPA needs to describe how the proposed text in §30.7 would affect current practice. The commenter asserts that, without this information, commenters are left to guess at the scope and potential impact of EPA's proposal as it applies to "independent review."
Response: In this rule, the EPA is not changing the pre-existing requirements of the OMB's Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review does not typically include reanalysis of the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information) and, thus, peer review is not considered a replacement for the data availability requirements of this rule.
The EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the OMB Bulletin for Peer Review and the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review. The EPA Peer Review Handbook, 4th Edition, describes the process for peer reviews conducted by a Federal Advisory Committee and the National Academy of Sciences. All Agency peer review follows the recommended procedures and approaches found in the EPA's Peer Review Handbook. This Peer Review Handbook is used as guidance by EPA staff and managers to ensure that the Agency's Peer Review Policy is implemented objectively and that the integrity of our peer review activities can be demonstrated transparently to the American public. If the Agency conducts peer review on pivotal science, the EPA shall ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied in analyzing dose-response data and the implications of those assumptions for the results.
It should also be noted that in this final rule, rather than considering all studies that support the assessment of the relationship of a dose or exposure of a pollutant, contaminant, or substance to the magnitude of a predicted health or environmental impact as "pivotal science," the EPA is balancing transparency and feasibility by focusing on those studies that inform the quantitative relationship between the dose or exposure of a pollutant, contaminant, or substance and an effect. Thus, "pivotal science" includes only those studies that are integral to characterizing dose-response relationships (e.g., identifying candidate PODs). These are the studies that drive the requirements or quantitative analyses of EPA significant regulatory actions or influential scientific information.
<IP_ADDRESS>.4 Other Comments
Comment: Commenter (0045) recommends that deliberations of the EPA include enough time for peer review of all sources to proceed before consideration and action. The commenter states that the ability of all constituents to participate in EPA activity comes with a commitment for an open and respectful process which will maintain the quality of all information used to make decisions at the EPA.
Response: This rulemaking is not intended to modify the Agency's interpretations of "best available science." The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. All Agency peer review follows the recommended procedures and approaches consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook, 4th Edition. The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect. The EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the OMB Bulletin for Peer Review and the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review. Further, the EPA plans to promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers.
Comment: Commenter (6143) suggests that EPA replace the proposed text: "EPA shall conduct independent peer review . . ." with: "EPA shall conduct and thoroughly document independent peer review . . ."
Response: EPA appreciates this recommendation. The EPA maintains that all Agency peer review follows the recommended procedures and approaches consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook, 4th Edition. The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect.
Comment: Commenter (6362) states that the language of 40 CFR 30.7 appears to be missing one or more words in the header to the section: "What role does independent peer review in this section?" The commenter expresses that they believe the missing word is likely "have," but suggests EPA should clarify and correct this section in the final rule.
Response: There was a typographical error in the header of Part 30.7 in the April 2018 notice of proposed rulemaking. The mistake was corrected in the supplemental notice of proposed rulemaking and in the final rule to read, "§30.6 What role does independent peer review have in this part?"
Comment: Commenter (6376) states that the ability to utilize peer-reviewed studies needs to be maintained as EPA moves towards creating more transparency. While EPA needs to evaluate and judge the merits of different data sets, the commenter suggests that it is impractical to expect the Agency to supplant the role of peer-review. The commenter adds that there should and can be a balance between transparency and the ability to draw from peer-reviewed scientific studies from outside sources.
Response: The EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the OMB Bulletin for Peer Review and the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review. All Agency peer review follows the recommended procedures and approaches found in the EPA's Peer Review Handbook.
Comment: Commenter (6362) notes that EPA uses the word "justify" frequently throughout the various sections of proposed regulatory text when referencing the use of regulatory science to make its decisions. The commenter provides, for example, §30.7 states: "EPA shall conduct independent peer review on all pivotal regulatory science used to justify regulatory decisions." The commenter suggests that there are more precise words that EPA should use to link "pivotal regulatory science" with "regulatory decisions," such as "underpin" or constitute the "foundation" of the "scientific basis" of its regulatory decisions.
Response: Part 30.6 in the final rule states, "EPA shall conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein. The EPA will evaluate whether or not to conduct additional peer review of individual studies identified as pivotal science if the studies have already undergone peer review." The word "justify" is not used in this provision.
<IP_ADDRESS> EPA's Review Shall Be Consistent with the Requirements of the OMB Final Information Quality Bulletin for Peer Review (70 FR 2664) and the Exemptions Described Therein.
 <IP_ADDRESS>.1 Time-Sensitive
Comment: Commenter (6137) suggests that, to be consistent with the OMB Bulletin, the independent peer review guidelines "do[] not cover time-sensitive health and safety disseminations, for example, a dissemination based primarily on data from a recent clinical trial that was adequately peer reviewed before the trial began. For this purpose, `health' includes public health, or plant or animal infectious diseases". The commenter provides that this encompasses most of the studies that EPA intends to review, including but not limited to, the Integrated Risk Information System (IRIS) assessments, TSCA risk evaluations, and National Emissions Standards for Hazardous Air Pollutants risk assessments. According to the commenter, in each case, EPA utilizes scientific data to make safety determinations for chemical pollutants and impacts on human health and the environment, and thus, timely review is tantamount to protecting public health. The commenter contends that the proposal to add another layer of review is antithetical to the time sensitive nature of these reviews.
Commenter (6137) states that the Bulletin makes clear that agencies should "ensure peer review does not unduly delay the release of urgent findings." The commenter asserts that, if EPA wants to independently peer review all pivotal science, then it must clearly outline how it will ensure the process will not lead to undue delay. The commenter notes that the Agency has failed to do so. The commenter provides, for example, there is nothing in the Proposed Rule outlining how EPA will conduct an independent review of the studies underlying the NAAQS standard, which will require review of hundreds if not thousands of science documents, within the NAAQS review cycle. The commenter adds that the standard review period in the independent peer review process (i.e., from submission of a manuscript to final review) is time intensive. The commenter states that, if it can take months to review one manuscript, and that it is difficult to envisage how EPA could subject every piece of pivotal regulatory science to the same standard of review and still complete scientific assessments in a timely manner.
Response: The EPA finds that comments received from the 2018 proposed rule and the 2020 SNPRM have merit, in part. However, in this rule, the EPA is not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review does not typically include reanalysis of the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information) and, thus, peer review is not considered a replacement for the data availability requirements of this rule.
Part 30.6 in the final rule states, "the EPA shall conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein. The EPA will evaluate whether or not to conduct additional peer review of individual studies identified as pivotal science if the studies have already undergone peer review." The Agency will follow the requirements by the OMB Final Information Quality Bulletin for Peer Review ensuring that peer reviews do not unduly delay the release of urgent findings.
<IP_ADDRESS>.2 Studies that have Already Been Peer Reviewed
Comment: Commenter (2337) expresses concern regarding the Proposed Rule's requirement that EPA conduct independent peer review of scientific evidence on which it relies. The commenter states that the Proposed Rule applies this peer review requirement uniformly and does not exempt studies that have already been independently peer reviewed, as many health and scientific studies are. The commenter notes that EPA has not explained why EPA must conduct additional peer review of data that has already been reviewed. The commenter adds that this is of particular concern as many environmental statutes require EPA to periodically review, and where appropriate revise, regulatory standards. The commenter notes that, if EPA were now to add independent peer review to its periodic review of standards, it would be even less able to meet this mandated timeframe.
Comment: Commenter (5165) asks why should scientific literature that has already undergone peer review and been vetted by EPA's science advisory panels be subjected to an additional layer of government peer review? Commenter states that these key questions should have been considered, and the answers made public, prior to the rule's proposal.
Comment: Commenter (9227) asserts that EPA must clarify that studies that have already been adequately peer reviewed by third parties need not be re-reviewed by EPA. The commenter notes that, because the proposed language of §30.7 expressly incorporates the OMB Peer Review Bulletin "and the exemptions described therein," it appears that EPA intends to incorporate the OMB Peer Review Bulletin provision providing that "agencies need not have further peer review conducted on information that has already been subjected to adequate peer review." According to the commenter, however, there is some ambiguity due to the proposed language in §30.7 instructing that EPA must "ask peer reviewers to articulate the strengths and weaknesses of EPA's justifications for the assumptions applied and the implications of those assumption for the results." The commenter provides that peer review conducted prior to EPA's reliance on a study would not have involved review of the strengths and weaknesses of EPA's justifications. According to the commenter, if EPA were required to re-peer review all influential scientific information, this rulemaking would burden EPA with needless and significant costs that likely would bring many EPA rulemakings to a standstill, preventing EPA from fulfilling its statutory mission of protecting public health and the environment. The commenter asserts that, to prevent this from happening, EPA must clarify that the proposed rule will not supplant EPA's existing authority under the OMB Peer Review Bulletin not to conduct further peer review where information has already been subject to adequate peer review -- and that such prior peer review is not subject to the requirement in proposed §30.7 that reviewers consider the strengths and weaknesses of EPA's justifications.
Response: This rulemaking is not intended to modify the Agency's interpretations of "best available science." The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. All Agency peer review follows the recommended procedures and approaches consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook, 4th Edition. The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect.
The EPA finds that comments received from the 2018 proposed rule and the 2020 SNPRM have merit, in part. However, in this rule, the EPA is not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review does not typically include reanalysis of the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information) and, thus, peer review is not considered a replacement for the data availability requirements of this rule.
The EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review. Implementation of this rule may increase the opportunity for independent validation of pivotal science by subject matter experts. As data are better understood through independent reanalysis, the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
<IP_ADDRESS>.3 Prevention of Conflict of Interest in Peer Reviewers
Comment: Commenters (0044, 0844, 2241, 4541, 6180, 6877, 6891, 6927) express concern about how to prevent conflicts of interest in peer reviewers from impacting decision-making.
Commenter (6891) supports EPA's recent policy decision to have external scientific advisory boards be represented by a diverse group of stakeholders, including those representatives from regulated industry.
Commenter (6927) recommends EPA ensure any third-party peer reviewers are truly independent and external from the agency staff, former staff or organizations that receiving EPA funding.
Commenter (6877) states that the proposed rule's statement, "EPA shall conduct independent peer review on all pivotal regulatory science used to justify regulatory decisions," risks the introduction of potential bias through the selection of the peer reviewers. The commenter notes that the reviewers should be individuals who have the relevant expertise and are impartial with full disclosure of any conflicts of interest. The commenter asserts that, having had industry, government, or other funding for work on the topic at hand should not by itself be disqualifying to serving in peer review of scientific research.
Commenter (0044) asserts that scientists from industries directly impacted by rulemaking on chemicals must not be involved in the peer review; they could serve as non-voting advisors to a review panel; they have an opportunity to submit their opinions under FIFRA applications. The commenter notes that, otherwise, the potential for conflict of interest is significant. The commenter states that the goal of rulemaking for chemical exposures is to protect the public health; it is not for putting industry concerns first.
Commenters (0844) states that when scientists who receive EPA grants also serve on one of the agency's advisory boards, a conflict is created. According to the commenter, these conflicts were eliminated when Administrator Pruitt issued a new directive in October 2017, that scientists who receive an EPA grant, or reap a substantial benefit from a grant, could not also serve on an advisory committee. The commenter notes that the scientists would be given the option of either ending their grants or to continue to advise EPA, but not both.
Commenter (4541) asserts that EPA needs to conduct its own peer review and remove from its advisory committees and peer review process individuals with conflicts of interest. The commenter states that it is always a concern when reviewers and their institutions are conflicted by their dependence on agency/government funding and yet expected to provide impartial, objective reviews for science that is funded by that same agency/government.
Commenter (6180) states that the proposed rule must require and ensure that peer reviews be comprised of independent experts and not only EPA staff or otherwise biased individuals. The commenter supports the OMB Final Bulletin which notes the importance of the independent nature of peer reviewers and that government employees are subject to federal ethics requirements (70 FR 2664).
Response: EPA is subject to the OMB Bulletin and its provisions have been incorporated into the Agency's peer review handbook (Ref. 44). EPA continues to believe that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will lead to consistent implementation of those procedures across the Agency. In the final rule, the EPA "shall conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein." All Agency peer review follows the recommended procedures and approaches found in the EPA's Peer Review Handbook. This Peer Review Handbook is used as guidance by EPA staff and managers to ensure that the Agency's Peer Review Policy is implemented objectively and that the integrity of our peer review activities can be demonstrated transparently to the American public.
Comment: Commenter (1972) requests that EPA clarify the use of peer review in proposed 40 CFR 30.7. The commenter provides that the OMB Bulletin states that, for more important work, "independent" peer review should involve "[r]reviewers" that "are generally not employed by the agency or office producing the document. As NAS has stated, `external experts often can be more open, frank, and challenging to the status quo than internal reviewers, who may feel constrained by organizational concerns.'" The commenter adds that EPA's own guidelines similarly note that "[f]or those work products that are intended to support the most important decisions or that have special importance in their own right, external peer review is the procedure of choice." The commenter expresses that, given the credibility bestowed by both the courts and the public upon the studies that are "peer reviewed," EPA should clarify the conditions under which it will use truly independent external peer reviewers. The commenter recommends that this include some parameters on how external peer reviewers are selected, such as taking care to ensure that reviewers are not those working under, or applying for, EPA grants or otherwise have long-standing relationships with EPA, such as former EPA officials now serving in academia. The commenter suggests that independent external peer review be used for all "significant regulatory actions" as that term is defined under Executive Order 2866. The commenter contends that avoiding internal reviews by other EPA officials for significant regulatory actions will help restore public trust in the quality of information relied upon by EPA.
Response: This rulemaking is not intended to modify the Agency's interpretations of "best available science." The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. All Agency peer review follows the recommended procedures and approaches consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook, 4th Edition. The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect.
The EPA finds that comments received from the 2018 proposed rule and the 2020 SNPRM have merit, in part. However, in this rule, the EPA is not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review does not typically include reanalysis of the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information) and, thus, peer review is not considered a replacement for the data availability requirements of this rule.
Comment: Commenter (0555) states that, when engaging experts in peer review, EPA should consider the recommendations of recent interdisciplinary efforts regarding scientific advisory panels. The commenter adds that such advisors can provide a necessary and valuable source of information and peer review for agency science, but care should be taken in both the composition of the panels and the charges they are given. According to the commenter, an important 2012 Keystone Center report offers a series of recommendations on "the composition of committees that are empaneled to review the science behind a regulatory decision." The commenter notes that the report acknowledges the importance of choosing panelists that "have the knowledge, training, and experience needed to address the charge to the panel," and it encourages agencies "to recognize that all potential panelists will have conscious and unconscious biases." The commenter adds that it recommended, "because biases exist, an agency should strive to engage a wide range of perspectives of qualified scientific experts." The commenter notes that both the Keystone group and a group convened by the Bipartisan Policy Center (BPC) in 2009 recommend that scientific peer reviewers restrict their advice to matters of science, and not be asked to recommend regulatory policies. The commenter states that EPA's Clean Air Scientific Advisory Committee (CASAC), for example, is tasked with advising on policy choices, which creates incentives to present policy views as scientific recommendations.,  The commenter recommends, when drafting charge questions for individual peer reviewers and scientific advisory committees, EPA be careful to solicit their scientific expertise without encouraging them to blur the lines between scientific expertise and policy judgment. According to the commenter, as both the BPC and Keystone reports emphasized, the questions posed to experts "should be clearly articulated, and `explicitly differentiate, to the extent possible, between questions that involve scientific judgments and questions that involve judgments about economics, ethics, and other matters of policy.'"
Response: This rulemaking is not intended to modify the Agency's interpretations of "best available science." The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. All Agency peer review follows the recommended procedures and approaches consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook, 4th Edition. The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect. However, as detailed in OMB's Final Information Quality Bulletin for Peer Review, "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary". Peer review does not typically include reanalysis of the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information) and, thus, peer review is not considered a replacement for the data availability requirements of this rule.
In the final rule, the EPA "shall conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein." All Agency peer review follows the recommended procedures and approaches found in the EPA's Peer Review Handbook. This Peer Review Handbook is used as guidance by EPA staff and managers to ensure that the Agency's Peer Review Policy is implemented objectively and that the integrity of our peer review activities can be demonstrated transparently to the American public.
Comment: Commenter (1390) asserts that scientists within government agencies and outside of government agencies should be involved in reviewing scientific information. The commenter states that they should be in continual dialogue about strengths and limitations of available scientific research. According to the commenter, government agencies should not be permitted to submit, approve, or implement policies until it is clear that agencies have exhausted all possible methods to review and dialogue about all scientific information relevant to a policy. The commenter adds that it will be impossible for all public policies to be completely aligned with all relevant scientific information, however, government agencies should not be permitted to submit, approve, or implement policies that are incongruent with relevant scientific information until they have transparently acknowledged the scientific information that pertains to the policy and transparently justified the reasons for which a policy is incongruent with specific scientific information.
Response: All Agency technical work products go through rigorous internal review and, when appropriate, peer review. The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. In the final rule, the EPA shall conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein. All Agency peer review follows the recommended procedures and approaches found in the EPA's Peer Review Handbook [4[th] Edition]." The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect. Further, the EPA plans to promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers.
Comment: Commenter (2241) recommends that the rule indicate how transparency will be used in the appointment of peer reviewers. The commenter expresses concern that the rule does not describe how this will be implemented. The commenter adds that, in particular, 70 FR 2664 clearly states that reviewers shall be selected for expertise, balancing of perspectives, independence, and freedom from conflicts of interest (which are to be disclosed).
Response: This rulemaking is not intended to modify the Agency's interpretations of "best available science." The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. In the final rule, the EPA "shall conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein." All Agency peer review follows the recommended procedures and approaches found in the EPA's Peer Review Handbook. All Agency peer review follows the recommended procedures and approaches consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook, 4th Edition. The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect.
The EPA finds that comments received from the 2018 proposed rule and the 2020 SNPRM have merit, in part. However, in this rule, the EPA is not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary."
Comment: Commenter (4595) expresses concern that the proposal does not articulate whether the "peer reviewers" would be the same peer reviewers or scientific advisory committee that conducted the original scientific assessment. The commenter states that this introduces an unnecessary phase of peer review that risks introducing regulatory inertia and the ability of the agency to utilize the best available science in its decision making.
Response: The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. In the final rule, the EPA "shall conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein." All Agency peer review follows the recommended procedures and approaches found in the EPA's Peer Review Handbook. All Agency peer review follows the recommended procedures and approaches consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook, 4th Edition.
The EPA finds that comments received from the 2018 proposed rule and the 2020 SNPRM have merit, in part. However, in this rule, the EPA is not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." The EPA maintains that all Agency peer review follows the recommended procedures and approaches consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook, 4th Edition. The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect.
<IP_ADDRESS>.4 Other Comments
Comment: Commenter (6375) states that the exemptions described under OMB's Information Quality Bulletin for Peer Review to ensure compliance with privacy and national security laws and regulations are appropriate. The commenter recommends that EPA handle these exemptions in a similar manner to the case-by-case exceptions for studies: The commenter further recommends that EPA provide a robust and transparent explanation in the Federal Register early in the rulemaking process to explain why the provisions of this rule do not apply to the rulemaking in question (e.g. which laws would be in conflict).
Response: All Agency peer review follows the recommended procedures and approaches found in the EPA's Peer Review Handbook. This Peer Review Handbook is used as guidance by EPA staff and managers to ensure that the Agency's Peer Review Policy is implemented objectively and that the integrity of our peer review activities can be demonstrated transparently to the American public. Part 30.6 in the final rule states, "the EPA shall conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein. The EPA will evaluate whether or not to conduct additional peer review of individual studies identified as pivotal science if the studies have already undergone peer review. As part of promulgating a significant regulatory action or drafting influential scientific information the Agency will document its decisions and reasoning."
Comment: Commenter (9227) suggests that, if EPA moves ahead with these proposed provisions, EPA revise the proposed regulatory language to clarify that all of the exemptions set forth in section IX of the OMB Peer Review Bulletin remain in effect regardless of whether they pertain to feasibility. The commenter further suggests that EPA clarify what, if any, additional effect is intended by the exemption provision in proposed 40 CFR 30.9. The commenter provides that proposed 40 CFR 30.9(b) provides that the Administrator may grant an exemption from the peer review requirements if he or she determines that "[it] is not feasible to conduct independent peer review on all pivotal regulatory science used to justify regulatory decisions for reasons outlined in OMB Final Information Quality for Peer Review (70 FR 2664), Section IX." The commenter notes, however, only two of the seven enumerated exemptions in Section IX of the OMB Peer Review Bulletin pertain to feasibility -- Exemption 1 governing "national security, foreign affairs, or negotiations involving international trade or treaties" and Exemption 3 governing time-sensitive health or safety disseminations.
Response: The requirements in 40 CFR 30.3 apply to influential scientific information, unless the influential scientific information is exempted from peer review requirements as described in Section IX of the OMB Bulletin for Peer Review (Ref. 8). Consistent with this approach, the EPA is finalizing the definition of "influential scientific information" as proposed in the 2020 SNPRM.
Part 30.6 in the final rule states, "the EPA shall conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein. The EPA will evaluate whether or not to conduct additional peer review of individual studies identified as pivotal science if the studies have already undergone peer review."
Section 30.7 (formerly 40 CFR 30.9 in the 2018 proposed rule and the 2020 SNPRM) of the final rule requires the Agency to document the rationale for any exemptions granted by the Administrator in the significant regulatory action or influential scientific information. This documentation would typically be provided as part of the proposed rulemaking given this will be part of the decision of what is the pivotal science for the rule. The final rulemaking will also provide clear documentation.
<IP_ADDRESS> EPA Shall Ask Peer Reviewers to Articulate the Strengths and Weaknesses of EPA's Justification for the Assumptions Applied and the Implications of those Assumptions for the Results
Comment: Commenter (6150) supports the appropriateness of requesting that peer reviews focus on the strengths and weaknesses of the assumptions and their implications.
Commenter (6927) requests EPA more clearly define and clarify what is meant by asking "peer reviewers to articulate the strengths and weaknesses of EPA's justification for the assumptions applied and the implications of those assumptions for results.
Conversely, commenters (4592, 4595, 6133) express concerns about the requirement that peer reviews focus on the strengths and weaknesses of the assumptions and their implications.
Commenter (4595) expresses concern that the proposed rule requires EPA to ask peer reviewers to articulate the strengths and weaknesses of EPA's assumptions, implying that the agency would be implementing a separate and secondary level of peer review to address assumptions.
Commenter (6133) states that the final paragraph of 40 CFR 30.7 appears to suggest that EPA should conduct peer review of the proposed agency action itself, rather than of the science underlying that action, stating that "EPA shall ask peer reviewers to articulate the strengths and weaknesses of EPA's justification for the assumptions applied and the implications of those assumptions for the results." According to the commenter, it is entirely unclear how "peer review" could be applied to EPA's reasoning itself, rather than the cited science, and the Proposal contains no further clarification. The commenter recommends that the EPA abandon the unlawful Proposal altogether but, if EPA does finalize any rule based on the Proposal, EPA still should abandon the unlawful approach reflected in proposed 40 CFR 30.7.
Response: EPA agreed with commenters that the statement in §30.7 of the notice of proposed rulemaking, "implications of the assumptions on the results," is unclear about how peer review could be applied to EPA's reasoning itself, rather than the pivotal science supporting the regulatory decision. Therefore, §30.6 (formerly 40 CFR 30.7 the 2018 proposed rule and the 2020 SNPRM) of the final rule was changed to "Because transparency in pivotal science includes addressing issues associated with assumptions used in analyzing dose-response data, the EPA shall ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied and the implications of those assumptions for the results." Further, as detailed in OMB's Final Information Quality Bulletin for Peer Review, "There will be cases in which an agency determines that a more rigorous or transparent review process is necessary. For instance, an agency may determine a particular journal review process did not address questions (e.g., the extent of uncertainty inherent in a finding) that the agency determines should be addressed before disseminating that information. As such, prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." For these reasons, EPA may determine that additional peer review is necessary.
Comment: Commenter (0560) requests that EPA add the following to §30.7: "EPA shall particularly request that reviewers address alternate models and ignored or contrary evidence."
Response: EPA appreciates the recommendation. The EPA has finalized language in §30.6 (formerly 40 CFR 30.7 the 2018 proposed rule and the 2020 SNPRM) of the final rule to, "Because transparency in pivotal science includes addressing issues associated with assumptions used in analyzing dose-response data, the EPA shall ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied and the implications of those assumptions for the results."
13.1.3 Comparison of the Approach in the Proposed Rule to Existing Scientific Review Processes
<IP_ADDRESS> Existing Peer-Review Mechanisms
 <IP_ADDRESS>.1 Peer Review is Not Adequate
Comment: Commenter (0040) states that study results need to go through the peer review publication and replication processes so that the conclusions are based on strengths of evidence and are weighed according to their merit. The commenter asserts that it is through this laborious process that science progresses.
Commenters (0563, 2142, 4534, 6045, 6176, 6365, 6867) suggest that peer review, in and of itself, is not sufficient to validate studies.
Commenter (0563) notes that, while some opponents of the proposal argue that peer review ensures the quality of studies published in refereed journals, this widespread perception is false. The commenter quotes Richard Smith:
      ... almost no scientists know anything about the evidence on peer review. It is a process that is central to science -- deciding which grant proposals will be funded, which papers will be published, who will be promoted, and who will receive a Nobel prize. We might thus expect that scientists, people who are trained to believe nothing until presented with evidence, would want to know all the evidence available on this important process. Yet not only do scientists know little about the evidence on peer review but most continue to believe in peer review, thinking it essential for the progress of science. Ironically, a faith based rather than an evidence-based process lies at the heart of science.
Commenter (2142) asserts that the peer review system is flawed because reviewers do not dig deeply into the data or analysis as they have little incentive to do so. The commenter states that reviewers often do not have access to the data or algorithms, making it impossible to truly check the results. The commenter states that the high percentage of invalidated peer-reviewed studies in the medical field, where one would think the scrutiny to be intense, shows the failure of relying solely on peer review.
Commenter (4534) states that, if the data has been peer reviewed and is available for the regulated parties to validate during the review and comment period, then all parties should be satisfied with the process. According to the commenter, utilizing data that has not been peer reviewed or validated could lead to incorrect conclusions, which impact recommendations to maintain or further improve the environment that we live in.
Commenter (6045) asserts that peer-reviewed scientific research data that are not publicly available are not rigorous enough for use in decision making. The commenter asks why should a secretly "insider-peer-reviewed" elitist group evaluate and approve something for public policy that affects millions of citizens, as well as billions of U.S. dollars in taxpayers' money?
Commenter (6176) states that journal peer review is not a substitute for making data available to the public. The commenter states that not all peer review processes are the same or of adequate quality. The commenter adds that, unless relevant underlying data and methodology are made available, the public has no way to know whether a specific peer review process has been effective and properly addressed potential problems.
Commenter (6867) asserts that pre-publication peer review is not an adequate substitute for public availability of data, methodology, and computational code. The commenter states that, in light of overwhelming evidence that a significant portion of studies published in prominent peer reviewed journals are not reproducible, it is dubious to claim that any particular research is valid and relevant unless -- at a minimum -- other researchers and the public have access to the underlying data, methodology, and computational code. The commenter states that empirical evidence indicates that peer reviewers routinely fail to identify even major errors and peer review in less prominent journals may often occur in name only. The commenter states that the best available metascience -- science about science -- indicates that pre-publication peer review is not adequate to ensure the validity of published scientific claims.
Commenter (6365) states that the experience of the lead NAAQS rulemaking (73 FR 66964) demonstrates the salutary purpose that transparency would serve in the regulatory process. The commenter provides that, in that rule, EPA relied heavily on Lanphear et al. (2005) although the underlying data were not available to EPA or to stakeholders for replication and validation at the time of publication. The commenter notes that, due to the lack of availability, there was no complete reanalysis of that data for nearly a decade and EPA relied on the study in both the 2008 and 2016 lead NAAQS reviews. The commenter states that the lack of available data has been noted in the scientific literature as preventing alternative statistical analyses to be performed, which is particularly important given that key confounding factors such as parental education, intelligence, birthweight, smoking, and race were not controlled in several of the Lanphear cohorts and may contribute to an overestimation of health effects. The commenter provides that, when the data underlying Lanphear was reanalyzed numerous errors that were not discovered in the peer-review process and that were not previously known to EPA were uncovered and analyzed. According to the commenter, although EPA ultimately determined that these errors did not materially affect the NAAQS review documents that relied on the Lanphear study, the Agency included a memorandum in the subsequent NAAQS rulemaking docket that acknowledged those errors and considered their impact on the regulatory process.
Response: The purpose of this action is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions.
The EPA finds that these comments have merit, in part. However, in this rule, the EPA is not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review does not typically include reanalysis of the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information) and, thus, peer review is not considered a replacement for the data availability requirements of this rule.
The EPA would also like to clarify that peer review is but one aspect that is considered in evaluating studies for potential use in EPA products. In addition to determining whether a study has been adequately peer reviewed, the EPA also independently evaluates study quality using the following factors: soundness, applicability and utility, clarity and completeness, uncertainty and variability, and evaluation and review. The EPA may also consider additional factors based on the purpose of the assessment. For example, when selecting studies for the dose-response assessment, the EPA would evaluate the exposure situation being addressed, the quality of the studies, the reporting adequacy, and the relevance of the endpoints.
Comment: Commenter (6362) highlights the NAS review of the 2010 draft formaldehyde IRIS assessment peer review as an example of where the IRIS assessment failed to reflect the best available science. The commenter expresses that the NAS stated that EPA's claims regarding all leukemias, myeloid leukemia or related hematopoietic cancers were not supported. The commenter also provides that EPA's preliminary conclusions appeared subjective and that no clear scientific framework had been used by EPA to reach its conclusion. The commenter adds that the NAS recommended that EPA revisit its determination of causality for specific cancers, using a methodology that integrates lines of evidence and addresses the specific criticisms in the NAS report. The commenter notes that the NAS also made numerous recommendations for improving the overall process and application of science used in all assessments generated by the IRIS program. Now, according to the commenter, seven years since that NAS report was published, EPA continues to revise its assessment while not disclosing how emerging scientific evidence or modern risk assessment methods are being employed.
Response: These comments are not directly relevant to this rulemaking. Any concerns about specific projects should be directed to the appropriate EPA programmatic or regional office.
<IP_ADDRESS>.2 Appropriate Review Processes Already Exist
 Overall Support for Current Review Processes
Comment: Commenters (0061, 1364, 1433, 1773, 1943, 1973, 2249, 2408, 2425, 2905, 2908, 2911, 4842, 4845, 4884, 5164, 5165, 6103, 6122, 6133, 6135, 6149, 6153, 6164, 6171, 6174, 6188, 6191, 6358, 6367, 6379, 6446, 6464, 6868, 6876, 6898, 6915, 6922, 6974, 6901, 6916, 6920, 8272, 8275, 8801, 9227) indicate that, in peer review, appropriate processes and procedures for designing and conducting studies already exist, including existing peer review procedures, binding contracts, science assessments, use of peer-reviewed literature and collaborative methods. Comments include:
 EPA's existing processes, including peer review, already help ensure that studies used by EPA are scientifically sound, the proposed rule is not needed to add credibility or reliability to the development of EPA models and standards. (6915)
Commenters (2425, 5165, 6358, 6868, 8272) state that the process of peer-reviewed research is already sufficiently transparent by design and that there are specific existing rigorous mechanisms already in place, including Institutional Review Board (IRB) review, peer review of funding requests, and peer review by journals before publication:
 Regarding the IRB, commenter (2425) states that EPA requires an IRB to review all scientific studies occurring in an academic institution prior to beginning the study. The IRB closely reviews all study methods to ensure that the methods are sound and study subjects are protected and provides details of requirements. Once a study is complete, data involving human subjects must always be kept for a minimum of five years after study completion and be available for IRB review. (2425)
 Commenters (6358, 8272) state that peer review processes for funding are often highly competitive and involve critiques of study design that, if addressed, can lead to research with fewer potential biases or unmeasured confounders. (6358, 8272)
 Commenters (2425, 5165, 6358, 8272) contend that in order to publish the data in a scientific journal, scientists not associated with the research study must review the study design to ensure that it is scientifically sound and that there are many rigorous aspects of this review.
 Commenter notes that this includes assessing for "good laboratory practices" and "standardized test methods." In this rigorous peer-review process, which is already required for studies to be considered by EPA, the commenter states that experts in the field of study not only assess the study's design and methods, but its statistical analysis and conclusions in an objective manner to determine the study's merit and significance. The commenter asserts that, if any of these areas are deemed to be unsatisfactory, the study will not be published. Additionally, the commenter states that, when studies are published, the authors must detail all methods and statistical tests used and this information is presented to the public. (2425)
 Commenter (5165) notes that when the results of a scientific study are submitted for publication, the uncertainties, assumptions, parameters and theories utilized by the scientists are laid out in the publication; peer review analyzes all these components to establish validity. Thus, commenters (6358, 8272) state that peer reviews for journal publications often involve critiques that lead to more nuanced discussions of a study's limitations and the conclusions that can be drawn given those limitations.
 Study manuscripts go through multiple rounds of rigorous peer review to ensure the validity of study methods, the fidelity of adhering to the scientific methods, and the added value of the findings yielded from the study. The commenter provides that, during the peer review process, scientists are frequently asked to perform additional analyses and present additional results to convince the peer reviewers that the findings are robust, and their conclusions are supported by the data. The commenter states that, for certain types of studies without identifying patient data, such as many clinical trials and basic laboratory research studies, it is becoming the norm among top tier journals to request data as part of the publication process. The commenter adds that many journals also use plagiarism detection software and submit images to manipulation detection software to ensure data integrity. (6868)
 Commenters note that there are also mechanisms for review and comment post-publication of studies in journals. Commenters state that investigators or institutions in possession of data sets often make them available to researchers who meet appropriate criteria, such as demonstrating the ability to safeguard privacy for research participants. The commenters express that other researchers may use the data either to reanalyze findings or to examine additional research questions. (6358, 8272)
 Commenters (6358, 6868, 8272) state that studies that generate unexpected findings and provocative implications provoke discussion. Commenter (6868) states that (1) letters to editors of science journals question the sample size, the study construction, the analytical method, the generalizability and nearly every conceivable aspect of a study; and (2) scientific meetings host pro/con debates about how to interpret provocative data and how far to apply its implications. The commenter (6868) states that this post-publication debate helps generate new questions to ensure that our understanding of the data is correct. Commenters (6358, 8272) state that re-analyses of data may be undertaken at times in order to strengthen confidence in particularly when there are unexpected or consequential findings, but typically confidence in findings is strengthened by having evidence from different studies accumulate.
 EPA already has in place a robust process for determining studies' reliability. The commenter states that EPA has spent decades creating and implementing an elaborate system to ensure the best scientific research serves as the basis for its regulatory actions. The commenter states that this system already includes vigorous peer-review requirements, ethical standards, and independent review by scientific professionals., The commenter states that according to EPA's own IQGs, EPA "maintain[s] a robust quality system that addresses [information collected through contracts with EPA; information collected through grants and cooperative agreements with EPA; and information submitted to EPA as part of a requirement under a statute, regulation, permit, order or other mandate] by including regulatory requirements for quality assurance for EPA contracts, grants, and assistance agreements." The commenter notes that this includes an Agency-wide Quality System that requires, among other things, a quality assurance manager and Quality Assurance Project Plan (QAPP) for projects and tasks that involve the use of environmental data, and a Peer Review System that requires major scientifically- and technically-based Agency work product to be peer reviewed. (6188)
 Commenters (0061, 1433, 1773, 1943, 6164, 6191, 6358, 6367, 6464, 6868, 6164, 8272, 8801) assert that the peer review process is the gold standard for scientific understanding and advancement for decades and assert that peer review is the integrity and reliability standard for ensuring unbiased, rigorous vetting of research and scholarly content prior to publication. Commenter (6164) states that the peer review process remains the global gold standard of academic achievement and affords the type of informed discourse necessary for the objectivity, rigor, and legitimacy of scientific information. Commenter (9227) states that publication in a peer-reviewed scientific journal is the way that scientists communicate their findings to other scientists and is considered the hallmark of scientific quality. Commenter (6103) contends that the peer review process is more than adequate to ensure studies are reliable to use in the policymaking process.
 EPA already uses collaborative methods to get to the truth of needed environmental protections. The commenter notes that the Volkswagen diesel passenger vehicle transgression was discovered by West Virginia University students operating in a laboratory using methods not then sanctified by the U.S. EPA Auto Emissions Laboratory. (4884)
 EPA has rightly continued to rely on the robust peer-reviewed literature to inform the air quality standard-setting process year after year, incorporating the best available scientific evidence in epidemiology, toxicology, and exposure assessment to set the outdoor air quality standards at levels that protect public health and the environment. (6133)
 Existing procedures and safeguards commonly used to conduct, disseminated, and synthesize scientific research are more than adequate in ensuring the validity and integrity of the science used to inform EPA decision-making. (6135)
 Existing policy vehicles under established statutory authority have proven effective in providing a consensus scientific basis through open and transparent processes that support a regulatory framework for radiation protection that balances the beneficial uses of ionizing radiation with the health and safety of workers, patients, and the public. (6149)
 The review process currently utilized by EPA in evaluation of scientific research to inform policy is already transparent and involves rigorous peer-review and evaluation by independent experts. (6171)
 EPA has a proven record of using the best available science and research to set standards under its foundational laws that protect public health and the environment, which the scientific peer review process and our transparent and participatory rulemaking process can well ensure. (6174)
 Commenter is confident that longstanding practices of peer review undertaken by leading journals that publish toxicology studies of public health importance have been and will continue to be of sufficient rigor to assure the integrity and reliability of such research. (6367)
 EPA-implemented statutes have established independent scientific oversight and scientific review processes to ensure that EPA standards and regulations are founded in the best available science. (6379)
 Procedures are already in place that assure use of the best available science for EPA's regulatory activity. The commenter states that these procedures include that much of the scientific research considered in the regulatory context, and all research published in reputable journals, has already been subject to extensive peer review. The commenter states that the method that is accepted in the scientific community for assessment of the strength of scientific research is peer review. The commenter notes that, since scientific studies are regularly peer reviewed, including peer review by expert panels such as the SAB, the proposed rule unnecessary. (6446)
 Commenter encourages EPA to continue the current open and science-based development process and continue to actively involve states. (6901)
 Formal peer review, whether or not prior to publication, also validates the outcome of scientific studies, as reflected in the OMB Guidelines presumption of objectivity and usefulness for scientific studies that have undergone formal peer review. (6916)
 The proposal calls into question established processes such as EPA's IRIS review program and NAAQS program. The commenter states that EPA more-than-adequately explains its decisions and analyses both in recommending NAAQS revisions and in quantifying chemical toxicities in IRIS. (6920)
 The proposed rule is unnecessary as there are processes in place to rigorously review the scientific integrity of the studies that are used in regulatory science. The commenter states that protocols and data analysis plans are published and publicly available. The commenter states that findings are peer reviewed by scientists prior to publication and additional data are available upon request for review. The commenter states that external advisory boards with relevant expertise review the studies that are being considered in the EPA regulatory processes. (6876)
 The processes for developing human health assessments and setting and reviewing standards by EPA have been routinely scrutinized by organizations such as the NAS and the Government Accountability Office (GAO). The commenter states that EPA has incorporated their recommendations and improved its approach over time. (6920)
Response: The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. EPA does not take the position that studies for which the underlying data are not publicly available are per se invalid. Rather, EPA acknowledges that where underlying dose-response data in pivotal science are available, subject matter experts could independently reanalyze the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of uncertainty. As discussed in the final rule preamble, the EPA clarifies that the rule requirements are not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Revie . The preamble to the Bulletin states that, "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review by either journals or independent advisory committees, such as the EPA's SAB, is typically conducted without access to the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information). Should peer reviewers have access to the underlying data, they could conduct a more in-depth review that may include independently reanalyzing the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of uncertainty in the original study. Thus, peer review is not considered a replacement for the data availability requirements of this rule and implementation of this rule could provide independent peer reviewers as well as subject matter experts within the EPA and the public with the opportunity to evaluate pivotal science in ways that otherwise would not have typically occurred with existing EPA or peer review processes.
The EPA is, therefore, finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate additional peer review, consistent with the OMB Bulletin for Peer Review, of individual studies identified as pivotal science if the studies have already undergone peer review. If the Agency conducts peer review on pivotal science, the EPA shall ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied in analyzing dose-response data and the implications of those assumptions for the results.
The purpose of this action is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions. Such independent reanalyses will subsequently enable the EPA to make stronger, data-driven decisions in future rulemakings or in revisions to existing rules or influential scientific information. Implementation of this rule will also increase the opportunity for independent validation of pivotal science by subject matter experts. As data are better understood through independent reanalysis, the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
Advisory Boards
Comment: Commenters (1973, 2336, 5165, 5172, 6046, 6119, 6121, 6164, 6188, 6367, 6373, 6446, 6894, 6920, 6922, 8801, 9227) suggest that advisory boards already provide assessment of the adequacy of the approaches taken in the studies, including the EPA SAB and EPA's CASAC. Commenter (6922) provides that the primary function of EPA's SAB is to review the quality and relevance of scientific and technical information being used by EPA or proposed as the basis for its regulations.
Commenter (6145) states that in addition to all of the already existing laws and guidance under which EPA is directed to review scientific information in its promulgation of regulations, EPA also has "several mechanisms for vetting science through several expert panels, including the EPA SAB, the EPA Clean Air Scientific Advisory Committee, and the EPA FIFRA Scientific Advisory Panel . . .."
Commenter (2336) asserts there are some aspects of EPA's duties that need to be insulated from political considerations, at least to the extent that the scientific integrity of a study can be judged independently of the political ramifications of using it to formulate regulations. The commenter states that, to fill that need, Congress provided for a 48-member SAB in the CAA. The commenter states that, although the Board's recommendations are advisory, the EPA Administrator, as a practical matter, is obligated to provide a cogent and convincing rationale for rejecting the Board's determinations.
Commenters (5165, 6164, 9227) express concern that the proposal does not acknowledge that EPA already has institutional mechanisms to review and vet scientific information through panels of scientific experts. The commenters assert that these expert panels were established to review the quality and relevance of scientific and technical information being used by EPA or proposed as the basis for EPA regulations. Commenter (5165) states that, by ignoring the existence of these bodies in the proposed rule, EPA suggests that it does not trust its own scientific advisors which tends to undermine public confidence in EPA decision making, rather than to bolster it. Commenter (6164) urges EPA to use these resources and clarify how this expertise will be incorporated into rulemaking under this proposal. Commenter (9227) adds that a SAB workgroup noted:
      The proposed rule fails to mention that EPA has mechanisms for vetting science through several expert panels . . . For example, the EPA CASAC routinely reviews and evaluates epidemiologic and toxicological studies that are the basis for dose-response relationships used in risk and exposure assessments for air pollutants regulated under the National Ambient Air Quality Standards. Although such mechanisms do not typically engage in reanalysis of original data using the same methods as the original investigators, they do entail a rigorous review process that goes beyond the typical journal peer review procedures.
Commenter (6446) states that procedures are already in place that assure use of the best available science for EPA's regulatory activity. The commenter provides that these procedures include the SAB which is required by its authorizing legislation to make every effort to maximize public participation and transparency, "including making the scientific and technical advice of the [SAB] and any investigative panels . . . publically available in electronic form on the website of the Environmental Protection Agency." The commenter notes that the proposed rule does not mention the SAB, or its important function in providing expert assessments of whether scientific research offered as the basis for regulatory action is, in fact, the best available science.
Commenter (6894) states that the CASAC was established in 1977 to explicitly provide a diverse, independent science peer review of all the considered science so the Administrator would have a direct report on the science and policy options to consider in parallel to those of staff. The commenter states that, along with public comment, this proven process has long provided the credible "process" only vaguely described in the proposal.
Commenter (1973) states that the EPA SAB review process already provides assessment of the adequacy of the approaches taken in the studies considered in setting a standard. The commenter provides the following example regarding the setting of air quality standards:
      The process involves a summary of the science by the EPA in the Integrated Science Assessment (ISA), which is reviewed by experts, and then provided to the external EPA Clean Air Science Advisory Committee (CASAC) for review. If, based on how the analysis was done, what confounders were controlled for, or other criteria, the reviewers of the ISA, members of CASAC, or the general public find a study wanting, they are free to raise that issue in public comments to EPA on the draft ISA, which EPA is required to respond to substantively. Subsequently they can again provide comments to CASAC when it reviews the draft ISA. This has happened routinely in the past, where many comments on the adequacy of individual studies, on possible flaws in measurement techniques or analytical methods have been raised and debated in past NAAQS reviews under the current review process for setting the NAAQS. There is no need to exclude most human studies to achieve a goal that is already being met.
Response: Peer review by either journals or independent advisory committees, such as the EPA's SAB, is typically conducted without access to the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information). Should peer reviewers have access to the underlying data, they could conduct a more in-depth review that may include independently reanalyzing the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of uncertainty in the original study. Thus, peer review is not considered a replacement for the data availability requirements of this rule and implementation of this rule could provide independent peer reviewers as well as subject matter experts within the EPA and the public with the opportunity to evaluate pivotal science in ways that otherwise would not have typically occurred with existing EPA or peer review processes.
The EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the OMB Bulletin for Peer Review and the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review. If the Agency conducts peer review on pivotal science, the EPA shall ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied in analyzing dose-response data and the implications of those assumptions for the results." The EPA Peer Review Handbook, 4th Edition, describes the process for peer reviews conducted by a Federal Advisory Committee and the National Academy of Sciences.
Peer Review is Rigorous
Comment: Commenter (6922) states that the EPA's Peer Review Handbook provides that if a regulation is supported by a scientific and technical work product, the underlying work product should be peer reviewed unless it meets listed exemption criteria. The Handbook explains that a critical element in ensuring that decisions are based on sound and defensible science is to have an open and transparent peer review process.
Commenters (1459, 1973, 2353, 2425, 2479, 4844, 5170, 5180, 6111, 6874, 6915, 6920, 6925) assert that scientific articles have already been subjected to a high degree of scrutiny and strong filters of quality control during the peer review process.
Commenter (6915) asserts that the existing peer review process ensures the reliability and validity of the scientific information relied upon by EPA in the regulatory process. The commenter states that existing policies and procedures for peer review include the following:
 EPA vets scientific studies through several independent expert panels, including the SAB, the EPA CASAC, the EPA FIFRA Scientific Advisory Panel, and the EPA Chemical Assessment Advisory Committee. The CASAC routinely reviews and evaluates epidemiological and toxicological studies that are the basis for dose response relationships used in risk and exposure assessments for air pollutants regulated under the NAAQS; the Chemical Assessment Advisory Committee reviews toxicological assessments of various chemicals for inclusion in EPA's IRIS database; and the NAS has reviewed EPA risk assessment practices numerous times.
 Each of these independent committees or panels is required to be staffed by a "fairly balanced" mix of regulators, academics, and industry/consultant representatives who bring a well-balanced perspective to the process. See Federal Advisory Committee Act, 5 U.S.C. App. 2 § (5)(b)(2), (c).
 OMB bulletin entitled "Final Information Quality Bulletin for Peer Review," 70 Fed. Reg. 2664-02 (Jan. 14, 2005), is applicable to all federal agencies, including EPA, and establishes government-wide guidance aimed at enhancing the practice of peer review of government science documents. The bulletin was subject to extensive public and agency comment on two prior draft versions. It includes guidance to federal agencies on what information is subject to peer review, the selection of appropriate peer reviewers, opportunities for public participation, and related issues. The bulletin also defines a peer review planning process that provides for public participation whenever possible and permits the public and scientific societies to comment about which scientific reports and studies merit especially rigorous peer review.
Commenter (1459) suggests that whether the data can be made publicly available is not as important as whether the data presented supports the conclusions and whether the experimental design and data analysis were free from error. The commenter states that, as such, the process of peer review by other experts in the field is a far more rigorous evaluation of a study's quality than the publish-ability of its data ever could be.
Commenter (6920) states that the current process is adequate. The commenter provides that the EPA currently uses robust, transparent processes to evaluate the best available scientific research, characterize the health hazards of chemicals and air pollution, and set standards to protect public health and the environmental. The commenter adds that EPA relies on peer-reviewed studies that describe the underlying data, methods, assumptions, sensitivity, and uncertainty of the results, and that these studies are cited and are published and available for public review.
Commenter (1973) states that EPA already has in place a detailed review process, that, together with the information already made public about the data and methods used in each study, allows EPA and its external science advisory committees, to reach that judgment.
Commenter (6925) states that the longstanding process of peer review includes an evaluation of the methodologies, assumptions, data, and inferences used in a study, which helps to ensure the appropriateness and validity of that study.
Commenters (2425, 4844) assert that scientific studies used by EPA to make regulatory changes are already rigorously examined prior to being published in peer-reviewed scientific journals. The commenters provide that EPA requires an IRB to review all scientific studies occurring in an academic institution prior to beginning the study. The commenters add that the IRB very closely reviews all study methods to ensure that the methods are sound and study subjects are protected. According to the commenters, once a study is complete, data involving human subjects must be kept for a minimum of five years after study completion and must be available for IRB review at all times. The commenters note that, in order to publish the data in a scientific journal, scientists not associated with the research study must review the study design to ensure that it is scientifically sound. The commenters add that this includes assessing for "good laboratory practices" and "standardized test methods." The commenters assert that, in this rigorous peer-review process, which is already required for studies to be considered by EPA, experts in the field of study not only assess the study's design and methods, but its statistical analysis and conclusions in an objective manner to determine the study's merit and significance. According to the commenters, if any of these areas are deemed to be unsatisfactory, the study will not be published, and that when studies are published, the authors must detail all methods and statistical tests used and this information is presented to the public.
Commenter (5170) states that that there are already transparent procedures that include review of research designs and findings by independent scientific experts and making the studies available to the public. The commenter notes that published research findings are evaluated and debated by scientists, health professionals, scientific organizations, industry as well as the public to analyze the appropriate conclusions that may be drawn. The commenter adds that, where studies make use of private data (for example, medical histories), independent scientists review and evaluate the studies.
Commenters (6153, 6874) state that the scientific community has developed robust mechanisms and procedures for designing and conducting studies in ways that minimize bias, drawing appropriate conclusions from research findings, and advancing knowledge. The commenters note that peer review processes for funding are often highly competitive and involve critiques of study design that, if addressed, can lead to research with fewer potential biases or unmeasured confounders. The commenters contend that peer reviews for journal publication often involve critiques that lead to more nuanced discussions of a study's limitations and the conclusions that can be drawn given those limitations.
Commenter (6111) asserts that there are ample and adequate safeguards in place at the leading journals to ensure "transparency"  -  the ability of other researchers to question, challenge, and validate the results of published studies. The commenter notes that this would include the journals' policies on treatment of data from research published years and even decades ago. Similarly, commenter (5180) asserts that the scientific studies published in scientific journals have already been critiqued and validated by experts familiar with the relevant scientific field as part of the scientific peer-review process.
Response: The EPA agrees with commenters that the scientific community and government agencies are making great strides in data transparency; however, improvements can still be made over existing policies and mechanisms. The requirements provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical considerations in order to ensure the Agency maintains a strong scientific basis for its decision-making. The incremental progress made possible by this rule provides an important step towards prioritizing transparency in particularly impactful EPA rules and assessments and will inform future statute-specific rulemakings.
Further, the EPA would like to clarify that the rule requirements are not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review.The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review by either journals or independent advisory committees, such as the EPA's SAB, is typically conducted without access to the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information). Should peer reviewers have access to the underlying data, they could conduct a more in-depth review that may include independently reanalyzing the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of uncertainty in the original study. Thus, peer review is not considered a replacement for the data availability requirements of this rule and implementation of this rule could provide independent peer reviewers as well as subject matter experts within the EPA and the public with the opportunity to evaluate pivotal science in ways that otherwise would not have typically occurred with existing EPA or peer review processes.
The EPA is, therefore, finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the OMB Bulletin for Peer Review and the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review. If the Agency conducts peer review on pivotal science, the EPA shall ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied in analyzing dose-response data and the implications of those assumptions for the results.
Existing Scientific Review Processes are Effective Even Without Availability of Raw Data
Joint Statement
Comment: Commenters (1306, 6110, 6122, 6876, 6909, 6915, 9227) refer to language in a Joint Statement by editors of several scientific journals and assert that effective peer review of scientific studies does not require that all underlying data be made public.
Commenters (1306, 6122, 6876, 9227) generally concur with statements from the editors, including the following:
 The merits of studies relying on data that cannot be made publicly available can still be judged.
 Reviewers can have confidential access to key data and as a core skill, scientists are trained in assessing research publications by judging the articulation and logic of the research design, the clarity of the description of the methods used for data collection and analysis, and appropriate citation of previous results.
Commenter (6110) states that the editors noted that many scientific journals already have policies to ensure transparency as much as possible and, in cases where such transparency is not possible, reviewers can be given confidential access to the raw data so that they can check and replicate the findings. The commenter asserts that the EPA proposal does not allow for such situations.
Commenter (6122) states that, because they conduct their own research and extensive review of others' research, scientists do not need the raw data to determine if a study was conducted correctly, but rather are trained to do so by examining data collection and analysis methods and to determine whether the results can be replicated. The commenter notes that when doctors prescribe a medicine that they have deemed effective, it is not because they have reviewed and analyzed the raw data of patients in the clinical trials themselves; instead, they have reviewed the published papers on the medication and have seen both that the trials were done correctly and that the results were significant enough to prescribe the treatment. The commenter states that this is no different with research scientists.
Response: The final rule does not categorically exclude studies based on the availability of the underlying dose-response data. The EPA would also like to clarify that the final rule does not require the EPA to collect, store, or publicly disseminate data and models underlying pivotal science. Further, EPA does not take the position that studies for which the underlying data are not publicly available are per se invalid. Rather, EPA acknowledges that where underlying dose-response data in pivotal science are available, subject matter experts could independently reanalyze the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of uncertainty.
Further, the EPA would like to clarify that the rule requirements are not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review by either journals or independent advisory committees, such as the EPA's SAB, is typically conducted without access to the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information). Should peer reviewers have access to the underlying data, they could conduct a more in-depth review that may include independently reanalyzing the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of uncertainty in the original study. Thus, peer review is not considered a replacement for the data availability requirements of this rule and implementation of this rule could provide independent peer reviewers as well as subject matter experts within the EPA and the public with the opportunity to evaluate pivotal science in ways that otherwise would not have typically occurred with existing EPA or peer review processes.
The EPA is, therefore, finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the OMB Bulletin for Peer Review and the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review. If the Agency conducts peer review on pivotal science, the EPA shall ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied in analyzing dose-response data and the implications of those assumptions for the results.
Study Validation is Addressed Through the Peer Review Process
Comment: Commenters (0569, 1481, 1793, 1796, 4590, 5173, 5180, 6103, 6109, 6357) assert that the design, methods, and assumptions of research studies are vitally important for validating scientific results and are typically addressed through peer review prior to publication.
Commenters (0569, 4590, 5173) state that once the results of the research study are published, the scientific community and the public at large may respond to the study and challenge the premises, methodology, analyses, and/or conclusions. Commenters (0569, 4590, 5173) state that such challenges could include other research aimed at testing the validity of the findings. Commenters (0569, 4590, 5173) state that this process is one of the long-standing fundamental checks on the validity of scientific research.
Commenter (5180) asserts that the scientific studies published in scientific journals have already been critiqued and validated by experts familiar with the relevant scientific field as part of the scientific peer-review process.
Commenter (4590) states that current scientific practice recognizes that providing publicly available data and models is desirable, where doing so is consistent with researchers' legal and ethical obligations. The commenter notes that, in contrast, the proposed EPA rule requires public availability of data and models from scientific studies as the basis of decision-making, but does not recognize adequately either the value of peer review and subsequent research in addressing validity, the balancing of privacy concerns against public disclosure, or the importance of other context in evaluating research.
Commenter (1481) asserts that the appropriate use of statistics can be assessed as accurately and more efficiently from the published methodology and statistics than from the overwhelmingly large original data sets. The commenter states that open raw data sets offer no benefit to the public, since proper statistical methodology requires field specific expertise. The commenter adds that, thus, there is no useful transparency added to the studies but the policy comes at the cost of confidentiality.
Commenter (1793) states that decision makers do not need to see raw data that went into studies in order to trust scientific evidence. The commenter notes that they can look at the methods, design, and results in order to assess the quality of the science. According to the commenter, this rule fundamentally misrepresents how science works. The commenter asserts that scientific studies already undergo intense scrutiny and peer-review from experts in the field before they are published in a scientific journal.
Commenter (4838) asserts that studies with confidential data can still be appropriately peer-reviewed through the use of confidentiality agreements, and subject to rigorous scientific scrutiny over their methods and conclusions. Commenter (1796) states that alternate methods exist to ensure that expert reviewers can evaluate the studies for methodological rigor, as well as for obtaining confidential access to data where necessary and appropriate. Commenter (4595) asserts that there are credible procedures for testing results and verifying outcomes with methodologies that do not require access to raw data.
Commenter (6103) asserts that the value of any particular scientific study should be defined by the peer review process, not by whether the raw data are accessible. The commenter states that researchers have legal and ethical obligations to keep personal information private and businesses should not have to compromise their intellectual property. The commenter states that the public already has access to the science EPA relies on and can fully access methods, summary data, results, and interpretation as can all peer reviewers. The need for access to raw data is entirely unnecessary and would cause many problems by precluding the use of critical studies.
Commenter (6109) states that, while EPA appears to suggest that in the absence of publicly available raw data, there is no mechanism to verify a study's validity, this is incorrect. The commenter states that the peer review process is rigorous and allows scientists to examine each other's work and analytical methodologies. The commenter states that, in addition, independent verification and confirmation by other studies can lend credence to studies based on confidential information.
Response: EPA does not take the position that studies for which the underlying data are not publicly available are per se invalid. Rather, EPA acknowledges that where underlying dose-response data in pivotal science are available, subject matter experts could independently reanalyze the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of uncertainty.
Further, the EPA would like to clarify that the rule requirements are not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review by either journals or independent advisory committees, such as the EPA's SAB, is typically conducted without access to the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information). Should peer reviewers have access to the underlying data, they could conduct a more in-depth review that may include independently reanalyzing the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of uncertainty in the original study. Thus, peer review is not considered a replacement for the data availability requirements of this rule and implementation of this rule could provide independent peer reviewers as well as subject matter experts within the EPA and the public with the opportunity to evaluate pivotal science in ways that otherwise would not have typically occurred with existing EPA or peer review processes.
The EPA is, therefore, finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the OMB Bulletin for Peer Review (Ref. 8), of individual studies identified as pivotal science if the studies have already undergone peer review. If the Agency conducts peer review on pivotal science, the EPA shall ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied in analyzing dose-response data and the implications of those assumptions for the results As discussed in the preamble to the final rule, the rule requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions. Such independent reanalyses will subsequently enable the EPA to make stronger, data-driven decisions in future rulemakings or in revisions to existing rules or influential scientific information. Implementation of this rule will more effectively share pivotal science for external consideration and increase the opportunity for independent validation of pivotal science by subject matter experts. As data are better understood through independent reanalysis, the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
Release of Raw Data (Including Privacy Data) is Not Required Under Existing Rigorous Peer Review Processes
Comment: Commenters (1461, 1973, 2910, 5418, 6103, 6111, 6922) contend that existing peer review processes that do not require the release of privacy data are sufficient.
Commenter (1461) states that in cases where data cannot be made available to the public, scientific journals and standards can maintain rigor by careful peer review, including allowing confidential access to key data to reviewers. Commenter (1973) asserts that the existing system has worked well and has provided the EPA with studies of mortality and hospital admissions studies from the same U.S. data by multiple investigating teams, while still maintaining data privacy. Commenter (2910) notes that pooled anonymous data has been used successfully for years to advance medical science. Commenter (5180) states that federal agencies like the EPA do not need to see the raw data that went into studies in order to trust the scientific evidence and consider it in their rulemaking.
Commenter (6111) asserts that the public health, medical, and scientific research community does not regard the public disclosure of all raw data as necessary. The commenter states that, for example, the Committee on Publication Ethics (COPE), which has over 12,100 member journals and editors covering all areas of scholarly inquiry, has established 10 core practices. The commenter provides that COPE's core practice #5 on data and reproducibility provides that "[j]ournals should include policies on data availability and encourage the use of reporting guidelines and registration of clinical trials and other study designs according to standard practice in their discipline." The commenter adds that the simplicity and generality of this core practice statement signals that the question of standards for data transparency, data access, data sharing, data peer review, and replication and reproducibility practices are far from settled. The commenter asserts that there is no one-size-fits-all approach to the critical questions of data transparency, data sharing, and reproducibility.
Commenter (5418) notes that the approach advocated in the proposed rule is inconsistent with professional best practices in their respective disciplines for conducting, reviewing, and confirming the results/findings of studies, especially those based on confidential personal health data of study participants. The commenter notes that they publish their research results in the most reliable, highest-quality, peer-reviewed medical and scientific journals, including Lancet, Nature, Science, New England Journal of Medicine, Journal of the American Medical Association, Cell, and Environmental Health Perspectives and they conduct peer reviews of the work of other researchers.
Commenters (6915, 9227) assert that, in a letter filed in this docket, the Presidents of the National Academies of Science, Engineering, and Medicine (NASEM) observe that the public availability of data is not necessary to ensure the integrity of regulatory science and is not a sufficient criterion for excluding a particular study from consideration. Commenter (9227) states the Presidents' letter notes: "The National Academies have developed a long-standing body of work that demonstrates scientific literature can be evaluated in a transparent and objective manner without complete disclosure of the underlying data." Commenter (9227) states the letter goes on to explain: "If the study data are not available, their absence may affect how the study is rated and used in the [agency's] analysis, but the study should not necessarily be eliminated from the assessment."
Commenter (6925) states that the complete public release of data is not necessary to ensure validity or transparency in science nor the regulatory process. The commenter notes that transparency has long been a requirement in science, with the understanding that the methodology is described in a manner that would allow the study to be reproduced.