Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." EPA researchers, grantees and contractors are required to make peer reviewed journal articles and the underlying data resulting from EPA-funded research accessible to the public. This final rule applies to pivotal science that was not funded by the EPA and requires the EPA to evaluate the availability of an earlier stage of data (the data that are ready to be analyzed to extract relevant information) than the stage of data required to be available under the 2016 EPA plan. As discussed more fully in the preamble of the final rule, the rule requirements provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical considerations and privacy considerations in order to ensure the Agency maintains a strong scientific basis for its decision-making. The incremental progress made possible by this rule provides an important step towards prioritizing transparency in particularly impactful EPA rules and assessments and will inform future statute-specific rulemakings.
CONSORT, ARRIVE and STROBE
Comment: Commenter (6924) suggests EPA focus on implementing existing initiatives and guidelines for improving data sharing and transparency at federal agencies. The commenter states that, through collaborative efforts, academia, medical scientists, industry and publishers have developed several protocols and guidelines specifically designed to improve reporting and evaluation of studies that will improve the quality and transparency of the interpretation of findings. The commenter adds that protocols and guidelines, such as CONSORT, Animal Research: Reporting of In Vivo Experiments (ARRIVE) and Strengthening the Reporting of Observational studies in Epidemiology (STROBE), do not require public access to all study data and will still improve the scientific basis of evaluating studies. The commenter suggests that EPA utilize these existing tools to evaluate study quality and build on this work already done by the scientific community.
Response: The EPA continues to believe that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will improve the opportunity for the public to access the EPA's scientific analyses and resulting regulatory actions in a way that is beneficial to the scientific process, the Agency's mission, and the public's health and safety. This rule is designed to build upon OMB M-19-15, which highlights the need to characterize the sensitivity of an agency's conclusions to analytic assumptions, as well as other federal guidance documents that require greater data transparency. As discussed more fully in the preamble of the final rule, the rule requirements provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical considerations and privacy considerations in order to ensure the Agency maintains a strong scientific basis for its decision-making. The incremental progress made possible by this rule provides an important step towards prioritizing transparency in particularly impactful EPA rules and assessments and will inform future statute-specific rulemakings.
NIH Guidelines
Comment: Commenter (6924) suggests that data sharing guidelines from the NIH are applicable to the sharing of final research data for research purposes. The commenter states that, as the NIH guidelines have already undergone an extensive deliberative process with input from the scientific community, EPA should reference these guidelines in its grants and/ or cooperative agreements.
Commenter (6126) suggests that EPA consider the NIH guidance for grantees about protocols and practices for data sharing. The commenter states that the policy, which has now been in effect for 15 years, reflects another process that EPA could consider as a model for developing a thoughtful and informed regulation about agency and researcher data sharing.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." EPA researchers, grantees and contractors are required to make peer reviewed journal articles and the underlying data and models resulting from EPA-funded research accessible to the public. The final rule does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science, nor does it require any researcher or other outside entity to provide data or models to the EPA or the public. Therefore, the Agency will not be developing information or protocols on data sharing.
National Center for Health Statistics Research Data Centers
Comment: Commenter (6924) suggests that providing secure access to data for research purposes also improves data sharing and is an alternative to requiring publicly available data. The commenter recommends that EPA consider the National Center for Health Statistics Research Data Centers as a model -- these centers have successfully provided access to sensitive information without breaches of privacy/ confidentiality.
Response: The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
<IP_ADDRESS>.6 Methodology and Assumptions
Comment: Commenters (1796, 6111, 6121, 6935) suggest that review of a study's scientific methodology and assumptions can be sufficient and is important when validating scientific results.
Commenter (6935) asserts that robust discussions of methodology and assumptions can provide sufficient information for stakeholders to critique and evaluate the validity of studies and models, which may be sufficient for the purposes of determining if the study can and should be relied upon by the Agency in a regulatory context. The commenter states that, accordingly, final rules should reflect that some flexibility may be appropriate if sufficient other information about studies and models is provided or otherwise available. The commenter adds that it may not be necessary for EPA to complete a full, independent validation of a study or model if sufficient information about inputs, assumptions and methodology are made available. The commenter states that such information may provide adequate information for determining whether the study can and should be relied upon in the regulatory context, without the expense, both in terms of resources and time, of independent validation. The commenter states that recognizing this in any final rule could provide the Agency and regulated entities additional flexibility.
Response: The EPA agrees that existing methods of evaluating the methods and assumptions presented in a peer-reviewed study are useful. However, when subject matter experts have access to the underlying data, they could conduct a more in-depth review that may include independently reanalyzing the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of uncertainty in the original study.
Further, although the EPA is prioritizing transparency in pivotal science, the Agency also recognizes that there will be instances where the underlying dose-response data of pivotal science is unavailable for independent validation. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, the availability of underlying dose-response data should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Though not exhaustive or exclusive, the EPA has identified several factors in 40 CFR 30.5(d) that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. Additional discussion on these factors, which include consideration of assumptions and methodology, is provided in the preamble to the final rule.
<IP_ADDRESS>.7 Replication, Reproducibility and Reanalysis Processes of Science
 Replication
Comment: Commenters (5172, 6046, 6111, 6373) suggest EPA consider replication of scientific studies by different researchers using different data sets to validate studies that may be useful for regulatory purposes. Commenters (6111, 6373) suggest EPA corroborate results of a scientific study by conducting subsequent studies/validation studies.
Commenter (1973) states that the publication of similar studies, by other investigating teams, using equivalent data from other countries, also provides the EPA with the assurance it needs that the results they rely on have been replicated, and by different people using different approaches. The commenter notes that, for example, Medicare data was first used to look at air pollution by one investigator, first working at EPA and then at Harvard. The commenter states that then other investigators sought the data and published their own studies. The commenter adds that, while these data may not be made publicly available as EPA proposes, the data are available to investigators with the will to apply for access and adhere to the data use rules. The commenter provides that, similarly, many studies, by many different teams of investigators have used mortality data from state mortality or national records to examine the acute effects of air pollution. The commenter asserts that there is no reason to risk revealing identified confidential medical information to accomplish a task that is already accomplished.
Commenter (2431) asserts that replication of results using different data, not revelation of personally identifiable health data, is the strength by which we should measure our science. The commenter states that, beyond the HEI reanalysis, the findings of Six Cities have been replicated numerous times in many independent studies using different study designs, larger numbers of enrollees, broader geographical distributions across multiple regions of the world, and increasingly sharp analytical tools. The commenter notes that the number of studies replicating Six Cities' original findings demonstrate the soundness of the scientific process and have established an extensive foundation of literature whose conclusions are just as important to the weight of evidence as Six Cities itself.
Commenters (6153, 6874) contend that true replication requires not reanalyzing existing work but independent investigators producing independent data. The commenters state that the HEI work reanalyzing an existing body of evidence  was time-consuming and expensive and that it would not be a good use of resources to require this approach for all studies contributing to the regulatory actions, or even to a few key studies.
Commenter (6876) states that the proposed rule fails to recognize the power of replication, a key criterion for defining the strength of scientific evidence. The commenter adds that replication refers to the fact that consistent findings from studies in different populations in different places strengthens the likelihood of an effect.
Commenter (6916) states that the proposal's focus on public release of the underlying data used in studies  -  particularly "dose response data" and studies, evinces a clear lack of understanding of the way science is conducted and verified. The commenter notes that, while replicating the results of a study with the same data set certainly verifies it, as the National Research Council said in 2002, "[f]or large epidemiological studies, repeating a study is seldom either possible or desirable." The commenter asserts that more effective, instead, is some combination of new studies testing the same question in different places, and/or an independent analysis with the same data.
Commenter (8801) states that systematic replication by different researchers using different techniques collecting different data sets, yet addressing a similar fundamental health question, provides the strongest possible replication method.
Commenter 6139) states that EPA's transparency proposal is based on the inappropriate grafting of transparency rules appropriate for clinical research onto the much different world of human health environmental research on which regulation is often based. The commenter notes that environmental health studies in communities cannot and should not be held to the same gold standard for clinical research as a double-blind drug and placebo study. The commenter states that, instead we depend upon replication through consistency in the findings when investigated in other populations by different authors using different methodologies, and on biological plausibility in the form of toxicological studies.
Response: Although the EPA agrees with commenters that meaningful insights can be obtained through similar studies in different media or context, the EPA continues to find that independent validation of the study findings and conclusions driving the EPA's dose-response assessments would provide important information. As detailed in the preamble of the final rule, there is scientific support for the usefulness of reanalyzing data, and the EPA finds this to be especially true for data that drive the quantitative requirements or analyses of EPA significant regulatory actions or influential scientific information. Implementation of this rule will more effectively share pivotal science for external consideration and, thus, increase the opportunity for independent subject matter experts to validate pivotal science, and as the dose-response data are better understood through independent reanalysis the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
The EPA also wishes to clarify reanalysis is not required under this rule. The rule requires the EPA to give greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Reproducibility
Comment: Commenter (0040) states that the whole scientific and statistical process of drawing conclusions is based on the idea of reproducibility. The commenter notes that this duplication of research occurs a lot, especially if findings are unexpected, controversial, or groundbreaking. According to the commenter, if results are not duplicated, then the theories are rejected. According to the commenter, only after initial studies are replicated and extended upon, a body of evidence grows and the public policy shifts accordingly.
Commenter (6920) asserts that reproducibility and independent validation are critical aspects of the scientific method and have resulted in significant advancements in our understanding of the health effects of air pollutants at different exposures and thresholds.
Response: In the final rule the terms replicate and reproduce (or other forms of the word) are no longer used. In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze." In addition, the EPA is modifying the definition of "independent validation" in the final rule by replacing "capable of being substantially reproduced" with "produced." The EPA will not finalize the proposed 40 CFR 30.2 definition of "capable of being substantially reproduced" because the term is not used in the final rule's definition of "independent validation" or elsewhere in 40 CFR 30.
Reanalysis
Comment: Commenters (5172, 6046, 6111, 9227) suggest EPA consider reanalysis of scientific studies to validate studies that may be useful for regulatory purposes.
Commenter (6135) requests that EPA recognize that reanalysis of individual studies is not a replacement for replication of study findings in multiple distinct health studies. The commenter suggests that except in the case of exceptional circumstances, EPA follow the example of past EPA administrators in waiting for additional studies to be completed before taking regulatory action based on a single study finding.
Commenters (5017, 6111, 6133, 6137, 6351, 6373, 6916, 9227) assert that scientific results and methods can be questioned and validated without exposing protected underlying data to public scrutiny. As an example, the commenters point to the series of studies, re-analyses, verifications, and extensions of work begun in 1974 on the prospective "respiratory health effects of respirable particles and sulfur oxides on a sample of adults and children in six U.S. cities."
Commenter (6351) suggests that, instead of blocking studies, EPA refer studies to an independent third party such as the HEI. The commenter states that autonomous reviewers examined the data from the 1993 Harvard Six Cities Study and the 1995 ACS Study and developed a report on the two studies that confirmed their original findings. The commenter notes that since these studies, other research has confirmed their findings as well, including some studies that used publicly available datasets. The commenter states that similar third-party reviews could readily address concerns about existing or future studies as needed.
Similarly, commenter (9227) states that an important effort began with Harvard's "Six Cities" study, reported in (Dockery et al., 1993). The commenter explains that the researchers initially sought to reproduce their initial findings using a data base with a much larger number of subjects and cities and did indeed reproduce those findings (Pope et al., 1995). According to the commenter, by 2009 enough new evidence had accumulated for EPA's integrated assessment for particulate matter to conclude that the number of large U.S. cohort studies, together with supporting evidence from other epidemiology and toxicological studies were sufficient to infer a causal relationship between long-term PM2.5 exposures and mortality and cardiovascular effects. The commenter states that this conclusion regarding causality (the strongest finding possible under the causality classification methodology based on these studies was endorsed by the external CASAC, which noted: "The five-level classification of strength of evidence for causal inference has been systematically applied; this approach has provided transparency and a clear statement of the level of confidence with regard to causation, and we recommend its continued use in future ISAs." The commenter provides that the link between particulate matter exposure and mortality that was observed in the Six Cities study has been vetted through multiple mechanisms that have confirmed the validity of the findings without public access to the underlying data -- including extensive reanalysis using larger datasets with longer duration of follow up and different statistical methods; reproduction and corroboration with independent studies using distinct populations and methodologies; and rigorous external review by independent scientists.
Commenters (6111, 6137, 9227) refer to comments of a SAB workgroup which assert that removing confidential information so that it can be publicly disclosed is unnecessary, as studies can be validated without demanding access to confidential data. Commenters (6111, 9227) state that, as explained by the SAB, the HEI's reanalysis of the Six Cities Study, through "an unusually rigorous form of peer review and independent reanalysis, coupled with many follow-up studies, has accomplished a measure of confidence in findings without public access to data and analytic methods." Commenters (6137, 9227) note the SAB workgroup also states:
      The proposed rule fails to mention that there are various ways to assess the validity of prior epidemiologic studies without public access to data and analytic methods. For example, the HEI conducted a re-analysis of the influential Harvard Six Cities and ACS epidemiologic studies and was able to replicate its findings and to assess the robustness of the findings via sensitivity analysis.
Response: In the final rule the terms replicate and reproduce (or other forms of the word) are no longer used. In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze." In addition, the EPA is modifying the definition of "independent validation" in the final rule by replacing "capable of being substantially reproduced" with "produced." The EPA will not finalize the proposed 40 CFR 30.2 definition of "capable of being substantially reproduced" because the term is not used in the final rule's definition of "independent validation" or elsewhere in 40 CFR 30.
The EPA agrees with commenters on the utility of findings from reanalysis studies. Indeed, one criterion for the Administrator's consideration of an exemption under 40 CFR 30.7 is that a third-party has conducted independent validation of the study's underlying dose-response data through reanalysis.
Finally, although the EPA agrees with commenters that meaningful insights can be obtained through similar studies in different media or context, the EPA continues to find that independent validation of the study findings and conclusions driving the EPA's dose-response assessments would provide important information. As detailed in the preamble of the final rule, there is scientific support for the usefulness of reanalyzing data, and the EPA finds this to be especially true for data that drive the quantitative requirements or analyses of EPA significant regulatory actions or influential scientific information. Implementation of this rule will more effectively share pivotal science for external consideration and, thus, increase the opportunity for independent subject matter experts to validate pivotal science, and as the dose-response data are better understood through independent reanalysis the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
<IP_ADDRESS>.8 Systematic Review
Comment: Commenters (4831, 6125, 6158, 6915) suggest that EPA consider adoption of systematic-review methods. Commenter (6121) suggests that multiple steps of independent scientific review be conducted to ensure the integrity and objectivity of science.
Commenter (4831) notes that the National Academies have carried out numerous studies that advise EPA on the scientific bases of regulatory decisions related to human health and the environment. The commenter states that these reports encourage EPA to consider all available science in the rule-making process and provide guidance about how the agency could be more transparent in describing how evidence is gathered and evaluated. The commenter suggests that systematic-review methods be adopted to ensure objectivity, rigor, and transparency in performing literature-based reviews (IOM, 2011; NRC, 2011, 2014). The commenter provides that NASEM, 2017a, includes four case examples of systematic reviews.
Commenter (6125) asserts that the proposal ignores the existence of "systematic review" methods for review of evidence that are being developed within the agency, most notably within the IRIS program, prompted by criticisms and recommendations from the NAS. The commenter states that the development of such methods is informed by the National Toxicology Program's Office of Health Assessment and Translation (NTP/OHAT) guidance and other similar tools (UCSF's Navigation Guide). According to the commenter, no authoritative body of experts has ever recommended requiring "raw data" in order to perform or review dose response assessments.
Commenter (6915) states that recently a rigorous systematic review process has been developed and implemented by EPA's IRIS program to ensure even greater thoroughness and objectivity in hazard identification. The commenter asserts that, in this way, EPA already ensures that the studies and data upon which it relies are valid.
Commenter (6158) asserts that systematic review methods are one well-regarded means to develop comprehensive criteria to evaluate and "rank" different scientific studies. The commenter states that data availability is only one criterion considered and the absence of public data is not a worthy justification for the exclusion of any given study. The commenter states that study design, analytical techniques used, and statistical methods applied are all deemed to be more important criteria by which to evaluate the strength and utility of a scientific study.
Response: When evaluating potential links between exposure to a pollutant, contaminant, or substance and effects and the nature of the dose-response relationship, the EPA will follow best practices, which may include systematic review, and rely on the highest quality, most relevant studies in determining the potential for hazard due to exposure to a pollutant, contaminant, or substance. Where there is convincing and well-substantiated evidence (consistent with Agency guidelines on hazard identification and dose-response assessment) to support a relationship between exposure and effect, the EPA will identify a subset of those studies for use in characterizing the quantitative relationship between the amount of dose or exposure to a pollutant, contaminant, or substance and an effect. This will be based on the exposure situation being addressed, the quality of the studies, the reporting adequacy, and the relevance of the endpoints. From that subset, the specific dose-response studies or analyses that drive the requirements, quantitative analyses, or both will be identified as pivotal science.
Once the EPA has identified pivotal science -- for either significant regulatory actions or influential scientific information -- the EPA will then evaluate if the underlying dose-response data are available in a manner sufficient for independent validation. The EPA will give greater consideration to pivotal science for which the underlying dose-response data are either publicly available in a manner sufficient for independent validation or, in the case of PII, CBI, or proprietary data, available through restricted access that affords privacy in a manner sufficient for independent validation. The EPA may still use pivotal science when the underlying data are unavailable after either giving it lesser consideration or receiving an exemption from the requirements of this rule from the Administrator. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. Because the final rule does not categorically exclude any studies, including human health studies, from EPA's consideration, the Agency will not be prevented from evaluating and considering all quality, relevant studies when promulgating significant regulatory actions and developing influential scientific information. Further, the final rule only applies to dose-response data; therefore, all quality, relevant studies will continue to inform other aspects of the EPA's evaluations
Where underlying dose-response data in pivotal science are available, subject matter experts could independently reanalyze the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of the uncertainty in the assessment. Such independent reanalyses will subsequently enable the EPA to make stronger, data-driven decisions in future rulemakings or in revisions to existing rules or influential scientific information.
<IP_ADDRESS>.9 Transparency and Openness Promotion (TOP)
Comment: Commenters (1306, 5120, 6119, 6360) suggest that EPA consider the TOP standards and assert that the scientific community already supports TOP Standards which are able to accommodate research using strict confidentiality and private guidelines. Commenter (1306) states that these standards recognize the array of workflows across scientific fields and make the case for data sharing at different levels of stringency; and in cases where all data cannot be fully shared.
Commenter (6360) recommends that EPA draw from (or adopt) existing transparency screening tools developed by consensus in the scientific community to further define the scope and applicability of the rule's requirements. The commenter states that one important, open source system, the TOP guidelines, have been recently adopted by over 5,000 major scientific organizations and journals. The commenter notes that, published in Science in 2015, the TOP guidelines include eight standards, each with three levels of increasing stringency. The commenter provides that claims made in a journal requiring Level III disclosure are more likely to be recognized as scientific facts than those published to Level I standards. The commenter states that, since TOP and other systems are widely adopted now and are growing in their reach, scientists understand their obligations under these systems if they want to be published in certain journals. According to the commenter, if they must turn in their data to be published in Science, for example, turning it over to EPA for public review is not a significant additional burden.
Commenter (6360) recommends that EPA require TOP Level III transparency for all significant regulatory actions and all other EPA policy decisions have at least Level II standards for transparency to the public. The commenter states that, by "significant regulatory actions," they mean "regulatory actions" as defined in Executive Order 13771 and "significant" as defined in Executive Order 12866, OMB's Good Guidance bulletin, and other relevant guidance regarding regulatory planning and review. The commenter adds that, if researchers must comply with Level III procedures to receive one publication in Science, it seems reasonable to require the same level of disclosure for EPA decisions with millions (or billions) of dollars of annual economic impact. The commenter notes that, as EPA takes comment on a proposed decision and notes that the underlying judgements in the applied science meet Level II criteria, the public could evaluate the claims and provide valuable feedback to EPA regarding the appropriate rigor of its data transparency procedures. The commenter adds that, based on these comments, EPA could decide that the final agency action requires Level III transparency. According to the commenter, the Level II and Level III requirements mirror somewhat the applicability of the proposed standards in EPA's rulemaking. The commenter suggests that the tiered levels and specific requirements in TOP offer an existing, widely used approach for EPA to consider that would allow it to prioritize its resources and its requirements with respect to applied science developers. The commenter states that EPA can calibrate the transparency requirements of particular science applications with the magnitude of the policy decision in question.
Commenter (6360) states that the TOP criteria require researchers to post data, analytic methods, and research materials to a trusted depository. The commenter states that, for EPA applications of science, this depository should be the public docket of a proposed rule or the accompanying administrative record associated with a proposed decision. The commenter recommends that, for sensitive information, EPA adopt the depository standards in common use by other federal agencies, journals, and EPA itself. According to the commenter, for example, the Census Department has established stringent criteria for researchers seeking to review data that could allow individual responses to be identified. The commenter notes that similar procedures are in place to protect data from new drug trials in journal depositories. The commenter adds that EPA has procedures for its contractors to review CBI  -  procedures that could be modified for non-government public commenters to use to review applied science that relies on sensitive data.
Response: This final rule does not require the EPA to disclose or host data, but to determine if dose-response data are available and to give greater consideration to those studies for which such data are publicly available or available through restricted access. Restricted access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information. The EPA agrees that the scientific community and government agencies are making great strides in data transparency; however, improvements can still be made over existing policies and mechanisms. As discussed more fully in the preamble of the final rule, the rule requirements provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical and privacy considerations in order to ensure the Agency maintains a strong scientific basis for its decision-making. The incremental progress made possible by this rule provides an important step towards prioritizing transparency in particularly impactful EPA rules and assessments and will inform future statute-specific rulemakings.
<IP_ADDRESS>.10 Other Available Approaches
Comment: Commenters suggest EPA consider other available approaches that can ensure the validity of scientific studies that may be useful for regulatory purposes, including the following:
 Collaborative studies, where the original study authors employ data use agreements to share the data with other researchers, who can validate and expand upon the analysis. (6925)
 Comparisons of research findings with the results of other peer-reviewed research efforts, including through meta-analyses and literature reviews that are designed to shed light on consistent findings across studies. (9227)
 Competitive grant process and grant application review. (6121, 6373)
 Confidential sharing of data with independent research teams that are able to validate results. (9227)
 Data-use agreement application review. (6373)
 Detailed description of research methods, code and non-confidential data in their published articles. (6111)
 Expertise and judgment of career EPA scientists when considering the strength and relevance of studies included in EPA decisions. (6121)
 Federal regulations that include review requirements for research involving human subjects. (6119)
 IRB requirements. (6121, 6373)
 Risk assessment drafts are made public and can be commented on. (6046)
 Single research questions require multiple trials. (6373)
 Solicitation of expert opinion. (6153, 6358)
 Statutory requirements. (6373)
Response: Although the EPA agrees with commenters that meaningful insights can be obtained through similar studies in different media or context, the EPA continues to find that independent validation of the study findings and conclusions driving the EPA's dose-response assessments would provide important information. As detailed in the preamble of the final rule, there is scientific support for the usefulness of reanalyzing data, and the EPA finds this to be especially true for data that drive the quantitative requirements or analyses of EPA significant regulatory actions or influential scientific information. . Implementation of this rule will more effectively share pivotal science for external consideration and, thus, increase the opportunity for independent subject matter experts to validate pivotal science, and as the dose-response data are better understood through independent reanalysis the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
Further, although the EPA is prioritizing transparency in pivotal science, the Agency also recognizes that there will be instances where the underlying dose-response data of pivotal science is unavailable for independent validation. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, the availability of underlying dose-response data should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Though not exhaustive or exclusive, the EPA has identified several factors in 40 CFR 30.5(d) that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. Additional discussion on these factors is provided in the preamble to the final rule.
Finally, the EPA would like to clarify that the approach in the final rule gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access.
Comment: Commenter (6357) states that data from clinical trials are often not publicly available but are made available for confidential safety-monitoring and peer-review. The commenter states that policies for in-depth federal agency review also exist, as exemplified by the Food and Drug Administration's (FDA's) ability to audit clinical trials.
Response: The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Comment: Commenter (6357) states that the validity of scientific evidence is based not only on individual studies, but also on a collection of multiple sources of data that may confirm or refute individual findings. The commenter encourages EPA to consider multiple legitimate paths for validation of scientific findings to ensure that the science most relevant to the protection of human health is included in the decision-making process.
Response: Although the EPA agrees with commenters that meaningful insights can be obtained through similar studies in different media or context, the EPA continues to find that independent validation of the study findings and conclusions driving the EPA's dose-response assessments would provide important information. As detailed in the preamble of the final rule, there is scientific support for the usefulness of reanalyzing data, and the EPA finds this to be especially true for data that drive the quantitative requirements or analyses of EPA significant regulatory actions or influential scientific information. Implementation of this rule will more effectively share pivotal science for external consideration and, thus, increase the opportunity for independent subject matter experts to validate pivotal science, and as the dose-response data are better understood through independent reanalysis the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
Further, although the EPA is prioritizing transparency in pivotal science, the Agency also recognizes that there will be instances where the underlying dose-response data of pivotal science is unavailable for independent validation. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, the availability of underlying dose-response data should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Though not exhaustive or exclusive, the EPA has identified several factors in 40 CFR 30.5(d) that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. Additional discussion on these factors is provided in the preamble to the final rule.
Comment: Commenter (6941) states that the TSCA identifies independent verification as one of five scientific standards that EPA must consider (also the intended use of the information, its relevance for use in making regulatory decisions, the clarity and completeness of its documentation, and the extent to which variability and uncertainty are evaluated and characterized). The commenter notes that even when underlying data is not publicly available in a manner sufficient for independent verification, EPA's use of such information, methods, or models can be consistent with the best available science if these other standards are met.
Response: This final rule does not interpret or apply the provisions of any environmental statutes; such efforts will occur in the subsequent actions under the relevant statutes described in the preamble of this rule. As this rule makes clear, if implementing this procedural rule would result in conflicts with existing environmental statutes, and their implementing regulations, this rule will yield to the EPA statutes and regulations.
<IP_ADDRESS> Overall Comparison to Existing Transparency Mechanisms
 <IP_ADDRESS>.1 Are Existing Mechanisms Adequate?
Comment: Commenters (1905, 2472, 5170, 6135, 6152, 6194, 6446, 6894, 6915, 6916, 6920, 9227) express concern that the rule fails to analyze whether existing mechanisms for vetting science without publicly releasing subjects' personal data are adequate. Comments include:
 The proposed rule does not make a persuasive case for replacing the well-established and successful processes for determining scientific validity. The commenter asserts that, while there have been cases of intellectual fraud or non-replicable results, it is the scientific process and the scientific community itself that has discovered and corrected these problems. (5170)
 The proposed rule does not identify any deficiencies in the long-established approach of using scientific studies to inform agency decisions. (6135)
 The proposal does not justify the need for a change to its current transparency or reproducibility practices that include peer review and existing access to the science that EPA relies upon. (6152)
 EPA has not explained why existing federal and professional rules, standards, and policies, many of which promote the objective of transparency, cannot be used to attain the purported objective of the proposal. The commenter adds that the proposal also ignores, or presents no argument for rejecting, scientific practices embodied in the host of nongovernmental methods and best practices established and adhered to by the research community to ensure the transparency, objectivity, and validity of studies, analyses, models, and reports, including by reproduction and/or replication where appropriate. (6194)
 Commenter asserts that the lack of any explanation for a need for the proposed rule is doubtless because procedures are already in place that assure use of the best available science for EPA's regulatory activity. The commenter states that these procedures include that much of the scientific research considered in the regulatory context, and all research published in reputable journals, has already been subject to extensive peer review. (6446)
 Commenter contends that there are significant and viable and obvious alternatives to this proposal which the Agency has failed adequately to consider and explain why it believes such alternatives are not sufficient. (6916)
 While the proposed rule cites the need for increased transparency, it provides little evidence that this need is not being met. The commenter states that EPA models, for example, that serve as the basis for predicting cancer risk and blood lead levels in children are well presented, explained and supported. The commenter asserts that debate continues on the accuracy of such models but there is little debate that they are transparent. (6920)
 Commenter asserts that EPA fails to explain why EPA's existing mechanisms are inadequate to ensure the scientific integrity of its actions. The commenter states that the proposal ignores both the available approaches embraced by the scientific community and the record of past EPA assessments, which reveal alternative methods for ensuring the credibility of potentially useful scientific studies. The commenter notes that the proposal says virtually nothing about the level of confidence those mechanisms afford in EPA's regulatory science. (9227)
Response: The EPA continues to believe that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will improve the opportunity for the public to access the EPA's scientific analyses and resulting regulatory actions in a way that is beneficial to the scientific process, the Agency's mission, and the public's health and safety. This rule is designed to build upon OMB M-19-15, which highlights the need to characterize the sensitivity of an agency's conclusions to analytic assumptions, as well as other federal guidance documents that require greater data transparency. The EPA's attention to data transparency is also responsive to the broader interest in greater data and model transparency observed in the numerous transparency initiatives in the scientific community and federal government, as well as the criticism the EPA has received from members of the public, scientific community, and Congress on the transparency of the scientific basis for EPA's decisions in previous influential scientific information assessments and regulatory actions. The EPA's continued progress toward maximizing transparency is vital to building and maintaining trust with the public and credibility in the Agency's decisions. The EPA also agrees with commenters that the scientific community and government agencies are making great strides in data transparency; however, improvements can still be made over existing policies and mechanisms.
Comment: Commenter (9227) states that the proposal fails to offer a reasoned explanation for its departure from EPA's Scientific Integrity Policy that broadly requires the agency to consider all available scientific information when undertaking rulemakings. The commenter provides that EPA's own existing Scientific Integrity Policy states:
      To support a culture of scientific integrity within the Agency, this policy. . . [r]ecognizes . . . policy makers within the Agency weigh the best available science, along with additional factors such as practicality, economics, and societal impact, when making policy decisions.
The commenter adds that the proposal conflicts with this policy by restricting what may be the best available science on a given topic from EPA's consideration solely because the underlying data cannot be made public. The commenter contends that public availability of data is neither necessary nor sufficient to ensure that studies constitute "best available science."
Commenter (9227) states that the proposal fails to offer a reasoned explanation for its departure from EPA's which require EPA to ensure the objectivity of influential scientific information it disseminates by using "the best available science and supporting studies conducted in accordance with sound and objective scientific practices." The commenter provides that EPA considers information to be disseminated when EPA prepares and distributes information to support an Agency decision or regulation or when EPA distributes information in a way that suggests EPA agrees with it, that it supports EPA's viewpoint, or if in the distribution EPA proposes to use it to support or formulate a regulation or agency decision. According to the commenter, thus, the proposal conflicts with the Guidelines by restricting scientific studies that EPA may use to support regulations, which may cause it to disseminate other information to support its regulations that is not based on the best available science.
Commenter (9227) expresses that the proposal fails to offer a reasoned explanation for its departure from EPA's Peer Review Handbook which acknowledges that "EPA strives to ensure that the scientific and technical bases of its decisions meet two important criteria: (1) they are based upon the best current knowledge from science, engineering, and other domains of technical expertise; and (2) they are credible." The commenter notes that EPA's Science Policy Council Handbook on Risk Characterization also requires reasonableness in the agency's risk assessments, which is achieved when "the characterization is based on the best available scientific information." The commenter asserts that these policies clearly impact EPA's regulatory actions, and thus will be impacted by the proposal. According to the commenter, EPA completely fails to analyze the impact the proposal will have on its ability to comply with these policies and fails to explain why it is changing course or justify its decision to do so. The commenter contends that the proposal fails to even acknowledge that the agency is changing positions.
Response: In the final rule, the EPA will give greater consideration to pivotal science for which the underlying dose-response data are either publicly available in a manner sufficient for independent validation or, in the case of PII, CBI, or proprietary data, available through restricted access that affords privacy in a manner sufficient for independent validation. The EPA may still use pivotal science when the underlying data are unavailable after either giving it lesser consideration or receiving an exemption from the requirements of this rule from the Administrator. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. Because the final rule does not categorically exclude any studies from EPA's consideration, the Agency will not be prevented from evaluating and considering all quality, relevant studies when promulgating significant regulatory actions and developing influential scientific information. Further, the final rule only applies to dose-response data; therefore, all quality, relevant studies will continue to inform other aspects of the EPA's evaluations. The EPA continues to believe that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will improve the opportunity for the public to access the EPA's scientific analyses and resulting regulatory actions in a way that is beneficial to the scientific process, the Agency's mission, and the public's health and safety.
Comment: Commenter (6915) states that a fundamental premise of the proposed rule is that only studies for which the underlying data are publicly available are valid for decision-making. The commenter notes that this premise is inconsistent with generally accepted practices for conducting and evaluating scientific research. The commenter asserts that the rationale for the premise is not provided:
      EPA presents no evidence for the conclusion that its current criteria for selecting studies result in scientifically invalid conclusions or overly stringent regulations.
The commenter adds that the D.C. Circuit has already rejected EPA's proposed approach of excluding studies relying on non-public data as "impractical and unnecessary" when raised by a trade association as part of a challenge to an air quality standard. Am. Trucking, 283 F.3d at 372.
Response: The purpose of this action is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. The rule does not categorically exclude studies -- even studies where the underlying dose-response data are not available for independent validation and only applies to dose-response data. The rule requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions. Where underlying dose-response data in pivotal science are available, subject matter experts could independently reanalyze the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of the uncertainty. Such independent reanalyses will subsequently enable the EPA to make stronger, data-driven decisions in future rulemakings or in revisions to existing rules or influential scientific information. This could occur through standard cyclical reviews (e.g., revisions to NAAQS, risk and technology reviews, national primary drinking water regulations), ad hoc revisions, or revisions through the information quality guidelines or other petition processes. Implementation of this rule will more effectively share pivotal science for external consideration and increase the opportunity for independent validation of pivotal science by subject matter experts. As data are better understood through independent reanalysis, the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
<IP_ADDRESS>.2 How Would the Proposal Remedy Any Existing Deficiencies?
Comment: Commenters (6125, 6135, 6152, 6194, 6446, 6894, 9227) express concern that the proposed rule does not provide any rationale for how the proposed rule, if promulgated, would remedy any specific deficiencies. Comments include:
 Both the discussion of purportedly desirable attributes in studies to be used for assessment and how the agency should use alternative low dose models are far less detailed and nuanced than either EPA's voluminous existing guidance on risk assessments or recommendations on these topics contained in several NAS reports relating to risk assessment, including the 2007 "Silver Book." According to the commenter, it is impossible to tell from reading the proposal what the real problems, if any, might be, nor how commenters should respond to EPA's proposed fixes. In short, the proposal's description of the "fix" is as inadequate and off the mark as its description of the "problem." (6125)
 EPA has not pointed to a single regulatory action over the thousands it has implemented that was based on faulty science and that this proposal would remedy. The commenter states that, historically, EPA has stated that a review of raw data would be warranted "[o]nly in extreme cases -- for example where there are credible allegations of fraud, abuse or misconduct." The commenter notes that EPA has advocated this position in court proceedings. (6194)
 EPA does not explain why the proposal is preferable to existing recommendations and methods vetted and supported by the scientific community, which endeavor to promote transparency through open science policies and secure data-sharing systems. (6194)
 The proposed rule advances no reason why the accepted methods used in the scientific community to evaluate scientific research should be replaced with the single, threshold requirement in the proposed rule. (6446)
 The proposed rule seems not to trust the present process but offers no alternative or specific critique. (6894)
 EPA's proposal does not question whether this rule would add anything to the current efforts, or whether it would have any effect whatsoever in increasing public accessibility of data. (9227)
Response: The purpose of this action is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions. Where underlying dose-response data in pivotal science are available, subject matter experts could independently reanalyze the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of the uncertainty. Such independent reanalyses will subsequently enable the EPA to make stronger, data-driven decisions in future rulemakings or in revisions to existing rules or influential scientific information. This could occur through standard cyclical reviews (e.g., revisions to NAAQS, risk and technology reviews, national primary drinking water regulations), ad hoc revisions, or revisions through the information quality guidelines or other petition processes.
Implementation of this rule will more effectively share pivotal science for external consideration and increase the opportunity for independent validation of pivotal science by subject matter experts. As data are better understood through independent reanalysis, the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information. The EPA's attention to data transparency is consistent with open data initiatives in the federal government and responsive to the broader interest in greater data and model transparency observed in the numerous transparency initiatives in the scientific community and federal government, as well as the criticism the EPA has received from members of the public, scientific community, and Congress on the transparency of the scientific basis for EPA's decisions in previous influential scientific information assessments and regulatory actions. The EPA's continued progress toward maximizing transparency is vital to building and maintaining trust with the public and credibility in the Agency's decisions.
<IP_ADDRESS>.3 Existing Transparency Efforts
Comment: Commenters (2492, 6188, 9227) express concern that the proposal does not address the existing steps being taken by the scientific community to make underlying data publicly available where feasible.
Commenter (2492) states that there are now many effective protocols and technologies for making human subjects research highly transparent while protecting private data. The commenter states that EPA should consult with scientists and scientific institutions to learn more about the current best practices in promoting transparent research.
Commenter (9227) states that there are already various ongoing initiatives to make scientific data and models more commonly publicly available, including those noted by EPA. The commenter also refers to the ongoing implementation of the 2016 Plan to Increase Access to Results of EPA-Funded Scientific Research. The commenter states that this Plan aims to maximize access to "research data underlying a publication" resulting from EPA-funded research. The commenter states that it is worth emphasizing the Plan also exempts "research data [that] cannot be released due to one or more of constraints, such as requirements to protect confidentiality, personal privacy, proprietary interest, or property rights." The commenter states that the Plan also explicitly indicates that:
      [i]t is important to recognize that some research data cannot be made fully available to the public but instead may need to be made available in more limited ways, e.g., establishing data use agreements with researchers that respect necessary protections. Whether research data are fully available to the public or available to researchers through other means does not affect the validity of the scientific conclusions from peer reviewed research publications.
Response: The EPA agrees with commenters that the scientific community and government agencies are making great strides in data transparency; however, improvements can still be made over existing policies and mechanisms. The rule requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions. Such independent reanalyses will subsequently enable the EPA to make stronger, data-driven decisions in future rulemakings or in revisions to existing rules or influential scientific information. The EPA's continued progress toward maximizing transparency is vital to building and maintaining trust with the public and credibility in the Agency's decisions.
<IP_ADDRESS> Parallel Scientific Process at Odds with the Current Process
Comment: Commenter (1409) argues that this regulation reduces the methods of scientific research--often peer-reviewed research that has stood the test of time--and thus would diminish the EPA's understanding of science and its knowledge in making regulations.
Commenter (6125) states that the proposed rule opens the door to EPA's scientific practices being determined by politically appointed regulators, and not scientists. The commenter asserts that, by contrast, EPA's current science policy guideline-based approach provides the agency with the flexibility to move forward to incorporate new science information in a timely manner, whether it is developed by industry, the academic community, or the agency itself.
Commenter (6125) asserts that the proposed rule approach is a breach of the fundamental principle of separating risk assessment from risk management, as championed in the 1983 NAS report.
Commenter (6924) suggests that risk assessment and other scientific methods utilized by the agency should not be the subject of rulemaking, which would freeze those methods in time and prevent EPA from staying abreast of rapidly advancing scientific methods and thus access to the best available science.
Response: As more fully discussed in the preamble of the final rule, the EPA disagrees that this rule is politically motivated, as transparency assumes no political ideology, nor would this rule result in decreased human health or environmental protections, as the benefits of greater data transparency and the significance of reanalyzing and validating study results are well-documented in scientific literature. The EPA believes that data transparency is vital for individuals who have not contributed to the study to be able to verify the quality and strength of published studies and that the opportunity to independently validate the pivotal science that the EPA relies upon is important in furthering scientific understanding and the Agency's mission. When data are widely available, researchers can validate research results and help identify and correct unintended errors, as well as reanalyze the data for new and different purposes, examine novel questions, provide new scientific insights, and improve model development. The EPA continues to believe that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will improve the opportunity for the public to access the EPA's scientific analyses and resulting regulatory actions in a way that is beneficial to the scientific process, the Agency's mission, and the public's health and safety.
Comment: Commenter (6133) asserts that EPA, other federal agencies, EPA scientific advisory bodies, the NAS, and EPA's SAB have repeatedly and consistently relied upon the best available, peer-reviewed, independent, credible scientific studies -- for which the underlying data are not publicly available -- and found that science to be valid, reliable, trustworthy, and a reflection of the "best available science" that EPA claims as its concern in the proposal. The commenter states that the proposal arbitrarily excludes prior research, studies, and data that do not meet its applicability criteria based on concerns that were never announced to researchers or the public, or deemed necessary by any government agency, at the time the research, studies, or data-gathering were undertaken. The commenter asserts that the proposed rule is strikingly at odds with those scientific practices and their history, with nothing in the rulemaking docket to support casting aspersions on the practices or history sufficient to prohibit EPA from considering such science.
Response: Because the final rule does not categorically exclude any studies, including human health studies, from EPA's consideration, the Agency will not be prevented from evaluating and considering all quality, relevant studies when promulgating significant regulatory actions and developing influential scientific information. Further, the final rule only applies to dose-response data; therefore, all quality, relevant studies will continue to inform other aspects of the EPA's evaluations.
13.2 SNPRM Independent Peer Review
13.2.1 Existing Peer Review Processes and Protocols Are Sufficient
Comment: Commenters (10748, 10866, 11193, 11330, 11332, 11350, 11358, 11381, 11400, 11403, 11404, 11495, 11479, 11892, 11920, 12719, 12727, 13788, 14397) point out that there are already existing peer review methods in place and/or that the EPA already employs a number of methods for evaluating the quality of studies it uses in rulemaking and assuring that the Agency relies on the best available science; these include formal peer review, review by the SAB and other advisory committees and external bodies, systematic frameworks that govern the Agency's review and evaluation of scientific studies that inform its work, Data Quality Objective (DQO) requirements and other stipulations for grantees, quality assurance plans, and the public comment process. Commenters (11330, 12727, 13788) argue that EPA has failed to justify why these mechanisms are insufficient and/or are failing. Commenter (12727) adds that the EPA has also not sufficiently explained how the proposed rule would "improve transparency and the scientific integrity of the regulatory outcomes in an effective and efficient manner."
Commenter (10748) states that peer review processes are already in place as all published scientific studies must undergo a rigorous peer-review process to ensure sound science and the government should let scientific journals and experts continue to decide the quality of studies.
Commenter (11193) states that the scientific community has long established rules for evaluating science using processes like independent peer review, independent advisory boards, and institutions like the National Academies to review issues of scientific importance. The commenter asserts that these independent peer review methods have, over the decades, led to the greatest scientific advancements in the history of the world and notes that this has happened largely due to the efficacy and thoroughness with which this process evaluates all the available science, including the participation of agency, academic, and industry scientists meeting and using a thorough process for review.
Commenter (12719) states that publications in peer reviewed journals have had their protocols, data, and statistical analyses assessed during the peer review process leading up to publication. The commenter states that the EPA's current process for reviewing such data already allows for stakeholders to challenge the use of the studies in decisions without making the data publicly available for "independent validation."
Commenter (12727) states that EPA has developed a well-earned reputation for grounding its decisions in rigorous science over approximately five decades of operation spanning administrations of both parties. The commenter notes that this reputation has not come by accident -- rather, it has resulted from deliberate efforts by EPA to utilize time-tested tools including formal peer review; close scrutiny of agency policies, decisions, and studies by the agency's advisory committees and external bodies such as the National Academies; and systematic frameworks that govern the agency's review and evaluation of the scientific studies that inform its work (such as the Preamble to the Integrated Science Assessment and systematic review protocols accompanying IRIS toxicological reviews). The commenter adds that on top of these well-established mechanisms, the ability of the public to comment on the Agency's use of science -- and seek judicial review of arbitrary actions -- constitutes a further safeguard to ensure the Agency acts on the basis of the best available science. The commenter states that, like the original proposal, the Supplemental Notice fails to explain why these existing mechanisms are insufficient -- or to address other information that has emerged since the publication of the original proposal that underscores this basic flaw in the rule. The commenter notes that EPA's own SAB issued a final report on this proposal in April 2020 that sharply criticized EPA's failure to consider the effectiveness of existing transparency mechanisms -- and warned that the proposal could "reduce scientific integrity" and "decrease efficiency."
Commenter (11330) states that EPA presents no evidence at all that peer review, a system that has literally built American scientific might, is failing. In fact, the commenter notes that only two out of ten thousand papers are retracted in the United States. The commenter believes the system is strong, fair, and leads to positive scientific and public health outcomes.
Commenter (11350) states that EPA science assessments generally include an exhaustive and critical review of relevant studies and a full explanation of how they are being interpreted. The commenter notes that extensive information about each study is typically part of the public record even if all underlying data may not be included. The commenter adds that EPA assessments are normally subject to public comment and independent peer review and members of the regulatory community are free at any time to replicate studies they deem flawed, or to independently seek access to underlying data and reanalyze them. The commenter concludes that the so-called problem that the proposed seeks to fix is largely imaginary and that the stakes for EPA science and the protection of public health are simply too high to finalize this deeply problematic and unnecessary proposal.
Commenter (11479) notes that "evaluation and review" is where the agency already independently validates the research without using underlying data by asking the following questions:
      a) To what extent has there been independent verification or validation of the study method and results? What were the conclusions of these independent efforts, and are they consistent? b) To what extent has independent peer review been conducted of the study method and results, and how were the conclusions of this review taken into account? c) Has the procedure, method or model been used in similar, peer reviewed studies? Are the results consistent with other relevant studies? d) In the case of model-based information, to what extent has independent evaluation and testing of the model code been performed and documented?
The commenter (11479) states that these guiding questions ensure that the Agency does not have to fully reanalyze every study and can instead focus on the weight of the evidence approach, which "considers all relevant information in an integrative assessment that takes into account the kinds of evidence available, the quality and quantity of the evidence, the strengths and limitations associated with each type of evidence and explains how the various types of evidence fit together." The commenter concludes that including data availability in the weighting injects an arbitrary parameter, unrelated to study quality and reliability.
Commenter (11920) states that access to raw data at the stage described in the supplemental notice is typically not a requirement even for peer-review, as scientists can evaluate the robustness and accuracy of findings based on the presentation of processed data and description of the methods and analysis used. The commenter notes that the editors-in-chief of leading scientific journals, including Science, Nature, Proceedings of NAS, Public Library of Science, and Cell, have denounced the original proposed rule noting that "the merits of studies relying on data that cannot be made publicly available can still be judged...As a core skill, scientists are trained in assessing research publications by judging the articulation and logic of the research design, the clarity of the description of the methods used for data collection and analysis, and appropriate citation of previous results."
Commenter (11911) states that EPA does not assess the rule's redundancy given that many peer-reviewed journals have already updated their data transparency policies to encourage data availability.
Commenter (10866) states that truly transparent regulatory procedures should be identical or similar to the principles utilized by virtually all medical professional associations, utilizing a structured multi-step process that EPA could readily adopt. The commenter summarizes these steps as follows:
      a. Develop the question. Find all of the relevant evidence based on pre-established criteria for reviewing the literature; identify the methodologic characteristics of each publication -- e.g., anecdote (lowest form of evidence), randomized controlled trial with appropriate blinding (very high level of evidence). Determine risk of bias: high bias from industry-funded studies; low bias from NIH-funded clinical trials.
      b. Interpret the evidence. Understand association between variables and statistical precision. Synthesize evidence. Formulate evidence-based conclusion.
      c. Create rule in accordance with established procedures including substantial opportunities for public hearings and comments.
Commenter (11892) suggests that independent peer review would be impractical for some specific wetland data, as much of the data used to influence wetland management and regulatory decisions has undergone some sort of peer review through a journal or university committee, or was collected with approved quality control plans, often already approved by EPA. Therefore, the commenter asserts, independent peer review of wetland-related data is not necessary for all wetland data as current procedures and practices are sufficient. The commenter recommends that EPA provide written justification of its findings after EPA conducts an independent peer review to justify significant regulatory decisions.
Commenter (11403) states that there is no need for EPA to impose additional measures to ensure the reliability of science. The commenter states that peer review is a central component of the scientific process that creates transparent and valid science. The commenter notes that when leading scientific bodies and journals such as NAS, Public Library of Science, Science, and Nature expressed opposition to this type of rulemaking in 2018, EPA should have immediately scrapped this proposal in its entirety. The commenter states that the proposed rule has not addressed the core issue of undermining peer-reviewed science. The commenter notes that the editors of the esteemed journals said in a 2018 statement that the merits of research can still be judged without the sharing of raw data because scientists that conduct peer review are trained "as a core skill" to assess "research publications by judging the articulation and logic of the research design, the clarity of the description of the methods used for data collection and analysis, and appropriate citation of previous results." The commenter adds that in other words, because they conduct their own rigorous review of others' research, scientists do not need raw data to determine if a study was conducted correctly but rather are trained to do so by examining data collection and analysis methods to determine whether the results can be replicated. The commenter states that when doctors prescribe a medicine they have deemed effective, it is not because they have reviewed and analyzed the raw data of patients in the clinical trials themselves, but instead, they have reviewed the published papers on the medication and have seen both that the trials were done correctly and that the results were significant enough to prescribe the treatment. The commenter states that his is no different with research scientists.
Commenter (11404) states that the scientific community has its own process that has been thoroughly developed and extensively vetted. The commenter states that Dr. Nancy Cartwright, through her research, presents a list of criteria used to define "credible science": it is credible because 1) of the scientific method, 2) science is rigorous, 3) science produces general truths by generalizing from well-tested results, 4) science is objective 5) because of the integrity of the scientific community and 6) because interplay between overlapping areas of scientific research serves as an invaluable feedback mechanism. The commenter suggests that the six defining characteristics, at a high level, encompass the rigorous, multi-year processes of feedback and peer review, discussion with colleagues, replication, and publication and adds that science that meets the criteria is therefore credible and trustworthy. The commenter notes that this list does not include a requirement to make all data "publicly available in a manner sufficient for independent validation," nor is such a requirement included in the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) guidelines either, as that would be overly burdensome and unnecessary.
Commenter (13788) states that EPA has long relied on peer review and already has extensive policies in place regarding influential scientific information to assess the quality of the scientific research. The commenter states that EPA provides no explanation as to why those policies are not sufficient to allow validation of scientific studies and information relied on by the Agency. The commenter suggests that simply stating that "EPA shall conduct independent peer review on all pivotal regulatory science used to justify significant regulatory decisions and on all pivotal science underlying influential scientific information, consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review," fails to explain why this does not already occur under the Agency's past and current practice. The commenter concludes that if validation is truly the aim of this rulemaking, EPA must explain why existing practices for peer review are insufficient to validate scientific studies as EPA has failed to do so, indeed, the supplemental proposal contains very little discussion of peer review at all.
Response: The EPA agrees with the comments, in part. All Agency technical work products go through rigorous internal review and, when appropriate, peer review. The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. Adequate peer review is peer review that is consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook. The EPA is subject to the OMB Bulletin and its provisions have been incorporated into the Agency's peer review handbook (Ref. 44). The EPA continues to believe that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will lead to consistent implementation of those procedures across the Agency.
The EPA will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect. As discussed in the final rule preamble, the rule requirements are not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review does not typically include reanalysis of the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information) and, thus, peer review is not considered a replacement for the data availability requirements of this rule. Therefore, as detailed in the preamble of the final rule, EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review. If the Agency conducts peer review on pivotal science, the EPA shall ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied in analyzing dose-response data and the implications of those assumptions for the results.
If implementing this procedural rule would result in conflicts with the guidance found in the Agency's existing Peer Review Handbook, the conflicting requirements found in the Handbook will yield to this final rule. The EPA disagrees that the rule adds another layer of peer review that could result in unnecessary delays. EPA clarified in 40 CFR 30.6 that "the EPA will evaluate whether or not to conduct additional peer review of individual studies identified as pivotal science if the studies have already undergone peer review." This should ensure that no unnecessary peer reviews are conducted that could cause undue delay.
This rulemaking is not intended to modify the Agency's interpretations of "best available science." The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. Further, the EPA plans to promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers.
13.2.2 Concerns Related to Transparency in EPA's Peer Reviews
Comment: Commenter (11381) states that the absence of provisions requiring the Agency to obtain third-party information and ensure that their own work is reproducible undermines the effectiveness of EPA's highly developed peer review program. The commenter states that a key element in the supplemental proposal is continued reliance on peer review of pivotal science, but peer review of all types is shackled by a lack of transparency. The commenter states that EPA's own peer review program includes no expectation or requirement that Agency documents submitted for review will be transparent and the supplemental proposal maintains these deficiencies intact, doing nothing to ensure that the Agency will affirmatively comply with its own final rule with respect to even its own peer reviewers. The commenter notes that, instead, new Part 30 would assign to peer reviewers a limited and much less important task: "articulat[ing] the strengths and weaknesses of EPA's justification for the assumptions applied and the implications of those assumptions for the results," which is what Agency peer reviewers have done for years and does nothing to enhance transparency.
Commenter (11381) states that governments use peer review to show that the science behind their policy decisions is correct and notes that government peer review is always owned by the sponsoring agency and that agency chooses the reviewers, writes their charge, and decides whether to respond to their comments. The commenter states that government peer review panels often have limited expertise. The commenter adds that a panel consisting of 15 members may seem to be rich in expertise, but that expertise is rapidly diluted when the review task before them is broad or the available time is short. The commenter states that government-wide peer review guidelines require agencies to limit peer review to scientific and technical matters and avoid asking peer reviewers to opine on policy.
Commenter (11381) recommends that the quality of Agency-sponsored peer review be enhanced by ensuring that the information they provide reviewers is fully transparent. The commenter states that peer reviewers, such as members of EPA's SAB, are a scarce resource and their limited time would be much better spent if the Agency demonstrated compliance with the final transparency rule before seeking peer review. The commenter also recommends that the Agency should strengthen its peer review program by requiring Agency offices to comply with the IQG before seeking peer review. The commenter states that EPA's Peer Review Handbook includes no such requirement, a deficiency that has been as persistent as it is unjustifiable. The commenter recommends that the Handbook, and the policy on which it is based, should be immediately revised to make IQG compliance a prerequisite for peer review.
Response: The EPA is subject to the OMB Bulletin and its provisions have been incorporated into the Agency's peer review handbook (Ref. 44). The EPA continues to believe that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will lead to consistent implementation of those procedures across the Agency.
It should be noted that in this final rule, rather than considering all studies that support the assessment of the relationship of a dose or exposure of a pollutant, contaminant, or substance to the magnitude of a predicted health or environmental impact as "pivotal science," the EPA is balancing transparency and feasibility by focusing on those studies that inform the quantitative relationship between the dose or exposure of a pollutant, contaminant, or substance and an effect. Thus, "pivotal science" includes only those studies that are integral to characterizing dose-response relationships (e.g., identifying candidate PODs). These are the studies that drive the requirements or quantitative analyses of EPA significant regulatory actions or influential scientific information. If peer review under section 30.7 is warranted, it would be conducted on "pivotal science" and not on every dose-response study considered by the Agency in developing a significant regulatory action or influential scientific information. The EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the OMB Bulletin for Peer Review and the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review. If the Agency conducts peer review on pivotal science, the EPA shall ask peer reviewers to articulate the strengths and weaknesses of the justification for the assumptions applied in analyzing dose-response data and the implications of those assumptions for the results.
13.2.3 Supplemental Proposal Violates the Principles of the OMB Final Information Quality Bulletin for Peer Review
Comment: Commenter (12715) states that the supplemental violates the principles of OMB Memorandum M-05-03, Final Information Quality Bulletin for Peer Review, which establishes government-wide guidance "aimed at enhancing the practice of peer review of government science documents." The commenter states that contrary to this OMB memorandum, the SNPR would prevent or limit EPA's reliance on peer-reviewed research unless the underlying data can be made available for public review. The commenter notes that for studies with restricted data that would require a tiered access system to achieve public availability, EPA does not state how such systems will work in practice other than to note that it is conducting a pilot study using the Research Data Center's "secure data enclave" to host EPA datasets in a restricted use environment. The commenter states that development of standard data repositories "is still ongoing," and that EPA itself "does not intend to make all data and models underlying pivotal regulatory science and pivotal science publicly available. The commenter notes that it is not clear if EPA will make data and models publicly available via some tiered access system where it does have access to data and models, or if EPA intends that scientists and other researchers must themselves establish and maintain a tiered access system that ensures privacy protections for sensitive data. The commenter states that despite all these uncertainties regarding the mechanisms for public access, the supplemental proposal claims, with no support, that studies where the data is publicly available -- through a complex tiered access system or otherwise -- are superior to studies where the data is not publicly available, regardless of the rigor of the underlying science and the thoroughness of the peer review process. The commenter asserts that disregarding or giving less weight to studies simply because the underlying data is not publicly available undermines OMB principles of peer review. Rather than excluding evidence from consideration absent the raw data's availability and pursuing adoption of a tiered access system that has yet to be developed or shown to be workable, the commenter recommends that EPA should continue to adhere to existing, time-tested, and well-established peer review requirements and guidelines to ensure that a consistent method is used to evaluate the quality of any data utilized by the agency for finalizing influential scientific information and significant regulatory decisions.
Response: The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. Adequate peer review is peer review that is consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook. The EPA is subject to the OMB Bulletin and its provisions have been incorporated into the Agency's peer review handbook. The EPA continues to believe that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will lead to consistent implementation of those procedures across the Agency. The EPA disagrees that the final rule violates the principles of OMB's Bulletin for Peer Review (Ref. 8). The EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review.
Although not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounts for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
The EPA would like to clarify that the final rule does not require the EPA to disclose or host data, but to determine if dose-response data are available and to give greater consideration to those studies for which such data are available for independent validation. As discussed in the final rule, underlying dose-response data are considered available if they can be accessed through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information. The rule does not prescribe a type of data sharing mechanism. By placing greater emphasis on the availability of dose-response data underlying pivotal science, the rule will allow for greater scientific scrutiny as EPA decision makers are developing significant regulatory actions and influential scientific information and increases the likelihood that any errors will be identified and corrected. Greater transparency is also inherently valuable as a principle of good government and provides benefits to the public at large, including reducing the risk of errors in EPA analyses and in the science such analyses rely upon. The ability for independent subject matter experts to validate pivotal science will facilitate more effective comment and engagement with the public during development of future significant regulatory actions and influential scientific information.
13.2.4 Resource Burdens of Peer Review Requirements
Comment: Commenter (11193) suggests that if every comment made by a member of an advisory panel or every comment made by a peer reviewer required that the underlying data, models, and analysis for that comment to be made publicly available, this long-established process would grind to a halt, weakening the foundations of scientific decision-making.
Commenter (11919) asks that, as states rely on EPA research, EPA consider how it will meet the additional workload required in the proposed regulatory text at 40 CFR 30.7, calling on EPA to conduct independent peer review, without impacting the important work of EPA research scientists.
Commenter (11388) states that the proposal implies that EPA will have to conduct additional Agency-led peer review for every applicable study. The commenter does not agree with this provision as most of these studies have already undergone expert, independent peer review and creating an additional level of peer review to be conducted by the agency adds unnecessary hours and labor to EPA's decision-making procedures. The commenter recommends this requirement be removed from the final rule.
Commenter (12430) notes that as a result of the changes in the supplemental proposal, EPA would effectively have to conduct independent review for all regulatory or influential science, regardless of the peer review process to which that science has already been subjected, which comprises a comprehensive and highly burdensome new task for the agency. The commenter states that EPA would have to coordinate the peer review (or perhaps conduct the peer review itself), pay for it, wait for it to be completed, and ensure that it complied with the mandate under the supplemental proposal to "articulate the strengths and weaknesses of U.S. EPA's justification for the assumptions applied and the implications of those assumptions for the results." The commenter states that the supplemental proposal fails to acknowledge this burden or explain where EPA would find the resources and staff to perform these tasks. The commenter notes that California statute requires "external scientific peer review of the scientific basis for any rule proposed for adoption by any board, department, or office within" CalEPA. The commenter adds that in California statute, however, "scientific basis" means "those foundations of a rule that are premised upon, or derived from, empirical data or other scientific findings, conclusions, or assumptions establishing a regulatory level, standard, or other requirement for the protection of public health or the environment" The commenter states that in contrast, the supplemental proposal would require EPA to independently peer review, not the foundations of a science-based rulemaking, but the actual studies, data, and models underlying both every rule and every piece of "influential scientific information."
Commenter (10748) recommends that providing technical and financial support is essential for this rule, but states that it would also be costly to the Agency and is duplicative since there is already a peer review process in place within the scientific community that is trustworthy.
Response: The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. Adequate peer review is peer review that is consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook. The EPA is subject to the OMB Bulletin and its provisions have been incorporated into the Agency's peer review handbook. The EPA continues to believe that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will lead to consistent implementation of those procedures across the Agency.
Further, the EPA would like to clarify that the rule requirements are not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble of the bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review does not typically include reanalysis of the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information) and, thus, peer review is not considered a replacement for the data availability requirements of this rule.
Therefore, the EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will evaluate whether or not to initiate peer review, consistent with the the EPA's Peer Review Handbook, of individual studies identified as pivotal science if the studies have already undergone journal peer review.
The preamble of the final rule specifies, "given the costs of the current robust process for identifying and reviewing scientific studies and documentation that are existing Agency practice, as well as that the determination of dose-response data availability is limited to pivotal science underlying significant regulatory actions and influential scientific information, the EPA anticipates that the incremental costs of this rule will be small relative to current administrative costs for developing significant regulatory actions or influential scientific information."
13.2.5 Considerations of Bias and Independence of Peer Reviewers
Comment: Commenter (11193) asserts that putting the authority of the peer preview process in the hands of a political appointee, who would have the authority to waive requirements for the data to be public on a case-by-case basis, rather than in the hands of an apolitical team of scientists is, itself, indefensible scientifically, as this would allow politics, and not science, to determine the basis for regulatory policies designed to protect the environment and the American people.
Commenter (10866) suggests that an additional pertinent consideration and criteria, if EPA were to adopt procedures similar to the principles utilized by medical professional associations, includes whether the study is published in a peer-reviewed journal. The commenter observes that in this sphere it should be noted that there are industry-sponsored journals that are peer reviewed; these studies should not be considered because of the risk of bias. The commenter also suggests that the impact factor of the journal should be considered, noting that in general, the highest impact factor journals are the sources of the most carefully reviewed and reliable studies. The commenter recommends that these studies should receive careful attention but should not be immune to the deficiencies noted above.
Commenter (11920) asserts that this rulemaking must also be considered in the context of the Administration's broader assault on science. The commenter suggests that this supplemental notice is designed to cast unjustified doubt on science that has gone through the independent peer-review process, while promoting unscientific criteria for evaluating science. The commenter states that EPA has failed to show any compelling need or justification for the proposed regulation, nor has it demonstrated that the purported benefits of this regulation would outweigh the substantial costs in compliance as well as increased risks to human health and the environment. The commenter supports the language requiring independent peer review in §30.7, adding that it is imperative that the "peer review" be truly independent and that there is transparency as to the peer review process (i.e., who are the peer reviewers, do they have a direct or indirect interest in the subject matter being reviewed, do they have established positions that they have already taken on the "science/subject matter" that they are reviewing, etc.). The commenter recommends that EPA strengthen the "independence" and transparency of the peer review process by identifying who does the peer review, whether they have a direct or indirect interest in the subject matter, previous positions the reviewer may have taken on the matter being reviewed, and affiliation of the reviewer, including any connections between the reviewer and his/her organization and the subject matter.
Commenter (11381) asserts that government peer review panels often include stakeholders who have key policy interests, but little or no expertise with respect to the scientific or technical issues under review and that this practice is inconsistent with government-wide peer review guidance. The commenter states that stakeholders are essential for advisory groups chartered to give policy advice, but they do not belong on scientific peer review panels.
Commenter (11381) suggests that government peer review focuses inordinately on a select group of potential conflicts of interest, typically restricted to financial interests related to for-profit enterprises while other financial interests, such as financial interests related to non-profit enterprises and financial interests with respect to non-profit or agency funding, are ignored. The commenter states that conflicts of intellectual interest are desirable, for it is where such interests arise that peer review can be expected to be most rigorous. The commenter adds that on the other hand, scientists with intellectual interests aligned with an agency, or with researchers whose pivotal science is involved, are least likely to perform a review with care and are also are most likely to defend the scientific information under review for nonscientific reasons. The commenter concludes that the primary conflict of interest in government peer review is provided by employees of the sponsoring agency as they are by far the most personally invested in obtaining specific outcomes.
Response: This rulemaking is not intended to modify the Agency's interpretations of "best available science." The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. Adequate peer review is peer review that is consistent with the requirements of the OMB Peer Review Bulletin and EPA's Peer Review Handbook. The Agency will continue conducting peer reviews according to the peer review policies and practices that were in place before the rule goes into effect. The EPA Peer Review Handbook, 4[th] Edition, describes how peer reviewers are selected by the Agency. As discussed in the final rule preamble, the EPA would like to clarify that the rule requirements are not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." If implementing this procedural rule would result in conflicts with the guidance found in the Agency's existing Peer Review Handbook, the conflicting requirements found in the Handbook will yield to this final rule.
The EPA appreciates the recommendation related to selecting a balanced peer review panel; however, the recommendation is not relevant to this rule and will not be considered in finalizing the rule. The EPA will continue to conduct peer reviews according to the peer review policies and practices that were in place before the rule goes into effect. The EPA Peer Review Handbook, 4[th] Edition, describes how peer reviewers are selected by the Agency.
The EPA is finalizing the language at 40 CFR 30.6 (formerly 40 CFR 30.7 in the 2018 proposed rule and the 2020 SNPRM) to clarify that the Agency will conduct independent peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review and the exemptions described therein. The EPA will evaluate whether or not to conduct additional peer review of individual studies identified as pivotal science if the studies have already undergone journal peer review.
Although not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
    the quality of the study relative to other studies for which the dose-response data are available;
    the extent to which there are other studies for which the dose-response data are available;
    the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
    the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
    the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
    the extent to which the study is supported by other scientific evidence;
    the extent to which the study accounts for unique scientific considerations;
    the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
    the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
Further, the EPA plans to promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers.
The preamble of the final rule specifies, "given the costs of the current robust process for identifying and reviewing scientific studies and documentation that are existing Agency practice, as well as that the determination of dose-response data availability is limited to pivotal science underlying significant regulatory actions and influential scientific information, the EPA anticipates that the incremental costs of this rule will be small....relative to current administrative costs for developing significant regulatory actions or influential scientific information."
 Chapter 14: Implementation (Including Implementation Benefits and Costs)
The Agency states in the notice of proposed rulemaking (NPRM) that it believes the benefits of this proposed rule justify the costs. The proposal provides that the net benefits from wrongful cost estimations and greater reproducibility would cover the costs of implementation.
While the supplemental notice does not expand the benefits and costs discussion from the original rule, the expansion of the scope established by the supplemental notice of proposed rulemaking (SNPRM) drew many additional comments on the anticipated benefits and costs.
Several comments were received on both the NPRM and the SNPRM on the benefits and costs that would be incurred from the rule's implementation. In this section, implementation topics are organized under sections 14.1 NPRM Implementation and 14.2 SNPRM Implementation.
Section 14.1: NPRM Implementation
       Lack of Clarity How the Rule Would be Implemented
       Incorporating Stronger Data and Model Access Requirements in Grants and Cooperative Agreements
       Building Upon Other Federal Agencies' Policies Regarding Grantee and Cooperator Requirements
       Platform for Increasing Public Access to EPA-Funded Data
       Methodologies and Technologies Designed to Provide Protected Access to Data
       Other Implementation Issues that May Require Special Consideration
       Timing of Implementation
       Impacts on Other Federal Agencies, States and Tribes
       Communications
       Implementation Costs
Section 14.2: SNPRM Implementation
       Inter-organizational Efforts
       The EPA's Readiness to Implement the Rule
       Implementation Timing/Process
       Implementation Costs and Benefits
       Implementation Bias
       Rule Includes Insufficient Implementation Detail/Clarity
The specific comments provided in this Chapter reflect the comment letters that provide arguments supported by detailed rationale, examples, and references/documentation. The Docket ID numbers for these commenters are cited throughout this Chapter. In addition, some general comment letters may also include comments related to comments included and responded to in this Chapter. These comments are summarized in Appendix B. Note that points and issues raised in these general comment letters may also be raised by the more detailed comments summarized in this Chapter. The responses provided in this Chapter adequately address these more general comments.
14.1 NPRM Implementation
14.1.1 Lack of Clarity How the Rule Would be Implemented
<IP_ADDRESS> No Part of the Agency Should Be Responsible for Implementing Rule Requirements
Comment: Commenter (0671) contends that no part of the Agency should be responsible for implementing this proposed regulation because it runs counter to the text and intent of the Clean Air Act (CAA) and amendments. Commenter (8281-PH20) asserts that EPA should focus on implementing existing initiatives and guidelines for improving data sharing and transparency at the federal government.
Response: The EPA is authorized to issue this rule under its authority to promulgate housekeeping regulations governing its internal affairs. In this final procedural rule, the EPA does not interpret or apply provisions of a particular statute or statutes that it administers. The EPA will undertake such efforts in forthcoming actions, which will be either statute-specific science transparency regulations or programmatic regulations implementing this procedural rule. Some of these subsequent actions will be substantive rules issued under the associated environmental statutes and will be subject to judicial review.
<IP_ADDRESS> General Lack of Specificity of Implementation and Implementation Responsibilities
Comment: Commenters (5165, 6126, 6144, 6147) express concern regarding the lack of specificity of implementation responsibilities within the EPA in the proposed rule. Commenters (5165, 6144, 6351, 6909) assert that there is little detail on how EPA itself would implement this rule and/or how it would coordinate with other entities on shared responsibilities to review the best available science to protect public health and the environment.
Commenter (6351) states that the proposal fails to discuss how EPA would implement the proposal. The commenter notes that the proposal offers no process for public hearing or even consultation with the SAB over implementation. As written, the commenter suggests that the proposal would require review and assessment of volumes of existing research and revisions to internal processes yet to be determined. It also, according to the commenter, seems to give arbitrary decision-making authority to the Administrator to determine the fate of such research.
Response: Implementation guidelines are forthcoming. Further, the EPA plans to promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers.
<IP_ADDRESS> Disclosure Avoidance and De-Identification of Confidential Data
Comment: Commenter (6126) states that the proposed rule does not specify who will be responsible for disclosure avoidance and de-identification of identifiable, confidential data. The commenter provides that some research subjects have been promised confidentiality as a condition of participation in studies, and researchers may be legally and ethically obligated to provide it. The commenter asserts that there is no current organizational structure at EPA that is well-suited to implement this regulation and manage the important privacy and confidentiality issues that may arise. The commenter states that, similar to the recommendations of the Commission on Evidence-Based Policymaking (Commission), EPA currently lacks a leadership structure well-suited for providing oversight and coordination of data resources. The commenter makes the following recommendations:
 EPA should assign responsibilities to senior leaders for data and evidence activities.
 EPA should develop a Confidential Information Protection and Statistical Efficiency Act (CIPSEA)-approved statistical unit capable of managing the relevant de-identification and other confidentiality issues. If EPA were to be approved for a CIPSEA unit by Office of Management and Budget (OMB) and the agency was willing to meet the requirements of CIPSEA for a unit within the agency, strong confidentiality protections could be in place.
Commenter (6126) notes the following recommendations of the Commission and encourages EPA to identify individuals or develop a statistical unit that can perform the functions:
 Recommendation 3-3 suggests that a senior official in agencies be identified to ensure sufficient data stewardship and implementation of privacy-protective measures.
 Recommendation 5-1 suggests that agencies should identify a chief evaluation officer to help coordinate evidence-building activities.
Response: The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA to disclose or host data, but to determine if dose-response data are available and to give greater consideration to those studies for which such data are available. Hence, this rule does not impose obligations on the EPA or any other party to make data available or requirements to ensure that personally identifiable information (PII) and confidential business information (CBI) are not disclosed. The Agency will continue to follow existing procedures for protecting PII and CBI as appropriate. The Agency may opt, at its discretion, to incur the costs associated with making data available when it is in the public interest to do so, but that will be decided on a case-by-case basis and is not a requirement of the final rule.
<IP_ADDRESS> Review and Enforcement of Confidentiality Agreements or Memoranda of Understanding Established for Data Access
Comment: Commenter (6126) states that the proposed rule does not specify who will review and enforce any confidentiality agreements or memoranda of understanding established for data access. The commenter expresses concern that EPA's existing organizational infrastructure may be insufficient to fulfill such obligations and necessary activities to protect privacy and confidentiality.
Response: The final rule does not require the EPA to disclose or host data, but to determine if dose-response data are available and to give greater consideration to those studies for which such data are available.
<IP_ADDRESS> Adjudication of Issues that May Arise
Comment: Commenter (6362) suggests EPA may also want to consider establishing an office similar to that of National Institutes of Health (NIH) Office of Research Integrity to adjudicate any issues that may arise in the administration of its practices under this rule.
Response: The EPA has a Scientific Integrity Official who implements the Agency's scientific integrity policy. The Scientific Integrity Official also adjudicates any allegations of violation of the Agency's scientific integrity policy.
Comment: Commenter (4534) states that the process to deny a request for data should be substantial. The commenter states that, in recent years we have witnessed several branches of the government not release data or records that were requested by the public or other governmental agencies for political reasons. The commenter provides that the denial was framed that the release of information would be a national security risk.
Response: This is a rule of internal procedure and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
<IP_ADDRESS> Establishment of an Epidemiological Study Council
Comment: Commenter (6862) proposes that an Epidemiological Study Council be established with the goal of creating a transparent document reflecting a thorough review of the study. The commenter suggests there should be a middle ground where the Agency determines that the study can be used for regulatory decisions when all supporting data are not available. The commenter asserts that the decision to use the study in the absence of an independent review of the original data should not be the responsibility of one individual. The commenter provides that the responsible individuals would be identified and would defend their decisions from objections from either industry or public interest groups. The commenter suggests the following proposal to justify the use of an epidemiological study when not all of the supporting data are available or to otherwise state clearly why the study cannot be used without the submission of additional data:
      Each epidemiology study occurring in the open literature or otherwise needs to have a supporting formal Record of Epidemiological Study Review (ROESR) that clearly delineates the justification for a decision to include or not include the study in the regulatory decisions for the chemical or environmental situation. The production of the ROESR will consist of the separate and independent reports of several Sub-Committees that will be integrated by the Epidemiological Study Peer Review Council (ESPRC) and signed by the Council Chairperson and each Council member. It is important to have independent Sub-discipline committees review the study so that the biases of other sub-disciplines are minimized. The Sub-committee members should be drawn from the relevant staff throughout the Agency as well as other government agencies as needed. Each member of the Sub-committees will sign their respective Sub-committee reports. The roles of the six suggested Sub-Committees and Council Chairperson are as follows:
      1. Ethics Evaluation Sub-Committee. This sub-committee will evaluate all aspects of the ethical treatment of the individuals in the cohorts. This includes that it will identify how additional data can be provided to the Agency in a manner that will assure the identities of the individuals are protected.
      2. Endpoint Evaluation Sub-Committee. This evaluation would limit its conclusions with regard to how well the character of the endpoint itself was assessed for and whether or not the effect reported is plausibly related to treatment or within normal variation. The latter includes how many subjects in a cohort are needed to make a meaningful statistical difference for the particular lesion in question.
      This Sub-committee will provide commentary on the consistency of characterization of the endpoint, resolution of confounds as well as any known other chemicals or conditions that affect the normal distribution of the endpoint.
      The experts in this Sub-committee will vary depending upon the endpoint claimed by the epidemiological study and consist of experts in cancer, behavioral response, or other appropriate discipline for interpretation of the significance of the endpoint.