Commenters (11491, 11576, 11920) state that the EPA's own SAB highlighted that adoption of this rule would introduce bias by selectively excluding relevant data. The commenters states that, specifically, the Board found that "exclusion of segments of the scientific literature, with the possibility of inclusion of other selected information without pre-defined criteria, could allow systematic bias to be introduced with no easy remedy."
Commenter (12430) states that in the SNPRM the EPA disclaims any responsibility for obtaining, redacting, or publishing the restricted data, and the SNPRM has no provisions for compensating researchers for this activity. The commenter states that developing and maintaining a tiered access system could prove onerous and expensive. The commenter states that large industrial actors would be able to absorb the costs more readily than other researchers could. The commenter states that, as a result, industry-funded science would be more likely to satisfy the requirements of the SNPRM and qualify for EPA consideration than other science would.
Commenter (12546) states that it is well documented that financial sponsorship (i.e. source of funding) introduces a risk of bias in the results and conclusions in favor of the regulated industry's interests, including animal studies examining the effect of exposure to atrazine on reproductive or developmental outcomes. The commenter states that this has the impact of potentially reducing or eliminating regulations that apply to industry products and ultimately have the effect of increasing profits.
Commenter (12464) asserts that the fact that the proposal places the burden of making data publicly available on researchers systemically favors industry data over academic data. The commenter states that academic scientists have no incentives to go back to old studies and -- at a considerable cost of time and money -- identify the relevant data and make it available to the general public or through a tiered access system. The commenter states that industry scientists, however, have a financial incentive to make available data from studies that support those industries' goals. The commenter states that such an approach does not evenhandedly promote transparency in science.
 Commenter (11497) states that, in reality, "public scrutiny" would only serve industry's interest, either by eliminating the use of studies that contradict their claims of chemical safety or by allowing them to attempt to discredit studies and researchers by "reinterpreting" the data in their favor.
Commenter (12415) contends that, if the rule is adopted, the EPA may rely less on valid information and more on potentially biased data and research from the regulated community (e.g., polluting industries) that would ostensibly meet the standard of "public" while not meeting the standard of independent science.
Response: As described in the preamble to the final rule, the EPA is not categorically excluding any studies from consideration when promulgating significant regulatory actions and developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information. The EPA will subsequently consider underlying dose-response data availability when evaluating pivotal science and give greater consideration to studies where the underlying dose-response data are available in a manner sufficient for independent validation. Although the EPA is prioritizing transparency in pivotal science, the EPA acknowledges, and agrees with commenters, that there may instances where the underlying dose-response data are not publicly available due to technological feasibility or cannot be made available in a secure environment that still allows for independent analysis. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, the availability of underlying dose-response data should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. The EPA could also give pivotal science with underlying dose-response data that is unavailable for independent validation full consideration upon receiving an exemption from the requirements of this rule from the Administrator as detailed under 40 CFR 30.7.
The technical considerations in 40 CFR 30.5(d), the equal consideration of data available through restricted access, the Administrator's exemption in 40 CFR 30.7, and the EPA's incremental approach to maximizing transparency by first focusing on dose-response data are intended to minimize potential unintended consequences, such as the potential for bias suggested by the commenter. In addition, this final rule sets the overarching structure and principles for transparency in significant regulatory actions and influential scientific information. The EPA plans to develop internal implementing guidance and promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers. The EPA believes the rule requirements provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical and privacy considerations in order to ensure the Agency maintains a strong scientific basis for its decision-making.
The EPA received comments from the SAB on the 2018 NPRM and 2020 SNPRM on April 24, 2020. The EPA considered these comments in development of the final rule. In brief, the SAB consensus was that it supported the precept of enhancing public access to scientific data and analytical methods to ensure scientific integrity, consistency, and robust analysis and their consensus recommendations were that the EPA develop additional policy and/or guidance documents; further describe how pivotal science would be selected, made available, and peer reviewed; consider establishing an office on data sharing with the support of a peer review panel; provide greater clarity in the definitions of key terms, including data, independent validation, and publicly available; develop specific criteria for the Administrator's exemptions under the rule; consider applying rule requirements only to information developed after the effective date of the rule; seeking input from experts to identify best practices to ensure efficiency, utility, and protection of data that are made available; and consider funding reanalysis work.
As noted in the preamble for the final rule, this rule sets the overarching structure and principles for transparency of pivotal science in significant regulatory actions and influential scientific information and technical considerations, including those noted by the SAB, could be addressed in subsequent implementation guidelines and/or statute-specific rulemakings. Further, in response to comments from SAB and the public, the EPA clarified and modified several key definitions, including data, independent validation, and publicly available, in the final rule. The EPA also provided criteria for the Administrator's exemption in 40 CFR 30.7 of the final rule. The EPA would like to clarify that this rule does not direct or require any outside entity or the EPA to establish data sharing mechanisms and does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science. Comments related to making data available and future funding opportunities are not within the scope of this rulemaking. Finally, the EPA is maintaining that the rule provisions apply equally to all dose-response data underlying studies used as pivotal science, regardless of when the study or the data was created. Scientific transparency is important regardless of the age of the study or the dose-response data. However, in consideration of instances where data cannot be made available, the EPA amended the rule to provide technical considerations (40 CFR 30.5(d)) that can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability in the absence of maximum transparency and included an exemption criterion -- that the development of the dose-response data was completed or updated before the effective date of the rule -- to 40 CFR 30.7 to account for the concerns raised by the SAB and other public commenters.
Comment: Commenters (12727, 12731) note that SAB member Robert W. Merritt commented that data from older studies may be located in obsolete storage media or data formats that are either impossible or very costly to convert to a shareable form. Moreover, the commenter states that grants funding older studies have likely lapsed and little to no funding would be available to academic and non-profit institutions to make data available, potentially skewing the pool of research towards for-profit industry studies. The commenter also asserts that EPA has disclaimed any responsibility for this cost and is instead opting to discard studies out of hand.[,]
Response: As described in the preamble to the final rule, the EPA is not categorically excluding any studies from consideration when promulgating significant regulatory actions and developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information. The EPA will subsequently consider underlying dose-response data availability when evaluating pivotal science and give greater consideration to studies where the underlying dose-response data are available in a manner sufficient for independent validation.
Although the EPA is prioritizing transparency in pivotal science, the EPA acknowledges, and agrees with commenters, that there may instances where the underlying dose-response data are not publicly available due to technological feasibility or cannot be made available in a secure environment that still allows for independent analysis. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, the availability of underlying dose-response data should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. The EPA could also give pivotal science with underlying dose-response data that is unavailable for independent validation full consideration upon receiving an exemption from the requirements of this rule from the Administrator as detailed under 40 CFR 30.7.
Comment: Commenters (11576, 12720) notes that the proposal states that "the Agency does not intend to make all data and models underlying pivotal regulatory science and pivotal science publicly available (85 FR 15402). The commenters state that the rule fails to consider how the EPA plans to assess which studies and underlying data will be made publicly available, or how it plans to prevent potential bias in its decision to make certain data and models publicly available as opposed to other data and models.
Commenter (12720) states that there is a significant potential that the EPA could selectively release only specific types of studies to members of the public, including biased handling of data and models involving CBI. The commenter states that CBI is already protected from release under FOIA. The commenter states that, in 2019, the EPA finalized an update to its FOIA Regulations. Under this rule, the Administrator and other officials are allowed to review all materials that fit a FOIA request criteria and can decide "whether to release or withhold a record or a portion of a record on the basis of responsiveness or under one or more exemptions under the FOIA, and to issue `no records'' responses." The commenter states that, given the vague language in this proposal and existing FOIA regulations at EPA, there is a significant potential for political interference and arbitrary determinations within EPA for public release of data and models.
Response: Based on a consideration of these and other similar comments, the EPA clarified in the final rule how pivotal science would be selected. In addition, the EPA intends for the final rule to be implemented in a transparent manner. The final rule requires the EPA to document the rationale for exemptions granted by the Administrator to the requirements included in the final rule in the significant regulatory action or influential scientific information.
Comment: Commenter (11176) expresses concern that the rule will hamper the use of evidence in EPA work because of the discounting of important research and introduce potential bias in the rulemaking process, which in turn may have very detrimental health and environment effects.
Commenter (11186) asserts that the EPA should not pick and choose which studies go into laws based on public availability because this approach would not be telling all sides of the story: that would be biasing our laws to minimize the impacts of environmental exposures.
Commenter (11495) states that, in the alternative Option, the EPA is proposing to "give greater consideration," but this is absent of any scientific justification. The commenter states that the "greater consideration" Option is not fully discussed with no details provided about how the Agency would go about providing a weight to an individual study based on the availability of underlying data. The commenter states that this presents a considerable potential to introduce bias. The commenter states that, for instance, studies which contain PII are most likely to reflect specific health outcomes at an individual level, where the existing detail promotes mechanistic understanding and generalizability by clearly defining the impacted population. The commenter adds that, without CBI and proprietary data, analyses are not sufficiently informed of the true impact on business practices, management and economic outcomes.
Commenter (12430) contends the SNPRM would introduce a bias towards less sensitive studies. The commenter states that higher-quality study designs typically collect detailed demographic information and information on risk factors and medical conditions so that susceptible subpopulations can be identified. The commenter states that the more detailed the study, the more difficult it would be to remove PII and eliminate the risk of re-identification. The commenter states that, under the SNPRM, studies that omit personal information would be given greater weight, since this information can be released without linking to an individual. The commenter states that personal information is used to strengthen a study by providing a way to account for impacts due to characteristics other than the exposure effects, such as effects due to age, gender, smoking status, and preexisting medical conditions. The commenter states that it also allows an investigator to stratify the analysis by income and race/ethnicity to see whether any subgroups may be more susceptible to exposure to pollution. The commenter states that studies with individual personal information often are subject to agreements with the study subjects that limit who can view their personal information, making it impossible for this information to be released, even if it can be grouped in such a way as to not identify individuals. The commenter states that the studies that do not contain detailed personal level data are not as strong, since potential confounders cannot be corrected for, thereby limiting the ability of EPA to discover, and base its decisions on, the most accurate determination of the true causes and consequences of a health impacts.
Commenter (12458) contends that giving greater weight to studies where the underlying data is available will increase the relative weight of industry-funded research and research using non-human subjects, which are less likely to foreground questions about human health.
Response: The EPA considered these and other similar comments when developing the final rule. As described in the preamble to the final rule, the EPA is not categorically excluding any studies from consideration when promulgating significant regulatory actions and developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information. The EPA will subsequently consider underlying dose-response data availability when evaluating pivotal science and give greater consideration to studies where the underlying dose-response data are available in a manner sufficient for independent validation. Although the EPA is prioritizing transparency in pivotal science, the EPA acknowledges, and agrees with commenters, that there may instances where the underlying dose-response data are not publicly available due to technological feasibility or cannot be made available in a secure environment that still allows for independent analysis. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, the availability of underlying dose-response data should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. The EPA could also give pivotal science with underlying dose-response data that is unavailable for independent validation full consideration upon receiving an exemption from the requirements of this rule from the Administrator as detailed under 40 CFR 30.7.
The technical considerations in 40 CFR 30.5(d), the equal consideration of data available through restricted access, the Administrator's exemption in 40 CFR 30.7, and the EPA's incremental approach to maximizing transparency by first focusing on dose-response data are intended to minimize potential unintended consequences, such as the potential for bias suggested by the commenter. In addition, this final rule sets the overarching structure and principles for transparency in significant regulatory actions and influential scientific information. The EPA plans to develop internal implementing guidance and promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers. The EPA believes the rule requirements provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical and privacy considerations in order to ensure the Agency maintains a strong scientific basis for its decision-making.
Including this review of dose-response data availability for pivotal science is critical to the EPA's progress toward increased transparency and providing increased opportunity for scientific reanalysis and review by independent third parties. This approach will result in significant regulatory actions and influential scientific information based on the highest quality, most relevant studies and ensure adequate human health and environmental protections while maximizing transparency and maintaining consistency with the statutes the EPA administers. The EPA does not believe that this final approach will lead to bias towards any party.
<IP_ADDRESS> Retroactive Application Would be Burdensome
Comment: Commenter (11358) reiterates their 2018 comments that, regardless of the components of the final rule, EPA should seek to phase-in whatever requirements are justified to allow for sufficient time to respond and prepare for the implications of this rule. The commenter contends that this rule should also not apply to the previous record, and that trying to apply this proposal to models, rules, and research that has already begun or has concluded would only serve to set current work back and complicate work already done. The commenter provides that it makes sense to "grandfather" what has already been completed and to implement this rule in stages in order to not compromise or delay EPA's work.
Commenter (12720) states that the supplemental proposal "would apply to reviews of data, models, and studies at the time a rule is developed or influential scientific information is finalized, regardless of when the data and models were generated." According to the commenter, this requirement raises numerous serious implementation issues.
Response: The final rule applies prospectively to significant regulatory actions and influential scientific information; the final rule has no retrospective effect on either significant regulatory actions or influential scientific information.
Comment: Commenter (11391) states that applying the Proposed Rule retroactively could undermine the scientific bases supporting many of our most important environmental protection programs. According to the commenter, retroactive application would mean that when EPA is, for example, conducting a review of NAAQS or considering modifying the list of Hazardous Air Pollutants pursuant to the CAA, or reviewing Water Quality Standards under the CWA, it could refuse to consider valid, peer-reviewed epidemiological studies that have supported existing standards if all the underlying data in those studies has not been made public. The commenter contends that this could potentially decimate many of the existing pollution standards that unquestionably protect both the environment and human health.
Further, the commenter (11391) notes that in many cases, there will be valid reasons why the data underlying historic studies cannot be made public. In this regard, the SAB has noted that the underlying data "may have been discarded if they were deemed not necessary to maintain" and, even if they still exist, "the researchers may no longer be alive or in a position to assemble the data." The commenter provides that data assembly is time consuming and labor intensive; it is unclear whether EPA would perform the necessary work itself or require researchers to do it. The commenter states that either way, regulatory decision-making would be delayed, and scientific research hampered.
Response: The final rule applies prospectively to significant regulatory actions and influential scientific information; the final rule has no retrospective effect on either significant regulatory actions or influential scientific information.
Although the EPA is prioritizing transparency in pivotal science, the EPA acknowledges, and agrees with commenters, that there may instances where the underlying dose-response data are not publicly available due to technological feasibility or cannot be made available in a secure environment that still allows for independent analysis. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, the availability of underlying dose-response data should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. The EPA could also give pivotal science with underlying dose-response data that is unavailable for independent validation full consideration upon receiving an exemption from the requirements of this rule from the Administrator as detailed under 40 CFR 30.7.
Comment: Commenter (11576) contends that the proposed rule must not be applied retroactively to existing data, models, and studies -- indeed, it must not be applied at all. The commenter provides that complying with the requirements of the proposed rule would represent a significant departure from standard scientific practices, as data availability typically has no bearing on the accuracy or robustness of scientific findings. The commenter asserts that if EPA finalizes the proposed rule, EPA must provide adequate notice to allow researchers to attempt to comply with the rule's requirements, which will impact not only data reporting but privacy agreements for data collection. Furthermore, the commenter notes that without ensuring that infrastructure exists to make such data sharing feasible, the EPA cannot expect previously generated data and models to be able to be shared. The EPA's SAB has rightly warned that "retrospective application of the requirement would be difficult to implement, could be expensive with no clear responsibility regarding who would cover the added costs, and could arbitrarily impact the conclusions drawn."
According to the commenter (11576), retroactively applying the proposed rule to existing science would arbitrarily and unjustly eliminate a large collection of established, peer-reviewed, and verified science generated prior to the establishment of the rule's expectations. Furthermore, the commenter states that the proposed rule would impose new challenges for researchers related to data handling, storage, and the protection of sensitive information, and studies conducted prior to this rulemaking would likely not have made arrangements to allow for sharing of sensitive data publicly in the ways required by the proposal. Additionally, the commenter (11576) asserts that retroactive application of the proposed rule would conflict with the practice of building scientific studies and models iteratively over time in the context of existing literature, as it would essentially wipe out consideration of older and established studies conducted prior to the establishment of mechanisms for sharing data publicly. The commenter states that effectively excluding older studies based on this arbitrary criterion would undermine the most critical and foundational studies and impose tremendous challenges for more recent studies to re-establish foundational findings for acceptance under the guidelines described in the rulemaking.
Response: The EPA is finalizing the applicability of this rulemaking to significant regulatory actions and influential scientific information because of the potential broad impact of these actions and assessments on American lives and livelihoods. The provisions of the final rule apply to science that serves as the basis for informing a significant regulatory action or influential scientific information, as well as to dose-response data underlying pivotal science, regardless of the source of funding or identity of the party conducting the science.
Although the EPA is prioritizing transparency in pivotal science, the EPA acknowledges, and agrees with commenters, that there may instances where the underlying dose-response data are not publicly available due to technological feasibility or cannot be made available in a secure environment that still allows for independent analysis. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, the availability of underlying dose-response data should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. The EPA could also give pivotal science with underlying dose-response data that is unavailable for independent validation full consideration upon receiving an exemption from the requirements of this rule from the Administrator as detailed under 40 CFR 30.7.
The EPA would like to emphasize that the final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedures and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the rule does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science.
The EPA would also like to emphasize that the final rule applies prospectively to significant regulatory actions and influential scientific information; the final rule has no retrospective effect on either significant regulatory actions or influential scientific information.
Comment: Commenter (12720) contends that one important aspect of the retroactivity issue that EPA must consider is that it is not sufficient to apply the policy prospectively to new agency actions, because such actions may rely on scientific products that pre-date the new policy. The commenter refers to the CWA, which requires revision of standards every three years, and notes section 304(a)(1) of the CWA, where the water quality criteria that state standards include are commonly based on levels EPA determines to be safe enough to support various uses in reliance on available science.
According to the commenter (12720), EPA's criteria guidance is -- as the statute provides -- periodically updated, but any new update will be based on then-existing scientific studies (especially if there is no evidence to suggest that those studies should be ignored). The commenter notes that accordingly, EPA may be issuing revised criteria documents and making approval/disapproval decisions on newly-submitted state standards that rely on science products that pre-date this rule for many years into the future. The commenter asserts that any final rule must not limit EPA's ability to use these pre-existing studies for new Agency actions.
Response: The final rule applies prospectively to significant regulatory actions and influential scientific information; the final rule has no retrospective effect on either significant regulatory actions or influential scientific information.
Further, as described in the preamble to the final rule, the EPA is not categorically excluding any studies from consideration when promulgating significant regulatory actions and developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information. The EPA will subsequently consider underlying dose-response data availability when evaluating pivotal science and give greater consideration to studies where the underlying dose-response data are available in a manner sufficient for independent validation. Although the EPA is prioritizing transparency in pivotal science, the EPA acknowledges, and agrees with commenters, that there may instances where the underlying dose-response data are not publicly available due to technological feasibility or cannot be made available in a secure environment that still allows for independent analysis. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, the availability of underlying dose-response data should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. The EPA could also give pivotal science with underlying dose-response data that is unavailable for independent validation full consideration upon receiving an exemption from the requirements of this rule from the Administrator as detailed under 40 CFR 30.7.
<IP_ADDRESS> Benefits of Proposal
Comment: Commenter (11911) contends that the Agency does not adequately explain the expected benefits of the proposed rule. According to the commenter, applying this rule retroactively does not alter the incentive for papers that have already been published, while penalizing the Agency and public by excluding potentially valuable information from the regulatory process.
Commenter (10742) states that implementing this tiered system would likely require a significant investment of staff time and resources and it is not clear that the potential costs of the proposal outweigh the benefits.
Commenter (12727) contends that the Supplemental Notice does not cure EPA's failure in the 2018 proposal to articulate the benefits of its policy. The commenter provides that the error is even more glaring now that the scope of the proposal has significantly expanded. The commenter asserts that the SNPRM would have contained at least the seed of benefits that should be all the more obvious now that the proposal covers a wider range of studies and actions, yet EPA still has not characterized what positive outcomes this proposal would deliver. The commenter states that EPA's failure to characterize or analyze the benefits of its proposal -- or to acknowledge the absence of benefits -- is arbitrary and unlawful.
Response: The EPA agrees that neither the 2018 proposed rule nor the 2020 SNPRM included a characterization of costs or benefits of the proposed action. As discussed in the preamble to the final rule, this is a rule of internal procedure promulgated under the EPA's housekeeping authority and imposes no enforceable duty on any state, local, or tribal governments or the private sector. Additionally, this rule does not require the EPA or other parties to disclose or host data. Hence, this rule does not impose costs on the EPA or any other party to make such data available. The EPA recognizes there will be some incremental review and administrative costs to the Agency but anticipates these will be small. In addition, the Agency may opt, at its discretion, to incur the costs associated with making data available when it is in the public interest to do so, but that will be decided on a case-by-case basis and is not a requirement of the final rule.
The main benefits of this rule spring from greater transparency in significant regulatory actions and influential scientific information. By placing greater emphasis on the availability of dose-response data underlying pivotal science, the rule will allow for greater scientific scrutiny as EPA decision makers are developing significant regulatory actions and influential scientific information and increases the likelihood that any errors will be identified and corrected. Greater transparency is also inherently valuable as a principle of good government and provides benefits to the public at large, including reducing the risk of errors in EPA analyses and in the science such analyses rely upon. The ability for independent subject matter experts to validate pivotal science will facilitate more effective comment and engagement with the public during development of future significant regulatory actions and influential scientific information.

 Chapter 3: Capable of Being Substantially Reproduced, Independent Validation, and Reanalyze
In the notice of proposed rulemaking (NPRM), the Agency discusses reproducibility and ability to replicate findings as important qualities of information that fulfills the proposal's requirements.
The Agency is proposing a definition for the term "capable of being substantially reproduced" in §30.2 of the supplemental notice of proposed rulemaking (SNPRM) to read:
      Capable of Being of Substantially Reproduced means that independent analysis of the original or supporting data using identical methods would generate similar analytic results, subject to an acceptable degree of imprecision or error.
The EPA's interpretation of "capable of being substantially reproduced" as included in the proposed definition above builds on the description in the "Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies". The Agency states that these guidelines, which were issued by the Office of Management and Budget (OMB), are intended to help agencies ensure and maximize the quality, utility, objectivity and integrity of the information that they disseminate.
Several comments were received regarding reproduction, replicability, reanalysis, and independent validation. In this section the public comments are organized according to the following topics under sections 3.1 NPRM Replicability, Reproducibility, and Reanalysis and 3.2 SNPRM Capable of Being Substantially Reproduced; Replication and Reproduction:
Section 3.1: NPRM Replicability, Reproducibility and Reanalysis
          Proposed Rule Requirement to Make Available Information Necessary for the Public to Replicate Finding (§30.5)
          Clarification of the Meaning of Terms Used in the Proposal
          Replication versus Reproducible versus Reanalysis
          Whether the Ability to Reanalyze, Replicate and Reproduce a Study Should Determine the Quality of the Study
          Replicability-Specific Comments
          Reproducibility-Specific Comments
          Replication (or Reproduction) Crisis
          Reanalysis-Specific Comments
          Recommends Further Consultation to Develop New Rule Addressing Reproducibility and Replicability Concerns
Section 3.2: SNPRM Capable of Being Substantially Reproduced; Replication and Reproduction
 Capable of Being Substantially Reproduced; Replication and Reproduction
 Reanalyze
The specific comments provided in this Chapter reflect the comment letters that provide arguments supported by detailed rationale, examples, and references/documentation. The Docket ID numbers for these commenters are cited throughout this Chapter. In addition, some general comment letters may also include comments related to comments included and responded to in this Chapter. These comments are summarized in Appendix B. Note that points and issues raised in these general comment letters may also be raised by the more detailed comments summarized in this Chapter. The responses provided in this Chapter adequately address these more general comments.
3.1 NPRM Replicability, Reproducibility and Reanalysis
3.1.1 Proposed Rule Requirement to Make Available Information Necessary for the Public to Replicate Findings (30.5)
<IP_ADDRESS> Comments Supporting the Proposed Replicability Requirement
Comment: Commenters (1941, 2421, 6105, 6365, 6864, 6921) support the proposed requirement to make available information necessary for the public to replicate findings. Commenters (0039, 0563, 0844, 1486, 2850, 4541, 6867, 6921) refer to a replication (or reproduction) crisis as evidence that supports the need for the proposed rule (specific comments related to replication (or reproduction) crisis provided in section XIV.G.1). Specific comments supporting study reproducibility are provided in section XIV.F.2.
Commenter (2421) expresses support for the proposed rule because they believe it is common sense to base administrative policies, procedures and rules on valid, verifiable information.
Commenter (6864) states the public's interest is best served when science is replicable, transparent, and explainable with other toxicology information. The commenter states that, when studies cannot be replicated or when such studies are not consistent or explainable with other information, using such studies then depends on having access to the underlying data for independent analysis.
Commenters (6105, 6864) state that for many chemicals regulated by EPA, these data are often from dozens to hundreds of studies conducted according to federal guidelines. The commenters report that studies that are not replicated, or for which the results appear to be outlier(s), or that are not consistent within an overall pattern, are still considered by EPA. According to the commenters, EPA's scientists will often contact the authors to obtain additional information or data in order to conduct their own analysis. When such data are forthcoming, independent analyses can be conducted by EPA scientists without the need to disclose confidential information, and public health is better served. The commenters express concern that, when such information is withheld by the authors, EPA's scientists are left with a dilemma - whether and how to use the study.
Commenter (1941) specifically endorses the core precepts of section 30.5. The commenter reports that efforts to gain access to the NIOSH/NCI "Diesel Exhaust in Miners Study" ("DEMS") data, and the results obtained, serve as a clear-cut example of the vital importance of ensuring that pivotal scientific data are made available for review, replication and reanalysis by other independent scientists. The commenter states that, without that type of third-party expert review, the original results of pivotal regulatory science might not receive the scrutiny and reassessment that is warranted, and the appropriate corrections and amendments of those original results might not be made. The commenter asserts that in turn could lead to the implementation of mis-informed regulatory policy and inherently flawed regulations. According to the commenter, the notice of proposed rulemaking (NPRM), when properly implemented, can serve as a safeguard against that type of adverse outcome. The commenter provided additional information regarding the DEMS study, which is summarized below.
      The DEMS study covered more than 10-year period and published their results in 2012. DEMS' purpose was to assess the potential association between exposures to diesel engine exhaust (DEE) and lung cancer among a population of 12,315 mine workers employed in eight U.S. non-metal mines. The NIOSH/NCI investigators claimed to observe a positive exposure-response trend between estimated levels of respirable elemental carbon and lung cancer. There were a number of significant questions regarding the scientific robustness of the DEMS' investigators' methods and observations. Efforts to help facilitate independent investigators' access to the full DEMS data sets in a manner sufficing to allow for a thorough replication and reanalysis of the DEMS results took well over three years and well over $200,000. The investigators ultimately were able to complete their reanalyses, which in turn led to the publication of four peer-reviewed journal articles highlighting the significant questions and uncertainties that pertain to the original DEMS results. The independent investigators were able to demonstrate that: (i) when better estimates of exposure to DEE are utilized, the reported associations between occupational exposures to DEE and lung cancer are greatly attenuated, with most being statistically non-significant; (ii) the observed health impacts were associated principally with only one mine (the limestone mine); (iii) exposures to radon were significant cofounders, and appear to account for most of the remaining observed lung cancer associations; and (iv) the DEMS results are not sufficiently robust to serve as the basis for a quantitative risk assessment for occupational exposures to DEE and lung cancer.
Commenter (6365) contends that the experience of the lead national ambient air quality standards (NAAQS) rulemaking demonstrates the salutary purpose that transparency would serve in the regulatory process. The commenter states that, while EPA relied heavily on the Lanphear et al. (2005) study in its analysis of the health effects of lead exposure, the underlying data for the Lanphear study were not available to EPA or to stakeholders for replication and validation at the time of publication. The commenter reports that it took stakeholders years, including litigation under the Freedom of Information Act (FOIA), to obtain that data. According to the commenter, due to the lack of availability, there was no complete reanalysis of that data for nearly a decade and EPA relied on the study in both the 2008 and 2016 lead NAAQS reviews. The commenter states that the lack of available data has been noted in the scientific literature as preventing alternative statistical analyses to be performed, which is particularly important given that key confounding factors such as parental education, intelligence, birthweight, smoking, and race were not controlled in several of the Lanphear study cohorts and may contribute to an overestimation of health effects. The commenter reports that when the data underlying the Lanphear study was reanalyzed in Crump, KS et al. (2013), numerous errors that were not discovered in the peer-review process and that were not previously known to EPA were uncovered and analyzed. The commenter notes that, although EPA ultimately determined that these errors did not materially affect the NAAQS review documents that relied on the Lanphear study, the Agency considered the errors and their potential impact on EPA's regulatory efforts to be sufficiently substantial so as to include a memorandum in the subsequent NAAQS rulemaking docket acknowledging those errors and considering their impact on the regulatory process.
Commenter (6864) refers to an example supporting the need for replication where there was the publication of a series of studies on a single human group from exposure to a commonly used insecticide that showed an unexpected effect at extremely low exposures. The commenter notes that the finding of these studies has not been replicated in other human studies and is in contrast to extensive animal and other limited human studies. The commenter reports that, while EPA scientists asked the authors of the studies for the underlying data in order to confirm the results at the lower exposure, the authors refused the release of data citing confidentiality concerns. According to the commenter, since EPA had neither confirmatory studies, nor consistent information that made sense in light of the multitude of available studies of the chemical, nor the ability to conduct its own analysis, it chose not to use the incompletely published information from this single human group. The commenter contends that EPA's decision not to use studies with incomplete information is correct. The commenter asserts that when the underlying data are not provided to EPA scientists, especially those adept in the receipt and protection of confidential information, it is difficult to use these studies with confidence to make a credible risk judgment for public health, much less national rulemaking.
Response: The EPA found that comments regarding confusion around the definition of "replicate" in the 2018 proposed rule have merit and is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze." Therefore, in the final rule the term replicate is not used.
<IP_ADDRESS> Comments Opposing the Proposed Replicability Requirement
Comment: Commenters (1409, 1973, 4894, 6117, 6144, 6170, 6880, 6895, 6915, 8276, 9227) oppose the proposed requirement for information necessary for the public to replicate findings. Specific comments supporting their opposition and others' opposition to the proposed study replicability requirement, where specifics provided, are included under relevant topics under sections XIV.B through I.
Response: The EPA found that comments regarding confusion around the definition of "replicate" in the 2018 proposed rule have merit and is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze." Therefore, in the final rule the term replicate is not used.
3.1.2 Clarification of the Meaning of Terms Used in the Proposal
<IP_ADDRESS> Recommendation that EPA Better Define Terms Used in the Rule
Comment: Commenters (6114, 6880, 6915, 6909, 8281-PH20) recommend that the EPA better define the terms used in the rule. Commenter (8281-PH20) states that the terms, "pivotal regulatory science", "replication", "reproducible", and "research data" are not defined or are problematic.

Response: In this final rule, in Part 30.2, the EPA is defining the following terms: "data," "dose-response data," "independent validation," "influential scientific information," "pivotal science," "publicly available," "reanalyze," "science that serves as the basis for informing a significant regulatory action," and "significant regulatory actions." The terms "pivotal regulatory science", "replication", "reproducible", and "research data" are not used in this final rule.
<IP_ADDRESS> Request that EPA Clarify/Define Replicability and Reproducibility
<IP_ADDRESS>.1 Clarify/Define Replicability and Reproducibility Differences
Comment: Commenters (6114, 6125, 6351, 6915, 6909) request that EPA clarify or include definitions to better inform the meaning and differences between "replicability" and "reproducibility" in the rule. Commenter (6351) requests clarification on what the EPA considers "replication".
Commenter (6114) states that this proposal calls for "replication" and "reproducibility"' of studies, yet there are questions within the scientific community as to what these terms mean, and to what degree they are needed.
Commenter (6125) asserts that, in context, EPA is using "replicate" not in the sense of conducting a new study to determine if the original conclusions also are supported by analyses of new data, but whether the results are "reproducible" by other analysts using the original data. The commenter reports that neither the proposal, nor the legislation it is based on, define "reproducible," or "replicable" as used in the rule and are in fact not clear on how broadly this goal might be applied (information must be sufficient to "replicate" findings).
Commenter (6909) reports that EPA has seemingly used the words "reproducibility" and "replicability" interchangeably when in fact they do not mean the same thing. According to the commenter, the EPA's lack of clarity here is one example of the numerous ways in which this proposed rule is so vaguely written and uses terms which are so ill-defined that it is difficult to comment effectively  -  it is unclear whether EPA is concerned about only using studies that can be "reproduced" or only using studies that can be "replicated".
Commenter (6880) recommends EPA consider sources discussing the difference between "replicable" and "reproducible" which include:
 http://www.replicability.tau.ac.il/index.php/replicability-in-science/replicability-vs-reproducibility.html [Note: link does not work  -  double-checked comment letter-correct cite],
      https://jblevins.org/log/rep, and
      https://osf.io/tvyxz/wiki/6.%20Future%20Directions/.
Response: In the final rule the terms replicate and reproduce (or other forms of the word) are no longer used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze." In addition, the EPA is modifying the definition of "independent validation" in the final rule by replacing "capable of being substantially reproduced" with "produced." The EPA did not finalize the proposed 40 CFR 30.2 definition of "capable of being substantially reproduced" because the term is not used in the final rule's definition of "independent validation" or elsewhere in 40 CFR 30.
<IP_ADDRESS>.2 Interpretation and Definition of Replicability
Comment: Commenters (1973, 6114, 6116, 6125, 6131, 6876, 6880, 6895, 6909, 6924, 9227) who note that the term "replicate" needs to be defined or clarified in the rule refer to literature references defining and interpreting "replication" and provide their interpretation of the term.
   Definitions/Interpretations in the Literature
 Commenters (6924, 9227) refer to a 2016 NAS workshop report and suggest that the EPA should use the term as defined in the 2016 NAS workshop report:
            Replication means that a scientific experiment or a trial to obtain a consistent result is repeated. The repeated experiment uses exactly the same protocols and statistical programs but with data from a different population. The goal is to see if the same results hold with data from a different population.
 Commenters (6114, 6116, 6131) refer to an opinion piece by Dr. Bernard Goldstein (Goldstein, B., 2017)  which states that replication is best achieved, when the findings of the initial study are supported by other studies approaching the same question in different ways.
   Commenter Interpretation of the Term
 Commenter (6876) states that replication refers to the fact that consistent findings from studies in different populations in different places strengthens the likelihood of an effect.
 Commenter (6880) states that to "replicate" findings means to repeat an entire experiment, generate new data, and get the same results.
 Commenter (6895) states that replicability is one of the principles of scientific research and in environmental exposure studies that is used to determine if linkages to health effects caused by environmental pollutants can be confirmed in similar or in other populations. The commenter provides that replication means that any scientist studying the same population-wide health effects and pollutant concentrations in an environment should come to the same conclusion regarding linkages between those two factors.
 Commenter (6909) states that, to replicate a study is to repeat the entire study, independent of the original researcher, using the same methods (but without the use of the original data).
 Commenter (6125) states that, under standard usage, a study can be replicated without access to the underlying data and this standard usage would apply.
 Commenter (9227) states that replicating studies does not require access to underlying study data, but rather details regarding the methodological design.
 Commenter (1973) states that replication tests whether different populations recruited in different ways in different locations, with different data collected on them, or in a different manner, and potentially analyzed in a different manner, produces similar results. The commenter provides that this is why scientists believe that replication is the key approach to reaching conclusions.
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze."
<IP_ADDRESS>.3 Interpretation and Definition of Reproducibility
Comment: Commenters (6117, 6362, 6868, 6875, 6880, 6909, 9224, 9227) who note that the term "reproducibility" needs to be defined or clarified in the rule refer to the NAS workshop report definition of "reproduction" and provide their interpretation of the term.
Commenters (6875, 9224) express concern that, according to Goodman, Fanelli, and Ioannidis (2016), there is no scientific consensus for what "methodologically reproducible" is. The commenters suggest that, if EPA wants to build a rule around transparency in regards to methodology, "methodologically reproducible" must be defined.
   Definition/Interpretation in the Literature
 Commenters (6924, 9227) refer to the 2016 NAS workshop report and suggests EPA should use the term as defined in the 2016 NAS workshop report:
            Reproduction means producing something that is very similar to the original research, but in a different medium or context. A researcher who is reproducing an experiment addresses the same research question but may use different methodologies or experimental measurements than the original researcher did.
   Commenter Interpretation of the Term
 Commenter (6880) states that to "reproduce" findings, which requires publicly available data, means to analyze the data provided and come to the same conclusions.
 Commenter (6909) states that to "reproduce" a study is to take the original data from a study and to reproduce its findings by analyzing the data in the same way or using the same code.
 Commenter (9227) states that reproduction refers to a body of evidence addressing the same hypothesis, but using different populations, methods, etc.
 Commenter (6117) states that "methods reproducibility" is when a study provides enough information about the experimental and/or computational procedures so that future authors can repeat the study using the same data to obtain the same results.
Response: In the final rule the term reproduce (or other forms of the word) is no longer used. The EPA is modifying the definition of "independent validation" in the final rule by replacing "capable of being substantially reproduced" with "produced." The EPA did not finalize the proposed 40 CFR 30.2 definition of "capable of being substantially reproduced" because the term is not used in the final rule's definition of "independent validation" or elsewhere in 40 CFR 30.
<IP_ADDRESS> Interpretation and Definition of Reanalysis
Comment: Commenters (6924, 9227) refer to the 2016 NAS workshop report and suggest EPA should use the term as defined by the NAS:
      Reanalysis is conducting a further analysis of data. A person doing a reanalysis of data may use the same programs and statistical methodologies that were originally used to analyze the data or may use alternative methodologies, but the point is to analyze exactly the same data and see if the same result emerges from the analysis.
Commenter (9227) states that reanalysis is a tool to demonstrate the robustness of an effect to changes in the statistical model underlying an analysis of a single data set. The commenter contends that reanalysis does not validate or invalidate a study's findings.
Commenter (1973) states that, compared to replication, reanalysis is a thin reed. The commenter provides that reanalysis can determine whether the same data, analyzed in a different manner (which may or may not be appropriate) produces similar results.
Response: The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze." In the final rule, Part 30.2, the EPA defines reanalyze to mean to analyze exactly the same dose-response data to see if a similar result emerges from the analysis by using the same methods, statistical software, models, or statistical methodologies that were used to analyze the dose-response data, as well as to assess potential analytical errors and variability in the underlying assumptions of the original analysis.
<IP_ADDRESS> Interpretation and Definition of Validation
Comment: Commenters (0671, 6915) express concern that "validation" of scientific studies is mentioned in the proposed rule, but the term is not defined.
Commenter (0671) states that the proposed rule is using a very narrow definition of "validation"; in reality the best validation is not from the same data already used but from new data on other populations using other methods. The commenter asserts that the best replication happens when there is agreement across populations and methodologies and study designs. The commenter reports that his broader type of replication is what EPA already highlights when making rules.
 Response: As discussed in the preamble to the final rule, EPA defines "validation" as to substantiate or confirm. In the final rule, EPA defines "iindependent validation" as "the reanalysis of study dose-response data by subject matter experts who have not contributed to the development of the study to evaluate whether results similar to those reported in the study are produced."
 In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze." Given that proposed 40 CFR 30.5 also included the term "independent validation" and that this term directly relates to "replicate," the EPA also proposed a definition at 40 CFR 30.2 for this term. The proposed definition of "independent validation" included the term "capable of being substantially reproduced." The EPA also defined this term because it was an important component of the definition of "independent validation." The EPA modified the definition of "independent validation" in the final rule by replacing "capable of being substantially reproduced" with "produced." The EPA did not finalize the proposed 40 CFR 30.2 definition of "capable of being substantially reproduced" because the term is not used in the final rule's definition of "independent validation" or elsewhere in 40 CFR 30. The EPA is also modifying the definition of "reanalyze" to specify the use of the same methods because the proposed rule specified the use of the "same or different" methods. This change was made so that the definition would be consistent with the final rule's definition of "independent validation."
3.1.3 Replication versus Reproducible versus Reanalysis
Comment: Commenters (1973, 4840, 6132, 6158, 6170, 6874, 6880, 6893, 6916, 6924, 6925, 8771-PH56, 9227) contend that while replication is important, replication is not done by making underlying data available to reanalyze but is by determining whether the same effects occur and that making data publicly available is insufficient to allow someone to replicate findings. Commenters explain that validation in science is usually accomplished not by reanalyzing published studies' data, as proposed by this draft rule, but instead by other scientists conducting separate studies in other populations to test if the study results are replicable elsewhere or not.
Commenter (1973) states that it has been argued that public data would allow reanalysis of data to ensure conclusions of the studies EPA relies on are valid. The commenter agrees that reanalysis has a role to play, however, the commenter states that the key determination of the consistency of scientific evidence comes from replication, not reanalysis. The commenter asserts that replication is not the re-analysis of previously published studies. It occurs when other researchers, independently conduct similar research on different populations in different locations and come to similar conclusions. Science reaches consensus when competing hypotheses have been tested in multiple independent replications with broadly consistent results, not when one data set has been reanalyzed many times. The commenter asserts that the only thing that would ostensibly be gained by EPA's rule would be, were investigators to comply with EPA's request, the ability for others to re-analyze the same data.
Commenter (4840) states that the scientific enterprise involves approaching the same question in different ways to determine if the results support each other. The commenter asserts that reanalyzing the same study over and over is little different from checking on a surprising newspaper article by buying additional copies of the same paper to see if it says the same thing.
Commenter (6158) states the proposed rule would only address the issue of reanalysis (i.e. analyzing exactly the same data and determining if the same results emerges from this new analysis), but this would do nothing to address replication (i.e. doing the same exact experiment on a different population to see if the same result occurs) or reproducibility (i.e. addressing the same research hypothesis in a different study system or with different methodologies to see if similar findings occur). The commenter provides that, to the extent that there is a replication/reproducibility issue in science generally, the solution is additional, different, independent studies on the same general issue, not essentially rechecking the math in previously published studies that have already undergone extensive peer-review.
Commenter (6170) states that it is much more important in our science to replicate results with new data and in new settings than to re-analyze existing data. This is the type of replication that needs to be encouraged and not the type of data sharing the proposed rule would seek. Many such replication studies have been and will be done outside the United States and these data would never become available to EPA  -  or being made publicly available  -  but some may represent the best studies worldwide, due to high level human exposures in such countries (arsenic in Bangladesh and Taiwan) and the national health system records that are available (in Denmark, Sweden, Norway and Taiwan for example) and much more comprehensive than anything currently available for conducting human studies of this kind in the US.
Commenter (6880) states that it is clear that making data publicly available is insufficient to allow someone to replicate findings and, therefore, EPA's insistence that data must be made public in a manner that allows the findings to be replicated is fundamentally unworkable.
Commenter (6893) reports that reanalysis of data may be undertaken at times to increase confidence in particularly unexpected or consequential findings, but typically confidence in findings is strengthened by having evidence from different studies accumulate.
Commenter (6916) states that the Proposal's focus on public release of the underlying data used in studies  -  particularly "dose response data" and studies, evinces a clear lack of understanding of the way science is conducted and verified. The commenter provides that, while replicating the results of a study with the same data set certainly verifies it, as the National Research Council said in 2002, "[f]or large epidemiological studies, repeating a study is seldom either possible or desirable." According to the commenter, a more effective approach is some combination of new studies testing the same question in different places, and/or an independent analysis with the same data. The commenter asserts that formal peer review, whether or not prior to publication, also validates the outcome of scientific studies.
Commenter (6924) states that, because the proposed rule argues that data and methods should be made available in sufficient detail that one could redo the analysis, the proposed rule appears to be primarily concerned with reanalysis rather than replicating experiments even though the citations for this section of the rule are focused on issues with replication of experimental findings, not reanalysis. The commenter provides that a re-analysis of a study's data alone cannot verify or invalidate scientific findings without the requisite attention to evaluating the research design, methods, uncertainties, biases, and consistency with existing publications.
Commenter (6925) states that our scientific understanding of how air pollution affects health outcomes is based on evaluating a range of high-quality studies conducted in different populations, using a variety of appropriate methods, and evaluating a variety of air pollution mixtures. The commenter provides that the concept of "repeatability" in the evaluation of whether an exposure causes a particular health outcome refers to the notion that many different studies on different populations show consistent findings, and that such studies offer complementary data to draw this conclusion. The commenter contends that this type of repeatability provides a stronger evidence base from which to conclude that there is a link between the exposure (e.g. air pollution) and the health outcome. The commenter asserts that reevaluating the same study using the same data and methods simply helps to confirm that specific analysis but does not help to expand the evidence base and our understanding of the science of how pollution impacts human health.
Commenter (8771-PH56) states that most often, reanalysis only reveals that different levels of precaution are consistent with the available scientific information; the choice among them is not a job for science, but for policy. The commenter provides that if EPA wants to advocate for a policy of taking more risks with public health in order to impose fewer costs on industry, it should do so forthrightly, rather than hiding the policy change under the banner of "better science." The commenter asks what if someone did reanalyze a health study and got a very different answer, and one that suggests the first study had exaggerated the harm? According to the commenter, in such a case, the "second" study would be "right" and the first one "wrong" only if both of the following conditions held true:
      (1) The difference in the results was not already acknowledged (or contained but unacknowledged) within the uncertainties inherent in each answer! After all, if the first investigators claimed that banning a chemical would save somewhere between 500 and 1500 lives across the country, and EPA chose to estimate its benefits as the average of 1000 lives saved, a second study estimating only 600 lives saved would not be "different" at all; and
      (2) The first study was not just different, but wrong. Anyone can take the same data and botch the risk analysis of it, making it seem like they have a better answer. Just as there are potential problems with an analysis that doesn't control for some variable, it can be a mistake to control for a variable that shouldn't be included -- in other words, more pseudo-sophistication doesn't necessarily mean better.
Commenter (9227) states that reanalyzing a study using publicly available data is not necessary to ensure valid science nor sufficient to ensure against invalid results. The commenter reports that, to ensure the validity of scientific research, the scientific community relies most heavily upon peer review. The commenter reports that studies used by EPA are often evaluated by one of EPA's scientific advisory boards, such as the Clean Air Science Advisory Committee (CASAC) or the Science Advisory Board. (SAB) The commenter provides that these types of reviews do not depend on a study's data being made publicly available.
Commenter (9227) provides that reanalyzing a study is just one of many ways the scientific community ensures integrity, and it is not, in fact a widely used mechanism. The commenter contends that reproducing study results using a different population or method is generally considered a stronger validation than simply reanalyzing the results using the same data, as it shows that the results hold across a different population.
Response: Although the EPA agrees that meaningful insights can be obtained through similar studies in different media or context, the EPA continues to find that independent validation of the study findings and conclusions driving the EPA's dose-response assessments would provide important information. As detailed in the preamble to the final rule, there is scientific support for the usefulness of reanalyzing data, and the EPA finds this to be especially true for data that drive the quantitative requirements or analyses of EPA significant regulatory actions or influential scientific information. Implementation of this rule will more effectively share pivotal science for external consideration and, thus, increase the opportunity for independent subject matter experts to validate pivotal science, and as the dose-response data are better understood through independent reanalysis the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
The Agency recognizes that the 2018 NPRM used terms that were not fully defined or not defined at all. The 2020 SNPRM addressed a number of these comments. To address the concerns expressed by commenters, particularly concerning "reproducible" vs. "replicable" vs. "re-analysis", in this final rule, in Part 30.2, the EPA is defining the following terms: "data," "dose-response data," "independent validation," "influential scientific information," "pivotal science," "publicly available," "reanalyze," "science that serves as the basis for informing a significant regulatory action," and "significant regulatory actions." The EPA is also clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." Therefore, in the final rule the terms "pivotal regulatory science", "replication", "reproducible", and "research data" are not used or defined.
3.1.4 Whether the Ability to Reanalyze, Replicate and Reproduce a Study Should Determine the Quality of the Study
Comment: Commenters (6111, 6127, 6144, 6921, 9227) generally argue that, while many environmental studies simply cannot or should not be replicated or reproduced does not determine the quality of the study.
Commenter (6111) states that it is not necessary to reproduce a study to validate it. The commenter provides that many of the studies that cannot be reproduced are valid but would be excluded by the proposed rule.
Commenter (6127) states that, although this rule aims to propose a solution to a replication crisis, there is no mention in the rule, or its citations, or of a study that was shown to be flawed due to a lack of raw data.
Commenter (6144) states that the ability of a study to be reproduced or replicated should not be held as the gold standard for checking the credibility of a study. The commenter asserts that, while it is impossible or simply unethical to recreate certain studies, this lack of recreation does not invalidate the findings in any way. The commenter recommends that EPA should have the flexibility to use its own criteria to judge the rigor and validity of the science informing rules as applicable. The commenter contends EPA already does this well and the rule does not point to any evidence to the contrary.
Commenter (6144) reports that, throughout the proposed rule, EPA argues that part of the reason for the policy is to allow regulators to better determine that key findings are "valid and credible." According to the commenter, the benchmark upon which validation and credibility are measured for the purposes of this rulemaking are through the relative reproducibility and replication of studies, even though there are certainly other criteria that agency scientists currently use to evaluate a study's credibility. The commenter asserts that the EPA's argument that requiring public disclosure is the way to achieve validity and credibility is inconsistent with provisions in OMB's Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies that indicate an agency can verify the independence and credibility of a study by evaluating the research design, methods, data summaries, and how closely aligned the study is with similar literature in the field. The commenter also notes that the lack of consideration for sensitive information in the proposed rule is inconsistent with provisions in the Guidelines document that note that, while transparency of data is important, it does not override other compelling interests such as privacy, trade secrets, intellectual property, and other confidentiality protections.
Commenter (9227) provides that the Harvard Six Cities Study has been reanalyzed and reproduced -- the underlying data is not publicly available because of patient confidentiality protections bound by individual contractual agreements between the scientists and the research participants and by the HIPPA. The commenter asserts that the unavailability of the underlying data is unrelated to the validity, integrity or quality of the Harvard Six Cities Study.
Commenter (9227) states that the results of the Health Effects Institute (HEI) reanalysis of the Harvard Six Cities Study suggests that routine assessment of quality indicators such as methodology, confounding and bias routinely evaluated in the peer review process are generally enough to confirm a study's validity.
Similarly, commenter (8771-PH56) states that the history of calls for reanalysis of major environmental health studies such as the Harvard Six Cities Study led to an expensive and independent total reanalysis convened by the HEI which found that the Harvard researchers had only slightly underestimated the relative risk of a unit of fine particle exposure.
Response: In the final rule the terms replicate and reproduce (or other forms of the word) are no longer used. In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze." In addition, the EPA is modifying the definition of "independent validation" in the final rule by replacing "capable of being substantially reproduced" with "produced." The EPA will not finalize the proposed 40 CFR 30.2 definition of "capable of being substantially reproduced" because the term is not used in the final rule's definition of "independent validation" or elsewhere in 40 CFR 30.
The EPA is a leader in the evaluation of human and ecological impacts of environmental pollution and methods to cost-effectively mitigate these impacts. The selection and use of quality, peer-reviewed literature is a critical part of this work. However, the strength of the review conducted as part of the study's acceptance and publication in a scientific journal has received increased attention in the scientific community due to the lack of transparency and consistency, potential for bias, and the difficulty that independent researchers have had in reproducing published results.,,,,, Jamieson et al. (2019) state, "Journals play a key role in ascertaining the quality of the evidence in work submitted for publication through peer review. However, that practice is neither uniform nor free of exploitation. While it can select for trustable results, its existence alone is not sufficient to verify the quality of the evidence used to back claims." Because of the potential impact of significant regulatory actions, independent validation of studies that are pivotal to the EPA's significant regulatory actions and influential scientific information may increase public confidence in the findings and conclusions presented in those studies. As Jamieson et al. (2019) states, publication alone (including the typical peer review necessary for publication) is insufficient to verify a study's quality, and additional confidence could be gained from third-party reanalyses that independently validate the study results.
The EPA agrees that data transparency is vital for individuals who have not contributed to the study to be able to verify the quality and strength of published studies and agrees with commenters that the opportunity to independently validate the pivotal science that the EPA relies upon is important in furthering scientific understanding and the Agency's mission. A presenter in a 2016 NAS workshop on Principles and Obstacles for Sharing Data from Environmental Health Research stated more directly that "for environmental policy making to be legitimate, the scientific reasoning behind a given decision -- including the data supporting it -- must be transparent" (NAS Workshop Report, Ref. 26). When data are widely available, researchers can validate research results and help identify and correct unintended errors, as well as reanalyze the data for new and different purposes, examine novel questions, provide new scientific insights, and improve model development.
This final rule is not intended to address a specific instance but many commenters have provided examples of Agency actions that could have benefited from the provisions of this final rule.
Comment: Commenters (1364, 1973, 6153, 6351, 6874, 8272) assert that replicating findings across similar studies is more informative than reproducing a study using the same data.
According to commenter (1364), support for the findings of studies, especially pivotal regulatory science, is best done if more than one group is studying the same thing, replicating the study simultaneously. The commenter provides that those studies could then be averaged, compared and further analyzed. The commenter inquires that, to zoom the principle of sample size out to the bigger picture, would it not benefit the EPA to have a bigger sample size of studies to amass data from?
Commenter (1973) states that the replication process of science provides protections against inappropriate methods because dozens of studies can be done, in different ways, by different teams of investigators. The commenter provides an example where an analysis includes a wide variability in statistical approaches and concludes that, along with the transparency in methods and with the expert peer review of the CASAC for air and other SAB committees, any reasonable goals of assurance that the standard is set based on valid science can be met with the current process.
Commenter (6351) states that, while many studies cannot be specifically repeated, especially those that examine the impacts of historic events, subsequent, similar studies from around the world have echoed their findings on health impacts.
Commenter (9227) asserts that the ability to replicate a study with very poor scientific quality does not strengthen the scientific belief that any toxicity is present. According to the commenter, studies that attempt to reproduce the same findings must have their quality clearly established before comparisons can be made across the multiple studies. The commenter states that "reproducing" studies is generally viewed as a more informative and resource efficient approach to validation of research.
As an example, commenters (6153, 6874, 8272, 9227) assert that the findings of the Harvard Six Cities and American Cancer Society (ACS) cohorts have been replicated and supported by follow-up research. Commenters (6153, 6874, 8272) state that, since publication of these two landmark studies, dozens of additional peer-reviewed studies, as well as follow-up research involving the Harvard Six Cities and ACS cohorts, have replicated and expanded upon their findings and provided insights into specific effects of particulate pollution on certain groups. Commenter (9227) states that, through reproduction and reanalysis, the original findings of the Harvard Six Cities Study have been validated many times over.
Commenter (6117) states that the goal of repeating a study should be to increase the total amount of evidence.
Response: In the final rule the terms replicate and reproduce (or other forms of the word) are no longer used. In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze." In addition, the EPA is modifying the definition of "independent validation" in the final rule by replacing "capable of being substantially reproduced" with "produced." The EPA will not finalize the proposed 40 CFR 30.2 definition of "capable of being substantially reproduced" because the term is not used in the final rule's definition of "independent validation" or elsewhere in 40 CFR 30.
As Jamieson et al. (2019) states, publication alone (including the typical peer review necessary for publication) is insufficient to verify a study's quality, and additional confidence could be gained from third-party reanalyses that independently validate the study results.
Comment: Commenters (6162, 6874, 6875, 6921) agree that any data, methodology, or models produced by EPA should be transparent and available to the public in a reproducible manner. Commenters (6162, 6874, 6921) contend that replication and validation of results (whether the results are reliable and consistent when using different assessment methods) are key to determining whether the science is accurate and of high quality.
Commenter (6921) recommends that, in setting standards, EPA should include consideration of whether the underlying study data is available for review and analysis. The commenter states that current concerns over low replication rates should make EPA hesitate to identify studies as representing the "best available science" if the underlying data and test methods are not released to allow review and replication using the same data or alternative data sets. The commenter asserts that concealing information that would allow the replication and validation of key research findings not only undermines the validity of the results, it also undermines public support for the policies. The commenter concludes that EPA has an obligation to consider transparency in making any judgments on what is the best available science for making policy decisions.
Commenter (6162) states that studies that are reproducible allow independent researchers to evaluate the myriad of choices and assumptions the researchers have made regarding the data and statistical models and the potential introduction of any sources of bias. The commenter reports that bias in research studies can arise from a variety of sources, including publication bias, model uncertainty, model selection issues, lack of adequate control for confounding variables such as other pollutants and weather, and exposure misclassification arising out of the poor correlation between ambient monitors and personal exposure. According to the commenter, privacy concerns with making accessible the data of individuals can be fully addressed using techniques already employed by the U.S. Census Bureau and other agencies, as EPA noted in its proposal.
Response: The Agency agrees that greater transparency is necessary. The scientific significance of reanalyzing and validating study results is not a recent development. McNutt (2014) noted, "Reproducibility, rigor, transparency, and independent verification are cornerstones of the scientific method." The National Academies of Sciences workshop on Reproducibility and Replicability in Science also noted that "certainly, reproducibility and replicability play an important role in achieving rigor and transparency."
Comment: Commenter (9227) provides that the proposed concepts of replication, reproduction and reanalysis do not include a discussion of the scientific quality of a study. According to the commenter, the proposal also does not address other key aspects of scientific quality such as generalizability and bias; and how these characteristics of any scientific study are assessed by the EPA and directly relate to the transparency of any decisions they might make.
Response: The EPA disagrees with the commenter. Data transparency is vital for individuals who have not contributed to the study to be able to verify the quality and strength of published studies and agrees with commenters that the opportunity to independently validate the pivotal science that the EPA relies upon is important in furthering scientific understanding and the Agency's mission. A presenter in a 2016 NAS workshop on Principles and Obstacles for Sharing Data from Environmental Health Research stated more directly that "for environmental policy making to be legitimate, the scientific reasoning behind a given decision -- including the data supporting it -- must be transparent" (NAS Workshop Report). When data are widely available, researchers can validate research results and help identify and correct unintended errors, as well as reanalyze the data for new and different purposes, examine novel questions, provide new scientific insights, and improve model development.
3.1.5 Replication-Specific Comments
<IP_ADDRESS> Replication Is Primarily Limited to Laboratory Studies
Comment: Commenter (9227) states that "replicating findings" is essentially limited to laboratory animal and randomized controlled trials and does not capture the vast majority of human epidemiological studies. The commenter reports that even in laboratory experiments, replication can be difficult due to uncontrolled factors like genetic drift in cell lines and animal strains. According to the commenter, if you do have replicate studies and one has a positive finding and another has a negative finding, there would have to be additional criteria used to determine which study was correct; thus, a failure to replicate should not immediately lead to the conclusion that there is no effect. The commenter asserts that rather than replicating a study, it is far better to develop a better study that replicates the results while providing greater insight into the basis underlying any toxicity.
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze."
<IP_ADDRESS> Replicability Is Not Always Possible
Comment: Commenters (0563, 0671, 1335, 1346, 1364, 1385, 1409, 1481, 1486, 1973, 1999, 2004, 4845, 4848, 4933, 5164, 6046, 6105, 6111, 6114, 6116, 6125, 6127, 6130, 6131, 6132, 6133, 6144, 6153, 6158, 6170, 6196, 6351, 6450, 6932, 6939, 6864, 6874, 6876, 6880, 6895, 6909, 6915, 6916, 6924, 8272, 8801, 9227) that oppose the proposal and report that the replication of studies is not always possible.
<IP_ADDRESS>.1 One-Time Events (Accidents, Disasters)
Comment: Commenters (0559, 0563, 1335, 1364, 1385, 1409, 1999, 4840, 5164, 6111, 6114, 6116, 6153, 6158, 6450, 6880, 6924, 6932) provide that it would be unrealistic and unethical to replicate exposure conditions of one-time events (accidents, disasters) that many public health studies have been based on. Most of these commenters also state that not being able to replicate a study because of does not invalidate the scientific research.
Commenter (0563) states that a common objection is that "many public health studies cannot be replicated, as doing so would require intentionally and unethically exposing people and the environment to harmful contaminants or recreating one-time events (such as the Deepwater Horizon oil spill)." According to the commenter, what needs to be replicable in studies of such events are not the events themselves, but the procedures used to collect and analyze data and make inferences from them. The commenter provides that, for a study that used tree-rings as proxy temperature measurements, no one needed to use a time machine to return to the 11th through 20th centuries and regrow trees to recognize that the authors had committed confirmation fallacy by excluding certain data and misused a statistical procedure, resulting in false results. The commenter states that all anyone needed was access to the raw data and the computer code used to analyze it.
Commenter (1409) states that the requirement for replicability could proscribe the use of studies of real-time science during environmental disasters, such as toxic spills, since it would be both unethical and infeasible to replicate such studies.
Commenter (6111) asserts that in some circumstances it would be inhumane, immoral, or physically impossible to replicate studies since some studies involve natural disasters, or one-time events and cannot be reproduced or replicated.
Commenter (6450) provides that many real-world field studies cannot be realistically or ethically replicated because doing so would require recreating unique events or exposing populations to hazardous situations and materials.
Commenters (0563, 1335, 1385, 1364, 1999, 4840, 5164, 6111, 6114, 6116, 6153, 6158, 6450, 6880, 6924, 6932) provide the following examples of one-time events (accidents, disasters) that would be unrealistic and unethical to replicate:
 Studies of Hiroshima and Nagasaki survivors that underlies the SDWA radionuclides regulation. (6111)
 Studies based on the massive oil leak at Deepwater Horizon. (0563, 1385, 5164, 6111, 6158, 6450, 6924)
 Wildfire in the Great Smoky Mountains National Park in 2016. (1364)
 Studies of Minamata residents impacted by contaminated wastewater discharged into Minamata Bay. (6153)
 September 11, 2001 World Trade Center disaster. (6120, 6915, 8913)
 The 1979 Three Mile Island accident. (1335, 6450)
 Hurricanes. (6120, 6915, 8913)
 Studies look at health impacts on people exposed to high levels of smoke from wildfires. (6114)
 The Flint water crisis. (6450)
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze."
<IP_ADDRESS>.2 Exposure Conditions No Longer Exist and Would Be Unethical to Replicate
Comment: Commenters (6111, 6133, 6153, 6170, 6939, 6450, 6153, 6874, 6932, 6939, 8272) contend that in some cases, the original studies cannot be replicated because the exposure conditions no longer exist and/or would be unethical to replicate. Commenter (6111) states that in some circumstances it would be inhumane, immoral, or impossible to replicate studies since some studies involve exposures and conditions that no longer exist and cannot be reproduced or replicated. According to the commenter, such incidents could not be replicated from practical, safety, or ethical points of view, yet the information they provide regarding real time exposure and dose-response interactions is invaluable. Commenter (2004) states that, while many environmental studies cannot be morally or ethically repeated due to further harm to people, these studies are peer reviewed to ensure that the results are both objective and accurate.
Commenters (5178, 6874, 6939, 8272) state that knowledge about impacts of neurotoxicants has often begun with findings of severe impacts in groups exposed to abnormally high levels of a substance  -  such as developmental delays in babies born to mothers who ate seafood from mercury-contaminated Minamata Bay17  -  and then progressed over future decades as researchers have identified subtler impacts at lower exposure levels.18 The commenters add that studies that rely on health effect data from "natural experiments," such as those experienced by Minamata residents because of the contaminated wastewater discharged into Minamata Bay, cannot ethically be replicated and may not have publicly available data, but should not be disregarded given that they can serve as an early signal of the need for regulation.
Commenters (6153, 6939) state that studies that rely on health effect data from "natural experiments," cannot ethically be replicated and may not have publicly available data but should not be disregarded given that they can serve as an early signal of the need for regulation. Commenters state that knowledge about impacts of neurotoxicants has often begun with findings of severe impacts in groups exposed to abnormally high levels of a substance.
Commenter (6170) provides that the exposures that environmental and occupational epidemiologists investigate are, by definition, considered potentially toxic to humans and thus can and should not be tested experimentally on humans.
Commenters (6111, 6450, 6153, 6932) provide the following examples of studies/exposure conditions of studies that would be unethical to replicate:
          Studies of the effects of lead from 1970s, when blood lead levels were higher than they are now. (6111)
          Studies of worker exposure to polychlorinated biphenyls (PCBs) before PCBs were banned; these studies formed the basis of water quality standards for PCBs under the Clean Water Act (CWA). (6111)
          Long-term cohort studies of benzene exposure in workers which formed the basis of EPA's 2007 CAA regulation for emissions of hazardous air pollutants from mobile sources. (6111)
          Studies that analyze the health impacts resulting from asbestos exposure, exposure to Dichloro-diphenyl-trichloroethane (DDT), and birth defects caused by thalidomide. (6450)
          Studies of developmental delays in babies born to mothers who ate seafood from mercury-contaminated Minamata Bay. (6153)
          The chlorpyrifos risk assessment relied on an epidemiological study that was done while chlorpyrifos was still allowed for household use; this study could not be repeated today, as indoor use of chlorpyrifos is no longer allowed in the U.S. (6133, 6932)
Commenter (1335) states that knowingly allowing similar conditions to continue in spite of the broad consensus of scientists - or promoting rules that would permit the return of similar situations - would be downright murderous.
Commenter (6133) states that the mercury EPA Reference Dose (RfD) is based on recommendations of the National Research Council (NRC) of the NAS, that conducted an extensive analysis and calculations derived from three longitudinal epidemiologic studies: the Seychelles Islands, the Faroe Islands, and the New Zealand studies. The commenter states that the use of these studies to set EPA exposure limits was the result of a years-long transparent process of expert scrutiny, public engagement, inter-agency cooperation, and publication in scientific journals. The commenter notes, however, the studies can no longer be reproduced, particularly the Faroe Islands study in which the exposure to the community was a result of eating whales, a practice that has since declined due to public alerts about the hazards of eating the mercury-tainted meat particularly for children and pregnant and breastfeeding women. The commenter adds that it would take decades to repeat the studies, which took decades to conduct in the first place. According to the commenter, EPA estimates that more than 75,000 newborns each year may have increased risk of learning disabilities associated with in-utero exposure to methylmercury, based on maternal blood levels exceeding the EPA RfD.
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze."
<IP_ADDRESS>.3 Long Term Epidemiological Studies
Comment: Commenters (0559, 6170, 6880, 6895, 6932, 8801, 9227) assert that long term epidemiological studies cannot be replicated because the temporal exposure conditions of the studies and other variables (e.g., population exposed) would be impossible to replicate. Commenters generally state that epidemiological studies are scientifically rigorous/valid studies.
          Commenter (6880) provides that these long-term studies involve a specific mix of individuals with their own medical histories and chemical exposures (e.g., National Institute of Health's (NIH's) Agricultural Health Study) that is not replicable.
          Commenter (6932) states that restricting studies that are not "replicable" is shortsighted because longitudinal epidemiological studies are a primary example of scientific studies that can be scientifically rigorous but are not replicable.
          Commenter (6170) states that by disqualifying high quality longitudinal epidemiological and clinical studies - often the most direct and relevant evidence of chemical impacts on humans - the proposed rule would diminish not strengthen the science underlying regulations.
          Commenter (6170) states that observational epidemiology studies are the only possible setting to gain knowledge about the potential of many agents in the most relevant population: in humans.
          Commenter (8801) states that on moral, ethical, and legal grounds, it is impossible to intentionally expose humans to known or suspected harmful substances; however, it would be absurd for EPA to argue that these types of epidemiological studies should not be considered when developing policies that affect the health of millions of Americans.
          Commenter (9227) notes that in environmental epidemiology, randomized control trials are not feasible or ethical, and replication of observational studies is virtually impossible since it is not possible to create the same conditions as seen in the original study.
Commenter (6895) generally supports the need for replication but states that in terms of epidemiology, replication does not require that every piece of data be available. Rather, based on the principles of scientific research, replicability is achieved when links between environmental pollutants and differing populations are seen in multiple studies by the same or different investigators.
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze."
<IP_ADDRESS>.4 Need to Protect Individual Medical Records/Information and Confidential Business Information
Comment: Commenters (4845, 6116, 6130, 6170, 6450) provide that studies may not be able to be replicated due to ethical and legal reasons protecting individual medical records/information and confidential business information (CBI):
          Privacy protections will often block release of individual medical records, especially for epidemiology and other studies of human cohorts. (4845)
          Industry-conducted studies may contain CBI required to be withheld by law. (4845)
          Studies that contain confidential business information, such as product formulations. (6116)
          The original raw data must be withheld to comply with ethical and legal requirements of epidemiological research (e.g., requirements of an Institutional Review Board (IRB) and/ or the Health Insurance Portability and Accountability Act of 1996 (HIPAA)). (6130, 6450)
          IRBs do not allow for such data to be made publicly available. (6170)
Commenter (6116) states that many studies performed using data from industrial sources would be excluded because they contain CBI. The commenter provides that, under federal law, there are criminal penalties for disclosing CBI  -  a serious risk of liability for EPA that the agency does not address except to say generically that all such laws will be followed, without demonstrating how compliance will be achieved in tandem with making research data publicly available.
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze." The final rule does not require EPA to obtain, store or disclose dose-response data underlying pivotal science used by the Agency.
<IP_ADDRESS>.5 Old Data May No Longer Be Available
Comment: Commenters (6130, 6450) provide that studies may not be replicable because the data may no longer be accessible.
Commenter (4845) reports that, for older studies predating digital technology, retrieving full study records may be difficult or impossible, making it impossible to replicate a study.
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze."
<IP_ADDRESS> Replication of Studies Would be Burdensome
Comment: Commenters (1481, 4848, 4933, 6116, 6127, 6153, 6874, 8272) express concern that replication of studies would be prohibitively expensive and time consuming.
Commenter (1481) states that the required replication of data and longitudinal studies, while worth pursuing and funding, would take decades to confirm results which currently point out significant and likely threats to public health, resulting in preventable danger to and increased health costs to the public. The commenter asserts that seeking to increase replication and longitudinal studies is an excellent goal which should be accomplished through earmarked funding rather than standards which cannot efficiently serve Americans.
Commenter (4848) asserts that EPA is not and has never been in the regular business of replicating studies. The commenter reports that the timing and the cuts in EPA funding makes replicating studies (as a condition of promulgating regulations) an impossibility.
Commenter (4933) reports that the very nature of longitudinal public health studies where health and toxins intersect are by design, large, expensive and require years or decades before results are found. The commenter provides that sample sizes can often number in the tens of thousands to millions of data points and may need to be collected over many years before a statistically significant finding is identified. According to the commenter, because these studies are so big, they are often too expensive to repeat. The commenter asserts that this transparency rule would destroy the confidential nature of research and make the burden of conducting research more difficult and expensive.
Commenters (6153, 6874, 6916, 8272) cite (in part or in total) that in a look back at the evolution of research on the health effects of particulate pollution, lead Harvard Six Cities investigator Douglas W. Dockery commented:
      True replication requires not reanalyzing existing work but independent investigators producing independent data. The HEI process made important contributions in evaluating, advancing, and applying new methods which led to significant additional analyses. But the HEI work reanalyzing an existing body of evidence was time consuming and expensive. It would not be a good use of resources to require this approach for all studies contributing to the regulatory actions, or even to a few key studies.
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze."
<IP_ADDRESS> Other Comments Regarding Replicability
<IP_ADDRESS>.1 Public Health/Precautionary Principle Relies on More than Data that can be Replicated
Comment: Commenter (8276) states that the protection of public health relies on more than data that can be replicated. The commenter reports that local health departments are obliged to employ the precautionary principle, which asserts that when health or safety is threatened, preventive steps must be taken even in the face of scientific uncertainty. The commenter asserts that the proposed rule will only perpetuate barriers to the protection of public health. The commenter describes three environmental issues which occurred in Los Angeles County where the Department of Public Health (DPH) demonstrated the need to exercise the precautionary principle:
      1. In response to a natural gas leak, DPH ordered the temporary relocation of over 11,000 residents that were experiencing symptoms, largely related to the sulfur odorants. Available health studies on human exposures to complex mixtures of oil and gas were lacking and insufficient to guide public health interventions. In a major oil or gas release, where toxicity information may be vital yet lacking, action based on the precautionary principle may be the only option to protect the health of nearby residents.
      2. The Exide Battery Recycling Plant released harmful toxins, including lead and arsenic, into the surrounding communities for decades. These emissions contaminated the soil of homes across Los Angeles and increased the risk of cancer, respiratory diseases, and learning disabilities in affected families. DPH immediately stepped into action and could not wait for the State's large-scale, time-consuming approach to soil sampling to be complete. This experience demonstrated the continued need for DPH to make critical program decisions in the absence of complete environmental data.
      3. In recent years, DPH has responded to an increasing number of community complaints of odors and noise surrounding oil and gas drilling facilities near homes and schools. Peer-reviewed literature remains largely inconclusive regarding health risks of living close to oil and gas operations; however, DPH has documented symptoms, such as headaches, nausea, and respiratory irritation, among residents near oil drilling sites. In two cases, DPH acted to reduce exposures and associated impacts in the adjoining community.
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze."
<IP_ADDRESS>.2 Issue of Replicability Not Resolved by Making Data Publicly Available
Comment: Commenter (6046) argues that, to assume that just by making data publicly available issues of replicability will be solved is simplistic and a false assumption. The commenter reports that it may often take researchers with expertise in certain statistical methods and having the appropriate statistical software to do so. According to the commenter, epi analyses may need to evaluate complex inter-relationships between variables using time-series, regression, Markov Chain Monte Carlo, Bayesian analysis, and other techniques. The commenter reports that, in contrast, analysis of non-epi datasets typically is completed using summary statistics, and perhaps basic methods to analyze associations between two variables in which case confidential data is more easily removed without affecting the analysis.
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze."
<IP_ADDRESS>.3 Use of Complex Tools and Materials in Science Can Make Replicability of Studies Infeasible
Comment: Commenter (8913) states that many avenues of inquiry which advance modern science use complex tools and materials (like human cell culture models and specially engineered animals) which are not feasibly replicable.
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze."
3.1.6 Reproducibility  -  Specific Comments
<IP_ADDRESS> Reproducibility Is Not Always Possible
<IP_ADDRESS>.1 Reproducibility as a Standard is Difficult (and Sometimes Impossible)
Comment: Commenter (6144) states that reproducibility as a standard is difficult, and sometimes impossible, to uphold for many types of studies that are done in the environmental health field and that requiring reproducibility is inconsistent with provisions in the OMB Guidelines or Information Quality Guidelines.
Commenter (6915) states that, to the extent the proposal seeks to enable third parties to "re-run" an analysis using the same supporting data and the same models, this may not be possible where proprietary models, methods, designs, and/or data were used in the study. The commenter reports that, as EPA points out, based on the Information Quality Guidelines, in cases where the Agency relies on proprietary models that cannot be made publicly available, the model applications are subject to EPA's peer review policy and other validation checks. The commenter provides that the Information Quality Guidelines indicate that "[t]hese steps, along with transparency about the sources of data used, various assumptions employed, analytic methods applied, and statistical procedures employed should assure that analytic results are `capable of being substantially reproduced.'"
Commenter (6362) states that it is unclear what constitutes a reproducible versus a non-reproducible finding. The commenter asserts that it is important to consider that there are different types of reproducibility, such as methods reproducibility, results reproducibility, and reproducibility of conclusions. The commenter reports, for example, OMB's Information Quality Guidelines define "capable of being substantially reproduced" as "independent reanalysis of the original or supporting data using the same methods would generate similar analytical results, subject to an acceptable degree of imprecision." According to the commenter, however, the inability to reproduce research studies can be related to issues of study design, variability or differences in biological test systems, data integrity, data analyses, and in some cases, scientific misconduct. The commenter asserts that new or novel findings that purport to indicate effects that have little or no biological basis, based on the weight of the evidence coupled to first principles of relevant scientific disciplines, should be subjected to suitable reproducibility requirements, which could include causal analytics.
Response: The Agency recognizes that the 2018 NPRM used terms that were not fully defined or not defined at all. The 2020 SNPRM addressed a number of these comments. To address the concerns expressed by commenters, in this final rule, in Part 30.2, the EPA is defining the following terms: "data," "dose-response data," "independent validation," "influential scientific information," "pivotal science," "publicly available," "reanalyze," "science that serves as the basis for informing a significant regulatory action," and "significant regulatory actions." The terms "pivotal regulatory science", "replication", "reproducible", and "research data" are not used in this final rule.
This final rule establishes how the Environmental Protection Agency (EPA) will consider the availability of dose-response data underlying pivotal science used in its significant regulatory actions and influential scientific information. When promulgating significant regulatory actions or developing influential scientific information, the EPA will give greater consideration to studies where the underlying dose-response data are available in a manner sufficient for independent validation. The final rule does not require the EPA reproduce or evaluate the reproducibility of a dose-response data.
<IP_ADDRESS>.2 Examples Where Reproducibility Is Not Always Possible
Comment: Similar to concerns expressed by many commenters regarding replicability, several commenters (1346, 3131, 4840, 4844, 5170, 6109, 6120, 6144, 6133, 6196, 6868, 6915, 8913) oppose the proposal and assert that the reproduction of studies is not always possible. These commenters generally argue that, while many environmental studies simply cannot or should not be reproduced, the studies provide key insights that should be taken into consideration.
One-Time Events
Commenter (5170) expresses concern that, while emergency events are essentially not reproducible, the information that is gained from studying these events is incredibly valuable to understanding the health impacts of environmental emergencies as well as non-emergency emissions and discharges.
Commenters (0563, 4840, 4845, 5170, 6111, 6114, 6116, 6133, 6144, 6196, 6868, 6915, 6924, 8913) provide the following examples of studies and studied events that are not reproducible or would be unethical to reproduce:
          Unique and catastrophic events, such as the Chernobyl accident. (4840, 6144)
          A steel mill shut down. (6868)
          Natural disasters, power plant closures, changes in traffic patterns during events like the Olympic games. (6868)
          Deepwater Horizon. (0563, 4840, 6111, 6924)
          Studies attempting to capture the impacts of one-time events like spills or plant explosions. (4845)
          Emergency events. (5170)
          Oil spill or volcanic eruptions. (5170)
          High levels of smoke from wildfires or high levels of chemicals resulting from industrial accidents or exposures. (6114, 6116)
          Katrina pollutant disaster, the Hurricane Harvey-related toxic chemical discharges, and the Church Rock uranium mill spill. (8913)
Exposure Conditions No Longer Exist and would be Unethical to Reproduce
Commenters (3131, 6133, 6874, 6932, 6939, 8272) provide the following examples of studies and exposures that would be unethical and immoral to reproduce:
          Public health studies that would subject participants to risk cannot and should not be reproduced. (3131)
          Poor air quality conditions experienced in the past. (6133)
          Epidemiologic studies of chlorpyrifos can no longer be reproduced. (6133, 6932)
          The Mercury EPA Reference Dose (RfD) is based on recommendations of the National Research Council (NRC) of the NAS, that conducted an extensive analysis and calculations derived from three longitudinal epidemiologic studies: the Seychelles Islands, the Faroe Islands, and the New Zealand studies. The commenter states that the use of these studies to set EPA exposure limits was the result of a years-long transparent process of expert scrutiny, public engagement, inter-agency cooperation, and publication in scientific journals. The commenter notes, however, the studies can no longer be reproduced, particularly the Faroe Islands study in which the exposure to the community was a result of eating whales, a practice that has since declined due to public alerts about the hazards of eating the mercury-tainted meat particularly for children and pregnant and breastfeeding women. The commenter adds that it would take decades to repeat the studies, which took decades to conduct in the first place. According to the commenter, EPA estimates that more than 75,000 newborns each year may have increased risk of learning disabilities associated with in-utero exposure to methylmercury, based on maternal blood levels exceeding the EPA RfD. (6133)
          Knowledge about impacts of neurotoxicants has often begun with findings of severe impacts in groups exposed to abnormally high levels of a substance  -  such as developmental delays in babies born to mothers who ate seafood from mercury-contaminated Minamata Bay17  -  and then progressed over future decades as researchers have identified subtler impacts at lower exposure levels.18 Studies that rely on health effect data from "natural experiments," such as those experienced by Minamata residents because of the contaminated wastewater discharged into Minamata Bay, cannot ethically be replicated and may not have publicly available data, but should not be disregarded given that they can serve as an early signal of the need for regulation. (6874, 6939, 8272)
Observational Research
Commenter (6117) states that "methods reproducibility" may be realistic for laboratory-based studies, where it is possible to implement the exact experimental and computation procedures with the same (or very similar) tools to obtain the same results. However, according to the commenter, new clinical or environmental studies may use procedures that are closely matched, but not identical, to previous evaluations. The commenter reports that when it comes to observational research, it is well known that some weaknesses are unavoidable. The commenter explains that it is often difficult to eliminate all potential sources of confounding, or to adjust for the same confounders, across studies from different time periods. The commenter asserts that therefore, it would be unfair to argue that all aspects of a study, including the design and analyses characteristics, need to be reproduced exactly.
Need to Protect Individual Medical Records/Information
          Exposing the identities of participants in the interest of full transparency would be unethical (3131).
          Researchers cannot ethically randomize people to harmful exposures in order to tackle confounding, nor violate informed consent agreements that prohibit open sharing of private data from past studies (6133).
          Commenter (3131) states that failing to utilize data gathered in responsible studies of health records of anonymous participants would be unethical.
Old Data May No Longer Be Available
Commenters (6196) provide that in some cases data is unavailable because the data is old and predates digital technology where retrieving full study records may be difficult or impossible.
Response: The Agency recognizes that the 2018 NPRM used terms that were not fully defined or not defined at all. The 2020 SNPRM addressed a number of these comments. To address the concerns expressed by commenters, in this final rule, in Part 30.2, the EPA is defining the following terms: "data," "dose-response data," "independent validation," "influential scientific information," "pivotal science," "publicly available," "reanalyze," "science that serves as the basis for informing a significant regulatory action," and "significant regulatory actions." The terms "pivotal regulatory science", "replication", "reproducible", and "research data" are not used in this final rule.
This final rule establishes how the EPA will consider the availability of dose-response data underlying pivotal science used in its significant regulatory actions and influential scientific information. When promulgating significant regulatory actions or developing influential scientific information, the EPA will give greater consideration to studies where the underlying dose-response data are available in a manner sufficient for independent validation. The final rule does not require the EPA reproduce or evaluate the reproducibility of a dose-response data.
<IP_ADDRESS> Comments Supporting Reproducibility as Described in the Proposal
<IP_ADDRESS>.1 Recent EPA Action(s) Support the Need for the Proposal
Comment: Commenter (6366) states that recent EPA action on the retrospective review of the Lead Renovation, Repair and Painting Program further reinforces the need for these actions. The commenter reports that, in finalizing the review, EPA claimed a new economic analysis had been completed, yet the data provided cannot be reproduced and is insufficient to fully evaluate the decisions EPA made. The commenter states that because all of the data used to conduct EPA's analysis has not been transparently presented in the associated docket, stakeholders have been unable to fully evaluate the report's underlying economic claims or their resulting impact on the review's conclusion. According to the commenter, this creates continuing challenges for stakeholders as they seek to work with EPA to ensure the program is effective and robust and that all impacts on small entities have accurately been taken into consideration.
Response: The Agency recognizes that the 2018 NPRM used terms that were not fully defined or not defined at all. The 2020 SNPRM addressed a number of these comments. To address the concerns expressed by commenters, in this final rule, in Part 30.2, the EPA is defining the following terms: "data," "dose-response data," "independent validation," "influential scientific information," "pivotal science," "publicly available," "reanalyze," "science that serves as the basis for informing a significant regulatory action," and "significant regulatory actions." The terms "pivotal regulatory science", "replication", "reproducible", and "research data" are not used in this final rule.
This final rule establishes how the EPA will consider the availability of dose-response data underlying pivotal science used in its significant regulatory actions and influential scientific information. When promulgating significant regulatory actions or developing influential scientific information, the EPA will give greater consideration to studies where the underlying dose-response data are available in a manner sufficient for independent validation. The final rule does not require the EPA reproduce or evaluate the reproducibility of a dose-response data.
<IP_ADDRESS>.2 Inability to Reproduce 2010 IRIS Assessment of Formaldehyde
Comment: Commenter (6369) states that transparency, reproducibility and application of current scientific knowledge are paramount to providing the foundation required for sound regulation. The commenter reports that, using the data obtained from the National Cancer Institute (NCI), in combination with data provided in the draft 2010 Integrated Risk Information System (IRIS) assessment for formaldehyde, further detailed analyses were conducted to validate pivotal scientific findings used by EPA's IRIS program to draw conclusions. The commenter provides that the results have been found to disagree with the original study findings and/or the findings cannot be replicated due to methodological issues.