Response: The EPA disagrees with this commenter. Peer review does not typically include reanalysis of the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information) and, thus, peer review is not considered a replacement for the data availability requirements of this rule. Jamieson et al. (2019) state, "Journals play a key role in ascertaining the quality of the evidence in work submitted for publication through peer review. However, that practice is neither uniform nor free of exploitation. While it can select for trustable results, its existence alone is not sufficient to verify the quality of the evidence used to back claims." The authors go on to say that additional confidence could be gained from third-party reanalyses that independently validate the study results. In addition, as detailed in OMB's Final Information Quality Bulletin for Peer Review, "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." The scientific community has also embraced greater data transparency, as evidenced by data sharing and availability requirements for many high-impact journals and the emergence of organizations, such as the Center for Open Science, and international initiatives like Findable, Accessible, Interoperable, and Reusable (FAIR) data principles; Facilitate Open Science Training in European Research (FOSTER); and Guidelines for Transparency and Openness Promotion (TOP) in Journal Policies and Practices that incentivize greater transparency in research.[,] The EPA supports these efforts and believes that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will improve the opportunity for the public to access the EPA's scientific analyses and resulting regulatory actions in a way that is beneficial to the scientific process, the Agency's mission, and the public's health and safety.
<IP_ADDRESS> Feasibility of Reproducing Certain Studies
<IP_ADDRESS>.1 Capability to Obtain Scientific Information
Comment: Commenter (11381) provides that EPA often relies on scientific studies published in peer-reviewed journals. According to the commenter, obtaining and disclosing all information necessary to assure independent reproducibility is required to fulfil the IQG.[,] Without a provision to obtain this information, the commenter asserts that EPA program offices will be expected to rely on the limited information disclosed in peer-reviewed journals, which would prevent the achievement of the goal of transparency.
Response: The Agency recognizes that the 2018 NPRM used terms that were not fully defined or not defined at all. The 2020 SNPRM addressed a number of these comments. To address the concerns expressed by commenters, in this final rule, in Part 30.2, the EPA is defining the following terms: "data," "dose-response data," "independent validation," "influential scientific information," "pivotal science," "publicly available," "reanalyze," "science that serves as the basis for informing a significant regulatory action," and "significant regulatory actions." The terms "pivotal regulatory science," "replication," "reproducible," and "research data" are not used in this final rule.
As further described in the preamble to the final rule, based on public comments and other considerations, the EPA is concentrating its efforts in the final rule to increase transparency of dose-response data, as the dose-response data are discrete and the dose-response assessment is a well-defined and impactful step in the quantitative assessment of risk. The EPA is pursuing an incremental approach to maximizing transparency in the science that it relies upon to ensure that implementation is done in a thoughtful and deliberate way that focuses on the EPA's most impactful actions, minimizes unintended consequences, and informs future transparency requirements. The incremental progress made possible by this rule provides an important step towards prioritizing transparency in particularly impactful EPA rules and assessments and will inform future statute-specific rulemakings. The final rule does not require the EPA to reproduce or evaluate the reproducibility of dose-response data. This rule is apart from and not intended to address potential issues with IQG. Future statute-specific rulemakings may be more expansive as the EPA continues to make incremental progress toward maximizing transparency.
Although the EPA is prioritizing transparency in pivotal science, the Agency also recognizes that there will be instances where the underlying dose-response data of pivotal science are unavailable for independent validation. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, a study with underlying dose-response data that are not available should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. The EPA could also give pivotal science with underlying dose-response data that is unavailable for independent validation full consideration upon receiving an exemption from the requirements of this rule from the Administrator as detailed under 40 CFR 30.7.
<IP_ADDRESS>.2 Ethical Issues
Comment: Commenters (11360, 11403, 11479, 11900, 12422, 12546, 12720) note that some public health studies are not reproducible due to ethical concerns. Specific concerns include:
Commenter (12422) queries whether studies on disasters like the Deepwater Horizon or Chernobyl would be discarded as they are not reproducible.
Commenter (11360) states that certain situations are either impossible to recreate or should not be due to the severity of health outcomes or the circumstances surrounding the exposures. The commenter provides that emergency responses to the 2010 Deepwater Horizon explosion and oil spill or a decade-long prospective cohort study on lead exposure in drinking water and adverse effects on childhood IQ would be difficult and unethical to replicate or reproduce.
Similarly, commenter (12546) provides that situations such as the 2010 Deepwater Horizon explosion and oil spill, health impacts from exposures during hurricanes such as Katrina and Rita, or a decade-long prospective cohort study on lead exposure in drinking water and adverse effects on childhood IQ would be impossible and unethical to replicate. According to the commenter, considering the severity of the health outcomes, dismissing or downgrading this critical research due to the Agency's proposed rigid transparency rule may result in regulatory delays in EPA's charge to protect human and environmental health.
Commenter (11403) provides that many public health studies -- including those focusing on acute environmental impacts -- are not reproducible because reproducing them would require intentional, unethical exposure of human beings, other living beings, or the environment to toxins. The commenter notes that this is not a limitation, nor does it discount the importance and reliability of those studies. According to the commenter, ignoring or giving low priority to peer-reviewed studies because authors are unable to publish raw data or because it is not reproducible is arbitrary and capricious.
Commenter (11479) notes that in addition to privacy, trade secret, intellectual property, and other confidentiality concerns with the underlying data, certain studies, such as observational studies, must be used and it is often not possible or ethical to recreate the conditions under which people were exposed to a contaminant. The commenter contends that the provisions in the proposed rule that require reproducibility are inconsistent with provisions in OMB guidance that state the reproducibility standard cannot be applied to all science.
Commenter (12720) states that the reference dose for mercury is based on an extensive analysis and calculations derived from three longitudinal epidemiologic studies: the Seychelles Islands, the Faroe Islands, and the New Zealand studies. The commenter notes that the studies can no longer be reproduced, particularly the Faroe Islands study in which the exposure to the community was a result of eating whales, a practice that has since declined due to public alerts about the hazards of eating the mercury-tainted meat particularly for children and pregnant and breastfeeding women. In addition, the commenter provides that it would take decades to repeat the studies, which took decades to conduct in the first place. According to the commenter, if EPA may no longer rely on these studies under the supplemental proposal, the Agency will not have access to the information it needs to ensure its regulations are sufficiently protective of public health.
Commenter (11900) contends that many foundational human environmental public health studies cannot be replicated because people are no longer exposed to the hazards at high levels -- often thanks to public health action taken in response to these same studies. The commenter provides that replicating the high levels of exposure observed in these studies is unethical and impossible, because it would involve intentionally exposing people to a harmful chemical. Even if these studies could be repeated, the commenter states that doing so would be a tremendous waste of time and funds. As such the commenter asserts that research initiated prior to the effective date of this rule be exempt from the requirement of the rule.
Response: The Agency recognizes that the 2018 NPRM used terms that were not fully defined or not defined at all. The 2020 SNPRM addressed a number of these comments. To address the concerns expressed by commenters, in this final rule, in Part 30.2, the EPA is defining the following terms: "data," "dose-response data," "independent validation," "influential scientific information," "pivotal science," "publicly available," "reanalyze," "science that serves as the basis for informing a significant regulatory action," and "significant regulatory actions." The terms "pivotal regulatory science," "replication," "reproducible," and "research data" are not used in this final rule.
This final rule establishes how the EPA will consider the availability of dose-response data underlying pivotal science used in its significant regulatory actions and influential scientific information. When promulgating significant regulatory actions or developing influential scientific information for which the conclusions are driven by the quantitative relationship between the amount of dose or exposure to a pollutant, contaminant, or substance and an effect, the EPA will give greater consideration to studies where the underlying dose-response data are available in a manner sufficient for independent validation. The final rule does not require the EPA to reproduce or evaluate the reproducibility of dose-response data.
<IP_ADDRESS>.3 Availability of Resources
Information from Previous Studies is Unavailable
Comment: Commenter (11576) notes that there are studies for which the availability of information is limited and cannot be replicated.
Response: In the final rule the term replicate is not used. The EPA is clarifying that the intent of the term in the proposed regulatory text at 40 CFR 30.5 was "reanalyze" rather than "replicate." In the 2020 SNPRM, the EPA proposed using the term "reanalyze" instead of "replicate" and proposed at 40 CFR 30.2 a definition for "reanalyze."
Further, although the EPA is prioritizing transparency in pivotal science, the Agency also recognizes that there will be instances where the underlying dose-response data of pivotal science are unavailable for independent validation. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, a study with underlying dose-response data that are not available should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. The EPA could also give pivotal science with underlying dose-response data that is unavailable for independent validation full consideration upon receiving an exemption from the requirements of this rule from the Administrator as detailed under 40 CFR 30.7.
Certain Studies May Not be Reproducible with Standard Resources
Comment: Commenter (11389) provides that some data, such as the Tropospheric Monitoring Instrument (TROPOMI) methane concentration dataset, are not easily manipulated without significant computing resources unavailable on a typical personal computer. The commenter states that, as a result, studies conducted using these data are not necessarily reproducible by a person without these resources.
Response: The Agency recognizes that the 2018 NPRM used terms that were not fully defined or not defined at all. The 2020 SNPRM addressed a number of these comments. To address the concerns expressed by commenters, in this final rule, in Part 30.2, the EPA is defining the following terms: "data," "dose-response data," "independent validation," "influential scientific information," "pivotal science," "publicly available," "reanalyze," "science that serves as the basis for informing a significant regulatory action," and "significant regulatory actions." The terms "pivotal regulatory science," "replication," "reproducible," and "research data" are not used in this final rule.
This final rule establishes how the EPA will consider the availability of dose-response data underlying pivotal science used in its significant regulatory actions and influential scientific information. When promulgating significant regulatory actions or developing influential scientific information for which the conclusions are driven by the quantitative relationship between the amount of dose or exposure to a pollutant, contaminant, or substance and an effect, the EPA will give greater consideration to studies where the underlying dose-response data are available in a manner sufficient for independent validation. The final rule does not require the EPA to reproduce or evaluate the reproducibility of dose-response data. Further, the EPA includes the term "subject matter expert" in its definition of independent validation to acknowledge that this work requires some level of expertise.
<IP_ADDRESS>.4 Depersonalized Data Does Not Allow for Reproduction
Comment: Commenter (11921) asserts that it is not possible to conduct a high-quality air pollution and health study without knowing the locations of those being studied, i.e. where they live, the sources and levels of their air pollution exposure, as well as their specific covariates, and specific health outcomes. The commenter states that, since the goal should be to find ways to share data that enables full replication and sensitivity analysis of original studies, it is valuable to consider several aspects of large population air pollution studies that have moved them towards utilizing data at smaller spatial scales, including:
          the utilization of land use regression and other methods that can account for the distance of a subject's home from roadways, industrial facilities, and other sources of air pollution and finer-grained community-level covariates to address earlier air pollution study reliance on central air quality monitoring data,
          as well as address two potentially significant sources of uncertainty in results: the so-called "ecological confounding"29 and "spatial autocorrelation."30, by using data at smaller scales.
The commenter notes that while these have generally enhanced the quality and sensitivity of the studies, these characteristics increase the difficulty of providing a fully "de-identified" data set while also enabling a different investigator to conduct a full replication and sensitivity analysis of the original study results. As such, the commenter asserts that analyses using more limited data sets may well therefore result in incomplete and misleading results.
Response: The Agency recognizes that the 2018 NPRM used terms that were not fully defined or not defined at all. The 2020 SNPRM addressed a number of these comments. To address the concerns expressed by commenters, in this final rule, in Part 30.2, the EPA is defining the following terms: "data," "dose-response data," "independent validation," "influential scientific information," "pivotal science," "publicly available," "reanalyze," "science that serves as the basis for informing a significant regulatory action," and "significant regulatory actions." The terms "pivotal regulatory science," "replication," "reproducible," and "research data" are not used in this final rule.
This final rule establishes how the EPA will consider the availability of dose-response data underlying pivotal science used in its significant regulatory actions and influential scientific information. When promulgating significant regulatory actions or developing influential scientific information for which the conclusions are driven by the quantitative relationship between the amount of dose or exposure to a pollutant, contaminant, or substance and an effect, the EPA will give greater consideration to studies where the underlying dose-response data are available in a manner sufficient for independent validation. The final rule does not require the EPA to reproduce or evaluate the reproducibility of dose-response data. Further, the EPA includes the term "subject matter expert" in its definition of independent validation to acknowledge that this work requires some level of expertise.
3.2.2 Reanalyze
<IP_ADDRESS> EPA's Position on Reanalysis
Comment: Commenter (12645) states that on a particularly important point, the definition of the science subject to EPA's new requirement, EPA has a lengthy paragraph in the Federal Register notice that is almost word-for-word from the National Academies of Science (NAS) Workshop report's account of what was said by Dr Lynn Goldman who chaired the workshop. The commenter notes that no citation is given to the speaker. The commenter provides below the wording in the workshop report to which the commenter has added track changes to demonstrate how EPA has edited these remarks to remove any language that would make it seem other than an NAS consensus report. The commenter provides EPA's language from the Federal Register following the statements from the NAS workshop.
      [The commenter provides Dr. Goldman's statement with tracked changes from a NAS Workshop on Principles and Obstacles for Sharing Data from Environmental Health Research. The tracked changes are shown here with strikethrough. The Federal Register language is provided afterwards and is underlined for comparison.]
       There are various approaches to testing and validating previous scientific work, she said." A reanalysis is when you conduct a further analysis of data." A person doing a reanalysis of data may use the same programs and statistical methodologies that were originally used to analyze the data or may use alternative methodologies, but the point is to analyze exactly the same data and see if the same result emerges from the analysis.
      "Replication means that you actually repeat a scientific experiment or a trial to obtain a consistent result," she continued,. The second experiment uses exactly the same protocols and statistical programs but with data from a different population. The goal is to see if the same results hold with data from a different population.
      "And then, finally, When you reproduce, you are producing something that is very similar to that research, but it is in a different medium or context," she said. In other words, a researcher who is reproducing an experiment addresses the same research question but from a different angle than the original researcher did. "Most of us, when we are doing systematic reviews, are more convinced that something is going on when we see reproducibility as well as replicability
      [Federal Register language]
      A reanalysis is when you conduct a further analysis of data. A person doing a reanalysis of data may use the same programs and statistical methodologies that were originally used to analyze the data or may use alternative methodologies, but the point is to analyze exactly the same data and see if the same result emerges from the analysis.
      Replication means that you actually repeat a scientific experiment or a trial to obtain a consistent result,. The second experiment uses exactly the same protocols and statistical programs but with data from a different population. The goal is to see if the same results hold with data from a different population.
      When you reproduce, you are producing something that is very similar to that research, but it is in a different medium or context. In other words, a researcher who is reproducing an experiment addresses the same research question but from a different angle than the original researcher did.
Commenter (12645) states further that by removing the quotation marks and "she said" language from the Workshop statement, EPA hides the fact that this was the opinion of an individual speaker rather than the consensus of an NAS Committee. The commenter suggests that EPA's omission of the last sentence from the workshop paragraph which clearly is at odds with EPA's goals was also highly misleading. The commenter notes that stated in that summary sentence is that environmental health science inherently moves forward by "reproducibility as well as replicability" which is contrary to EPA's focus on reanalysis of existing data by industry consultants. The commenter notes that had it been known to the readers of the Federal Register announcement that it was Dr Goldman who was being directly quoted they could have looked up her views on EPA's transparency proposal where they would have found that she is in fact strongly opposed to this action. The commenter states that the standard way of deciding what influences a decision in both science and law is to look at the references cited which clearly have influence, or they would not have been cited. The commenter suggests that another reasonable inference is that, "pivotal" findings should certainly be among these references, and would likely be among the most highly referenced. The commenter adds that in both science and law, the references chosen to be cited are crucial determinants of the probity and validity of the argument advanced, so it is highly ironic that EPA has misled the public in this disgraceful way.
Commenter (12645) asserts that EPA's heavy reliance on the NAS is also ironic in that the current EPA leadership has steadfastly refused to consult the Academies on any of the issues related to its attempts to change the scientific processes at EPA. The commenter states that, similarly, in violation of congressional mandates, EPA has not involved its external scientific committees in considering the rationale for any of its major proposed changes in EPA's scientific processes.
Commenter (12645) states that it is unclear as to whether this deception is unintentional or intentional. The commenter asserts that if unintentional, it is one more piece of evidence as to how out of touch EPA's policy leadership is with its own scientists, let alone the scientific community. The commenter argues that presence of appropriate quotation marks in a footnote in the Federal Register notice describing an OMB directive supports the likelihood that the intention was to deceive.
Response: First, EPA disagrees with the commenter's implication that EPA was being misleading. EPA explicitly explained the source of the information it was using for the definitions in the preamble. Indeed, EPA clearly identified it as an NAS workshop report. Further, the document is easily obtainable, so that any reader of the Federal Register notice for the supplemental proposal could easily obtain the document. Second, the commenter is correct that anyone could easily retrieve Dr. Goldman's opinions as well as her extensive body of work in this area. In summary, EPA disagrees that the Agency was being deceptive.
As to the commenter's comment about Dr. Goldman's opinions, her participation on the 2016 NAS Workshop predates 2018 proposed rule. Thus, her opinions on the 2018 proposed rule are not relevant to her participation in the 2016 NAS workshop.
The EPA consulted with the Science Advisory Board (SAB) on the 2018 proposed rule in August 2019. In addition to the consultation, the SAB elected to review the scientific and technical basis of the 2018 proposed rule and provided a report to the EPA in April 2020. In the report, the SAB recognized the importance of the purpose of this rule and provided recommendations to improve its technical feasibility. Comments related to EPA's consultation with NAS or SAB on issues unrelated to this rule are outside the scope of this rulemaking.
Finally as to the comment that summary sentence is that environmental health science inherently moves forward by "reproducibility as well as replicability," EPA agrees. However, the focus of this rule is transparency which does not conflict with this statement from the 2016 NAS workshop report.
Comment: Commenter (11381) suggests that if the Information Quality Guideline (IQG) framework were used as the baseline, reanalyze/reanalysis has no value-added. The commenter states that under the IQGs, it does not matter whether information is original or derivative; it is all covered if it is disseminated by an Agency. The commenter adds that under the IQGs, a "high degree of transparency is required if the information is influential," again, independent of whether the information or original or derivative.
Response: While this rule builds on the IQG, it is not equivalent. The rule applies to a different stage of data than that covered by the IQG. This rule applies to an earlier stage of data than discussed in the IQG. In particular, see the 2020 SNPRM Section III.B for a discussion of stage of data and Section III.C of the preamble to the final rule. Reanalysis of the earlier stage of data provides for a greater degree of transparency.
Commenter (12699) states that it is unclear why EPA only mentions "data and models" at the start of § 30.5, as this is just a subset of information that underlies pivotal regulatory science and pivotal science. The commenter suggests that EPA appears to recognize this in both versions of § 30.5, when it states, "information is considered `available in a manner sufficient for independent validation' when it includes the information necessary to understand, assess, and reanalyze findings." The commenter recommends that EPA should be focusing § 30.5 on all information necessary to understand, assess, and reanalyze findings (basically, in plain English, as a general rule, the public should have access to whatever information is necessary to evaluate a study). The commenter states that the list of examples in § 30.5 includes some of the additional information that is necessary, including computer codes and "associated protocols necessary to understand, assess, and extend conclusions."
Response: This rule applies to dose-response data not data and models. EPA has modified 40 CFR 30.5 to incorporate many of the suggestions by the commenter. This includes by adding the following l language to 40 CFR 30.5(b): "The EPA will use the following factors to assess the quality of studies identified in the systematic review: soundness, applicability and utility, clarity and completeness, uncertainty and variability, and evaluation and review". The information needed to assess these factors are generally included in a study when published and available to the public.
Comment: Commenter (11892) suggests that the proposed rule should consider alternative sites, or methods, rather than the same data for a reanalysis, and these can also be corroborative findings or show limitations or expansion of using results. The commenter notes that this is particularly for field studies in wetlands and waters, where regional site characteristics may provide differing results and that reasons for the difference can be further explored, but some results may be more applicable for certain uses or areas than others and should be considered.
Response: The commenter describes essentially reproducing results as described in the 2016 NAS Workshop report: "When you reproduce, you are producing something that is very similar to that research, but it is in a different medium or context." While reproducing a study is helpful, it does not result in increasing the transparency of a study that EPA uses as pivotal science which is the purpose of the rule.
Commenter (12712) does not support the modified rule to the extent that it suggests that significant EPA regulations may give "consideration" to studies based on data that is neither publicly available nor available through tiered access. The commenter states that this is too vague. The commenter believes that no such study, by definition unavailable for reanalysis by independent researchers, should even be a factor in an EPA decision to regulate. The commenter believes that the requirement in many of the statutes that EPA administers to the effect that regulations must be based on the "best available science" fully supports, and indeed may be argued to require the transparency rule. The commenter asserts that science proceeds by replication -- and what the rule calls reanalysis -- and without data availability, this process cannot happen.
Response: EPA disagrees that its approach is too vague. EPA has listed factors at 40 CFR 30.5 that it shall consider when determining the consideration to afford pivotal science for which the dose-response data are not available for independent validation. These are:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other pivotal science for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounted for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.

Further, as EPA has described in Section III.E of the preamble, only using studies in which the underlying dose-response data are available for independent validation would categorically exclude many valid studies. How EPA and the courts have interpreted the requirement to use best available science under the environmental statutes EPA administers is not consistent with a categorical exclusion of studies that include dose-response data that cannot be made available for independent validation.
Comment: Commenter (12715) provides that in some cases, the data and models used in a reanalysis would not be available to the public. Therefore, the commenter states that the EPA could use a reanalysis, its own or that of another party, which differs from the analysis provided in the original publication of the study, and the data and models underlying this reanalysis would not be available to the public. According to the commenter, this conflicts with EPA's stated goal of increasing transparency.
Response: Upon consideration of the comments, the EPA agrees that it is important not to categorically exclude any study because the data underlying a study at the stage required by this rulemaking may not be available for independent validation. EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Further, the rule does not require any entity, including EPA, to conduct independent validation.
<IP_ADDRESS> Definition of "Reanalyze" and Related Terms
Comment: Commenters (9587, 11401, 12406) support the clarity achieved by defining "Capable of being substantially reproduced", "Data", "Independent validation", "Influential scientific information", "Model", "Pivotal science", ``Publicly available" and "Reanalyze".
Response: EPA notes the commenters support. However, as described in the preamble to the rule, EPA is not finalizing the definitions of "capable of being substantially reproduced" or "model."
Comment: Commenter (9587) notes that EPA would be well-served by a systematic reform to provide clear definitions of all terms relevant to policymaking. The commenter suggests that EPA, following government best practices in the Department of Defense, institute a formal ontology to establish terminological exactitude in all policymaking.
Response: This comment is outside of the scope of this rulemaking.
Comment: Commenter (12454) is concerned the proposed language could be interpreted as excluding subsequent reanalysis of the data used to develop a rule during the period between the proposed rule and the final rule. The commenter requests that EPA include language explicitly addressing data reanalysis in its final rule.
Response: The rule does not require that a reanalysis be done at any time or even at all. The rule requires that EPA give greater consideration to pivotal science where the underlying dose-response data are publicly available in a manner sufficient for independent validation.
Comment: Commenter (12727) suggests that EPA's proposed definition for "capable of being substantially reproduced" attempts to make "reanalysis" synonymous with "reproduce," which is wholly inappropriate. The commenter states that to use the term "reproduced" in this context is to mislead the public into believing that some new scientific insights have been applied to an original set of data such that the original research findings are made stronger. The commenter asserts that this definition highlights a fundamental misconception underpinning the premise of the proposal -- that somehow independent validation is accomplished through reanalysis of study data by other experts. The commenter argues that the strength of scientific findings are not determined by endless reanalysis of a single data set, but rather gained over time through the conduct of separate investigations utilizing different methodologies that yield corroborating conclusions around a hypothesis being explored.
Response: EPA has not finalized "capable of being substantially reproduced". In his rule independent validation of pivotal science occurs through the reanalysis of the underlying dose-response data. This rule does not define independent validation as reproducing results of a study in another context or with another population.
Comment: Commenter (12720) states that in the definition for "reanalyze", EPA does not restrict reanalysis to "subject matter experts" as it did for "independent validation." The commenter notes that EPA does not describe any assumptions underlying analyses that may lead a person to fail to arrive at the "same result" despite analyzing "exactly the same data" and does not clarify what the "same result" is to be interpreted as. The commenter states that this definition focuses on using the "same" data but allows for "the same or different statistical software, models, and statistical methodologies," yet expects "the same result" to emerge, which is problematic. The commenter asserts that this definition clashes with the reality that, even using the same underlying data, changes in software, modeling approach, and statistical techniques would reasonably be expected to deliver results that are not exactly "the same" as results produced in an original analyses. The commenter concludes that EPA's definition is unreasonable, illogical, and flawed.
Response: EPA has finalized the definition of "reanalyze" at 40 CFR 30.2 to specify the use of "the same methods, statistical software, models, or statistical methodologies that were used to analyze the dose-response data."
There is no need to restrict "reanalysis" to "subject matter expert" because "reanalysis" is done for purposes of "independent validation." It is sufficient that the latter definition restricts it.
Comment: Commenter (11381) states that the supplemental proposal definition for "reanalyze" (and its cognate noun, reanalysis) is narrow; to be covered, a reanalysis must use "exactly the same data". The commenter suggests that this definition is incompatible with one of the Agency's stated purposes  -  "to assess potential analytical errors and variability in the underlying assumptions of the original analysis." The commenter states that such assessments typically require additional data, different data, or alternative formulations of the same data, none of which can be exactly the same. The commenter asks: What would happen under new Part 30 to information that is neither data nor models, or even if within those terms, not exactly the same? The commenter believes such information would be exempt from the provisions applicable to that which is included within reanalyze/reanalysis.
Response: Information that is not dose-response data would be outside the scope of the final rule.
Comment: Commenter (12715) is concerned that the supplemental proposal's new definition of "reanalyze" could ultimately result in scientific decisions being made based on reanalysis of data with different models and assumptions than those used by the original researcher(s), thereby altering the scientific conclusions presented in the originally published and peer-reviewed scientific study.
Response: The rule does not require that EPA or any other party reanalyze dose-response data before EPA can consider it in its decision making.
Comment: Commenter (11575) states that EPA's definition of reanalysis should not include the use of different statistical software, models, and methodologies, as it should limit the definition to only the use of the same methods. The commenter notes that the use of different statistical tests to assess the same data is a new analysis rather than a reanalysis. The commenter asserts that EPA has failed to specify the stage of data required for reanalysis and has not considered the implications of this definition on all types of data and models expected to be impacted by the expanded scope of this rulemaking. The commenter states that in the absence of specifying a consistent framework for the types of statistical analyses required for specific types of data to ensure consistency across reanalyses, the ambiguity in the definition of reanalysis invites opportunities for manufacturing uncertainty through conflicting results from alternative statistical tests. The commenter asserts that this approach inherently violates standard scientific process as attempts for reanalysis will disproportionately be targeted towards findings some would like to undermine. Commenter (11495) states that EPA has clarified in the supplemental proposal that the proposed actions are no longer premised on "replication" but rather on "reanalysis" of published scientific studies. The commenter adds that the existing discussion of reanalyzing data is problematic, with potentially severe consequences to quality and pragmatism. The commenter suggests that using the same data, program and statistical methodologies should yield similar results to published studies and is a viable definition of what it could mean to perform a reanalysis of a published study. The commenter notes that using different statistical software while using the same data and methodologies could also be a potentially useful form of reanalysis to ensure that results are not driven by quirks in the specific software used by study authors but this rule goes far beyond that definition.
Commenter (11495) states that in terms of the provided definition in the supplemental proposal, reanalysis using "alternative methodologies" should not necessarily be expected to deliver the same results as the original study. The commenter states that even in cases when alternative methodologies do provide the same results, these results would need to be interpreted differently given that an alternative methodology was employed. The commenter believes this is a far cry from what is traditionally referred to as a reanalysis in which new information or variables are added to a previous analysis to test new research questions. The commenter states that reanalysis studies of this type can be informative, but it has never been the expectation that someone interested in conducting such a study is entitled to having someone else do the majority of the work in preparing all of the data for continued study.
Response: EPA has finalized the definition of "reanalyze" at 40 CFR 30.2 to specify the use of "the same methods, statistical software, models, or statistical methodologies that were used to analyze the dose-response data."
EPA has explicitly explained the stage of data that should be considered. In particular, see Section III.B. of the 2020 SNPRM which discusses different stages of data. Also, see Section III.C of the preamble to the final rule.
EPA disagrees that reanalysis of data is equivalent to having someone else do the study for them. The study will be conducted and published by the first author(s). The person doing the reanalysis did not conduct the study, generate the data or publish the study. The person doing the reanalysis will use the data from the first study, reanalyze the data and determine if the same results were achieved.
Comment: Commenter (11479) states that EPA's draft rule, as modified in the supplemental notice, reveals that the Agency wants to use original data and methods to reanalyze rules and prove that they are reproducible before using them to support a rule. The commenter claims that the supplemental notice still has not clarified what it means by "capable of being substantially reproduced," and it contradicts itself. The commenter states that in the notice, EPA defines "capable of being substantially reproduced" as "independent analysis of the original or supporting data using identical methods would generate similar analytic results, subject to an acceptable degree of imprecision or error" but goes on to cite a NAS workshop report that defines reproducibility as "producing something that is very similar to that research, but is in a different medium or context. In other words, a researcher who is reproducing an experiment addresses the same research question from a different angle than the original researcher did." The commenter believes that EPA is disguising its quest to cherry-pick evidence with unclear definitions of reanalysis, independent validation, and "capable of being substantially reproduced."
Response: EPA agrees with commenters that "capable of being substantially reproduced" is confusing and inconsistent with the other defined terms. EPA has not finalized a definition of "capable of being substantially reproduced."
EPA disagrees that its definitions of "reanalyze" and "independent validation" are an attempt to cherry pick evidence. The definition of "reanalyze" is based on and indeed is very close to the definition in the 2016 NAS Workshop report cited and discussed in Section III.B of the 2020 SNPRM. The definition of "independent validation" hinges in large part on the definition of "reanalzye."
Comment: Commenter (11392) states that the proposed definition for "reanalyze" would introduce the risk of data manipulation into the Agency's regulatory process. The commenter states that by naming "different programs and statistical methodologies" as legitimate methods for the reanalysis of a body of data, the Agency would permit external entities with vested interests to deliberately deploy biased methodologies in order to reach different results from the same data. The commenter states that analyzing data with different methodologies will frequently lead to different results, but the Agency does not explain how the existence of different results would help improve the Agency's policymaking. The commenter suggests that EPA lacks any objective means to evaluate the superiority of one methodology over another as a part of the reanalysis process. The commenter asserts that the proposed definition would therefore result in confusion about how a study should be interpreted and undermine the role of scientific data in the Agency's regulatory process. The commenter recommends that EPA clarify why its proposed definition of "reanalyze" permits different methodologies to be used in reanalyzing study data, describe the process that it will utilize to assess divergent methodologies, and justify how the definition is consistent with the Agency's obligation to consider the best available science.
Commenter (11892) recommends developing different terms for analyzing the exact same data and method vs. different method or model for analysis. The commenter notes that models can be biased toward certain outcomes and recommends measures be in place to prevent rejection or acceptance of results due to bias in analysis.
Commenter (11576) states that the EPA's definition of reanalysis should not include the use of different statistical software, models, and methodologies, as it should limit the definition to only the use of the same methods. The commenter asserts that the use of different statistical tests to assess the same data is a new analysis rather than a reanalysis.
Response: EPA has taken into consideration the comments and has finalized the definition of "reanalyze" at 40 CFR 30.2 to specify the use of "the same methods, statistical software, models, or statistical methodologies that were used to analyze the dose-response data."
<IP_ADDRESS> Issues Associated with Reanalysis
<IP_ADDRESS>.1 Protocols are Needed for Reanalysis
Comment: Commenter (11495) states that the supplemental proposal only requires data be made publicly available to support reanalysis but does not discuss circumstances in which reanalysis might actually occur and how the Agency would respond to such activities.
Commenter (12617) asserts that the supplemental proposal's expansion of research that may have to be reanalyzed is unrealistic and vulnerable to manipulation. The commenter notes that when imagining what reanalyzing even a portion of relevant studies would look like, it sounds like a plan to hopelessly bog down the regulatory process. The commenter states that the Agency gets to choose which studies must be reanalyzed -- by unknown criteria -- by "subject matter experts" and "stakeholders." The commenter adds that, according to the proposal, reanalysis means not just performing the same data processing as the study authors but also experimenting with adding additional variables to models and additional assumptions and methods in the statistical analysis. The commenter asserts that a single study could hold up crucial rulemaking for years, perhaps indefinitely, as the Agency's chosen subject matter experts and stakeholders go through these layers of reanalysis --  especially if a stakeholder has an interest in slowing down or preventing a rule that, say, limits pollution.
Commenter (12720) reiterates comments provided in the SAB's review of the supplemental proposal:
      The SAB finds that requiring the identification of all studies and regulatory science supporting regulatory actions and making them available for reanalysis will be a complex process. As previously discussed, identifying and making "pivotal science data or studies" available could present challenges if some studies were considered in the regulatory decisions but were not used to determine the point of departure (POD) or reference dose (RfD) or other regulatory/technical level. It is not clear how much information and which studies should be included in the requirement to identify and make studies available. As further discussed in Sections 3.2 and 3.3 of this report, if the intent of the rule is to make available for reanalysis "all underlying pivotal science supporting influential scientific information and/or pivotal regulatory science" that contributed to the ultimate regulatory decision, then practical procedures must be established for an independent validation.
Response: The rule does not require that EPA or any other party reanalyze dose-response data and consider it in decision making. Thus, this rule does not include requirements for how a reanalysis should be conducted or how it should be used.
EPA will identify the pivotal science that includes dose-response data available for independent validation at the proposed rule stage for significant regulatory action and when influential scientific information is submitted for peer review which will provide the public the opportunity to comment on EPA's characterization and choice.
Comment: Commenter (11186) states that the rule does not provide information on the logistics of reanalysis. The commenter questions who would do the reanalysis and how much would this cost. The commenter notes that to access data, there would need to be a process involving an extensive protocol, servers to store data, and trained and qualified "re-analyzers", which would result in tremendous costs and time that would be better spent in research studying the prevention and treatment of adverse health outcomes.
Response: The rule does not require that EPA or any other party reanalyze dose-response data and consider it in decision making.
This rule does not require that any party either access data or make data available for independent validation.
Comment: Commenter (11496) asserts that re-analysis of studies already peer-reviewed and published in the scientific literature is not justified. The commenter does not consider the reanalysis of peer-reviewed and published data a legitimate purpose since pivotal regulatory decisions are already peer-reviewed and supported by a broad weight and range of evidence. The commenter states that even if distribution of private health information for reanalysis is a valid research proposal, EPA has not articulated how reanalysis requests and plans will be vetted or peer-reviewed and how the results of these exercises will be published or shared in a transparent process.
Response: The rule does not require that EPA or any other party reanalyze dose-response data before EPA can consider it in its decision making.
Comment: Commenter (10339) states that practical details about the implementation of the rule remain unresolved, including many issues raised by the EPA's Science Advisory Board draft report on the scientific and technical basis for the rule. The commenter notes that it is not clear precisely if/how EPA will pursue tiered access to data or make a final determination on whether the level of access is sufficient to be compliant with the rule.
Response: EPA will address how it will determine whether dose-response data are available for independent validation as it develops its implementation processes for the rule. The final rule does not require EPA to obtain, store or disseminate dose-response data nor does it require EPA to conduct independent validation of pivotal science.
<IP_ADDRESS>.2 Resources Required for Reanalysis
Comment: Several commenters (11921, 11408, 11492) state that is would be costly to EPA and external parties to reanalyze all the studies given the large number of studies with data and models.
Commenter (11495) states that the Supplemental Notice creates a new, complex and potentially resource- and time-consuming process at EPA without any discussion of parameters by which they would respond to the very activity that they are hoping to enable.
Response: The rule does not require that EPA or any other party reanalyze studies before EPA can consider it in its decision making.
<IP_ADDRESS>.3 Conflicting Reanalysis Results
Comment: Commenter (11361) notes that a crucial piece of omitted information in the supplemental proposal and the 2018 proposal relates to what will happen in the event of conflicting reanalyses of data. The commenter states that there is a complete lack of information provided about what could happen if an attempted independent validation conflicted with original study results. The commenter states that EPA provides no metrics for evaluating the appropriateness of methods used in a reanalysis, and conspicuously, provides no information about how a conflicting reanalysis result would impact the EPA's consideration of a study. The commenter observes that this obvious omission introduces another way in which this proposed rule could be manipulated and result in the dismissal of credible scientific studies.
Response: This rule is not intended to address the results of the independent validation through reanalysis. The purpose of the rule is transparency. EPA is increasing the transparency of its significant regulatory actions and influential scientific information by giving greater consideration to more transparent studies.
<IP_ADDRESS>.4 Potential for Bias
Comment: Commenter (11479) states that reanalyzing every study used by the EPA is unattainable and setting a goal of reanalysis may give stakeholders excessive power to weaken EPA's regulatory decisions. The commenter suggests that under the supplementary proposal, industry and others could pick apart the modeling done by the agency to assess residual risk of air pollution impacts of industrial facilities on communities and then challenge the agency's assessment of risks to communities.
Commenter (9455) states that allowing regulated industry-funded scientists to flood the regulatory process with "reanalyses", new "studies", and "independent validations" invites corruption of the regulatory process and failure to generate needed standards and policies. The commenter states that rather than introducing greater transparency and better science, this unethical program will muddy the process and introduce biased pseudoscience.
Commenter (9455) states that EPA wants all studies that support public policies and/or standard-setting to be posted for re-analysis and alternative modeling. The commenter is concerned that this opens the regulatory process to infinite delays and infinite debate without any clear scientific advantage. The commenter notes that the regulated industries have large resources to fund analyses and critiques that "prove" there is no need for regulation or other policy-making. The commenter states that in this proposed rule, EPA has failed to develop mechanisms for preventing unnecessary delays in setting needed standards or generating needed policy, as well as mechanisms to prevent conflict-of-interest bias from entering into data assessment, re-analyses, and alternative modeling. The commenter adds that nothing is said about assessing the quality of the re-analyses, alternative modeling, or independent validations. Commenter (11575) notes that several examples exist within public-health and epidemiological studies where reanalysis funded by industry groups resulted in the emergence of less protective results related to toxicity or mortality estimates for different compounds. The commenter concludes that without consistent guidelines for when reanalysis is required and what statistical tests are utilized, the proposal risks introducing bias through "statistical fishing expeditions."
Response: EPA disagrees that the rule will lead to the weakening of EPA's regulatory decisions. The rule does not require EPA or any other party, including stakeholders, to pivotal science based on dose-response data before EPA can use those studies. Stakeholders and other interested parties can raise challenges to the science underlying EPA's significant regulatory actions and influential scientific information regardless of whether they reanalyze the studies in question.
Comment: Commenter (11495) states that there is a lack of clarity over who the "stakeholders" involved in data reanalysis may be, and the suggestion that they may cherry pick data to force into models that support "alternative assumptions" goes directly against the scientific process and certainly looks at least superficially like exactly the sort of bias the EPA is hoping to avoid with their rule.
Response: It is not necessary or possible for EPA to describe the universe of stakeholders that may decide to conduct independent validation of pivotal science used in significant regulatory actions and influential scientific information. Also, the rule does not require any entity, including EPA, to conduct independent validation before EPA can consider pivotal science based on dose-response data.
Comment: Commenter (12715) states that, although the supplemental proposal requires that data be available in a manner that allows reanalysis, it does not require that EPA perform a reanalysis of the data of every study that is used as the basis for decision-making, and no criteria for determining when a reanalysis would be performed are provided. The commenter states that, therefore, EPA could selectively decide whether to rely on a reanalysis based on how the results of the reanalysis compare with the original analysis performed by the researcher and EPA's preferred outcome. The commenter notes that it is not clear whether the sitting Administrator -- a political appointee -- could require independent validation of studies that he or she finds objectionable. The commenter suggests that all of this is contrary to the proposed rule's stated goal of increasing transparency.
Response: The rule does not require that reanalysis be conducted nor does it designate that a reanalysis be given preferential treatment as pivotal science. Any reanalyses would be considered with other information provided to EPA as it develops significant regulatory actions and influential scientific information.
<IP_ADDRESS>.5 Reanalysis Requirements May Restrict Use of the Best Science
Comment: Commenter (11392) states that the proposed definition for "reanalyze" does not address how EPA would incorporate any completed reanalyses into its policymaking. The commenter adds that a number of critical questions remain undiscussed in the supplemental proposal: Would the Agency be required to consider a reanalysis alongside the original study, or would it pick and choose analyses to consider? Will the Agency publicly explain its decision to incorporate a reanalysis into a work product? How would the Agency compare a reanalysis to the original study? Will a reanalysis be required to go through peer review? If a reanalysis does not undergo peer review, would it receive equal consideration to an original peer-reviewed study, or would the Agency consider the absence of peer review as a factor in determining the value of the reanalysis? Would the Agency be compelled to delay the finalization of a rulemaking until the completion of a relevant reanalysis? The commenter states that all of these questions represent important practical considerations for the Agency. The commenter observes that if the Agency affords equal weight to any non-peer reviewed reanalysis as it does for a peer reviewed study, it calls into question whether Agency rulemakings are truly considering the best available science. The commenter notes that this also introduces questions about how EPA is satisfying the Information Quality Act, which requires that all significant regulatory information from federal agencies be peer reviewed, as well as EPA's own information quality guidelines.
Response: If a reanalysis of a study were considered pivotal science, it would need to be peer reviewed consistent with 40 CFR 30.6. Nothing in this rule requires independent validation through reanalysis, nor does the rule specify giving greater weight to the reanalysis of studies. Thus, the rule does not require that the timing of a significant regulatory action or influential scientific information be dependent on independent validation through reanalysis of pivotal science. This rule does not require that a reanalysis be considered alongside the original study.
Comment: Commenter (11395) states that reanalysis of data is only one of several mechanisms available for validating studies. The commenter notes that the scientific community uses numerous tools to validate studies, including rigorous peer review, replication of a study using the same methodology but with different data sources, or reproduction of a study's conclusions using different methodologies and data, and notes that many of these tools do not require access to all underlying study data. The commenter states that under the proposed rule, a peer reviewed study that has been replicated many times by different investigators using different data may be excluded from consideration, while one that uses an inferior database that is publicly available would be included.
Response: There is no categorical exemption for pivotal science for which the underlying dose-response data are not available. Further, as discussed in Section III.E of the preamble to the rule, in determining the consideration to give pivotal science for which the underlying dose-response data are not available, EPA will consider a range of factors in addition to the availability and the quality of the study.
<IP_ADDRESS>.6 Tiered Access and Restricted Data Issues Related to Reanalysis
Comment: Commenter (11921) states that although the proposed supplemental proposal acknowledges that personal information may not be able to be protected in all cases, and therefore "tiered access" to these data may be suitable, it is critical to recognize that while "depersonalized" data sets can be created, in many instances they will not allow for full replication and reanalysis  -  and could lead to incomplete and misleading results. The commenter notes that some have argued that it should be possible to create a "depersonalized" data set by stripping all personal identifiers such as address, date of birth, medical history, etc. and making such a data set widely available. The commenter suggests that it is not possible to conduct a high-quality air pollution and health study without knowing the locations of those being studied, i.e. where they live, the sources and levels of their air pollution exposure, as well as their specific covariates, and specific health outcomes, and unfortunately, should such information be made available at smaller spatial scale, it is possible to disclose extensive personal and medical information for individual study subjects, raising privacy and ethical concerns.
Commenter (12720) states that the claim that publicly available dose response data and models would allow for independent validation stands in direct contradiction to the legal privacy protections that apply to key data necessary for re-analysis. The commenter states that the proposed partial redaction of sensitive information poses a cascading set of problems, because the statistical models characteristic of epidemiologic investigations rely on the inclusion of potentially confounding variables (e.g., age, sex, home address, health status, diet and alcohol consumption, smoking history) in order to properly isolate the pollution-health relationship with precision. The commenter states that these studies analyze detailed health, demographic, spatial, and behavioral information from thousands of people to understand the dose-response connection and this information is extremely sensitive and collected at the individual level. The commenter notes that as such, our nation's health privacy laws and IRB protocols require researchers to keep the data secure and confidential to prevent misuse. The commenter states that collectively, these data points help researchers understand and isolate the cause-effect relationship between exposure to air pollution and risks for various health problems and notes that it would be extremely difficult if not impossible for anyone using partially redacted data sets derived from epidemiologic cohort studies to "validate" the results of the original studies, because such investigators would not be working with complete data sets. The commenter adds that as further demonstration, the 2009 Integrated Science Assessment for PM2.5 notes that "[a]ppropriate statistical adjustment for confounders requires identifying and measuring all reasonably expected confounders," and, therefore, exclusion of some potentially sensitive confounding variables from an underlying dataset likely would lead a different team of investigators to a different result. The commenter asserts that causing this wrongheaded and indefensible outcome results from the core approach and conceit in the Proposal, revealing it to be yet again, arbitrary and capricious and an abuse of EPA discretion. The commenter states that, put another way, the quantitative findings of dose-response relationships would almost certainly differ -- not as a result of any true difference in the quantitative exposure-effect relationship, but because the original work relied on complete data sets and the new analyses would not -- due to the Proposal. The commenter concludes that the resulting discrepancies in quantitative findings could serve as motivation to call the original study results into question due to faulty and incomplete re-analyses.
Response: Nothing in this rule require the disclosure of personally identifiable information (PII) or other sensitive information. Rather, this rule describes how EPA will handle studies based on whether the underlying dose-response data are available. Further, as stated elsewhere, this rule does not require that EPA or any other party conduct reanalysis of the dose-response data underlying pivotal science before EPA can use the data.
Comment: Commenter (11900) questions who will be able to access restricted data. The commenter states that the Supplemental Notice described the restricted access tier as "limited to authorized researchers and not possible for the general public." The commenter notes that EPA does not define a process for vetting applicants or criteria for granting access. The commenter states that restricted access must be limited to researchers with a legitimate scientific interest in the data, otherwise restricted access is simply `accessible to the public by application.' The commenter states that the Research Data Center (RDC) run by the National Center for Health Statistics (NCHS)/CDC -- identified as a possible model by the Supplemental Notice -- requires researchers "submit a research proposal outlining the need for restricted-use data." The commenter notes that a stated primary goal of the Supplemental Notice in making data available is to support reanalysis of study data. The commenter suggests that it is unclear on what basis applications seeking to perform reanalysis would be judged, because reanalysis is not typically considered research. The commenter recommends that EPA clearly define how applications to access restricted-access data will be evaluated.
Response: EPA does not intend to make all dose-response data underlying pivotal science available for independent validation. There may be instances where EPA does not own the dose-response data, lacks access to part or all of the dose-response data or does not have the authority to provide access to part or all of the dose-response data. Rather, this rule describes how EPA will handle studies based on whether the underlying dose-response data are available. Further, as stated elsewhere, this rule does not require that EPA or any other party conduct reanalysis of the dose-response data underlying pivotal science.
Comment: Commenter (12715) states that the supplemental proposal acknowledges that, in some cases, the data and models used in a reanalysis would not be available to the public (e.g., instances where EPA does not own the data and models, lacks access to part or all of the data and models, or does not have the authority to provide access to part or all of the data and models). The commenter suggests that this could result in EPA using a reanalysis, its own or that of another party, which differs from the analysis provided in the original publication of the study, and the data and models underlying this reanalysis would not be available to the public. The commenter states that this conflicts with EPA's stated goal of increasing transparency.
Response: In determining whether the underlying dose-response data are publicly available in a manner sufficient for independent validation, EPA will consider whether availability is such that a subject matter expert who has not contributed to the development of the study can evaluate whether results similar to those reported in the study are produced.
3.2.3 Independent Validation
<IP_ADDRESS> Definition of "Independent Validation"
Comment: Commenter (11192) states that the definition the EPA introduces for "independent validation" narrows validation by using the same data and the same methods to see if similar results are produced. The commenter suggests that while one can deduce that independent validation is a subtype of reanalysis, that isn't detailed in the supplemental proposal, and could lead to confusion.
Response: For the purposes of this rule, independent validation is not a subtype of reanalysis. Independent validation may be the result of reanalysis as it is defined in this rule.
Comment: Commenter (11405) states that the term "robustness" is not defined in the supplemental proposal; based on the examples it requires qualitatively similar findings using the same data but different analytical methods. The commenter states that because it involves different analytical methods and only qualitatively similar findings (not quantitatively identical outputs), robustness is not the same as reproducibility; it is closer to replicability. The commenter notes that, as defined in the supplemental proposal, the purpose of "independent validation" is to check reproducibility "using identical methods". The commenter concludes that, using these definitions strictly, an independent validation should not check for robustness by varying the methods.
Response: While robustness was used in the preambles of the 2018 proposed rule and the 2020 SNPRM, there is no requirement for robustness in this rule. Therefore, a definition of robustness is not included in 40 CFR 30.2.
Comment: Commenter (11381) states that the supplemental proposal's new term "independent validation", is unhelpful. The commenter notes that it focuses only on the adjective (independent) and ignores the noun (validation). The commenter adds that it conflates fundamentally different concepts: the "reanalysis of study data" and the independence of such a reanalysis ("by subject matter experts who have not contributed to the development of the study"). The commenter concludes that the definition permits obvious conflicts of interest to be ignored which could occur when the "subject matter experts" involved are not genuinely independent, or when they are employees of the Agency proposing to rely on study data as pivotal scientific information. The commenter states that while internal Agency reanalysis may not be sufficient, there is no rational basis for excluding it from a transparency rule.
Response: EPA disagrees that the definition does not focus on the noun "validation". The concept of validation is captured by the phrase "to evaluate whether results similar to those reported in the study are produced." The use of the term "independent" refers to being independent from the original study, that is by subject matter experts who have not contributed to the development of the study.
Comment: Commenter (12471) suggests clarifying the definition of "independent validation" as follows (underlined text is new text and struck out text is deleted text):
      Independent validation means the reanalysis of study data by subject matter experts who have not contributed to the development of the original study to demonstrate that the same analytic results observed reported in the study are capable of being substantially reproduced.
Response: "Reported" is more appropriate than "observed" because the reanalysis would be conducted by someone other than the one who conducted the original study. The person who conducts the reanalysis can only compare her/his results to what the original researcher(s) reported, not what was observed.
Comment: Commenter (11381) states that the supplemental proposal cites some external authorities approvingly for definitions, including for example National Research Council (NRC) (2007) and NAS (2016). The commenter suggests that the 2007 report offered promiscuous advice on validation but never defined it, and the 2016 report did not address validation at all. The commenter notes that other NRC reports have addressed validation but were not cited in the supplemental proposal, including NRC (2012) and NAS (2017). The commenter states that the 2012 report is a 145-page treatise on verification, validation, and uncertainty quantification (VVUQ) that contains a definition that EPA could have adopted and the 2017 report included a definition of validation that is consistent with the information quality paradigm; it utilizes reproducibility and replicability (as configured in the Information Quality Guidelines (IQGs)) in the pursuit of reliability and accuracy. The commenter asserts that it is therefore hardly surprising that the Science Advisory Board (SAB) was spun up about the Agency's proposal to mandate validation without defining it.
Response: As discussed in the 2020 SNPRM, EPA evaluated the literature in addition to the NAS and NRC reports to aid in developing appropriate definitions. EPA also considered the many public comments received on the terminology used in the 2018 proposed rule, most notably the comment on replicate. Other than noting that there are other documents which have definitions for related terms which EPA could have adopted, this comment does not provide constructive comment on how to improve the proposed definitions.
Comment: Commenter (12720) states that EPA's definition of "independent validation" in the supplemental proposal is inconsistent with the remainder of the proposal because it restricts the concept of "independent validation" to "subject matter experts who have not contributed to the development of the study." The commenter states that EPA restricts "independent validation" to an activity only conducted by "subject matter experts," a distinction in conflict with EPA's definition of "publicly available," which is more expansive and as such, it is unclear whether validation of study data by any member of the public is consistent with EPA's definition. The commenter notes that EPA fails to define "subject matter experts" nor explain how expertise is to be evaluated.
Response: EPA disagrees that there is a conflict. A subject matter expert is not someone who deals only with sensitive data. A subject matter expert is someone who has the education, experience and knowledge of a particular area or subject. The rule does not restrict activities of the public or subject matter experts. The sections cited by the commenter are definitions. The definition of independent validation includes subject matter expert because a subject matter expert would know for the particular type of dose-response study the appropriate methods to use to reanalyze the data to evaluate whether results similar to those reported in the study are produced. The definition of "publicly available" describes the availability of data and not the persons who might conduct independent validation.
Comment: Commenter (12720) states that the SAB's criticism underscores EPA's failure to adequately define terms in the Supplemental Proposal within the context of its scientific activities. The commenter states that, for example, the Supplemental Proposal makes no mention of internal and external validity. The commenter states that the proposal does not make any mention of the need to assess study validity, nor does it distinguish between internal validity (reduction of error and bias) and external validity (applicability of results to other populations, exposure settings, or time periods). Commenter (12720) states that it is highly problematic that EPA has not adequately defined "independent validation" within the proposal nor outlined clear criteria for assessing whether or not a study can be independently validated. The commenter adds that for meta-analyses and other studies that rely on multiple component studies, it is not clear what threshold would have to be met for the overall study to qualify as one that can be independently validated. The commenter asserts that the proposal does not address these difficulties and should be withdrawn.
Response: The determination of whether the dose-response data underlying a study will be available for independent validation will depend upon the specific study and the parameters of that study. Note that the rule does not require any entity to perform independent validation, including EPA. The threshold would be that a subject matter expert would have the dose-response data of the appropriate stage of data to be able to reanalyze the data.
<IP_ADDRESS> Issues Associated with Independent Validation
<IP_ADDRESS>.1 Implications for Researchers
Comment: Commenter (12727) states that EPA has not identified what information would be needed for independent validation of different areas of research. For example, the commenter suggests that the Final SAB Report raises additional points regarding the deficiencies in EPA's proposed definition for independent validation, highlighting the significant ambiguity that remains and insufficient attention to other, related considerations:
      EPA's supplemental proposal indicates that independent validation means the reanalysis of study data by subject matter experts who have not contributed to the development of the study to demonstrate that the same analytic results reported in the study are capable of being substantially reproduced. However, the specific definition of independent validation drives the feasibility of whether EPA can make data and models available for independent validation. For example, the EPA should consider the following questions: How much information is sufficient for independent validation? Will this information consist of equations where reviewers can verify the math, more detailed models where assumptions and limitations are described, or code to allow the public to evaluate and run the models if desired? Is this information simply the dose-response data for the endpoint of concern driving a regulatory limit, or is it availability of all data from a pivotal study to allow reviewers to examine the potential contributions of other variables on the primary endpoint of concern? Endpoint data are seldom evaluated in isolation so providing sufficient study information to allow an independent assessment seems important to meet the goals of the Proposed Rule. For example, an effect on pup body weights in a toxicology study should be examined with knowledge of maternal gestational body weight gains, litter size, food consumption, maternal/litter clinical signs, etc. Sample size and variability also play a key role in data interpretation.
Response: The determination of whether the dose-response data underlying a study will be available for independent validation will depend upon the specific study and the parameters of that study. The threshold would be that a subject matter expert would have the dose-response data of the appropriate stage of data to be able to reanalyze the data.
Regarding endpoints, EPA agrees that endpoint data are seldom evaluated in isolation. This rule would not require that the dose-response data be only for the endpoint. It would be the dose-response data underlying the study.
Comment: Commenter (11921) states that a requirement that any pivotal study shall ensure that "data and models are available in a manner sufficient for independent validation" implies a complex and potentially costly and time-consuming set of steps that include making available two detailed and complex sets of information:
 The workflow that was implemented to create the analytical data set used for the analysis (e.g. harmonized in time and space) from the original and often heterogenous data sets (individual electronic health records, gridded exposure data, confounders etc.), and
 The data analysis (statistical software) that was used to analyze the analytical data set.
The commenter states that creation of both of these in a manner that enables even highly-qualified investigators to reproduce the same results is difficult and time consuming. The commenter suggests based on HEI's hands-on experience that this is likely to be costly to the researchers and their institutions  -  both in the creation of the data and methods hubs, and in the continuous requests for further explication from those seeking the data. The commenter notes that in contrast to the proposal's statement in response to Executive Order 13771: Reducing Regulations and Controlling Regulatory Costs: "This action is not expected to be an Executive Order 13771 regulatory action because it relates to ``Agency organization, management or personnel,'' the proposal would clearly impose substantial additional costs on a wide range of research institutions.
Response: The EPA disagrees that this rule would impose costs on third-party researchers. This is a rule of internal procedure that does not impose requirements on any party other than the EPA. This rule imposes no costs on researchers or their institutions, and the EPA will consider and evaluate all relevant and appropriate science in its significant regulatory actions and influential scientific information. The EPA recognizes that researchers and other third parties may voluntarily consider the EPA's position on data availability, as described in this rule, as they make their own decisions about how to conduct research and the extent to which they make data available. Researchers may choose to make more data available, but the EPA recognizes that these parties will weigh their own benefits and costs and make choices that they deem appropriate.
Comment: Commenter (12430) asserts that the supplemental proposal would require publishing data so thoroughly redacted as to be effectively useless for validation.
Response: EPA disagrees with the commenter that the rule would require that thoroughly redacted data be published. This rule does not include requirements for publishing studies or data.
Comment: Commenter (12422) states that some of the findings on which our public health protections are based are the result of studies done across the globe. The commenter suggests that it is unlikely that Canadian or European scientists would turn over their raw data to a U.S. Agency for "independent validation."
Response: Nothing in the rule requires scientists to provide raw data to EPA for independent validation. Further, the rule does not require any entity, including EPA, to conduct independent validation of studies
<IP_ADDRESS>.2 Conflicting Conclusions
Comment: Commenters (11192, 11361) state that a crucial piece of omitted information in the supplemental proposal and the proposed rule relates to the complete lack of information provided about what could happen if an attempted independent validation conflicted with original study results. The commenters suggest that this obvious omission introduces another way in which this proposed rule could be manipulated and result in the dismissal of credible scientific studies. The commenters state that EPA omits this pivotal information in the proposed rule and the supplemental proposal.
Response: The rule does not require that EPA or any other entity conduct independent validation before EPA can use pivotal science. Nor does it establish requirements for the use of the independent validation if it is conducted.
<IP_ADDRESS> Current Processes Do Not Need Data to Be Publicly Available for Validation
Comment: Commenter (11479) states that EPA already independently validates the research without using underlying data by asking the following questions:
      a) To what extent has there been independent verification or validation of the study method and results? What were the conclusions of these independent efforts, and are they consistent? b) To what extent has independent peer review been conducted of the study method and results, and how were the conclusions of this review taken into account? c) Has the procedure, method or model been used in similar, peer reviewed studies? Are the results consistent with other relevant studies? d) In the case of model-based information, to what extent has independent evaluation and testing of the model code been performed and documented?
The commenter states that these guiding questions ensure that the Agency does not have to fully reanalyze every study and can instead focus on the weight of the evidence approach. The commenter argues that including data availability in the weighting injects an arbitrary parameter, unrelated to study quality and reliability.
Response: The rule complements the activities that EPA currently conducts by giving greater consideration to studies for which the underlying dose-response data are available for independent validation. For studies in which the dose-response data are available for independent validation, this will increase the transparency of EPA decision making in significant regulatory actions and independent scientific information.
Comment: Commenter (12715) states that independent validation and reanalysis can often be conducted with more processed form of data than the stringent stage of data EPA is requesting. Instead, the commenter suggests the EPA justify the rule's requirements in specific contexts and for specific types of data and models to avoid eliminating studies where access to less processed data would jeopardize privacy or lead to logistical challenges for data sharing.
Response: EPA has defined "data" in part to mean "the set of recorded factual material commonly accepted in the scientific community as necessary to validate research findings." Thus, the stage of data that is available for independent validation would be that stage of data necessary to accomplish the independent validation. The rule does not include a categorical exclusion of studies where the underlying data cannot be made available for independent validation.
<IP_ADDRESS> Independent Validation Does Not Imply or Improve Validity or Accuracy
Comment: Several commenters contend that independent validation does not imply or improve validity or accuracy and therefore there is no justification for this rule.
Commenter (11176) states that independent validation as EPA defines the term in the supplement is a rather narrow method of evaluating research and assessing a scientific conclusion that could ultimately hamstring the work of EPA scientists to determine the best science to inform EPA work. The commenter suggests that the focus on independent validation puts undeserved emphasis on an individual study and that such a focus is contrary to the science community's practice of generally not being interested in a specific study but rather on its underlying hypotheses. The commenter states that no hypothesis can be conclusively addressed by an individual study; instead, a hypothesis' validity should be assessed on a body of evidence.
Response: While EPA agrees that the body of evidence is important, this rule does not conflict with the importance of transparency of the dose-response data that EPA uses in significant regulatory actions and influential scientific information. The rule does not require any entity, including EPA, to conduct independent validation of studies before EPA can use those studies in significant regulatory actions and influential scientific information.
Comment: Several commenters (11176, 11496, 11399, 12727) contend that independent validation does not imply or improve validity or accuracy and therefore there is no justification for this rule.
Commenter (11176) states that the proposed rule also seems to assume that a study that can be independently validated is of higher quality. The commenter argues that there is no evidence to support such an assertion. The commenter notes that Leek and Peng discussed in further depth the lack of a connection between research that has been reproduced and quality in a 2015 article in Proceedings of the NAS. The commenter states that EPA also acknowledged this point in the document, Plan to Increase Access to Results of EPA -Funded Scientific Research where EPA states "Whether research data are fully available to the public or available to researchers through other means does not affect the validity of the scientific conclusions from peer-reviewed research publications." Commenter (11186) states that independent validation as described is not a sound method for assessing accuracy. The commenter notes that changes in statistical methodologies for the same sample can result in a different research question answered and this practice will undermine an established practice that scientists already use: the peer-review process.
In addition, commenter (11496) states that the need for independent validation is not justified. The commenter states that the most important of its concerns is EPA's stated desire to exclude any pivotal regulatory science which cannot be "independently validated" beyond the existing and well-established peer review process. The commenter asserts that this is an inherently flawed perspective on how science, peer review and weight of evidence are used to protect public health. The commenter states that if EPA identifies peer-reviewed pivotal science that is insufficient, EPA or partner agencies like the Agency for Toxic Substances and Disease Registry or National Institute of Health should fund follow-up confirmation studies by independent researchers, not seek to disclose personal information to third parties tasked with independently reanalyzing existing datasets. The commenter requests that EPA articulate why independent validation of peer-reviewed science is even necessary for regulatory decisions; this type of redundant statistical analysis has never been a requirement of the OMB or any other federal Agency. Commenter (12727) states that the definition of "independent validation" suffers from EPA inappropriately conflating "validation" with "reanalysis." The commenter states that validation of scientific data is not accomplished by simply independently reanalyzing the data. The commenter notes that false or inappropriately obtained data will be invalid regardless of whether it can be independently reanalyzed. The commenter suggests that reproducing study results using a different population or method is generally considered a stronger approach to validation, rather than simply reanalyzing the results using the same data, as it shows that the results hold across different populations and different study designs.
Commenter (11399) states that additional "independent validation" that goes above and beyond the peer review process and EPA existing science advisory infrastructure is not necessary to assure scientific validity and accuracy. The commenter argues that this new procedural hurdle limits the universe of scientific studies that EPA may consider. The commenter asserts that creating such an impediment undermines EPA's legal obligation to use the best-available science to guide its decision-making processes and portends a weakening of the scientific underpinnings of health-based environmental regulations. Commenter (12427) states that scientific rigor depends on elements of scientific inquiry and judgments regarding the validity, value, and utility of science. The commenter states that the elements of scientific inquiry include data, methods, logic connecting data, and methods to conclusions. The commenter states that to judge validity, value, and utility of science, it is necessary to know how one got the answer. The commenter states that adequacy for "conclusion-making" occurs when scientific evidence supports a sufficient degree of certainty that rules out alternative explanations and that "Transparency" is not a determinant of scientific rigor. The commenter argues that judgments regarding validity, value, and utility of science can be and are made via peer review not through the approach taken in this rulemaking.
Response: Although the EPA agrees with commenters that meaningful insights can be obtained through similar studies in different media or context, the EPA continues to find that independent validation of the study findings and conclusions driving the EPA's dose-response assessments would provide important information. The benefits of greater data transparency and the significance of reanalyzing and validating study results are well-documented in scientific literature. McNutt (2014) noted, "reproducibility, rigor, transparency, and independent verification are cornerstones of the scientific method." The National Academies of Sciences, Engineering, and Medicine (NAS) workshop on Reproducibility and Replicability in Science also noted that "certainly, reproducibility and replicability play an important role in achieving rigor and transparency." Munafò et al. (2017) state, "the credibility of scientific claims is rooted in the evidence supporting them, which includes the methodology applied, the data acquired, and the process of methodology implementation, data analysis and outcome interpretation. Claims become credible by the community reviewing, critiquing, extending and reproducing the supporting evidence. However, without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator. Transparency is superior to trust." The EPA continues to believe that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will improve the opportunity for the public to access the EPA's scientific analyses and resulting regulatory actions in a way that is beneficial to the scientific process, the Agency's mission, and the public's health and safety.
This rule is designed to build upon OMB M-13-13. It is also consistent with OMB M-19-15, which highlights the need to characterize the sensitivity of an agency's conclusions to analytic assumptions, as well as other federal guidance documents that require greater data transparency. The EPA's attention to data transparency is also responsive to the broader interest in greater data and model transparency observed in the numerous transparency initiatives in the scientific community and federal government, as well as the criticism the EPA has received from members of the public, scientific community, and Congress on the transparency of the scientific basis for EPA's decisions in previous influential scientific information assessments and regulatory actions.
The scientific community has also embraced greater data transparency, as evidenced by data sharing and availability requirements for many high-impact journals (Ref. 15) and the emergence of organizations, such as the Center for Open Science, and international initiatives like Findable, Accessible, Interoperable, and Reusable (FAIR) data principles; Facilitate Open Science Training in European Research (FOSTER); and Guidelines for Transparency and Openness Promotion (TOP) in Journal Policies and Practices that incentivize greater transparency in research.

.

 Chapter 4: Data and Models
Section 30.2 of the notice of proposed rulemaking (NPRM) defines dose response data and models as follows:
      Dose response data and models means the data and models used to characterize the quantitative relationship between the amount of dose or exposure to a pollutant, contaminant, or substance and the magnitude of a predicted health or environmental impact. Such functions typically underlie pivotal regulatory science that drives the size of benefit- cost calculations, the level of a standard, and/or the points of departure from which reference values (reference doses or reference concentrations) are calculated.
Section 30.5 includes the following requirement:
      When promulgating significant regulatory actions, the Agency shall ensure that dose response data and models underlying pivotal regulatory science are publicly available in a manner sufficient for independent validation. Where the Agency is making data or models publicly available, it shall do so in a fashion that is consistent with law, protects privacy, confidentiality, confidential business information (CBI), and is sensitive to national and homeland security.
In addition, the NPRM (§30.6) states that, when available, EPA shall consider high quality studies that explore:
 Appropriateness of a linear, no-threshold dose response;
 A broad class of parametric dose-response or concentration-response models;
 A robust set of potential confounding variables;
 Nonparametric models that incorporate fewer assumptions;
 Threshold models across the dose or exposure range; and
 Models that investigate factors that might account for spatial heterogeneity.
The NPRM (§30.6) also requires EPA to describe and document any assumptions and methods used and should describe variability and uncertainty.
In the SNPRM, the EPA proposes revising the NPRM regulatory text at 40 CFR 30.3, 30.5, 30.6, and 30.9 so that the provisions would apply to data, not only dose-response data. The SNPRM also proposes to delete the definition of "research data" and to add a definition for "data" at 40 CFR 30.2 as follows:
      Data means the set of recorded factual material commonly accepted in the scientific community as necessary to validate research findings in which obvious errors, such as keystroke or coding errors, have been removed and that is capable of being analyzed by either the original researcher or an independent party.
The SNPRM proposes a broader applicability with this definition as transparency of EPA's science should apply to other types of data that drive the requirements and/or quantitative analysis of EPA final significant regulator decisions and influential scientific information. The SNPRM specifically requests comment on whether the definition of "data" should apply to another stage of data and on whether there is another definition of "data" that should be considered.
Regarding the NPRM, several comments were received concerning dose response data and model proposal requirements, including the requirements of §30.5 and §30.6. Additionally, several comments were received regarding data and the use of data within the SNPRM. In this Chapter, these comments are organized under sections 4.1 NPRM Data and Models  -  Related Comments and 4.2 SNPRM Models, Variability/Uncertainty, and Assumptions/Defaults.
Section 4.1: NPRM Data and Models  -  Related Comments
 Data and Models; Dose response data and models (§30.2), requirements to the use of dose response data and models underlying pivotal regulatory science (§30.5), and additional requirements (§30.6)
 Definitions of Research Data (§30.2) and Data
Section 4.2: SNPRM Models; Variability/Uncertainty, and Assumptions/Defaults
          Modeling Definitions
          Models (General) and Best Science Practices
          Public Availability of Models/Data
          Weighting of Models
          Scope of Models Included
          Assumptions/Defaults-Variability/Uncertainty
          Impacts on Older/Existing Models
          Linear vs Nonlinear Modeling
          Impacts on Specific Models
Section 4.3 SNPRM Data, Variability/Uncertainty, and Assumptions/Defaults
          Clarity of Proposed Definitions
          Application of the Rule in Respect to Data
          Linear vs. Nonlinear Models
          Models (General) and Best Science Practices
          Assumptions/Defaults-Variability/Uncertainty
The specific comments provided in this Chapter reflect the comment letters that provide arguments supported by detailed rationale, examples, and references/documentation. The Docket ID numbers for these commenters are cited throughout this Chapter. In addition, some general comment letters may also include comments related to comments included and responded to in this Chapter. These comments are summarized in Appendix B. Note that points and issues raised in these general comment letters may also be raised by the more detailed comments summarized in this Chapter. The responses provided in this Chapter adequately address these more general comments.
4.1 NPRM Data and Models  -  Related Comments
4.1.1 Data and Models; Dose response data and models (§30.2), requirements to the use of dose response data and models underlying pivotal regulatory science (§30.5), and additional requirements (§30.6)
<IP_ADDRESS> Dose Response Data
<IP_ADDRESS>.1 Availability of Underlying Data
Support the Proposed Rule Requirements that Data Be Made Publicly Available
Comment: Commenter (6369) asserts that data access is crucial to reproducibility when conducting independent validation of EPA results as outlined in the proposed rule. The commenter adds that public access to information that the Agency deems critical to future or pending regulations is vitally important to the scientific process. The commenter also asserts that timely access to that information is just as important and provides examples to demonstrate how the lack of availability of underlying science used in decision-making can have significant impacts on EPA's chemical assessment conclusions.
For example, commenter (6369) states that they engaged directly with EPA's Integrated Risk Information System (IRIS) program to better understand the scientific information being relied upon to draw conclusions regarding formaldehyde carcinogenicity and had considerable difficulty in not only communicating with the IRIS scientists on the data that supports the program's conclusions regarding the carcinogenicity of formaldehyde and accessing the underlying data from studies that the IRIS program used in its draft chemical assessment. The commenter provides a chronology detailing their efforts and notes that, overall, it took six years to obtain all the necessary underlying data for evaluation/validation of the results from this study. The commenter provides that an evaluation of the study results determined that the changes reported by the study authors to be associated with formaldehyde exposure were not exposure dependent. The commenter contends that this highlights the critical need for public availability of results from studies that are key not only in dose-response modeling, but also in drawing conclusions regarding the potential health effects associated with exposure to formaldehyde. The commenter adds that data obtained via a Technology Transfer Agreement with NCI in 2014 to obtain information related to the Bean-Freeman (2009) study that was relied upon by the IRIS program to conduct dose response modeling to determine the potential for carcinogenic effects following formaldehyde exposure took several years before they were able to obtain the data. The commenter states that it is important to note that, in this case, the scientists gained access to underlying data related to individual subjects in a cohort. According to the commenter, this was accomplished while still allowing the protection of personal identifiers and other confidential information. Thus, the commenter concludes that, while concerns over confidential information are valid, they can be addressed while allowing transparency of critical scientific data.
Response: Categorically excluding studies because the underlying dose-response data are not publicly available would exclude valid studies. Thus, rather than categorically excluding studies, EPA will give greater consideration to studies where the underlying dose-response data and models are available for independent validation.
 Types of Studies and Examples for which Underlying Data May Not Be Available/Accessible
Other Federal Agency Data
Comment: Commenter (6373) suggests that the EPA state whether it will decline to consider scientific studies that are based on data collected by federal agencies that require privacy protection. The commenter notes that EPA research and studies that inform EPA regulatory activities often rely on data collected by other federal agencies, including but not limited to Medicare and Medicaid, the National Institute of Health (NIH), the Census Bureau, and the Social Security Administration. The commenter asserts that the Rule will not bind these agencies' data collection and research policies and practices.
Response: EPA will consider these studies if it determines they are pivotal science but will give them greater consideration if the underlying data and models are available for independent validation.
Old Studies - Original Data Does Not Exist or is No Longer Accessible
Comment: Commenters (1945, 4596, 5176, 6125, 6362, 6873, 6898, 6914, 8317) provide that, for older studies, authors or data and information sources may no longer exist, or may not be accessible.
Commenter (4596) asserts that EPA regulations that reduce exposure to pollutants and chemicals that are toxic to the developing brain have for decades relied on studies that have included data and information that was not made fully public, including longitudinal epidemiologic studies, human clinical studies, and animal toxicologic studies. In many cases, the commenter provides that the original raw data does not exist or is no longer accessible.
Commenter (6914) notes that epidemiological evidence and other accounts related to work-related illnesses and toxic releases often come from older, yet meaningful studies and sources, where the authors or data sources may not be accessible.
Commenters (1945, 5176) contend that there are legitimate reasons why data underlying a study's findings are not readily publicly available but should not be excluded, including reasons related to the technology available at the time of data collection, outdated formats in which data sets from older studies are stored. Commenter (6898) states that the Proposed Rule could allow the elimination of historical data sets that are not stored in currently used electronic formats. The commenter adds that the availability of historical public data is essential to scientists' understanding of the health impacts of environmental exposures over time.
Commenter (6125) states that, because many of the foundational lead studies analyzed children with higher exposure and blood lead levels than are commonly seen today, it would be unethical to run the same tests today, exposing children to levels of lead that we know are unhealthy. The commenter provides that, due to numerous factors related to their age, it may not be possible to recover the original data. The commenter adds that, if use of those older studies were successfully challenged because they do not meet the data requirement policy, they could not be used. The commenter contends that their exclusion would weaken the scientific support for EPA regulations that have clearly proved their worth in the dramatic declines in blood lead levels in American children.
Response: This rule does not categorically exclude any study. The rule requires that EPA give greater consideration to pivotal science where the underlying dose-response data are publicly available in a manner sufficient for independent validation. Thus, these studies will still be considered.
Confidentiality Requirements Legally Mandated by Institutional Review Boards (IRBs) and the Need for Reconsent of Privacy Agreements
Comment: Commenters (0017, 1999, 2283, 2909, 4591, 4595, 4596, 4848, 5013, 5015, 5115, 5165, 6111, 6351, 6897, 6905, 6919, 6922, 8277, 8281-PH26) state that, as articulated by numerous science and health organizations, many health studies, especially those that uncover the effects of exposure to toxins and pollutants/epidemiological studies, rely on medical records/location information that are sensitive covered information, and are confidential and private based on federal law and other patient privacy policies (i.e., including confidentiality agreements). Commenters (0106, 2259, 4591, 4596, 5015, 5180, 6873, 6905, 6919) provide that, as with all data (historic or otherwise), there may be circumstances where an individual participant's data cannot be made public because of confidentiality requirements legally mandated by IRBs and Health Insurance Portability and Accountability Act of 1996 (HIPAA). Commenter (6897) provides that, epidemiologic data often involves personal identifiers that can make full disclosure challenging  -  particularly retroactively, if participants have already agreed to be involved under more restrictive disclosure terms.
Commenter (6873) states that IRBs cannot release private data without consent, and getting this consent from study participants where the researcher has long since retired and participants have either died or moved multiple times would be extremely challenging and costly at best, and effectively impossible at worst, therefore possibly rendering "old" studies unusable.
Commenter (6919) provides that current federal and state law both require personally identifiable data to be de-identified due to the HIPAA and Colorado regulations protecting the confidentiality of Vital Statistics birth and death certificate data. The commenter provides that government agencies have a legal responsibility to protect residents from any risks associated with sharing personal health information  -  this includes discrimination. According to the commenter, researchers can only conduct high-quality and reproducible research by protecting the personal health information of study participants. Likewise, public health professionals must retain private personal data in their own study records in order to conduct accurate and useful surveillance to inform local, state, and national policies.
Commenter (5015) states that access to private medical information is essential to conducting high quality and reproducible air quality and health research:
 There are of course longstanding federal rules for protecting the privacy of individual medical information of the subjects of studies (HIPAA, Common Rule, etc.)
 Gaining access to data from older studies may be difficult, given the privacy commitments that were made to study subjects in the past.
 However, there are today several means to make such data available to investigators with appropriate privacy protections (e.g. Medicare, Federal Research Data Centers) and many investigators have been taking advantage of these.
 Although it is possible, as some have suggested, to create a "depersonalized" data set by stripping all personal identifiers, such as address, date of birth, etc., it is not possible to conduct a high-quality air pollution and health study without knowing the locations of those being studied, i.e. where they live, and what are the sources and levels of their air pollution exposure?
Commenter (8317) asserts that there is already a lack of scientific information about the occupational and environmental health risks farmworkers face. The commenter notes that because farmworkers used in occupational and environmental research are often migratory, moving for work across domestic and international borders, researchers may be unable to locate farmworkers they last encountered as study participants years ago, and thus unable to re-negotiate privacy agreements struck at the time the research was conducted. The commenter states that the EPA should base its regulatory decisions on the credibility of scientific evidence, not on arbitrary factors like the public availability of research data.
Response: This rule does not categorically exclude any study. The rule requires that EPA give greater consideration to pivotal science where the underlying dose-response data are publicly available in a manner sufficient for independent validation. Thus, these studies will still be considered.
Comment: Commenter (6125) expresses concern that lawsuits filed based on this proposal might force EPA to roll back current drinking water standards and drinking water health advisories for pollutants such as arsenic and nitrate. The commenter also expresses concern that EPA has not assessed how the requirements in the new policy might affect drinking water standards and advisories. The commenter states that both drinking water standards rely on epidemiological data, including confidential patient information, as well as on older studies. The commenter adds that data from such studies may have been discarded or lost or be in an unreadable form. The commenter adds that none of these studies could realistically or ethically be reproduced since they derive from a unique cohort, and it would be unethical to expose people to ingested pollutants.
Response: If a study is identified as pivotal science, EPA will still consider the study but may give it lesser consideration if the dose-response data are not available for independent validation.
Data Reduction Time Limits
Comment: Commenter (6362) states that, invariably there will be circumstances where underlying data no longer exist for studies and/or models that are high quality and reliable. For example, the commenter provides that most organizations have data retention policies that have resulted in the disposal of underlying data. Furthermore, according to the commenter (6362), Good Laboratory Practices (GLP) regulations include defined periods of time to retain data and study records. The commenter requests that EPA address how it will continue to use those studies and models considering these policies.
Response: If a study is identified as pivotal science, EPA will still consider the study but may give it lesser consideration if the dose-response data are not available for independent validation.
Proprietary Data
Comment: Commenter (4595) provides that some studies may rely on public and private-sector funding sources that limit access to underlying data for proprietary reasons.
Response: If a study is identified as pivotal science, EPA will still consider the study but may give it lesser consideration if the dose-response data are not available for independent validation.
Studies are Often the Property of Scientific Investigators
Comment: Commenter (6922) notes that the Courts have recognized that raw data underlying studies are often the property of scientific investigators and are often not available because of reasonable confidentiality arrangements with study participants.
Response: If a study is identified as pivotal science, EPA will still consider the study but may give it lesser consideration if the dose-response data are not available for independent validation.
Non-U.S. Studies
Comment: Commenter (1973) asserts that if the rule is finalized as proposed, all non-U.S. studies would be legally unable to meet EPA's requirements. The commenter provides that, for example, Canadian Community Health Study data would remain safe behind protected servers at Statistics Canada. Additionally, the commenter states that while the protocols, methods, and all other information necessary to judge the study are available, EPA would nevertheless ignore it. The commenter states that this would be the same for the ESCAPE study, the UK national cohort study, the Netherlands National Cohort study, the Danish National Birth Cohort and the Norwegian Birth Cohort (both funded by NIH), etc. The commenter adds that U.S. cohorts, given a choice between violating their promises in the informed consent documents they signed or their Data Use Agreements, or not making their data public, will choose the latter. The commenter contends that the EPA will achieve no greater "transparency", they will only deny themselves the ability to set standards based on the studies that the rest of the world relies on. The commenter provides that the EPA's own Science Advisory Board (SAB) has commented that this proposed rule" could have the effect of removing legal, ethical, and peer-reviewed studies of health effects as sources to support the agency's regulatory efforts".
Commenter (1657) states that if the proposed rule applies to data on human subjects, it could prevent the EPA from using any human subjects data collected from European Union (EU) member countries because after May 25, 2018 such data will be subject to the EU general data protection regulation (GDPR), which enshrines a "right to be forgotten." According to the commenter, while the results of analyses using EU human subjects data can be archived permanently, after that date the data itself could likely not legally be permanently and immutably archived.
Response: If a study is identified as pivotal science, EPA will still consider the study but may give it lesser consideration if the dose-response data are not available for independent validation.
Comment: Commenter (4593) provides that the U.S. is obliged to accept data generated in accordance with the Organisation for Economic Cooperation and Development (OECD) Test Guidelines and Good Laboratory Practice (GLP) Standards. The commenter notes that the U.S. and other OECD member countries realized that differences in testing requirements among countries meant that companies would in some cases have to retest a chemical in order to market it in other areas. According to the commenter, this was needlessly costly and resulted in a delay in obtaining information needed for regulatory assessment. As a result, the commenters states that OECD member nations agreed to accept for regulatory purposes data generated in accordance with the OECD test guidelines, GLPs and quality assurance program. The commenter adds that GLPs specify the information that must be reported; and underlying data is not one of them. Therefore, the commenter is concerned that the proposed policy which requires underlying data to be made available would run counter to the obligations under the Mutual Acceptance of Data treaty.
Response: If the study is identified as pivotal science, EPA will still consider the study but may give it lesser consideration if the dose-response data are not available for independent validation.
Specific Examples of Studies and Supporting Data Sets Would Be Eliminated from Consideration
Comment: Commenters (6378, 8275) provide two examples of studies that may not be able to be used under the proposed requirements. They are papers from the University of Washington that involved studies that ensure confidentiality of the participants (Miller KA et al, Long-term exposure to fine particulate matter air pollution and cardiovascular events in women. New England Journal of Medicine 2007; 356:447-58; and Kaufman JD et al, Association between air pollution and coronary artery calcification within six metropolitan areas in the USA [The Multi-Ethnic Study of Atherosclerosis and Air Pollution]: a longitudinal cohort study. The Lancet 2016; 388:696-704). Commenter (8275) notes that both studies contain dose response data and models as anticipated in the proposed rule, and for which the proposed rule creates an expectation that data required to replicate the analysis be made available. In each case, the commenter states that the underlying studies are conducted with strict rules under the auspices of the NIH (National Heart, Lung, and Blood Institute), that preclude any potential identification of subject identity or confidential information. The commenter provides that the NIH and the approving IRBs would not permit release of the data in a way that would adhere to the letter of the rule. According to the commenter, it is possible that limited datasets could be created for replication analysis that would protect participant identity and confidential information, but funds are not available to create these datasets, and the rule is not clear that such a limited dataset would be acceptable. The commenter states that, as a result, the rule would lead to arbitrary exclusion of pertinent scientific evidence.
Commenter (4591) notes that data from the landmark "Six Cities" study that established a link between air pollution and human mortality, could not be made publicly available for several reasons, including promises that were made to the study participants at time of enrollment. Notably, the commenter provides that the outcomes of this study have been validated in several scientifically rigorous ways and by several independent studies. Commenter (8281-PH38) states that when the published landmark Harvard Six Cities Study was challenged, the EPA took a logical step and referred them to an independent third-party, the Health Effects Institute, for a deep dive review. The commenter reports that the autonomous reviewers examined the data and developed a report that confirmed the original findings. The commenter notes that there has been other research since then that has confirmed similar findings, including some studies that use publicly available data sets.
Response: If a study is identified as pivotal science, EPA will still consider the study but may give it lesser consideration if the dose-response data are not available for independent validation.
Comment: Commenters provide and mention specific studies (not including the national ambient air quality standards (NAAQS) Supporting Studies) that would likely be impacted if the Proposed Rule were to apply.
Commenter (6170) provided a list of 50 example science references to some of the publications from their research (out of 265) that would not qualify for use in EPA rule making due to privacy concerns for human subjects if the Proposal is finalized.
Commenter (5012) asserts that the Proposal would eliminate consideration of many major studies on the health impacts of environmental pollutants and cites the Children's Health Study conducted in southern California as an example of a study that would probably be eliminated from consideration under the Proposed Rule.
Commenter (6919) notes that Colorado is one of the top-producing states for oil and gas. The commenter provides that, as such, we are observing and recording exposures and possible health effects from oil and gas development that has increased dramatically near residential areas. According to the commenter, under this proposed rule, it is possible that studies derived from data on individual exposures could be suppressed since these studies do or may involve taking water and air quality measurements at people's homes, the use of personal air monitors, possibly even blood and urine samples, and portions of the data collected would need to be redacted to protect the study participants. The commenter asserts that it would be a disservice to public health if valuable information from these types of studies were not considered in rulemaking actions because personally identifiable health data could not be protected under the proposed transparency requirements. The commenter provided one example of a study by the Colorado School of Public Health that evaluated participants ranging in age from 0-24 years to study the rates of childhood hematologic cancer and residential proximity to oil and gas development. The commenter contends that, because oil and gas production is proliferating in Colorado, studies like these are critical for understanding the industry's impacts on public health, including the detrimental effects of elevated ozone levels and associated air toxic emissions along the Colorado Front Range.
Commenter (6111) provides a 9-page partial list of studies that may contain protected health information that have been relied on by the EPA and cited in EPA documents. The commenter adds that there are many EPA rulemakings and decisions that have relied on studies that cannot be replicated and whose data likely could not be made publicly available. For example:
 PCBs: EPA's regulations establishing water quality standards for polychlorinated biphenyls ("PCBs") under the Clean Water Act (CWA) were based in part on long-term epidemiological studies of cancer rates in workers exposed to PCBs.
 Radionuclides: EPA's SDWA regulation for radionuclides relied on epidemiological studies of survivors from the Hiroshima and Nagasaki atomic bomb attacks.
 Particulate matter: EPA's 1997, 2006, and 2012 NAAQS for fine particulate matter all relied on studies using confidential data, such as the Six Cities Study.
 Methylmercury: EPA's reference dose for methylmercury in fish that will be consumed by humans relied on data from human exposures in the Faroe Islands.
The commenter contends that precluding reliance on these and other studies for the sole reason that the underlying raw data has not been or cannot be released to the public is arbitrary, capricious, contrary to professional best practices, and antithetical to protection of public health and safety as required by the statutes.
Commenter (6113) states that scientific evidence concerning the damage caused by pesticides was not generated by studies that were verifiable and public in the sense contemplated by EPA's proposal. (See generally Congressional Research Service, Pesticide Law: A Summary of the Statutes, Nov. 14, 2012 (overview of pesticide law). Little, if any of such evidence was generated by studies that were verifiable and public in the sense contemplated by EPA's proposed Anti-Environment Regulation.) According to the commenter, while much of the data included private medical data protected by privacy laws such as HIPAA, much of the data was private and or difficult to reproduce for other reasons (for example, endangered wildlife surveys may not be able to be repeated without impermissibly harming the wildlife).