Commenter (11358) states that this alternative approach will give the Agency appropriate flexibility when implementing this new policy. The commenter states that the EPA should refrain from banning all studies from being utilized within the regulatory process solely due to data availability. The commenter states that, for data that is not produced by EPA, such as data used for studies, which have been published in scientific journals, the process of peer reviewing may often be sufficient. The commenter states that, as the EPA notes in the SNPRM, raw data may not be available for a variety of reasons including privacy, age of the data, etc. The commenter states that this proposed alternative will ensure that the EPA is in a position to use all available data to inform the Agency's decision-making process.
Commenter (11395) states that, of these two alternatives, Option 2 is preferable because it does not automatically eliminate a wide range of well-conducted studies that cannot comport with the data availability specifications.
Commenter (12406) states that Option 2 is consistent with their 2018 comments, specifically that scientifically relevant and reliable studies should not be eliminated from consideration, but rather, accorded less weight when integrating evidence from multiple studies and across different lines of evidence and that any guidance and other relevant documents developed to assist EPA staff to comply with this rule should include specific examples and/or case studies, perhaps drawing from recent EPA rulemakings, to demonstrate what constitutes regulatory science that is material to EPA's significant regulatory decisions. The commenter recommends EPA revise the regulatory text in Option 2 to omit the word "short" to characterize the narrative that EPA will provide to describe the studies that are provided greater consideration along with its explanation why the studies received greater consideration. The commenter asserts that the Agency should provide sufficient information and explanation as may be necessary in any given circumstance.
Commenter (12471) states that, while the most robust standards for science transparency are clearly desired, the tools to achieve this worthwhile objective may not yet be in place. The commenter states that agencies and research institutions may need time to develop the tools and protocols to allow greater public access at the appropriate tiered access level considering the need to protect privacy and other confidential information. The commenter states that the EPA should note in this rulemaking that it will be revisiting this issue as the appropriate tools are developed, with the objective of increasing transparency and public access as expeditiously as possible consistent with federal law.
Response: The purpose of this action is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions. Where underlying dose-response data in pivotal science are available, subject matter experts could independently reanalyze the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of the uncertainty. Such independent reanalyses will subsequently enable the EPA to make stronger, data-driven decisions in future rulemakings or in revisions to existing rules or influential scientific information. This could occur through standard cyclical reviews (e.g., revisions to NAAQS, risk and technology reviews, national primary drinking water regulations), ad hoc revisions, or revisions through the information quality guidelines or other petition processes. Implementation of this rule will more effectively share pivotal science for external consideration and increase the opportunity for independent validation of pivotal science by subject matter experts. As data are better understood through independent reanalysis, the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
The transparency provisions in this final rule are intended to build upon existing federal government efforts and provide incremental progress toward the Agency's goal of greater transparency. The EPA and the federal government have long encouraged open data initiatives, as the principle of transparency in regulatory decision-making and the other operations of government agencies is a fundamental behavior of good government that is inherently valuable to the public.
Comment: Commenters (11922, 12471, 12612, 12699) support Option 2 but encourage limited use of data and models that cannot be independently validated.
Commenter (11922) supports the SNPRM's proposed modifications to 40 CFR 30.5 that at the very least give extremely strong preference for data and models that can be independently validated. The commenter supports the use of information that cannot be validated only in rare circumstances. The commenter states that studies and data that cannot be validated under Option 2 should be utilized only if adequate studies that can be validated are not available to develop the necessary underlying influential or pivotal information. The commenter states that, in the rare circumstance described above, when information that cannot be validated is used in the decision-making process, the data that cannot be validated should always be afforded less consideration, if any consideration at all, than data that can be validated.
Commenter (12471) recommends the EPA give preference to studies that are fully available to the public if they are of equal (generally commensurate with) or better quality than the non-available studies. The commenter states that the EPA could note that transparency, in and of itself, is a study criterion that enhances study quality by allowing greater independent review and reanalysis. The commenter states that, if the only studies available provide limited or no access to the underlying data or models, the EPA could still consider the data and models in the rulemaking if they are otherwise of sufficient quality, but note the weakness and additional uncertainties inherent in relying on studies that are not available at any protected tiered level.
Commenter (12612) states that, if data and models are not made public, then the study should not be ignored but should be considered secondary to other comparable studies where data and models are made public.
Commenter (12699) states that, when there are genuine privacy, confidentiality, and CBI concerns, the EPA should not exclude studies, but give lesser weight to the studies when underlying information is unavailable. The commenter states that, when the EPA provides a legal justification to not disclose underlying information, there should be clear documentation provided to the public in support of the legal justification. The commenter states that, when there are subjective privacy or confidentiality concerns unrelated to law, this can be tricky to ensure the authors or agency are stating a genuine concern. The commenter states that the EPA should view and treat such assertions with skepticism, and deem the concern to be genuine only when disclosure would violate widely accepted practices or norms. The commenter states that, if the concerns are deemed genuine, then the study should not be excluded,  however, it still should be provided less weight than comparable studies. The commenter states that it should be rare for the EPA to solely rely upon studies without underlying data, and the EPA should clarify in the rule that the Agency shall take all reasonable steps to avoid such an outcome. The commenter states that the EPA should certify in the administrative record of a rulemaking that it did a full literature review and no legitimate studies without such problems exist that could have been used in their place.
Response: The purpose of this action is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions. Where underlying dose-response data in pivotal science are available, subject matter experts could independently reanalyze the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of the uncertainty. Such independent reanalyses will subsequently enable the EPA to make stronger, data-driven decisions in future rulemakings or in revisions to existing rules or influential scientific information.
<IP_ADDRESS>.2 Oppose Option 2
Comment: Commenter (9587) endorses the previous, unmodified transparency requirements, and states that transparency should take priority over privacy and confidentiality in EPA policymaking. The commenter states that the EPA's proposed modification gives claims of privacy and/or confidentiality priority over transparency requirements. The commenter states that this modification practically will impose substantial limits on the effect of the rule since privacy, confidentiality, and tiered access are all concepts and practices that inhibit full transparency.
Commenter (12712) states that the SNPRM's "consideration" to studies based on data that is neither publicly available nor available through tiered access is too vague. The commenter states that no such study, by definition unavailable for reanalysis by independent researchers, should even be a factor in an EPA decision to regulate.
Response: The EPA is finalizing a rule of internal agency procedures, including how the Agency will consider the availability of dose-response data underlying pivotal science used in its significant regulatory actions and influential scientific information for independent validation. The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Comment: Commenters (11182, 11362, 11390, 11393, 11396, 11407, 11489, 11576, 11890, 11895, 12404, 12412, 12442, 12463, 12474, 12715, 13549) oppose Option 2.
Commenters (11362, 11890) oppose the weighting of publicly available data and down-valuing of confidential data. The commenters contend that the EPA must prioritize public health in its decision making, therefore data from public health and epidemiological studies must be taken into consideration for influential scientific information and regulatory actions.
Commenter (11390) suggests that Option 2 is designed to create the appearance of flexibility but flouts the EPA's mission and settled scientific principles. Commenter (11890) states that giving less consideration to epidemiological studies conducted in the U.S. and other countries that protect private health data solely because those data aren't publicly available is unscientific. Commenter (12442) states that this rule shifts the Agency's approach by injecting a non-scientific metric that credits a study for disclosing private individual information used.
Commenter (11489) asserts that the value or merit of peer-reviewed science is not determined by the public availability of the underlying research data, and there is no scientific justification for a rule that directs the Agency to selectively weight certain studies over others based on this factor alone. Commenter (12463) does not believe that public availability and independent validation are qualities sufficient to mean research should be weighted more heavily than research where data must remain confidential.
Commenter (11396) states that the alternative to tiered access is equally arbitrary as no standards or criteria are provided to guide the Agency's discretion under this provision and to ensure that the best available science drives EPA's decision-making. The commenter questions how much less consideration would be given to research that relies on confidential patient information and who would make these decisions? The commenter contends that placing a higher value on certain scientific studies over others without clear criteria risks politicizing EPA's decision-making process.
Commenter (11576) asserts that EPA must use the best available science and it cannot exclude or down-weight a study that has undergone rigorous peer review that sufficiently analyzed the quality of the study. The commenter notes that in their joint statement regarding the proposed rule, the editors-in-chief of six highly regarded journals voiced disagreement with the EPA's Proposal by noting that "[d]iscounting evidence from the decision-making process on the basis that some data are confidential runs counter to the EPA stated mission `to reduce environmental risks ... based on the best available scientific information.'" Additionally, the commenter states that the Agency has failed to articulate a rational connection between the availability of underlying data and the quality, significance, and scientific soundness of a study, nor could it. Therefore, the commenter asserts that by reducing the weight given to studies on the basis of data availability, the EPA would wrongly downgrade the importance of study findings, which would ultimately affect the EPA's ability to address well-documented health harms.
Commenter (12404) states that while Option 2 would not totally eliminate a study from consideration for failure to share identifying information, it would downgrade its weight in a rulemaking or as "influential science." The commenter maintains that this could easily lead to a scientifically perverse result of putting greater weight on a methodologically inferior study just because the researchers shared personally identifying information from their subjects. According to the commenter the rule would incentivize not the best science, but that which most violates privacy. Additionally, the commenter provides privacy protections are in place to prevent a repetition of Nazi experiments and of cruel, racist research conducted on African American citizens of the United States. As such, the commenter asserts it is the regulatory process itself and the American people who will suffer by being denied regulations based on the best available science if violating human subject protections is made a condition of giving a study full weight in rulemaking or "influential science."
Commenter (12463) provides that the qualities of public availability and independent validation are not sufficient to mean research should be weighted more heavily than research where data must remain confidential.
Commenter (12715) states that Option 2 is flawed not only because it weighs the value of scientific studies using a non-scientific and largely irrelevant criterion -- the public availability of data for independent validation -- but also because it identifies no criteria for when or how EPA would exercise the vast discretion the Agency creates for itself. Quite the opposite, the commenter provides, the EPA has instead requested comment on "how much consideration should be given to studies where there is limited or no access to the underlying data and models." Thus, according to the commenter, Option 2 does nothing to ensure that important and valid scientific studies for which the data may be unavailable receive any consideration at all.
Response: The EPA is finalizing a rule of internal agency procedures, including how the Agency will consider the availability of dose-response data underlying pivotal science used in its significant regulatory actions and influential scientific information for independent validation. The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
<IP_ADDRESS> Definition of Greater Consideration
Comment: Commenters (9587, 10748, 10339, 11397, 11176, 11492, 12720) express concern that, while the SNPRM states that "the Agency will give greater consideration to studies where the underlying data and models are publicly available in a manner sufficient for independent validation," "greater consideration" is not defined.
Commenter (10748) states that the level of difference in "consideration" is not defined. The commenter states that this could mean that a study that provides its data but is poorly designed would be held in higher consideration than a study that is well-designed but is unable to provide all its data. The commenter states that this is disconcerting and could lead to serious health consequences by implementing rulemaking based on poor science solely because higher-quality studies were unable to provide all their data.
Commenter (11176) notes that for the term, "greater consideration," the supplement does require "a short description of why greater consideration was given" when a study is given greater consideration. However, the commenter points out there is no (i) guidance for what "greater consideration" means when weighting the results of one study with that of another; and (ii) direction that EPA must explain the weighting given to various relevant studies. The commenter contends that such transparency here and in the previous paragraph is critical to rigorous scientific consideration of the relevant evidence.
Commenter (12720) states that the EPA has not defined what "greater consideration" means in different contexts.
Commenter (9587) suggests the EPA define greater consideration by definitions for all relevant policymaking subcategories.
Commenter (9587) suggests the EPA distinguish between publicly available research and research available only via tiered access, and give greater consideration to studies whose underlying data and models are publicly available than to studies whose underlying data and models are only available via tiered access.
Commenter (11401) suggests that "give greater consideration to" should be clearly defined to mean whether the studies can or cannot be used primarily as the basis of a regulatory action, based on their level of transparency.
Response: If researchers want to increase the likelihood that their studies receive greater consideration by the EPA, they may take steps to ensure that the underlying dose-response data are available to the greatest extent possible. But any such response to this final rule would be purely voluntary. The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
<IP_ADDRESS> Other Things Equal
Comment: Commenter (10972) asks how the weighting of studies will be determined. Commenters (9362, 10972, 11484, 12715) question who is responsible for determining that "other things are equal" when deciding which studies should be given greater consideration.
Commenter (11176) states that scientific reasoning would not support the following aspect of the SNPRM: "other things being equal, the Agency will give greater consideration to studies where the underlying data and models are available in a manner sufficient for independent validation either because they are publicly available or because they are available through tiered access when the data includes CBI, proprietary data, or PII that cannot be sufficiently de-identified to protect the data subjects." The commenter states that the EPA has not scientifically justified that greater consideration should be given to such studies.
Commenter (11390) notes that the SNPRM suggests that "other things being equal," a "high-quality" study based on publicly available data would receive more weight than a study supported by non-publicly available data. The commenter states that that explanation leaves much undefined. The commenter states that, for example, the EPA does not explain what it would consider to be a "high-quality" study, or how it would determine whether "other things" are equal in any given circumstance. The commenter states that, in the likely event that the Agency is presented with multiple studies that could be considered "high-quality" but that are based on varying amounts of data and whose conclusions are supported with varying robustness, there are no criteria for the Agency to apply in determining which studies to consider, or how heavily to weigh them.
Commenter (11392) states that "other things being equal" is a nonsensical and arbitrary standard. Commenter (11894) states that the EPA's use of "other things equal" (85 FR 15405, section 30.5(a)) is unhelpful.
Commenter (12699) suggests that, instead of "other things equal" when comparing studies, a better measure would be to studies that are "comparable," which to this author, means something slightly less than "other things equal."
Response: The EPA clarified its intent and the process by which it would identify and determine the consideration to afford pivotal science in the final rule and preamble. The term "other things equal" is no longer used.
<IP_ADDRESS> No Access or Access is Limited
The SNPRM states "Where there is no access to data and models, or access is limited, the Agency may still consider these studies, depending on the other attributes of the studies."
Comment: Commenter (9362) expresses concern that the EPA may still consider studies where there is no access to data and models, or access is limited. The commenter states that using "dark data" from third party sources does nothing to promote science and good policy-making decisions. The commenter states that the EPA should be promoting the acquisition of scientific evidence and this can only happen if the EPA uses publicly available information and data in their decision-making. The commenter states that, if third parties are unwilling to share their data, their data should not be used to make public policy decisions.
Commenter (11892) notes that the SNPRM requests comment on how much consideration should be given to studies when there is limited or no access to underlying data or models. The commenter suggests that the EPA accept approved theses/dissertations from accredited colleges/universities; published peer reviewed articles; other studies conducted according to federal/state/local QA/QC, as well as citizen science if data were collected with approved QA/QC. The commenter states that all these sources of information follow established quality review processes, which impacts credibility to results.
Commenter (11186) suggests that studies with limited or no access to underlying data should be given equal consideration as studies with publicly available data.
Commenters (11399, 12456, 12715, 12715) express concern that the SNPRM does not include criteria regarding the EPA's consideration of "other attributes" and studies where there is no access or limited access to underlying data and models.
Commenter (11399) asks what "other attributes" will the EPA consider, and how much weight will it place on such attributes? The commenter states that, with no criteria to make these determinations, it would be extremely difficult to apply this provision uniformly or objectively.
Commenters (12456, 12715) express concern that, in the alternative proposal, the EPA would have broad discretion to use, or exclude, potentially relevant, high-quality studies where there is no access or limited access to underlying data and models.
Commenter (12715) asserts the proposal is flawed because it identifies no criteria for when or how the EPA would exercise the vast discretion the Agency creates for itself. The commenter states that the EPA has instead requested comment on "how much consideration should be given to studies where there is limited or no access to the underlying data and models." The commenter states that, thus, Option 2 does nothing to ensure that important and valid scientific studies for which the data may be unavailable receive any consideration at all.
Commenter (12731) states that, while the Agency suggests that it "may still consider studies where there is no access or limited access to underlying data and models" the Agency's consideration of relevant, valid studies is not optional.
Response: Though not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounts for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
<IP_ADDRESS> Short Description of Why Greater Consideration
Comment: Commenter (11358) supports the EPA's suggestion that the Agency would provide a short description of why greater consideration was given to a particular study or data set. The commenter states that this will help to ensure the greater transparency that this proposal aims to promote.
Commenters (11392, 11922, 11401, 12720) contend that a "short description of why greater consideration was given" to some studies over others is not sufficient.
Commenter (11922) suggests that the utilization of studies that cannot be validated under Option 2 should be accompanied with a comprehensive explanation detailing why the information was given consideration as pivotal or influential and its relative weight in the decision-making process.
Commenter (11392) states that the text of the SNPRM is vague when it comes to the proposal that the Agency will provide a "short description" of weighting decisions. The commenter asks the following questions:
 Who will write this description?
 Will it be qualitative or quantitative?
 Will it be based on a standardized rubric or case-by-case?
 Where will the description be published?
 Will there be an appeals process?
Commenter (12720) states that, given the potential for such a decision on a single study to affect its application across a range of different Agency activities, a "short description" is not nearly sufficient transparency from the Agency in describing such a significant decision with potentially far reaching and long lasting consequences. Rather than a short description, the commenter suggests the EPA address the following questions:
 What information could be included in such a short description?
 How would the Agency assess and confirm public availability or verify that data and models underlying pivotal science studies remained "publicly available" into the future?
 When would such a description be drafted or finalized by the Agency within the context of its rulemaking responsibilities?
 Would such a description be included in a central science data clearinghouse and potentially revisited and revised over time, or would it necessarily be included in individual science products published by EPA and potentially remain static.
 How would the EPA explain and document its deliberations and decision-making around the waiver process in a way that accounts for the complexity of any endeavor to make all "data and models" underlying key scientific information publicly available?
Response: Where underlying dose-response data in pivotal science are available, subject matter experts could independently reanalyze the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of the uncertainty. Such independent reanalyses will subsequently enable the EPA to make stronger, data-driven decisions in future rulemakings or in revisions to existing rules or influential scientific information. This could occur through standard cyclical reviews (e.g., revisions to NAAQS, risk and technology reviews, national primary drinking water regulations), ad hoc revisions, or revisions through the information quality guidelines or other petition processes. Implementation of this rule will more effectively share pivotal science for external consideration and increase the opportunity for independent validation of pivotal science by subject matter experts. As data are better understood through independent reanalysis, the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
Though not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounts for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
<IP_ADDRESS> Arbitrary Decision Making
Comment: Commenter (10752) contends the proposal's inherent ambiguity makes it unacceptable as a legal regulation. The commenter states that the EPA proposes to arbitrarily pick and choose which data to rely upon.
Response: The EPA disagrees that the selection of pivotal science is arbitrary. Title 40 CFR 30.5 provides sufficient guidance for EPA to use in the selection of pivotal science.
Comment: Commenter (11176) notes that the supplemental proposal does not reference the well-known and valued scientific practice of meta-analysis, which EPA staff regularly uses, in its discussion of weighting evidence. According to the commenter, how to weigh the consideration of studies is at the heart of meta-analysis and its exclusion from the supplemental proposal reinforces the point that EPA has much work to do in laying the foundation for accomplishing the stated goal of the proposed rule. Additionally, the commenter asserts that "a short description of why greater consideration was given" is required whenever a study is given greater consideration.
Commenter (11349) asserts that the "greater consideration" criterion is arbitrary as it does not judge a study by standard scientific evaluation, such as the strength of its design or methods which are current acceptable metrics used to evaluate the merit of a scientific study.
Commenter (11391) states that, under Option 2, the EPA has not explained why certain studies are inherently unreliable, and thus should be given less weight. The commenter states that, to unfairly downgrade some valid and relevant studies without any apparent rationale or specified procedure is arbitrary and capricious in violation of the APA.
Commenter (11399) asserts that Option 2 would impose a wholly subjective framework by which EPA would determine the weight that should be afforded to scientific studies based on their transparency. The commenter asks what "other attributes" will the EPA consider, and how much weight will it place on such attributes? The commenter states that, with no criteria to make these determinations, it would be extremely difficult to apply this provision uniformly or objectively. The commenter states that this provision is an invitation for arbitrary decision-making. The commenter states that the EPA should not serve as an "arbiter" of scientific studies in this manner.
Commenter (13788) states that, while the details of EPA's alternative approach are unclear, assigning preferential treatment to certain studies based on criteria that are unrelated to the studies' scientific merit and quality would be arbitrary and capricious. The commenter states that the SNPRM states that "[w]hen promulgating significant regulatory decisions or finalizing influential scientific information, the Agency will, other things equal, give greater consideration to studies where the underlying data and models are publicly available." The commenter states that, however, the Agency does not explain what it means by "other things equal" such that one study would be given greater weight, nor is the fact that data can be made publicly available tied in any way to the actual scientific merit or quality of the study results. The commenter states that the SNPRM places great importance on having publicly available data for reanalysis of peer-reviewed studies, but the Agency provides minimal detail regarding how it will determine whether researchers have actually fulfilled this reanalysis requirement. The commenter states that giving the Administrator or staff leeway to "consider" one study more valuable than another based on criteria unrelated to quality clearly provides opportunity to inject politics into public health and environmental decision-making. The commenter states that, while no two studies are likely to be identical, this vague approach would actually force the Agency to rely on one more than another (whether or not the study receiving the greater weight is actually more accurate or up to date), and is arbitrary and capricious.
Commenter (11492) states that the Option 2 approach arbitrarily devalues scientific work that relies on health data containing personal identification. The commenter states that the EPA is charged with protecting public health and the environment, and as such, studies that involve health data are critical. The commenter states that, giving less consideration, or no consideration, to studies where underlying data are not publicly available is a political decision that has nothing to do with science. The commenter states that this is contrary to EPA's mission to protect human health and the environment based on best-available science.
Response: Though not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounts for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
Comment: Commenter (11495) states that weighing studies based on their ability to comply with EPA Federal Housekeeping internal requirements is unscientific, arbitrary, burdensome and capricious.
Response: The EPA disagrees with the commenter. As discussed more fully in the preamble to the final rule, the EPA and the federal government have long encouraged open data initiatives, as the principle of transparency in regulatory decision-making and the other operations of government agencies is a fundamental behavior of good government that is inherently valuable to the public. This rule builds on the work that is already being done to increase the availability of underlying data in the scientific community and in the federal government. The EPA also disagrees with the commenter that the requirements of this rule will result in any meaningful delay in promulgating regulations. While this final rule requires the Agency to evaluate the availability of dose-response data for pivotal science, the incremental burden to the Agency to carry out these requirements is expected to be small given (1) the extensive scientific review the EPA already conducts regularly and (2) that the requirement is limited to pivotal science (i.e., typically a small, though highly important, subset of the studies the EPA would review).
Comment: Commenter (11576) asserts the EPA cannot arbitrarily give greater or lesser weight to studies solely because of the public availability of underlying data. The commenter states that the Agency has failed to articulate a rational connection between the availability of underlying data and the quality, significance, and scientific soundness of a study. The commenter states that, by reducing the weight given to studies on the basis of data availability, the EPA would wrongly downgrade the importance of study findings, which would ultimately affect the EPA's ability to address well-documented health harms.
Commenter (12430) states that there is simply no appropriate and non-arbitrary method for U.S. EPA to deprioritize studies or models relying on non-public data. Commenter (12688) expresses concern that Option 2 is based on arbitrary criteria.
Commenter (12731) states that, while the EPA indicates the Agency could downgrade its consideration of many valid studies and seeks input on how much to diminish their weight,  discarding or discounting otherwise valid studies, for no reason other than an a priori judgment made without regard to study quality, is arbitrary and illegal, for the same reason that disregarding such studies altogether is illegal.
Commenter (13788) states that there are critically important questions that neither the proposal nor the SNPRM answers or explains. The commenter states that the limited information provided about the alternative approach suggests that it would be arbitrary because, like the original approach, it would force the Agency to distinguish between studies and treat them differently based on arbitrary criteria. The commenter states that data availability has no connection to the validity and quality of science.
Commenter (12715) asserts that the EPA's "alternate approach" to 40 CFR 30.5 fails to address concerns regarding the arbitrary down-weighting of relevant scientific studies. The commenter states that, under Option 2, the EPA could consider studies for which the data is not available or where access is limited. The commenter states that, however, the proposal identifies no criteria for when or how the EPA would exercise the vast discretion the Agency creates for itself.
Response: Though not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounts for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
<IP_ADDRESS> Implementation of Option 2
<IP_ADDRESS>.1 Criteria
Comment: Commenters (11192, 11361, 11390, 11391, 11399, 11911, 12720, 12731, 13788) express concern that, while the SNPRM states that "the Agency will give greater consideration to studies where the underlying data and models are publicly available in a manner sufficient for independent validation," the EPA provides no information about potential weighting criteria or calculations.
Commenter (11390) expresses concern that, in the likely event that the Agency is presented with multiple studies that could be considered "high-quality" but that are based on varying amounts of data and whose conclusions are supported with varying robustness, there are no criteria for the Agency to apply in determining which studies to consider, or how heavily to weigh them. The commenter states that, at best, the lack of clear guidance on how studies should be weighted would lead to inconsistent and unclear application of the proposed rule.
Commenter (12731) states that the EPA provides no details as to how the differential weighting alternative approach would work in practice. The commenter states that this description is entirely ambiguous, lacking sufficient detail not only as to how the EPA would intend to apply this alternative approach in practice (including how much weight it would give to the availability of data and models in assessing studies), but also under what circumstances the Agency would or would not decide to apply this alternative approach as the EPA maintains it may consider studies where underlying data is not available or only available on a limited basis. The commenter states that, in the EPA's briefing to House Science, Space, and Technology Committee staff, the Agency "was unable to expand on how this potential weighted system would operate. The Agency has not identified implementation details for the weighting approach, including any concrete ideas about how the scale of a weighted system would be structured."
Commenter (12731) states that the fact that the EPA itself does not understand how the alternative would work confirms that the Agency has not provided adequate notice for public comment, and it cannot finalize any such alternative without first examining the details of its implementation.
Response: Though not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounts for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
<IP_ADDRESS>.2 Detailed Description of Option 2
Comment: Commenters (9587, 11390, 11391, 11392, 11495, 11576, 11894, 12464, 12720, 13788) contend the proposed revisions should include a more detailed description for implementation of Option 2 as described below. The commenters suggest the following implementation questions need to be addressed:
 Are 10 studies that do not meet the data sharing requirements, but reach a similar finding, outweighed by 1 study that does meet the data sharing requirement, but has a contrary finding? (11495)
 Could the EPA explain away ignoring a pivotal public health study by simply saying it was given "less consideration," implying that it did in fact inform a problematic new pollution standard? (11392)
 How the EPA would apply such weighting? (11390, 12464)
 How would the EPA determine whether "other things" are equal in any given circumstance? (11390)
 How will the EPA create the necessary infrastructure or capacity to handle such decisions in a timely manner? (11391)
 How much greater consideration will the EPA give to studies where the underlying data and models are available for independent validation? (9587)
 How much additional weight will certain studies receive? How will that be determined? (13788)
 How would the weighting system impact existing weighting metrics used, such as the greater consideration given to human health studies over animal studies in NAAQS reviews? (11392)
 When the EPA would apply such weighting? (11390)
 What would the EPA consider to be a "high-quality" study? (11390)
 What are the parameters for making these decisions? (11391)
 While human health studies with the most medical information are currently given the most weight, animal studies will have no data-sharing complications  --  will the science-based metrics be overturned with a system based entirely on transparency of data? (11392)
 Who will have access to the information and what type of information they will have access to? (11576)
 Who will determine whether underlying data and models are available in a "sufficient" manner when, as EPA itself has pointed out, data are sometimes available in a graduated or tiered manner (i.e., data availability exists on a spectrum)? (11391)
 Will the EPA publish a regularly updated database indicating its determination about how much weight each study is given? If so, how will the public be able to access this information? (12720)
 Will the decisions made by other federal agencies using the same studies be necessarily affected by the determinations that the EPA makes? (12720)
 Will these determinations be final, or changeable? If final, will there be opportunities for the study authors or researchers to dispute the determination? If changeable, what is the process by which such a determination could be changed, and who within the EPA would have that authority? (12720)
 Would the EPA develop objective metrics to determine when and how -greater consideration" would be given to certain studies, or would decisions be made on a case-by-case basis without any established procedures? (11392)
 Would giving less "consideration" in assessments devolve into giving no weight? (11894)
Commenters (11392, 12431, 12720) contend that the EPA fails to provide sufficient description or necessary analysis of the Agency's Option 2 approach. The commenters suggest that the following information is necessary:
 The EPA must provide greater clarity on how such a weighted system would operate in practice, where decision making authority would reside, and what the implications would be for past studies that underpin the Agency's regulatory standards. (11392)
 The EPA needs to clarify whether a determination on degree of consideration would apply consistently across all the EPA staff, science advisors, and subject matter areas relevant to the EPA work. (12720)
 The EPA needs to clarify whether making data "available" for independent validation is sufficient for such increased consideration, or whether such independent validation must have actually occurred prior to the EPA making such a decision. (12720)
 Because the Agency relies on scientific studies across a range of disciplines and for a range of activities, the EPA needs to clarify how variations in weight given to specific studies would actually be implemented. (12720)
 The EPA needs to clarify what this greater consideration would entail. (12431)
Commenters (10752, 11192, 11361, 11399, 11911) contend that, as the EPA has proposed no criteria or oversight mechanisms, Option 2 is open to manipulation and, thus, is unacceptable.
Commenter (10752) states that the proposal's inherent ambiguity makes it unacceptable as a legal regulation. The commenter states that the EPA proposes to arbitrarily pick and choose which data to rely on with the presumption that the nonconfidential information is given more weight. The commenter states that it is safe to assume that, in the current regulatory climate, the EPA will always choose the industry data and models.
Commenters (11192, 11361) state that the Agency provides no example weighting criteria or maximum weighting ratios that might inform the public about what could be expected from this weighting proposal. The commenter states that, as a result ,this generalized suggestion could be used to mollify those that oppose dismissal of public health studies that couldn't make their data public, but in essence, still dismiss them by assigning them a very small weighting factor.
Commenter (11399) asserts that Option 2 would impose a wholly subjective framework by which the EPA would determine the weight that should be afforded to scientific studies based on their transparency. Commenter (11911) asserts that the SNPRM's Option 2 is open to manipulation without transparent criteria informed by independent scientific expertise.
Response: Though not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounts for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
<IP_ADDRESS>.3 Other
Comment: Commenter (10339) maintains that publicly funded, peer-reviewed studies should not be disadvantaged during Agency scientific reviews.
Response: The EPA disagrees that publicly funded, peer-reviewed studies would be disadvantaged in any way. The final rule includes sufficient flexibility in terms of the allowance for restricted access, considerations in 40 CFR 30.5, the Administrator's exemption in 40 CFR 30.7, and the EPA's incremental approach to maximizing transparency by first focusing on dose-response data to minimize such potential unintended consequences.
Comment: Commenters (11176, 11390, 11492, 11495, 12688, 12727) assert there is no scientific basis for employing the Option 2 weighting system.
Commenter (11176) states that the category of "non-federally funded and non-the EPA data" likely makes up the largest category informing the EPA's many actions. The commenter states that it is also the category that poses the most challenges for making data available. The commenter states that such studies generally provide important scientific contributions and should not be discounted because their data are not readily available. The commenter states that, in our 2018 comments, we included two examples of research that might be excluded: private-sector studies and meta-analyses. The commenter states that, while such work will not necessarily be excluded through the proposed rule with the SNPRM, it is likely to be de-emphasized to the detriment of evidence-based policymaking at the EPA. The commenter states that scientific reasoning would not support the following aspect of the SNPRM: "other things being equal, the Agency will give greater consideration to studies where the underlying data and models are available in a manner sufficient for independent validation either because they are publicly available or because they are available through tiered access when the data includes CBI, proprietary data, or PII that cannot be sufficiently de-identified to protect the data subjects." The commenter states that the EPA has not scientifically justified that greater consideration should be given to such studies.
Commenter (11492) states that the use of meta-analysis (when results from pertinent, multiple studies are combined) is becoming increasingly common, and they are crucial to understanding epidemiology and public health issues. The commenter states that the Supplemental Rule requires a brief description of why one study is given "greater consideration" over another, but does not give any direction as to how to weigh these different studies. The commenter states that failure of the Supplemental Rule to address basic questions such as these is indicative that the authors have no understanding of scientific analysis.
Commenter (11495) states that federal agencies should have the ability to individually evaluate and weight scientific studies based on the strength of the science using criteria such as participant selection, disclosure of methods and authentication of reagents, quality of exposure assessment, strength of association, consistency, specificity, temporality, evidence of dose-response, biological plausibility, and assessment of potential bias or confounding. The commenter states that weighing studies based on their ability to comply with the EPA Federal Housekeeping internal requirements is unscientific. The commenter states that it hinders the EPA by depriving the Agency of some of the best science on which it might make its decisions.
Commenter (12688) contends that Option 2 has no basis in scientific practice. The commenter states that it will hamper the Agency's ability to use high-quality science produced by researchers at institutions like UCLA. The commenter states that regulatory agencies should do what scientists do: consider studies in accordance with the robustness of their methodology and conclusions, and not based on arbitrary criteria.
Commenter (12727) states that the EPA proposes to "give greater consideration" when reviewing studies based on data's public availability, but this is not a science-based approach. The commenter states that the proposal to "downgrade" studies due to their public accessibility does not appear to improve any aspect of science-based decision making. The commenter states that public accessibility for the purpose of "independent validation" does not mean that the data are high quality, or even useful from a scientific perspective. The commenter states that, for example, epidemiological data on real-life human exposures are valuable for science-based decisions on chemical exposures, and yet much these data are rightfully protected as PII.
Response: Though not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounts for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
Comment: Commenter (11176) states that research that uses CBI, proprietary data, or PII is likely to provide greater insights into health effects pertinent to the EPA's work precisely because of these data. The commenter states that, because such data will not be "publicly available or available through tiered access," the potentially more insightful research that relies on these data is likely to be discounted by this proposed rule to the detriment of the EPA's important work. The commenter states that, for CBI, for example, the EPA uses such data to identify best available technologies (BAT) upon which many water regulations are based. The commenter states that companies only provide the data upon which BAT can be determined with the proviso that the data remain CBI. The commenter states that, without accurate data on the BAT currently in use, such regulations would be based on weaker science. The commenter states that the proposed rule poses three risks for this example: (i) businesses may have less confidence their CBI will be protected and may be reluctant to share their information; (ii) the EPA may discount research based on CBI weakening the scientific underpinning of a regulation; and (iii) water quality may suffer.
Response: The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Comment: Commenter (11391) states that implementing Option 2 would require so much additional complex review before important Agency rulemakings and other actions could be accomplished that a significant and deeply problematic regulatory bottleneck is easily foreseeable. The commenter states that the EPA has offered no indication that it has considered these problems nor how it plans to develop the necessary internal infrastructure or capacity to deal with them. The commenter states that the SAB raised concerns in its report on the proposed rule, noting that the requirement to make data available is unclear, "mak[ing] it difficult to understand the implications of the requirement." The commenter states that, according to the SAB, depending on how the requirement is interpreted, compliance with it could "be enormously expensive and time consuming." The commenter states that the SAB goes so far as to suggest that the EPA may have to establish an office on data sharing, or peer review panel or working group to implement the requirement. The commenter shares this view.
Response: The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Comment: Commenter (11395) does not agree that data accessibility should be the primary factor used to determine how a study is weighted relative to other studies.
Response: Though not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounts for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
Comment: Commenter (11495) states that, without detailed, consistent guidelines delineating exactly how studies will be weighted, as well as how missing data will be accounted for, the quality of this new document in no way approaches the quality of the EPA's existing documents that define inclusion and exclusion criteria, currently based on publication quality judged by rigorous scientific standards. The commenter states the SNPRM brings forward a new and complex process without any consideration to ensure that information is "accurate, clear, unbiased and complete", as mandated by the federal government, let alone how the policy will be practically implemented.
Commenter (12731) states that the EPA has failed to explain how differential weighting of studies based on data availability adds to, modifies, or is needed in light of existing, relevant frameworks at the Agency including, for example, frameworks developed for the EPA IRIS toxicological reviews, the development of Integrated Science Assessments, and ecological assessments.
Response: The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Though not exhaustive or exclusive, the EPA has identified several factors that balance some of the important technical considerations the EPA will consider in addition to data availability and that are particularly relevant to the stage of the analysis where dose-response data are used. The EPA will thus consider the following factors:
 the quality of the study relative to other studies for which the dose-response data are available;
 the extent to which there are other studies for which the dose-response data are available;
 the sensitivity of the conclusions in the significant regulatory action or influential scientific information based on the use of the study;
 the extent to which the study is fit for the purpose or intended use relative to other studies for which the dose-response data are available;
 the use of exposures or doses in a range and duration that is relevant for the intended use and that minimizes the need for extrapolations;
 the extent to which the study is supported by other scientific evidence;
 the extent to which the study accounts for unique scientific considerations;
 the extent to which the study minimizes the use of defaults and assumptions, uses appropriate and strong statistical methods, and includes a robust representation of uncertainty and confidence intervals; and
 the study's consideration of a broad range of parametric dose-response or concentration-response models, a robust set of potential confounding variables, nonparametric models that incorporate fewer assumptions, various threshold models across the dose or exposure range, and models that investigate factors that might account for spatial heterogeneity.
The flexibility provided in the rule in terms of the allowance for restricted access, considerations in 40 CFR 30.5, the Administrator's exemption in 40 CFR 30.7, and the EPA's incremental approach to maximizing transparency by first focusing on dose-response data will minimize any potential unintended consequences.
7.1.5 Ensure Over Time More Availability of Data
Comment: Commenter (11176) suggests that, by working together with other agencies to develop common standards and provide resources to researchers and research sponsors to create data governance systems that support tiered data access, the EPA can ensure that over time more of the data and models underlying the science that informs significant regulatory decisions are available publicly or via restricted access.
Response: The final rule does not regulate or impose requirements on any party outside of the EPA but rather exclusively governs the EPA's internal process for handling pivotal science. This rule does not require any researcher or other outside entity to provide dose-response data to the EPA. It also does not direct or require any outside entity or EPA to establish data sharing mechanisms. Further, the rulemaking does not require the EPA to collect, store, or publicly disseminate dose-response data and models underlying pivotal science. Instead, it governs internal agency procedures for weighing its consideration of various studies according to factors that include data and model availability.
Comment: Commenter (11576) states that seeking to require or promote greater public accessibility to models and data underlying studies is not the EPA's role; the Agency was established to protect public health and the environment using the best available science, not to attempt to tell scientists how to do their jobs. The commenter states that there is no way to ensure that the EPA's approach would honor "legal and ethical obligations to reduce the risks of unauthorized disclosure and re-identification[,]" nor does the EPA offer any explanation as to how it would address these issues. Commenter (11576) notes that the SNPRM states that:
      the EPA is requesting comment on how to ensure that, over time, more of the data and models underlying the science that informs significant regulatory decisions and influential scientific information are available to the public for independent validation in a manner that honors legal and ethical obligations to reduce the risks of unauthorized disclosure and re-identification". ... [And] the EPA is interested in comments about how to provide sufficient incentives and support to researchers to increase access to the data that may be used as pivotal regulatory science or pivotal science.
Response: The rule does not impose requirements on any entity outside of EPA. This is a rule of internal procedures and does not direct or require any outside entity or EPA to establish data sharing mechanisms. Further, the rulemaking does not require the EPA to collect, store, or publicly disseminate dose-response data for underlying pivotal science.
The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Comment: Commenter (11892) suggests that, because access to information can be limited by for-profit publications requiring payment to view full peer-reviewed articles, all work funded in part by the federal government should allow free full access to published articles.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." EPA researchers, grantees and contractors are required to make peer-reviewed journal articles and the underlying data and models resulting from EPA-funded research accessible to the public.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, the rule requires the EPA to evaluate the availability of the analyzable dataset, not the more limited dataset that accompanied the publication of the study or model. For this rulemaking EPA will not identify a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
Comment: Commenter (9587) suggests the EPA should provide a time limit for its willingness to accept privacy, confidentiality, and tiered access. The commenter states that scientific research can meet full transparency standards in time, even if it cannot meet it immediately. The commenter states that a time limit will provide the EPA researchers and independent scientists an incentive to change their practices. The commenter suggests a time limit of no more than 20 years -- say, 31 December 2040. The commenter states that this time limit might be subdivided, so that fields that require substantial longitudinal studies have longer time limits than fields that do not.
Response: The EPA clarified in the final rule that the public must be able to access the information on the date of publication of the proposed rule (or, as appropriate, a supplemental notice of proposed rulemaking, or notice of availability) for the significant regulatory action or on the date of dissemination of the draft influential scientific information for public review and comment.
Comment: Commenter (10974) states that, if the EPA were to spend their time and direct their resources towards ensuring that their own data is easily accessible in updated and user-friendly tools, they would model the way for, and incentivize and support, other researchers as they work to become more creative and thorough with data transparency. The commenter states that, rather than promulgating this new rule, the EPA should be taking immediate action to demonstrate data leadership  -  doing so would be a much more progressive and useful approach, and would model good governance and public service. Furthermore, the commenter provides that EPA would be better served supporting state efforts to make existing data easier to understand, to help better communicate research results and conclusions, and to help states develop third-party partnerships.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." EPA researchers, grantees and contractors are required to make peer-reviewed journal articles and the underlying data and models resulting from EPA-funded research accessible to the public.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, the rule requires the EPA to evaluate the availability of the analyzable dataset, not the more limited dataset that accompanied the publication of the study or model. For this rulemaking EPA will not identify a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
Comment: Commenter (11401) states that, applying the new standard to existing studies would encourage researchers to make their underlying data more transparent in accordance with the tiered approach, thereby benefiting not just the EPA rulemakings, but scientific knowledge to the broader society. The commenter states that encouraging researchers to release data for existing studies also addresses the EPA's question in the SNPRM on "how to provide sufficient incentive...to researchers to increase access to the data that may be used as pivotal regulatory science or pivotal science." The commenter states that allowing their existing, nontransparent studies to be used in the future would discourage greater openness and hinder the contribution to better science.
Response: The EPA is maintaining that the rule provisions apply equally to all dose-response data underlying studies used as pivotal science, regardless of when the study or the data was created. Scientific transparency is important regardless of the age of the study or the dose-response data. Although the EPA is prioritizing transparency in pivotal science, the Agency also recognizes that there will be instances where the underlying dose-response data of pivotal science are unavailable for independent validation. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, a study with underlying dose-response data that are not available should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. The EPA could also give pivotal science with underlying dose-response data that are unavailable for independent validation full consideration upon receiving an exemption from the requirements of this rule from the Administrator, as detailed under 40 CFR 30.7. One of the criteria the Administrator may use in his/her determination to grant a waiver is that the development of the dose-response data was completed or updated before the effective date of the rule. Because the final rule does not categorically exclude any studies from EPA's consideration, the Agency will not be prevented from evaluating and considering all quality, relevant studies when promulgating significant regulatory actions and developing influential scientific information. Further, the final rule only applies to dose-response data; therefore, all quality, relevant studies will continue to inform other aspects of the EPA's evaluations.
The considerations in 40 CFR 30.5, the Administrator's exemption in 40 CFR 30.7, and the EPA's incremental approach to maximizing transparency by first focusing on dose-response data are intended to minimize potential unintended consequences, such as the potential skew toward newer, domestic, and/or industry-funded studies suggested by the commenters. In addition, the EPA is maintaining language from the 2020 SNPRM in 40 CFR 30.3 stating that statutory requirements and corresponding implementing regulations will control in the event of any conflict, and clarifying in this preamble that the requirements in this final rule set the overarching structure and principles for transparency in significant regulatory actions and influential scientific information. The EPA plans to develop internal implementing guidance and promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers. The EPA believes the rule requirements provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical and privacy considerations in order to ensure the Agency maintains a strong scientific basis for its decision-making.
Comment: Commenter (12596) states that a Transparency Scoring System would provide incentives and support to researchers to increase access to the data that may be used as pivotal regulatory science or pivotal science. The commenter states that it would inform the public as to the methodologies underlying important pivotal regulatory science or pivotal science. The commenter states that those signals would be bi-directional, encouraging researchers to adopt transparency protocols with the incentive of having their studies receive greater consideration in the EPA rulemakings. The commenter states that such a framework would encourage researchers to fundamentally incorporate "transparency by design" into their studies.
Commenter (11386) contends the EPA should not incentivize researchers to provide public access to their data. The commenter states that consideration for rulemaking should not be used by the EPA as an incentive for researchers, nor should grant money be allocated on the basis of the feasibility of making data publicly available. The commenter states that both of these incentives potentially conflict with the primary purpose of the EPA and the EPA-funded research, which is to protect public health and the environment. The commenter states that it is admirable to make data public when possible, but the EPA should not pressure scientists to do so when it is not, for instance, in the best interests of the subjects in the study. The commenter states that the mission of the EPA is to protect human health and the environment, and providing regulatory incentives that influence the direction of scientific research does not fall within that mission. The commenter states that, while it would be inappropriate to base public health decisions on data transparency, it is appropriate and beneficial for the EPA to provide support and resources for increasing the access to scientific data. The commenter states that examples of ways in which the EPA can give this support include providing those carrying out the research with space on servers on which relevant data can be stored and made publicly accessible, providing access to computing power for running reanalysis, or providing a data-management team. The commenter states that these resources would lead to an increase in availability of the data and models underlying the science that informs significant regulatory decisions and influential scientific information. The commenter states that such support structures from the EPA would enable relevant data to be publicly accessible independent of the resources of the research team. The inherent value and validity of the data and of the conclusions drawn from its analyses would be unchanged by these methods of increasing transparency.
Response: The purpose of this action is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions. Where underlying dose-response data in pivotal science are available, subject matter experts could independently reanalyze the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of the uncertainty. Such independent reanalyses will subsequently enable the EPA to make stronger, data-driven decisions in future rulemakings or in revisions to existing rules or influential scientific information. This could occur through standard cyclical reviews (e.g., revisions to NAAQS, risk and technology reviews, national primary drinking water regulations), ad hoc revisions, or revisions through the information quality guidelines or other petition processes. Implementation of this rule will more effectively share pivotal science for external consideration and increase the opportunity for independent validation of pivotal science by subject matter experts. As data are better understood through independent reanalysis, the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
The EPA is not categorically excluding any studies from consideration when promulgating significant regulatory actions or developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information. The EPA will also continue to rely on the highest quality, most relevant studies available in determining the potential for hazard due to exposure to a pollutant, contaminant, or substance. Commenter (12464) states that, while academic researchers have no incentive to revisit their old research and make the raw data and models publicly available, industry scientists have a strong monetary incentive to do so if their research supports the industry's goals.
Commenter (11103) suggests that lowering barriers associated with making data available is a better mechanism for change than the supplemental proposal. The commenter references the 2019 NASEM report, which provides that the lack of a supportive framework for adequate recordkeeping disincentivizes researchers to create the conditions for reproducibility. The commenter states EPA can address this by fostering the development of new software tools, but by providing adequate training on how to use such tools and education in the importance of transparency in scientific studies. To accomplish this, the commenter recommends partnering with educational institutions and leveraging the working relationships EPA's Office of Public Engagement and Environmental Education (OPEEE) already has, such as the U.S. Department of Education, to create programs that promote a culture of openness in the scientific community The commenter provides the "Reproducible Research" course at John Hopkins University as such a program.
Response: The EPA would like to emphasize that this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedures and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (12465) expresses concern that the treatment of existing data or models may disincentivize the generation of new data under the new transparency requirements while reducing the importance of considering existing data, regardless of the scientific validity or merit of that data. Additionally, the commenter notes there are similar considerations between statutes and regulations that are outside of the scope of the supplemental proposal and recommends that these provisions be revisited in an effort to resolve the inherent conflict.
Conversely, Commenter (11401) notes that applying the new standard to existing studies would encourage researchers to make their underlying data more transparent in accordance with the tiered approach, thereby benefitting not just EPA rulemakings, but scientific knowledge to the broader society.
Response: The EPA is maintaining that the rule provisions apply equally to all dose-response data underlying studies used as pivotal science, regardless of when the study or the data was created. Scientific transparency is important regardless of the age of the study or the dose-response data. Although the EPA is prioritizing transparency in pivotal science, the Agency also recognizes that there will be instances where the underlying dose-response data of pivotal science are unavailable for independent validation. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, a study with underlying dose-response data that are not available should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. The EPA could also give pivotal science with underlying dose-response data that are unavailable for independent validation full consideration upon receiving an exemption from the requirements of this rule from the Administrator, as detailed under 40 CFR 30.7. One of the criteria the Administrator may use in his/her determination to grant a waiver is that the development of the dose-response data was completed or updated before the effective date of the rule. Because the final rule does not categorically exclude any studies from EPA's consideration, the Agency will not be prevented from evaluating and considering all quality, relevant studies when promulgating significant regulatory actions and developing influential scientific information. Further, the final rule only applies to dose-response data; therefore, all quality, relevant studies will continue to inform other aspects of the EPA's evaluations.
The considerations in 40 CFR 30.5, the Administrator's exemption in 40 CFR 30.7, and the EPA's incremental approach to maximizing transparency by first focusing on dose-response data are intended to minimize potential unintended consequences, such as the potential skew toward newer, domestic, and/or industry-funded studies suggested by the commenters. In addition, the EPA is maintaining language from the 2020 SNPRM in 40 CFR 30.3 stating that statutory requirements and corresponding implementing regulations will control in the event of any conflict, and clarifying in this preamble that the requirements in this final rule set the overarching structure and principles for transparency in significant regulatory actions and influential scientific information. The EPA plans to develop internal implementing guidance and promulgate either statute-specific transparency regulations or programmatic actions implementing this procedural rule, as appropriate, to clarify how the Agency will implement the provisions from this final rule for specific programs authorized under the statutes the EPA administers. The EPA believes the rule requirements provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical and privacy considerations in order to ensure the Agency maintains a strong scientific basis for its decision-making.
 Chapter 8: Research Data
Title 40, Code of Federal Regulations part 30.10 (40 CFR 30.10) of the notice of proposed rulemaking (NPRM) proposes to require the Environmental Protection Agency (EPA) to implement the rule consistent with the definition of research data in Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards, exemptions in Public Law 89 - 487, and other applicable federal laws. From the Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards:
      Research data means the recorded factual material commonly accepted in the scientific community as necessary to validate research findings, but not any of the following: preliminary analyses, drafts of scientific papers, plans for future research, peer reviews, or communications with colleagues. This "recorded" material excludes physical objects (e.g., laboratory samples). Research data also do not include: (i) Trade secrets, commercial information, materials necessary to be held confidential by a researcher until they are published, or similar information which is protected under law; and (ii) Personnel and medical information and similar information the disclosure of which would constitute a clearly unwarranted invasion of personal privacy, such as information that could be used to identify a particular person in a research study.
Several comments were received regarding the NPRM with regards to research data as defined under the Uniform Administrative Requirements. In this section these comments are organized as follows:
       Uniform Administrative Requirements
       Applicability of the Rule to Research Laboratory Samples
The specific comments provided in this Chapter reflect the comment letters that provide arguments supported by detailed rationale, examples, and references/documentation. The Docket ID numbers for these commenters are cited throughout this Chapter. In addition, some general comment letters may also include comments related to comments included and responded to in this Chapter. These comments are summarized in Appendix B. Note that points and issues raised in these general comment letters may also be raised by the more detailed comments summarized in this Chapter. The responses provided in this Chapter adequately address these more general comments.
8.1 Uniform Administrative Requirements
Comment: Commenters (6111, 6126, 6880, 6924) assert that the proposed requirements in §30.10 are inconsistent with aspects of the other proposed requirements. The commenters assert that, while proposed §30.10 requires EPA to implement the rule consistent with the definition of research data in the Uniform Administrative Requirements, the proposed rule appears to require all information be made publicly available, despite exclusions in the Uniform Administrative Requirements regarding medical information. The commenters refer to the Uniform Administrative Guidance which exempts from its definition of "research data" subject to disclosure any "medical information and similar information the disclosure of which would constitute a clearly unwarranted invasion of personal privacy, such as information that could be used to identify a particular person in a research study."
Commenter (6126) suggests that, to be consistent with the definition of "research data" in Office of Management and Budget (OMB) Circular A-110 and with EPA's existing guidance, the rule appears to preclude collection of identifiable information. The commenter states that, however, such information could be necessary to conduct replication and validation of studies. The commenter states that this would reflect the type of data available through the Federal Statistical Research Data Centers and referred to by the U.S. Commission on Evidence-Based Policymaking, which are cited in the preamble text. The commenter states that, if EPA's intent is to only consider replicable studies those with identifiable data but to preclude such information from the definitions, the rule's organizing logic must be deemed tautological. The commenter states that clarification is needed regarding whether EPA intends for identifiable data to be available to fulfill the regulatory requirements or if the intent is to rely exclusively on de-identified data.
Commenter (6880) states that the preamble and proposal are contradictory as to what data are subject to the proposed rule. The commenter states that, in the preamble, EPA makes the following statement that is unsupported by the text of the proposal: "Implementation of this rule will be consistent with the definition of `research data' in Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards." The commenter states that, in the cited regulation, the OMB defines "research data" within the context of identifying what information a federal agency must release to satisfy a Freedom of Information Act (FOIA) request for data generated by research funded through that agency. The commenter states that the definition specifically excludes certain confidential information from the definition of "research data" (§200.315(e)(3), (3)(i), and (3)(ii)).
Commenter (6924) asserts that the rule's requirement to make all information publicly available that would be needed to complete a reanalysis conflicts with the exclusions in the definition of "research data."
Response: Proposed 40 CFR 30.2 in the 2018 proposed rule included a definition of research data. In the 2020 SNPRM, the EPA deleted the proposed definition of research data. While one commenter on the 2020 SNPRM noted that the exclusions in the definition of research data of trade secrets and personal and medical information were not incorporated into the definition of data, commenters did not request that the EPA maintain a definition of research data. The EPA is not including a definition of "research data" in this final rule because it includes the definition of "data" in section 30.2 as follows:
      Data means the set of recorded factual material commonly accepted in the scientific community as necessary to validate research findings in which obvious errors, such as keystroke or coding errors, have been removed and that is capable of being analyzed by either the original researcher or an independent party.
In addition, the EPA would like to emphasize that this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedures and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science. The rule allows the EPA Administrator to grant an exemption to the transparency requirements if the Administrator determines that greater consideration is warranted when:
 Technological or other barriers render sharing of the dose-response data infeasible;
 The development of the dose-response data was completed or updated before the effective date of this final rule;
 Making the dose-response data would conflict with laws and regulations governing privacy, confidentiality, confidential business information (CBI), or national security;
 Third-party independent validation of the study's underlying dose-response data through reanalysis has been conducted; or
 The factors in 40 CFR 30.5 used in determining the consideration to afford to the pivotal science indicate full consideration is justified.
8.2 Applicability of the Rule to Research Laboratory Samples.
Comment: Commenter (6143) suggests that the Agency apply the proposed requirements to a limited and special class of laboratory samples, the analytical results of which (from non-destructive testing) may become an essential element of the research that underpins a given regulation. The commenter states that, not only can different laboratory analysts reach different conclusions by analyzing the same sample, it is common for analytical methods to improve over time. The commenter notes that improved analytical methods provide researchers with the opportunity to re-analyze properly archived samples, potentially yielding more accurate and even different results. According to the commenter, it should become part of the researchers' protocol to retain and properly archive this special class of samples when the analytical results are likely to inform or inspire the regulatory process. The commenter asserts that it is particularly true for research conducted by or for EPA and its contractors; and by any party conducting research with the intent to inform or inspire a specific regulatory action. (While it is true that certain types of samples cannot be retained because they are inherently unstable chemically and/or physically, other types of samples can remain stable for decades when properly archived.)
Response: EPA disagrees with the commenter's suggestion to extend the scope of the rule to laboratory samples. As stated in 40 CFR 30.3, the provisions of this part do not apply to physical objects (like laboratory samples). Physical samples are not needed to independently validate pivotal science in the context of this rule.
 Chapter 9: Science That Serves as the Basis for Informing a Final Significant Regulatory Actions
Section 30.1 of the proposed rule requires the Environmental Protection Agency (EPA) to ensure that the regulatory science underlying its actions be publicly available in a manner sufficient for independent validation. Section 30.2 of the proposed rule includes a definition of regulatory science as scientific information, including assessments, models, criteria documents, and regulatory impact analyses, that provide the basis for EPA final significant regulatory decisions. Several comments were received regarding "regulatory science". In this section these comments are organized as follows:
Section 9.1: NPRM Regulatory Science (section 30.2)
       Ensure that the Regulatory Science is Publicly Available in a Manner Sufficient for Independent Validation (§30.1)
       Regulatory Science Means Scientific Information that Provides the Basis for EPA Final Significant Regulatory Decisions (§30.2)
       EPA's Regulatory Science Should Be Consistent with the OMB's Final Information Quality Bulletin for Peer Review (83 FR 18770)
       EPA's Proposed Regulatory Science Requirements Consistency with OMB's Guidelines for Ensuring and Maximizing Quality, Objectivity, Utility and Integrity of Information Disseminated by Federal Agencies (OMB Guidelines)
       Regulatory Science that Cannot Meet the Requirements of §§30.5 and 30.6
       How Should EPA Address a Circumstance in which EPA Has a Statutory Requirement to Make a Determination for Which Scientific Information Publicly Available Sufficient for Independent Validation Does Not Exist? (83 FR 18772)
       How Does What this Rule Would Require Compare to Requirements of Environmental Statutes? (Relates to Chapter 1)
Comments that seek to clarify how the definition of "pivotal regulatory science" is distinct from the separately defined, "regulatory science" are primarily contained in the chapter of this response to comments (RTC) concerning "pivotal regulatory science" (Chapter 6).
The specific comments provided in this Chapter reflect the comment letters that provide arguments supported by detailed rationale, examples, and references/documentation. The Docket ID numbers for these commenters are cited throughout this Chapter. In addition, some general comment letters may also include comments related to comments included and responded to in this Chapter. These comments are summarized in Appendix B. Note that points and issues raised in these general comment letters may also be raised by the more detailed comments summarized in this Chapter. The responses provided in this Chapter adequately address these more general comments.

Note on Terminology
In the final rule, the EPA has replaced the term "regulatory science" with the term "science that serves as the basis for informing a significant regulatory action." The EPA made this change in response to comments that the term "regulatory science" may be confusing because it suggests either the term refers to a scientific discipline of regulatory decision-making, or that some science the EPA considers is inherently regulatory. Neither of these interpretations reflects the Agency's intent in defining this term. The EPA considers the breadth of scientific evidence in its rulemakings; while this scientific evidence informs policy decisions, the EPA's consideration of the science does not make it "regulatory science." To reflect this fact, in the final rule the EPA is changing the proposed term "regulatory science" to "science that serves as the basis for informing a significant regulatory action." While commenters on the 2018 proposed rule used the term "regulatory science," the EPA is responding to comments with the term "science that serves as the basis for informing a significant regulatory action" to reflect the contents of the final rule.
9.1 NPRM Regulatory Science (Section 30.2)
9.1.1 Ensure that the Regulatory Science is Publicly Available in a Manner Sufficient for Independent Validation (§30.1)
<IP_ADDRESS> General Opposition to Regulatory Science Being Required to be Publicly Available
Comment: Commenters (0044, 0671, 0732, 1943, 2687, 2909, 4591, 4595, 4848, 4894, 5181, 6111, 6125, 6147, 6158, 6897, 6904, 6907, 6924, 6940, 8281-PH5, 8281-PH29, 8281-PH46, 8281-PH84) generally oppose regulatory science underlying its actions being required to be publicly available in a manner sufficient for independent validation. Commenters generally express that, it is unnecessary and would limit the ability to use scientific data that is necessary for the EPA to meet its mission. Commenter (6105) asserts that not all scientific findings that are important are publishable, nor do scientific journals generally have sufficient space to include all data. Example comments include:
 Commenter (0732) states that there is a need to consider ALL scientifically well-planned, documented and reported studies about potential adverse health effects of chemicals. The commenter asserts that this serves to increase the likelihood that any regulatory standards developed will be based on the full and complete "weight of evidence" of data and helps reduce the uncertainty inherent in those numbers. According to the commenter, the ability of the EPA to protect health is directly connected to the quality and quantity of studies that underpin its standards.
 Commenter (4595) Commenter (4595) asserts that the EPA's Proposal risks rejecting valid scientific evidence and fundamentally mischaracterizes the way science is conducted and made available for decision-making. According to the commenter, by limiting the science EPA can use in policies and regulations, EPA will ultimately constrain itself to smaller groups of studies and risk biased outcomes. The commenter asserts that, if the pool of research is smaller - which is what this proposed rule will lead to - that smaller pool utilized creates an inherent bias and thus the rulemaking process will yield distorted results.
 Commenter (4848) contends that it is imperative that EPA allow consideration of all available scientific data pertinent to a proposed environmental rule or regulation  -  including randomly controlled human health trials and other epidemiological studies.
 Commenter (6924) In general, when assessing the body of evidence, EPA should consider both individual study quality and the overall quality of the body of evidence which requires consideration of all the relevant scientific literature. To not do so could potentially bias the assessment, as stated by the National Academies of Sciences (NAS).
 Commenter (0671) states that EPA does not need publicly available information to allow independent validation. The commenter states that any researcher who wishes to validate a study is already free to perform her or his own study asking the same scientific question. According to the commenter, this sort of replication is already performed hundreds of times a year across many areas of environmental science and is far more meaningful than the narrow and misguided notion of validation the EPA is implicitly using.
 Commenters (1943, 6940) assert that EPA needs to have access to data generated from modern, sensitive, and novel models and methodologies to protect the American public. Commenter (1943) states that any attempt to limit the applicability of scientific data will stifle innovation and progress toward a cleaner, healthier future for all Americans.
 Commenters (2687, 6147, 6158) express concern that this rule would greatly limit the data and models upon which EPA would rely when setting public health and environmental standards. Commenter (6158) also refers to the concerns expressed by Singla et al (2018) in their detailed comments to EPA submitted May 30, 2018.
 Commenters (2909, 5181, 8281-PH5, 8281-PH29) express concern that the proposed rule would limit the scientific research available to EPA policy makers to an extent that would undermine EPA's mission to protect public health and the environment.
 Commenter (6904) states that, if EPA excludes studies because the data cannot be made public, people may be exposed to real harm. The commenter states that the result would be decisions affecting millions based on inadequate information that fails to include well-supported studies by expert scientists.
 Commenter (4894) states that the proposed rule is based on a faulty premise, namely, that only studies whose underlying data are publicly available, sufficient to support replication, should be considered by the EPA as it develops regulations governing clean air, clean water, and exposure to toxic substances and pesticides.
 Commenter (4591) states that the goals of open science are not advanced through this proposal, which does not provide incentives, funding, or infrastructure for increasing access to data, but simply allows the Agency to disregard important, ethical, well-designed and executed studies.
 Commenter (6907) is concerned that the proposed rule on transparency would reduce the number of valid scientific studies utilized by EPA in conducting risk assessments and therefore, jeopardize the safety of pesticides and impact rulemaking under Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA).
 Commenter (6125) asserts that, while it is not flatly untrue that the proposal "does not compel" EPA to make the information available, it is worse than disingenuous not to mention also that unless the information is publicly available it cannot be used as a basis for the rulemaking decision unless EPA grants an exemption.
 Commenter (8281-PH84) expresses concern that, under this proposal, studies without a guarantee of full public access will be considered unreliable and will play no role in assessing a chemical's health effects on human health. The commenter states that this ignores the many ways in which the scientific community, regulators and the public have traditionally determined the quality and relevance of study results.
Response: In this final rule the EPA is not categorically excluding any studies from consideration when promulgating final significant regulatory actions or developing influential scientific information. The EPA has removed the term "science that serves as the basis for informing a significant regulatory action" from 40 CFR 30.1, and will not require that science that serves as the basis for informing a significant regulatory action is publicly available in a manner sufficient for independent validation. Instead, in the final rule, 40 CFR 30.4 requires that the EPA clearly identify all science that serves as the basis for informing a significant regulatory action, and make all such science that serves as the basis for informing a significant regulatory action publicly available to the extent permitted by law. The EPA clarifies that this requirement is consistent with the EPA's existing practice of making science that serves as the basis for informing a significant regulatory action available in the public docket as part of the rulemaking.
Comment: Commenter (4831) references several NAS reports and states that these reports encourage EPA to consider all available science in the rule-making process and provide guidance about how the agency could be more transparent in describing how evidence is gathered and evaluated.
Response: Consistent with existing Agency guidelines, the EPA will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information. The EPA will also continue to rely on the highest quality, most relevant studies available in determining the potential for hazard due to exposure to a pollutant, contaminant, or substance.
Comment: Commenters (4591, 6191) concur with the Association of American Medical Colleges, Association of American Universities, Association of Public and Land-grant Universities, and Council on Governmental Relations in their July 11, 2018 statement "Re: Docket Number EPA-HQ-OA-2018-0259-0025, Strengthening Transparency in Regulatory Science," which states that "[e]ven the strongest and most sincere supporters of the open science movement have recognized that there is value in research for which underlying data are not made publicly available and acknowledge an imperative to leverage all science to develop policies and regulations."
Response: EPA agrees with the commenter that there is value in research for which the underlying data and models cannot be made available. In consideration of this concern, in this final rule the EPA is not categorically excluding any studies from consideration when promulgating final significant regulatory actions or developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information. The EPA will also continue to rely on the highest quality, most relevant studies available in determining the potential for hazard due to exposure to a pollutant, contaminant, or substance.
Comment: Commenter (6135) opposes any effort by EPA, under the guise of transparency, to disregard the findings of scientific studies assessing the health impacts of air pollution when making regulatory decisions. The commenter states that, since there is no way to provide de-identified data in the public domain that would allow for secondary reanalysis of both the exposure assessment and health analysis in studies assessing the health impacts of air pollution, the EPA should instead continue to focus on assessing results across multiple studies to evaluate the veracity and general applicability of scientific findings.
Response: The EPA disagrees that this rule will cause the Agency to disregard the findings of scientific studies assessing the health impacts of air pollution when making regulatory decisions. The EPA is not categorically excluding any studies from consideration when promulgating final significant regulatory actions or developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information. The EPA will also continue to rely on the highest quality, most relevant studies available in determining the potential for hazard due to exposure to a pollutant, contaminant, or substance. The EPA acknowledges, and agrees with commenters, that there may be pivotal science for which the underlying dose-response data are not publicly available due to technological feasibility or cannot be made available in a secure environment that still allows for independent analysis. For example, dose-response data underlying older pivotal science may no longer be available or may not exist in a currently usable format. In these cases, the EPA may still use the pivotal science after either giving it lesser consideration or receiving an exemption from the requirements of this rule from the Administrator.
Comment: Commenter (6907) states that, while the proposed rule suggests that fraud is prevalent in studies where data are not published, this is not the case. The commenter states that, in many instances, these unpublished data protect the humans who are part of studies. The commenter states that the scientific process ensures that studies remain reproducible even when data cannot be made publicly available.
Response: The EPA is committed to protecting the privacy of study participants and will ensure that the implementation of this rule is consistent with law; protects privacy, confidentiality, and confidential business information (CBI); and is sensitive to national security interests. The EPA continues to believe that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will improve the opportunity for the public to access the EPA's scientific analyses and resulting regulatory actions in a way that is beneficial to the scientific process, the Agency's mission, and the public's health and safety.
Comment: Commenter (6915) states that, going forward, many academic scientists whose research is relevant to EPA regulations may not conduct and report their studies in a way that satisfies the requirements of the proposed rule. The commenter asserts that the proposed rule's provisions would require significant additional resources and could impose unreasonable and impractical requirements beyond those included in current protocols. The commenter states that academic researchers may not be concerned about or even consider whether their studies would qualify for use in establishing EPA regulations, and/or may not have the resources to reshape their approach to maintaining data. The commenter notes that, for those researchers who do attempt to comply with the proposed rule's requirements, the extent and nature of the data that must be maintained and made publicly available is vague and unclear, making compliance virtually impossible.
Response: EPA disagrees with the commenter that this rule would impose a burden of unreasonable or impractical requirements on researchers or other members of the public. Instead, this is an internal rule of agency procedure and thus puts no burden on any entity other than EPA. This rule is consistent with advances by the scientific community, which has also embraced greater data transparency, as evidenced by data sharing and availability requirements for many high-impact journals and the emergence of organizations, such as the Center for Open Science, and international initiatives like Findable, Accessible, Interoperable, and Reusable (FAIR) data principles; Facilitate Open Science Training in European Research (FOSTER); and Guidelines for Transparency and Openness Promotion (TOP) in Journal Policies and Practices that incentivize greater transparency in research[,].
<IP_ADDRESS> Generally Support Requirement to Make Regulatory Science Publicly Available
Comment: Commenters (0555, 0563) generally support regulatory science underlying its actions being required to be publicly available in a manner sufficient for independent validation The commenters generally express that often analyses underpinning regulations reflect policy preferences or are grounded on false information and that EPA's proposal to strengthen the transparency of its regulatory science would lead to better regulatory outcomes.
Commenter (0555) provides that regulatory impact analyses often hinge on assessments of risk that necessarily involve assumptions and judgments but often generate precise-sounding predictions that hide not only considerable uncertainty about the actual risk, but hidden policy judgements. The commenter states that too often scientific information presented to the public and decision-makers is infused with hidden policy preferences. The commenter asserts that EPA's proposal to strengthen the transparency of its regulatory science includes reasonable steps that could improve the evidential basis for its regulatory policies, and thus improve regulatory outcomes by targeting resources to where they can achieve the largest benefits. The commenter states that, as President Obama's science advisor observed, "open communication among scientists and engineers, and between these experts and the public, accelerates scientific and technological advancement, strengthens the economy, educates the Nation, and enhances democracy."
Commenter (0563) does not agree with the claim that peer review alone is sufficient quality control. The commenter asserts that using scientific research that is non-transparent and non-replicable increases the likelihood that policy will be grounded on false information. The commenter does not agree with the argument that EPA should be able to use all available scientific research and that excluding some may cripple its ability to protect us.
Commenter (0563) states that studies may rely on funding from sources that restrict disclosure of underlying data, and the rule would prohibit the EPA's use of those. The commenter states that is precisely why the rule is needed. The commenter adds that the funders should change their policies; and argues that their refusal suggests they have something to hide.
Commenter (0844) asserts that many of the arguments opposing the EPA's transparency rules are surreal. The commenter did not explain this position.