The commenter (12430) (representing California) notes that they rely on the legitimacy of this scientific review process for purposes of commenting on the NAAQS-setting process and developing and implementing its SIPs. The commenter states that the supplemental proposal does not address how the proposal would affect this scientific review process. If a finalized supplemental proposal prevents U.S. EPA and its CAA Advisory Committee from considering the best available science, the commenter contends that it would severely restrict U.S. EPA's decision-making when establishing, reviewing, or revising air quality criteria and NAAQS. According to the commenter, the resulting effect is likely to be confusion and disagreements between the California Air Resources Board (CARB) (and its counterparts in other states) and U.S. EPA about what studies can and should be considered. The commenter states that this is likely to harm the cooperative relationship between U.S. EPA and state air agencies, in addition to hindering the ability of U.S. EPA and CARB to meet the obligations of the CAA, and, ultimately, harming public health through the setting of substandard NAAQS.
The commenter (12430) states that the same dynamic would likely play out under other environmental laws that depend upon cooperative federalism, such as the CWA. Similarly, the commenter contends that this would risk damaging the cooperative federalism framework devised by Congress for these laws. The commenter asserts that the U.S. EPA's failure to consult with the states on the impact of this rule is also inconsistent with U.S. EPA's own primary goal set forth in its 2018-2022 Strategic Plan to create more effective partnerships with the states, among others, in carrying out shared responsibilities and communicating results to all Americans. Given the potentially significant impacts, the commenter states that the U.S. EPA's failure to analyze the cooperative federalism impacts of the SNPRM under E.O. 13132 further demonstrates the arbitrariness of U.S. EPA's rulemaking process.
Commenter (12435) states that this action will have substantial direct impact on the state's ability to use the best available scientific information to protect human health and the environment. The commenter adds that this action could also impact the state's ability to ensure more stringent cleanup requirements are used at EPA-lead Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA) sites when state standards and/or cleanup levels are based on best available science, but EPA standards and/or cleanup levels may be based on a limited set of data and models. Furthermore, the commenter states that the action could impact previous and forthcoming rules involving interstate air pollution flow and impede states with NAAQS nonattainment areas such as Colorado from legally addressing emissions in upwind states.
Response: This action does not have federalism implications as it imposes no enforceable duty on any state or local governments. It will not have substantial direct effects on the states, on the relationship between the national government and the states, or on the distribution of power and responsibilities among the various levels of government.
In addition, this final rule will not impact EPA's ability to rely upon best available science and does not establish an environmental health or safety standard. The final rule requires EPA to follow best practices to evaluate potential links between exposure to a pollutant, contaminant, or substance and the effect and the nature of the dose-response relationship. EPA will also use the following, well-established factors when assessing study quality: soundness, applicability and utility, clarity and completeness, uncertainty and variability, and evaluation and review. The EPA will continue to rely on the highest quality, most relevant studies available in determining the potential for hazard due to exposure to a pollutant, contaminant, or substance. As further detailed in the preamble to the final rule, only when characterizing the quantitative relationship between the amount of dose or exposure to a pollutant, contaminant, or substance and an effect, will the EPA identify pivotal science and give greater consideration to pivotal science for which the underlying dose-response data are available in a manner sufficient for independent validation. Title 40 CFR 30.5 also provides additional considerations for instances when underlying dose-response data are not available for independent validation. In addition, as this rule makes clear, if implementing this procedural rule would result in conflicts with existing environmental statutes, the procedural rule will yield to the existing statutes.
<IP_ADDRESS> Executive Order 13175: Consultation and Coordination with Indian Tribal Governments/Tribal Community Implications
Several comments (11407, 11895, 12415, 12423, 12474,) were submitted on the supplemental proposal related to Executive Order 13175, tribal government consultation requirements and tribal community implications of the proposal. This comments are provided below.
Comment: Commenters (11407, 12423, 12474, 13549, 14397) cite Executive Order 13175 legal obligations. According to commenters, the rule has potentially far-reaching and substantial impact on tribes and it should have triggered a tribal consultation process under the Executive Order.
Commenters (11895, 12474, 13549) states that they are disheartened by the supplemental proposal's lack of consideration and analysis relating to the proposal's impact on Indian Tribes and Environmental Justice communities: the supplemental proposal merely states, without further explanation, that "this action does not have tribal implications as specified in Executive Order 13175," and "EPA believes that this action is not subject to Executive Order 12898 (59 FR 7629, February 16, 1994) because it does not establish an environmental health or safety standard." 85 Fed. Reg. at 15,404. The commenters note that it is well deduced that the proposed rule will have a cascading effect on all future rules and as Tribes they will see it and will feel its repercussions.
Given the over-arching nature of the proposed rule's purview and particular impact on tribes, commenters (11895, 12423, 12474, 13549, 14397) state the Agency should have included a justification for not consulting with tribal governments under the Executive Order.
Response: This action imposes no enforceable duty on any state, local or tribal governments or the private Sector. Therefore, this action does not have tribal implications as specified in Executive Order 13175. Thus, Executive Order 13175 does not apply to this action.
Comment: Commenters (12423, 13549) express that tribal communities are likely to experience disproportionate adverse impacts as a result of the proposed rule/supplemental. Specific comments include:
Commenter (12423) states that many reservations are disproportionately impacted from higher levels of chemicals, pesticides, and contaminants, as well as from National Priority List Superfund Sites that release contaminants in the soils, water, and air, thus impacting tribal health.
The commenter (12423) asserts that tribal specific data are needed for tribes to be represented and protected by the EPA regulations that are intended to protect the environment tribes depend on for their very lifeways. According to the commenter, if this rule is enacted, tribes would need to alter their laws concerning data rights and management. For example, the commenter notes that some tribes only allow researchers access to tribal data for the duration of the study and require that, once the study concludes, the data be returned to the tribal nation. Additionally, the commenter adds that there are tribal sovereignty implications in protecting proprietary knowledge and substantial tribal member privacy concerns because of the relatively small populations sizes.
Commenters (12474, 13549) notes that it is well established that Tribal communities are likely to experience disproportionate exposure to harmful pollutants and chemicals. The commenters state that the proposal seeks to preclude the use of scientific research critical to establishing safeguards against this disproportionate exposure. Before finalizing any version of the proposed rule and/or supplemental proposal, the commenters assert that the EPA must listen to, analyze, and address Tribal concerns and impacts via public comments and government-to-government consultations.
Commenter (12415) contends that the rule's impact on tribes would be potentially significant and far-reaching, yet the EPA did not consult with the Confederated Tribes of the Umatilla Indian Reservation (CTUIR) (or any other tribe to our knowledge) about it. Furthermore, the commenter notes that given current circumstances regarding Coronavirus/COVID-19, particularly its acute, debilitating effects on Indian Country, there has not been adequate time nor resources available to fully engage in this rulemaking process, in addition to EPA's pervasive failures and shortcomings in terms of proper and meaningful government-to-government consultation.
Commenter (12474) states that they recently learned that the EPA is refusing Tribal consultation on this rulemaking. According to the commenter, one reason given is that this rule is being proposed under the EPA's housekeeping authority and is an internal rule. As they state in other comments on the proposal, the commenter states that this authority is not an appropriate vehicle to address a rule of this importance. The commenter asserts that this proposal will change the way scientific analyses are performed at the EPA. The commenter contends that it is wholly insufficient for EPA to force Tribes to save our comments for future rulemakings, which will be constrained by these problematic rules. The commenter notes that the proposed new rules will cite back to this proposed rule and will not have any recourse. The commenter states that another reason given by EPA for refusing consultation is the claim that this proposal does not have Tribal implications. According to the commenter, while this proposal may not impose direct regulatory requirements on Tribes, it will most certainly have implications on the ability of Tribes to protect the health of their citizens and therefore it has Tribal implications. The commenter asserts that if the EPA moves forward with this rulemaking, it must reverse its ill-conceived refusal to undertake the necessary and essential government-to-government Tribal consultations on this rulemaking. The commenter concludes that the EPA's refusal to offer Tribal consultation is a failure to uphold the 1984 Indian Policy.
Commenter (14397) states that in 2013, a two year review by the Department of the Interior's Secretarial Commission on Indian Trust Administration and Reform resulted in the recommendation that the U.S. government clarify that: all federal agencies have a trust responsibility to Indians; that this trust responsibility demands a high standard of conduct; and each Agency should place Indian interests before those of the Agency and outside parties. The commenter contends that, quite clearly, the EPA has not followed this doctrine.
Commenter (12423) urges the EPA to correct this by postponing any rulemaking until meaningful tribal consultation has been conducted.
Commenter (12415) states that the EPA has yet to explain how the proposed rule would implicate Treaty Rights, Treaty-secured resources, trust assets, how they would be safeguarded, and how they would or could be affected if the rule is adopted. The commenter notes that the fundamental question is, how will the EPA ensure that Treaty Rights are honored, and can be exercised, if you are less able to protect the resources on which those rights depend because you have adopted less-protective rules that excluded legitimate (although nonpublic) science and data?
The commenter (12415) encourages the EPA to follow its own Guidance, which to their knowledge has not been rescinded. This Guidance, "EPA Policy on Consultation and Coordination with Indian Tribes: Guidance for Discussing Tribal Treaty Rights," merits closer scrutiny, stating in part:
      EPA recognizes the importance of respecting tribal treaty rights and its obligation to do so. . . . EPA will . . . consider all relevant information obtained to help ensure that EPA's actions do not conflict with treaty rights, and to help ensure that EPA is fully informed when it seeks to implement its programs and to further protect treaty rights and resources when it has discretion to do so. [Footnote omitted]
The Guidance summarizes the significance of treaties:
      The U.S. Constitution defines treaties as part of the supreme law of the land, with the same legal force as federal statutes. Treaties are to be interpreted in accordance with the federal Indian canons of construction, a set of long-standing principles developed by courts to guide the interpretation of treaties between the U.S. government and Indian tribes. As the Supreme Court has explained, treaties should be construed liberally in favor of tribes, giving effect to the treaty terms as tribes would have understood them, with ambiguous provisions interpreted for their benefit. Only Congress may abrogate Indian treaty rights, and courts will not find that abrogation has occurred absent clear evidence of congressional intent. [Footnote omitted]
The Guidance specifically notes as an example that ". . . protecting fish may involve protection of water quality in the watershed[,]" and that:
       . . . [T]reaty rights most likely to be relevant to an EPA action are rights related to the protection or use of natural resources, or related to an environmental condition necessary to support the natural resource, that are found in treaties that are in effect. . . . "Treaties also may contain necessarily implied rights. For example, an explicit treaty right to fish in a specific area may include an implied right to sufficient water quantity or water quality to ensure that fishing is possible. Similarly, an explicit treaty right to hunt, fish, or gather may include an implied right to a certain level of environmental quality to maintain the activity or a guarantee of access to the activity site. [Emphasis added.]
A suggested question for EPA to explore is:
      How are treaty rights potentially affected by the proposed action? This question is designed to help EPA understand how a treaty right may be affected by the proposed action. EPA should explain the proposed action, provide any appropriate technical information that is available, and solicit input about any resource-based treaty rights. It is also appropriate to ask the tribe for any recommendations for EPA to consider to ensure a treaty right is protected.
The commenter (12415) states that they are not aware if EPA has addressed this question in this rulemaking process; if so, we do not feel that we have been provided a satisfactory answer to it.
The Guidance continues:
      EPA's next steps typically will involve conducting legal and policy analyses in order to determine how to protect the rights. These analyses are often complex and depend upon the context and circumstances of the particular situation. Issues that may arise often involve precedent-setting questions or warrant coordination with other federal agencies. It is expected that the EPA lead office or region that engaged in the tribal consultation about the potentially affected treaty rights will coordinate with the Office of International and Tribal Affairs, the Office of General Counsel, and appropriate Offices of Regional Counsel to conduct these analyses. Although the details of how to conduct such legal and policy analyses are not addressed by this Guidance, the EPA process may warrant continued or additional consultation with tribes.
Again, the commenter (12415) states that it is unclear whether EPA has performed such analyses for the proposed rule, which they contend that the EPA should.
Response: This is a rule of internal procedure promulgated under the EPA's housekeeping authority, and imposes no enforceable duty on any state, local or tribal governments or the private sector. In addition, the EPA does not agree that its approach will lead to systematic bias towards certain types of stakeholder goals of any party. As described in the preamble, the EPA is not categorically excluding any studies from consideration when promulgating significant regulatory actions and developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information. Statute-specific transparency regulations or programmatic regulations are forthcoming and EPA will conduct required analyses and consultations as appropriate in the course of those rulemakings.
Comment: Commenter (111407) provides the EPA with specific examples of ways the EPA must use best available science to fulfil its trust responsibilities to the Ute Mountain Ute Tribe to protect public health and the environment of the Ute Mountain Ute Tribe:
      1. EPA has a trust responsibility to intervene in disputes between delegated jurisdictions on surface water pollution and water quality standards issues, as well as approval of water quality standards. Much of the science behind those standards is peer reviewed science that goes back decades in time. flow can water quality managers cook up quick new science that complies with this proposed rule to properly regulate water quality? Is EPA going to reinvent that scientific wheel when the Ute Mountain Ute Tribe objects to a neighboring jurisdiction's standards or when an upstream entity objects to a regulatory action by the Tribe? These studies require years of research and complex formulas to protect human health and complex ecosystems from pollution in a way based on sound science.
      2. The Ute Mountain Ute Tribe has the only operating privately-owned conventional uranium mill in the U.S. next to its Reservation. The groundwater under and around the mill are being polluted. Innovative scientific research was undertaken in partnership with the U.S. Geologic Survey, Lawrence Livermore National Laboratory, U.S. EPA, and the Tribe to better understand the age and quality of the groundwater and to use the research as a baseline into the future for best available science in protecting the groundwater and identifying the sources of the pollution. Is EPA going to fund a new study? Is EPA going to debunk the peer-reviewed work of its own scientists and those from other leading scientific agencies with this rule making?"
Response: As described in the preamble, the EPA is not categorically excluding any studies from consideration when promulgating significant regulatory actions and developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information. Additionally, this rule does not require the EPA or other parties to disclose or host data. Hence, this rule does not impose costs on the EPA or any other party to make data available.
<IP_ADDRESS> Executive Order 13771: Reducing Regulations and Controlling Regulatory Costs
Comment: Commenters (12715, 12291) assert that pursuant to Executive Order 13771, agencies must assess and consider the costs of regulatory actions when making regulatory decisions. The commenter states that the supplemental proposal fails to assess costs to EPA and to researchers. Instead, the commenter notes that the EPA claims the supplemental proposal is not expected to be an Executive Order 13771 regulatory action because it relates to "agency organization, management or personnel." 85 Fed. Reg. at 15,404. Yet, the commenter asserts that EPA's proposed action does not fall within this category.
Response: This final rule does not regulate any entity outside the EPA. Rather, the requirements modify the EPA's internal procedures regarding the transparency of pivotal science underlying final significant regulatory actions and influential scientific information.
In addition, because this rule would not change the process by which the EPA already evaluates studies for scientific merit, the incremental costs of the rule would be for EPA to determine data availability for the limited subset of dose-response studies identified as pivotal science. This is a small increment in effort relative to the Agency's extensive existing scientific review process. Therefore, as described in the preamble discussion of costs and benefits, the EPA finds that the incremental costs of this rule related to the additional review of data availability are anticipated to be small.
<IP_ADDRESS> Executive Orders Require that the EPA Quantify Costs and Benefits of Regulatory Actions and Ensure the Public's Right to Participate in the Process
Comment: Commenter (11903) contends that the Trump Administration has assured the public, repeatedly, that it is committed to reduce the regulatory costs they will bear. The commenter provides that moreover, executive orders issued by previous Presidents address EPA's duty to identify and quantify the costs and benefits of its regulatory actions, and the public's right to participate in the process by which EPA does so. According to the commenter, those orders include Executive Order No. 12866, "Regulatory Planning and Review", 58 FR 51735 (1993), and Executive Order No. 13563, "Improving Regulation and Regulatory Review", 76 FR 3821 (2011).
The commenter (11903) provides that as Executive Order No. 13563 indicates, our regulatory system "must identify and use the best, most innovative, and least burdensome tools for achieving regulatory ends", and "[i]t must take into account benefits and costs, both quantitative and qualitative." The commenter notes that to that end, the order directs that
      to the extent permitted by law, each agency must, among other things: (1) propose or adopt a regulation only upon a reasoned determination that its benefits justify its costs (recognizing that some benefits and costs are difficult to quantify); (2) tailor its regulations to impose the least burden on society, consistent with obtaining regulatory objectives, taking into account, among other things, and to the extent practicable, the costs of cumulative regulations; (3) select, in choosing among alternative regulatory approaches, those approaches that maximize net benefits (including potential economic, environmental, public health and safety, and other advantages; distributive impacts; and equity); ....; and
      "In applying these principles, each agency is directed to use the best available techniques to quantify anticipated present and future benefits and costs as accurately as possible."
Additionally, the commenter (11903) notes that the Order specifically addresses the public's rights to participate in the process by which an Agency makes its "reasoned determination" that a rule's benefits justify its costs, tailors its rule "to impose the least burden on society", quantifies the benefits and costs "as accurately as possible", and chooses the approach that will "maximize net benefits":
According to the commenter (11903), the duties imposed by the Executive Orders do not exhaust the duties that EPA owes to the public, nor the duties EPA owes to the commenters in this rulemaking. The commenter notes that in the years since Chenery I, Overton Park, Industrial Union, and State Farm, the Federal courts have made amply clear that, when an Agency like EPA intends to promulgate a rule, it must meet the law's demands for reasoned justification. The commenter states that an Agency must provide: reasonable notice of its proposal; opportunity to comment on it; consideration of the comments received; appropriate response to comment; and a satisfactory justification for its final decision, including determinations based on evidence. The commenter contends that in sum, agencies such as EPA are obligated to avoid pretext and administrative evasiveness, and must instead engage in reasoned decision making, with comparative assessment to justify its choices, and develop a record showing that it did so. The commenter also maintains that accordingly, the law requires reasons, not just action, from EPA. The commenter asserts that the Agency's current managers should be fully familiar with the law's requirements by now, in part because they received frequent instruction from the Federal courts finding that EPA has, during the last three years, violated them.
Response: EPA has provided meaningful opportunities for input during the public comment period for both the 2018 proposed and the 2020 supplemental proposed rulemaking. The preamble for both these proposals includes a discussion of the importance of scientific integrity. The final rule further elaborates and provides the rationale for this action. In addition, EPA is providing a response to comments to address all the comments received during both the initial proposal and the supplemental.
This is a rule of internal procedure promulgated under the EPA's housekeeping authority, and imposes no enforceable duty on any state, local, or tribal governments or the private sector. Additionally, this rule does not require the EPA or other parties to disclose or host data. Hence, this rule does not impose costs on the EPA or any other party to make such data available. As described in the preamble discussion of benefits and costs, the EPA recognizes there will be some incremental review and administrative costs to the Agency but anticipates these will be small.
15.2.2 Paperwork Reduction Act
Comment: Commenters (11894, 12464) contend that the EPA has also failed to comply with the Paperwork Reduction Act (PRA), 44 USC §§ 3501 et seq. Commenter (11894) contends that the PRA imposes obligations on every federal Agency whenever the Agency proposes measures that will lead to the submission to the Agency of information from 10 or more individuals (or entities) and that the PRA requirements apply even when the submission of information by the public is voluntary. The commenter states that the proposed rule would plainly impact the practices and actions of scientists, economists, and a wide range of others with respect to the amount of information submitted to EPA.
According to commenter (12464), the proposal would create new reporting requirements that would apply to multiple persons who would have to make the underlying data available, either to the public in general, or through a tiered access system. According to the commenter, such reporting and data management procedures are costly and, in some instances, inconsistent with other federal policies governing information use and dissemination.
Commenter (11894) states that EPA should be required to prepare and submit an Information Collection Request (ICR) that estimates the burden expected to result from the proposed rulemaking. Further, the commenter states that the ICR should be available for public comment The commenter (11894) contends that the proposed rule totally ignores how it would impact existing reporting and recordkeeping practices. According to the commenter, if the current practices for reporting and recordkeeping of scientific research important for environmental regulation were deemed adequate, this proposed rule would be unnecessary; if they are not, EPA must intend the proposal to change those practices. The commenter notes that the PRA requires agencies to address how its proposal would modify those practices and to show why the proposal makes as little change as considered absolutely necessary. The commenter asserts that the EPA's proposal does not even begin to address this requirement.
Response: This action does not contain any information collection activities and therefore does not impose an information collection burden under the PRA. Further, as stated in the preamble, this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedures and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science. The EPA is only required to determine if such a mechanism was available, should a subject matter expert decide to use it.
15.2.3 Additional Related Comments
Comment: Commenter (12720) states that there is no legal authority under 5 U.S.C. 301; 5 U.S.C. App.; Pub. L. 98 - 80, 84 Stat. 2086 to adopt regulation under the proposed definition of "[i]nfluential scientific information", "data.", or capable of being substantially reproduced".
Response: As explained in the final rule preamble, EPA is not relying on 5 U.S.C. 301 in promulgating this internal procedural rule. Rather, EPA is promulgating this rule in accordance with its authority to promulgate regulations governing its internal affairs, as conferred upon EPA in the Reorganization Plan No. 3, 84 Stat. 2086 (July 9, 1970), which established the EPA.
Comment: Commenter (11482) notes that EPA's lack of analysis may constitute a violation of the APA and Executive Order requirements for promulgation of Agency rules.
Response: This is a rule of internal procedure promulgated under the EPA's housekeeping authority, and imposes no enforceable duty on any state, local, or tribal governments or the private sector. Hence, this rule does not impose costs on any party other than EPA. As described in the preamble discussion of benefits and costs, the EPA recognizes there will be some incremental review and administrative costs to the Agency but anticipates these will be small.
 Chapter 16: Federal Transparency and Data Integrity Laws
The 2018 proposal states that:
      The best available science must serve as the foundation of EPA's regulatory actions. Enhancing the transparency and validity of the scientific information relied upon by EPA strengthens the integrity of EPA's regulatory actions and its obligation to ensure the Agency is not arbitrary in its conclusions. By better informing the public, the Agency in enhancing the public's ability to understand and meaningfully participate in the regulatory process. In applying the best available science to its regulatory decision-making, EPA must comply with federal transparency and data integrity laws, and must also ensure that its decision-making is marked by independence, objectivity, transparency, clarity, and reproducibility. (83 FR 18769, April 30, 2018)
The EPA solicited comment on the proposal and how it can best be promulgated and implemented in light of existing law and prior Federal policies that already require increasing public access to data and influential scientific information used to inform federal regulation. (83 FR 18768, April 30, 2018)
This section provides responses to comments on existing transparency and data integrity laws. These comments are organized according to the following topics:
       Federal Laws and Law-Related Policies Already Address Data Concerns
       Information Quality Act (IQA), EPA's Information Quality Guidelines
       OMB Circular A-110, Freedom of Information Act (FOIA)
       OMB Memorandum 3-13
       Safe Drinking Water Act Regulations Have a Strong Scientific Basis and the Act Outlines How the Information is to be Shared
The specific comments provided in this Chapter reflect the comment letters that provide arguments supported by detailed rationale, examples, and references/documentation. The Docket ID numbers for these commenters are cited throughout this Chapter. In addition, some general comment letters may also include comments related to comments included and responded to in this Chapter. These comments are summarized in Appendix B. Note that points and issues raised in these general comment letters may also be raised by the more detailed comments summarized in this Chapter. The responses provided in this Chapter adequately address these more general comments.
16.1 Federal Laws and Law-Related Policies Already Address Data Access Concerns
Comment: Several commenters (2908, 6111, 6145, 6188, 6194, 6915, 6916, 9227) contend that federal laws and law-related policies already govern the quality of and access to raw data, including the Information Quality Act (IQA), Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity, or Information Disseminated by the Environmental Protection Agency (Oct. 2002) (EPA's Information Quality Guidelines), Office of Management and Budget (OMB) Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards ("OMB Uniform Guidance"), Data Access Act, OMB Circular A-110, Freedom of Information Act (FOIA), and OMB Memorandum 13-13.
Response: The EPA also agrees with commenters that government agencies are making great strides in data transparency pursuant to the laws and guidance cited by the commenters. However, EPA promulgated this rule because improvements can still be made over existing policies and mechanisms. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions.
Commenters (2249, 2908, 2906, 4931, 5120, 6145. 6147, 6149, 6168, 6194, 6373, 6899, 6925, 9227) suggest that the proposal is unnecessary, inconsistent and/or redundant considering existing laws, regulations, guidance or ongoing work by the scientific community that advance transparency-related goals. It is unclear to what extent this proposed rule would be redundant to or conflict with these existing initiatives. Commenters also assert that the proposed rule would unnecessarily add another layer of federal oversight to the vast collection of existing federal regulations.
Response: The EPA agrees with commenters that government agencies are making great strides in data transparency pursuant to the laws and guidance cited by the commenters. However, EPA promulgated this rule because improvements can still be made over existing policies and mechanisms. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions. The provisions of the final rule will not prevent consideration of the best available science and build upon, and do not supersede, existing executive orders and other policies that focus on transparency. Further, as stated in the preamble, this rulemaking is not intended to develop the Agency's interpretation of best available science. The EPA will continue to consider all peer-reviewed science evaluated in the EPA's influential scientific information and significant regulatory actions, consistent with existing study quality assessment factors and the statutory mandates specific to that rulemaking. Because the final rule does not categorically exclude any studies, including human health studies, from EPA's consideration, the Agency will not be prevented from evaluating and considering all quality, relevant studies when promulgating significant regulatory actions and developing influential scientific information. Further, the final rule only applies to dose-response data; therefore, all quality, relevant studies will continue to inform other aspects of the EPA's evaluations.
Commenter (6133, 6194, 6880) state that the provisions in the proposed rule that would exclude consideration of relevant scientific findings and conclusions if the underlying data is not publicly available are inconsistent with the 2009 Presidential Memorandum for the Heads of Executive Departments and Agencies on Scientific Integrity cited in the proposal.
Commenter (6133) states that the sentences quoted from the 2009 Presidential Memorandum in footnote 2 of the proposal are inaccurately used to support the proposed rule.
Response: In consideration of public comments, the EPA made several modifications of the original proposed text to more fully consider data privacy concerns. The EPA agrees that it is important not to categorically exclude any study because the data underlying a study at the stage required by this rulemaking may not be available for independent validation. Therefore, rather than finalizing an approach that would provide for a categorical exclusion of these studies, the EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information. The EPA is also including consideration of privacy concerns in the Administrator's exemption in 40 CFR 30.7. Because the final rule does not categorically exclude any studies from EPA's consideration, the Agency will not be prevented from evaluating and considering all quality, relevant studies when promulgating significant regulatory actions and developing influential scientific information. Further, the final rule only applies to dose-response data; therefore, all quality, relevant studies will continue to inform other aspects of the EPA's evaluations. The requirements in this final rule provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical considerations in order to ensure the Agency maintains a strong scientific basis for its decision-making. The incremental progress made possible by this rule provides an important step towards prioritizing transparency in particularly impactful EPA rules and assessments and will inform future statute-specific rulemakings. The EPA disagrees with the commenter's suggestion that implementation of the final rule would lead to suppression or alteration of scientific findings by political officials.
16.2 Information Quality Act (IQA) and Implementation Guidelines
16.2.1 Information Quality Act
Comment: Commenters (6126, 6194, 6916, 9227, 12727, 12430, 13788) state that the rule provisions are inconsistent with the IQA, FOIA, and OMB and EPA guidelines implementing the. The commenters contend that adding additional regulatory requirements and constraints is both impractical and unnecessary to achieve those goals.
Response: EPA disagrees that the final rule is inconsistent with the IQA. Implementation of the final rule will help to maximize both the quality, objectivity and utility of the pivotal science underlying the Agency's significant regulatory actions and influential scientific information. EPA's implementation of the final rule will be consistent with the Agency's obligations under the statutes cited by the commenter because the final rule does not require that the Agency disseminate information that is subject to limitations on disclosure such as confidentiality and privacy. The final rule also does not require the categorical exclusion of studies which was included in the 2018 proposal. The purpose of this action is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions.
16.2.2 Proposed Rule Consistency with OMB Information Quality Guidelines
Comment: Commenters (0555, 0844, 6141) support EPA's position that the proposal is consistent with or not radically different from the OMB Guidelines. Commenters (6102, 6150) assert that previous Administration commitments to improve transparency have included several policies that support increased transparency, including the OMB Guidelines, and that the proposed rule will help ensure that the policy goals reflected in the OMB Guidelines and memoranda are implemented as part of EPA's rulemakings.
Commenter (6364) emphasizes that EPA's regulatory science should be guided by and consistent with the OMB Guidelines, and they urge the EPA to reference this point in §30.10 of the rule to ensure that peer review satisfies the rigor of "independently validating key findings and ensuring that the quality of published information meets the standards of the scientific and technical community."
Response: EPA agrees that this rule is designed to build upon various guidelines on transparency and appreciates these comments. The role of independent peer review is addressed in 40 CFR 30.6 and it specifically directs the EPA to conduct peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review. The rule also gives the EPA discretion to conduct additional peer review of individual studies identified as pivotal science.
Comment: Commenters (1011, 4595, 6125, 6133, 6137, 6168, 6188, 6194, 6880, 6916, 9227, 12727) provide that EPA's assertion that the cited OMB Guidelines are consistent with the proposal is incorrect nor does the Agency explain why it believes the rule is consistent with these guidelines.
Response: EPA does not agree that this rule is inconsistent with the OMB Guidelines, rather it will complement and augment them and other transparency efforts. As discussed in the preamble to the final rule, the rule requirements are not changing the pre-existing requirements of the OMB Final Information Quality Bulletin for Peer Review. The preamble to the Bulletin states that "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review does not typically include reanalysis of the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information) and, thus, peer review is not considered a replacement for the data availability requirements of this rule. Therefore, the EPA is finalizing the role of independent peer review in 40 CFR 30.6. The provision directs the EPA to conduct peer review consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review. In order to account for the instances when the EPA determines that prior peer review was adequate, as defined by OMB, 40 CFR 30.6 gives the EPA discretion to conduct additional peer review of individual studies identified as pivotal science.
Additionally, in consideration of public comments, the EPA made several modifications of the original proposed text to more fully consider data privacy concerns. The EPA agrees that it is important not to categorically exclude any study because the data underlying a study at the stage required by this rulemaking may not be available for independent validation. Therefore, rather than finalizing an approach that would provide for a categorical exclusion of these studies, the EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information. The EPA is also including consideration of privacy concerns in the Administrator's exemption in 40 CFR 30.7. Because the final rule does not categorically exclude any studies, including human health studies, from EPA's consideration, the Agency will not be prevented from evaluating and considering all quality, relevant studies when promulgating significant regulatory actions and developing influential scientific information. Further, the final rule only applies to dose-response data; therefore, all quality, relevant studies will continue to inform other aspects of the EPA's evaluations.
Also, in consideration of public comments, the EPA provided a discussion of the costs and benefits of this rule in the preamble to the final rule. Briefly, this is a rule of internal procedure promulgated under the EPA's housekeeping authority and imposes no enforceable duty on any state, local, or tribal governments or the private sector. Additionally, this rule does not require the EPA or other parties to disclose or host data. Hence, this rule does not impose costs on the EPA or any other party to make such data available. The EPA recognizes there will be some incremental review and administrative costs to the Agency but anticipates these will be small.
Codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will improve the opportunity for the public to access the EPA's scientific analyses and resulting regulatory actions in a way that is beneficial to the scientific process, the Agency's mission, and the public's health and safety. The rule requirements provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical and privacy considerations in order to ensure the Agency maintains a strong scientific basis for its decision-making. The incremental progress made possible by this rule provides an important step towards prioritizing transparency in particularly impactful EPA rules and assessments and will inform future statute-specific rulemakings.
Finally, the EPA is relying exclusively on its housekeeping authority to promulgate this purely procedural rule. This final rule does not interpret or apply the provisions of any environmental statutes, nor does it modify the Agency's interpretations of "best available science." It therefore cannot conflict with the Data Quality Act. The EPA will continue to consider all peer-reviewed science, consistent with existing study quality assessment factors and corresponding statutory mandates. The EPA will then identify and consider "pivotal science in accordance with the provisions of this rule," unless the implementation of the rule conflicts with statutory requirements and associated implementing regulations. As this rule makes clear, if implementing this procedural rule would result in conflicts with existing environmental statutes, and their implementing regulations, this rule will yield to the EPA statutes and regulations.
16.2.3 EPA Information Quality Guidelines-Specific Comments
Comment: Commenter (5374) states that the proposed requirements are consistent with the EPA Guidelines and support for the proposed rule. The commenter provides that the EPA issued these guidelines to formalize and maximize the quality of disseminated information, particularly with respect to the objectivity, utility, and integrity of scientific data. The commenter asserts that no matter who conducts the work, the EPA should have access to all raw data so that it has the opportunity to review and independently verify data, results and conclusions before relying upon them in a pesticide risk assessment.
Response: EPA agrees that this rule is consistent with and augments EPA Guidelines on transparency. In this final procedural rule, the EPA does not interpret or apply provisions of a particular statute or statutes that it administers. The EPA will undertake such efforts in forthcoming actions, which will be either statute-specific science transparency regulations or programmatic regulations implementing this procedural rule. As this rule makes clear, if implementing this procedural rule would result in conflicts with existing environmental statutes, and their implementing regulations, this rule will yield to the EPA statutes and regulations.
As discussed in the preamble to the final rule, the EPA maintains that the appropriate stage of data is the stage consisting of the raw data modified to remove obvious errors. The EPA disagrees with the commenter that the raw data are the appropriate stage of data because third parties who are not familiar with the data collection process may not be able to appropriately identify errors in a raw dataset, which may inadvertently produce a skewed analysis.
Comment: Commenter (6903) asserts that the proposed rule is unnecessary because the EPA has already addressed the question of public data in its policies and guidelines. The commenter provides that the EPA Guidelines establish the Agency's culture and practice to ensure the quality of data used and disseminated by EPA and delineate clearly the Agency's commitment to public access to data. The commenter further states that the EPA Guidelines also establish a clear and effective policy that considers making data publicly available. For example, the commenter states that the EPA Guidelines provide that "[i]n cases where the Agency relies on proprietary models, these model applications are still subject to our Peer Review Policy."
Response: EPA does not agree that this rule is unnecessary. As detailed in the preamble to the final rule, the rule requirements for the EPA's independent evaluation of the availability of data are necessary and critical to prioritizing data transparency in the pivotal science underlying its significant regulatory actions and influential scientific information. The transparency provisions in this final rule are intended to build upon existing federal government efforts and provide incremental progress toward the Agency's goal of greater transparency.
Comment: Commenters (6111, 6133) provide that the EPA Guidelines took the position that the standard set forth in the Safe Drinking Water Act (SDWA) -  "the best available, peer-reviewed science"  -  should apply to all the Agency's risk assessments.
Commenters (6160, 6137) assert that, according to the EPA Guidelines (2005), EPA uses a "weight-of-evidence" approach that "considers all relevant information and its quality, consistent with the level of effort and complexity of detail appropriate to a particular risk assessment." EPA Guidelines 6.4. According to commenter (6137), the proposed rule contravenes these requirements and will prevent EPA from considering relevant information by precluding consideration of certain data. The commenter further states that, pursuant to the EPA Guidelines, EPA must ensure that the information it disseminates "is accurate, reliable and unbiased." EPA Guidelines 6.4(A). To do so, it uses:
      the best available science and supporting studies conducted in accordance with sound and objective scientific practices, including, when available, peer reviewed science and supporting studies; and (ii) data collected by accepted methods or best available methods (if the reliability of the method and the nature of the decision justifies the use of the data).
Id.
Commenters (6137, 9227) assert that, given that the Proposed Rule eliminates from consideration any scientific study where the underlying data cannot be made publicly available, it undoubtedly precludes the use of the best available science in certain situations.
Commenter (6194) states that EPA's guidance  directs that "influential scientific, financial, or statistical information" should be held to a higher degree of quality and transparency standards and that the EPA Guidelines clarify that "if access to data and methods cannot occur due to compelling interests such as privacy, trade secrets, intellectual property, and other confidentiality protections, EPA should, to the extent practicable, apply especially rigorous robustness checks to analytic results and carefully document all checks that were undertaken."
Commenter (6160) adds that the EPA Guidelines already address the transparency of data and methodology, including "(1) the source of the data used, (2) various assumptions employed, (3) the analytic methods applied, and (4) the statistical procedures employed." The commenter asserts that, because the EPA already has these methods in place, the proposed rule does nothing to improve upon them while creating unnecessary obstacles for EPA's consideration of the best available evidence for policy and rulemaking.
Response: This proposed rule will not prevent consideration of the best available science and will complement existing executive orders and other policies that focus on transparency. As stated in the preamble, this rulemaking is not intended to develop the Agency's interpretation of best available science. The EPA will continue to consider all peer-reviewed science evaluated in the EPA's influential scientific information and significant regulatory actions, consistent with existing study quality assessment factors and the statutory mandates specific to that rulemaking.
Additionally, EPA agrees that it is important not to categorically exclude any study because the data underlying a study at the stage required by this rulemaking may not be available for independent validation. Therefore, EPA is not finalizing the proposed option in the 2020 SNPRM that would categorically require that for studies to be considered pivotal science the underlying data would need to be available for independent validation. Because the final rule does not categorically exclude any studies from EPA's consideration, the Agency will not be prevented from evaluating and considering all quality, relevant studies when promulgating significant regulatory actions and developing influential scientific information. Further, the final rule only applies to dose-response data; therefore, all quality, relevant studies will continue to inform other aspects of the EPA's evaluations.
Comment: Commenter (6915) states that the rule provisions are inconsistent with the EPA's Information Quality Guidelines.  The commenter asserts that the guidelines direct the EPA to "ensure reproducibility according to commonly accepted scientific, financial, or statistical standards. . . . [I]f access to data and methods cannot occur due to compelling interests such as privacy, trade secrets, intellectual property, and other confidentiality protections, EPA should, to the extent practicable, apply especially rigorous robustness checks to analytic results and carefully document all checks that were undertaken." Id. at 21.
Commenter (6188) states that, according to EPA's own IQGs, EPA "maintain[s] a robust quality system that addresses [information collected through contracts with EPA; information collected through grants and cooperative agreements with EPA; and information submitted to EPA as part of a requirement under a statute, regulation, permit, order or other mandate] by including regulatory requirements for quality assurance for EPA contracts, grants, and assistance agreements. This includes an Agency-wide Quality System that requires, among other things, a quality assurance manager and Quality Assurance Project Plan (QAPP) for projects and tasks that involve the use of environmental data, and a Peer Review System that requires major scientifically- and technically-based Agency work product to be peer reviewed.
Commenter (6145) asserts that the EPA's Information Quality Guidelines demonstrate that EPA is already required to have transparency in its reliance on scientific studies used in the establishment of regulations. The commenter notes that these safeguards are sufficient to ensure that the science relied upon by EPA is the best available science and that it is subject to peer review and available to the public. According to commenter for example, with regard to ensuring and maximizing the quality of "influential" information (which includes information in support of agency actions including rules).
Commenter (9227) asserts that, pursuant to EPA's guidelines, EPA must ensure that the information it disseminates "is accurate, reliable and unbiased." EPA Guidelines 6.4(A). The commenter contends that, given that the Proposed Rule eliminates from consideration any scientific study where the underlying data cannot be made publicly available, it undoubtedly precludes the use of the best available science in certain situations.
Response: The EPA disagrees that the provisions of this rule are inconsistent with the EPA's IQGs or that the guidelines accomplish the same goals as the rule. As detailed in the preamble to the final rule, the rule requirements for the EPA's independent evaluation of the availability of data are necessary and critical to prioritizing data transparency in the pivotal science underlying its significant regulatory actions and influential scientific information. The transparency provisions in this final rule are intended to build upon existing federal government efforts and provide incremental progress toward the Agency's goal of greater transparency.
Comment: Commenter (6147) asserts that EPA already has many policies in place that assure the best available science is used in all elements of its analysis, including the following:
 Risk Characterization Policy (USEPA, 1995).
 USEPA's Information Quality Guidelines (IQG) (USEPA, 2002).
 USEPA Quality Policy (2008).
Response: The transparency provisions in this final rule are intended to build upon existing federal government efforts and provide incremental progress toward the Agency's goal of greater transparency. The EPA and the federal government have long encouraged open data initiatives, as the principle of transparency in regulatory decision-making and the other operations of government agencies is a fundamental behavior of good government that is inherently valuable to the public. For example, the EPA's 2016 Plan to Increase Access to Results of EPA-Funded Scientific Research noted that "transparency is a core EPA value" and that increased availability of research data would accelerate scientific breakthroughs that support the Agency's mission and policymaking efforts. The EPA's Open Government Plan 5.0 also details the EPA's progress in implementing the tenets of the numerous data transparency initiatives in the federal government prior to 2018, including OMB M-10-06 , the Office of Science and Technology Policy Memorandum of February 22, 2013 , and OMB M-13-13 .
The scientific community has also embraced greater data transparency, as evidenced by data sharing and availability requirements for many high-impact journals and the emergence of organizations, such as the Center for Open Science, and international initiatives like Findable, Accessible, Interoperable, and Reusable (FAIR) data principles; Facilitate Open Science Training in European Research (FOSTER); and Guidelines for Transparency and Openness Promotion (TOP) in Journal Policies and Practices that incentivize greater transparency in research.
The EPA supports these efforts and is pursuing an incremental approach to maximizing transparency in the science that it relies upon to ensure that implementation is done in a thoughtful and deliberate way that focuses on the EPA's most impactful actions, minimizes unintended consequences, and informs future transparency requirements. As more fully discussed in the preamble of the final rule, the EPA also agrees with commenters that the scientific community and government agencies are making great strides in data transparency; however, improvements can still be made over existing policies and mechanisms. Many scientific publications, for example, require authors to make a data availability or data access statement, which discloses where and under what conditions the underlying study data are available. Yet the EPA cannot solely rely on data availability statements made in published research because initiatives toward greater data sharing and transparency amongst scientific journals and international organizations are still being implemented, are inconsistently enforced, and the true accessibility of data in a public repository is still limited. Further, as detailed in OMB's Final Information Quality Bulletin for Peer Review, "the intensity of peer review is highly variable across journals" and "prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary." Peer review by either journals or independent advisory committees, such as the EPA's Science Advisory Board, is typically conducted without access to the underlying data (i.e., the proper stage of data where the data that are ready to be analyzed to extract relevant information). Should peer reviewers have access to the underlying data, they could conduct a more in-depth review that may include independently reanalyzing the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of uncertainty in the original study. Thus, peer review is not considered a replacement for the data availability requirements of this rule and implementation of this rule could provide independent peer reviewers as well as subject matter experts within the EPA and the public with the opportunity to evaluate pivotal science in ways that otherwise would not have typically occurred with existing EPA or peer review processes.
16.3 OMB Circular A-110, Freedom of Information Act
Comment: Commenters (0569, 6194, 6915, 9227) provide that, in 1999, the Shelby Amendment or Data Access Act (attached as a rider to the Omnibus Appropriations Act of fiscal year 1999, P.L. 105-277), mandated that OMB Circular A-110 be amended to ensure that all published data from federally sponsored research used for policy and rulemaking be made available through procedures established under the FOIA. Commenter (0569) asserts that this process accomplishes EPA's goal of allowing the public to verify the soundness of science underlying policy decisions. According to commenter (6915), the law promotes public access while protecting privacy by excluding medical and business-related confidential data from disclosure. See 2 C.F.R. § 200.315 (which superseded OMB Circular A-110, which has now been incorporated into the Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards). Specifically, commenter (6194) states that OMB Circular A-110 guidance defines "research data" in part as "recorded factual material commonly accepted in the scientific community as necessary to validate research findings," but exempts from disclosure any "medical information and similar information the disclosure of which would constitute a clearly unwarranted invasion of personal privacy, such as information that could be used to identify a particular person in a research study."
Response: EPA does not agree that FOIA mechanisms already achieve the goals of this rule. Availability via FOIA mechanisms does not mean that these data are immediately or easily available to the public. Some commenters cited the EPA's Freedom of Information Act Annual Report Fiscal Year 2019 (2020), which lists a median response time for "expedited processing" of FOIA requests by the EPA as 493 days (Ref. 42). Also, FOIA applies to federal agencies and cannot be used to compel other entities to release information. For these reasons, the EPA is modifying the definition in the final rule to add the following at the end of the definition: "the public must be able to access the information on the date of publication of the proposed rule for the significant regulatory action or dissemination of the draft influential scientific information for public review and comment" to increase the opportunity for independent validation to occur closer in time to the development of significant regulatory actions and influential scientific information.
Comment: Commenter (6955) states that there is no discussion of FOIA provisions of OMB Circular A-110 for obtaining research data. Hence, according to the commenter, there is no discussion of the differences between the two approaches and no claim that the new proposal offers any added value. The commenter asserts that the omission of discussion of OMB's FOIA provisions, as covered (briefly) in OMB Circular A-110 (OMB 1999, Fischer 2013), is an indication that the proposed rule is incomplete and another indication that it would be best if it were withdrawn. The commenter asserts that it appears that the proposed public-access part of the proposal is only needed, if the restriction of all studies to compliant ones survives to the final rule. In any case, according to the commenter, by omitting mention of the OMB and IQA FOIA options, the proposed rule gives the impression it is a public-access rule, rather than a study exclusionary rule.
Response: EPA disagrees that the final rule is incomplete because there is no discussion of FOIA. The final rule also does not require the categorical exclusion of studies which was included in the 2018 proposal. The Agency will consider all studies relevant to the development of a significant regulatory action or influential scientific information but will give greater consideration to the studies identified as pivotal science when the underlying dose-response data are available for independent validation. Also see the preceding comment response.
16.4 Open Data Policy
Comment: Commenter (6915) states that OMB Memorandum 13-13,  mandates, among other things, broader public access to federal and federally funded data and information and provides that information collection should be done in a way to support information dissemination. The commenter states that this includes building redaction, slicing, and exporting into how data are collected to reduce the cost of public access later on. The commenter states that the memorandum also requires agencies to create data catalogs to include datasets "that can be made publicly available but have not yet been released." Id.
Response: As EPA states in the preamble, this rule builds upon prior EPA actions in response to government-wide data access and sharing policies, including OMB Memorandum 13-13. However, improvements can still be made over existing policies and mechanisms. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions.

 Chapter 17: Consistency with Recommendations of Third-Party Organizations Who Advocated for Open Science and Policies Adopted by Major Scientific Journals
The proposal states that it "takes into consideration the policies or recommendations of third-party organizations who advocated for open science" and "the policies recently adopted by some major scientific journals, spurred in some part by the `replication crisis." 83 Fed. Reg. at 18,770. The proposal specifically cites to web sites for proceedings of the National Academy of Sciences, PLOS ONE (Public Library of Science), Science, Nature and The Economist. The proposal also states that "[t]hese include policies and recommendations from: The Administrative Conference of the United States' Science in the Administrative Process Project; National Academies' Reports on Improving Access to and Confidentiality of Research Data, Expanding Access to Research Data, and Access to Research Data in the 21st Century; the Health Effects Institute; Center for Open Science; members of the Risk Assessment Specialty Section of the Society of Toxicology, the Dose Response Section of the Society for Risk Analysis, and the International Society for Regulatory Toxicology and Pharmacology; and the Bipartisan Policy Center's Science for Policy Project." 83 Fed. Reg. at 18,770 n.10.
Comments were received on the notice of proposed rulemaking's claim that the EPA took into consideration the policies and recommendations of cited third-party organizations and the scientific community. Some commenters provide specific comments on third-party organizations and scientific journals cited as supporting the proposal where as others supported or opposed the proposal's claims more generally. Commenters also highlighted third-party and scientific community work and expertise and recommend that these experts be consulted in the development of the Policy on Regulatory Transparency. In this section these comments are organized according to the following topics:
       Clarification of "takes into consideration the policies or recommendations"
       Misrepresentation of Third-Party and Major Scientific Journal Recommendations
The specific comments provided in this Chapter reflect the comment letters that provide arguments supported by detailed rationale, examples, and references/documentation. The Docket ID numbers for these commenters are cited throughout this Chapter. In addition, some general comment letters may also include comments related to comments included and responded to in this Chapter. These comments are summarized in Appendix B. Note that points and issues raised in these general comment letters may also be raised by the more detailed comments summarized in this Chapter. The responses provided in this Chapter adequately address these more general comments.
17.1 Clarification of "takes into consideration the policies or recommendations"
Comment: Commenters (6133, 6137, 6194, 6448, 6915, 6916, 9227) request clarification of the basis of how the cited third-party organizations and major journals are considered in the proposal.
Commenters (6133, 6194, 6448, 6915, 6916, 9227) note that the proposed rule states that it "takes into consideration the policies or recommendations of third-party organizations who advocated for open science," but it is not clear which policies or recommendations were included as a basis for the rule beyond listing names of institutions.
Specifically, commenter (6133) states that the proposal does not explain how it takes into consideration the sources cited in footnotes 10-12. The commenter explains that, to the extent EPA is relying on these policies or recommendations, it has not provided enough information to evaluate that reliance and it must withdraw the proposed rule. The commenter further states that, consistent with the proposed rule's other citations, EPA points to nothing in the policies or recommendations from these third-party organizations that supports the proposed rule's preclusion of peer-reviewed science from consideration in regulatory decision making.
Commenter (6137) asserts that, not only is the proposed rule a drastic departure from EPA's prior policies, positions, and procedures, but it likewise is predicated on irrational and unsupported conclusions. The commenter states that a close examination of generally applicable data access policies and guidelines  -  including policies and recommendations of third-party organizations and major scientific journals upon which EPA allegedly relies  -  reveals that EPA's proposed rule is not based on a reasoned explanation and has no rational connection to facts in the record, but rather is entirely baseless.
Commenter (6194) argues that even when a specific report is referenced, such as the National Academies' Reports on Improving Access to and Confidentiality of Research Data, Expanding Access to Research Data, and Access to Research Data in the 21st Century, the proposed rule fails to cite the specific sections EPA purportedly considered or relied upon or the specific policies embodied in these reports that are allegedly advanced by the proposed rule. Id. at 18,770 n.10. The commenter states that other references are similarly vague.
Commenters (6915, 9227) state that the proposal does not identify any groups or reports that advocate precluding consideration of non-public data in regulatory decision-making.
Commenter (9227) expresses that the EPA does not explain how EPA's current use of science is inconsistent with any recommendations from third-party organizations or inadequate considering them. The commenter concludes that the EPA's failure to provide any specific information or citations in support of its conclusory statements make it impossible to meaningfully comment on the support for EPA's Proposal.
Response: EPA notes that the articles and third-party organizations cited in the April 30, 2018, notice of proposed rulemaking are not cited in the preamble of the final rule.
However, EPA referenced those sources in the April 30, 2018 notice of proposed rulemaking because EPA agrees with the general proposition advanced therein that increasing public access to the data underlying scientific studies is beneficial. The purpose of this action is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions.
17.2 Misrepresentation of Third-Party and Major Scientific Journal Recommendations
 17.2.1 General Comment Positions on Misrepresentation
Comment: Commenters (0670, 1589, 2787, 4591, 4831, 5172, 6046, 6125, 6126, 6133, 6137, 6155, 6166, 6168, 6880, 6912, 6915, 6916, 6924, 8281-PH20, 9227) assert that EPA misrepresents one or more of the third-party organizations and scientific journals/scientific sources cited by the EPA as supporting the proposed rule. One commenter (6155) notes that EPA does not actually state that the references support the rule, but only that they are "consistent" with it. Commenter (6125) notes that the proposal has included only an undifferentiated data dump of citations without identifying passages of relevance or explaining references to policies of other federal agencies. Commenters (6924, 6133, 6137, 6125, 6155, 6880) assert that the policies and recommendations of many of the cited third-party organizations do not support the EPA's proposal to preclude the consideration of peer-reviewed studies in regulatory decision making. They contend that in addition to misrepresenting these sources, the proposal rejects the policies or recommendations of many of the cited third-party organizations. Commenter (6133) asserts that all the cited scientific journal policies are for prospective publication, do not suggest disregarding consideration of studies without public data, and have exceptions to protect confidential or private information.
Commenters (2787, 6912, 6192, 6155) state that the proposal invokes policies from Science, and the Proceedings of the National Academies of Science as being consistent with the proposed rule, but each of these organizations has argued against, criticized, or distanced themselves from the rule. Commenter (6912) adds that, in addition to the criticism EPA received from the individuals, organizations, and publications it cited as supporting the proposed rule, virtually the entire scientific community opposed the proposed rule. The American Association for the Advancement of Science, sixty-nine public health and scientific groups, and almost 1000 individual scientists publicly opposed the rule. Many of these commenters note that the editors of all the scientific journals cited by EPA stated apropos of the proposed rule that "[e]xcluding relevant studies simply because they do not meet rigid transparency standards will adversely affect decision-making processes." Commenter (6912) further states that a co-author of the scientific articles cited by EPA stated that "[i]f the proposed rule is approved, science will be practically eliminated from all decision-making processes. Regulation would then depend uniquely on opinion and whim."
Commenter (6125) states that, upon examination, the other Executive Branch and EPA documents cited by EPA's proposal provide little support for its conclusions. According to the commenter, many of them expressly recognize that disclosure may not be possible, and no agency of the federal government that funds scientific research involving confidential health information provides for the automatic unfettered disclosure of data even when funding new studies. Instead, the commenter states that they make this a balancing judgment with the balancing factors being the added cost of disclosure, the feasibility of providing for disclosure, and the marginal gain in accuracy from disclosure. For example, the commenter states that the National Institutes of Health (NIH) policy allows investigators to promise subjects that their data will not be shared with other researchers, but only if the application provides an adequate justification.
Response: EPA agrees with the third-party organizations referenced in the April 30, 2018, notice of proposed rulemaking because EPA agrees with the general proposition that increasing public access to the data underlying scientific studies is beneficial. EPA notes that the articles and third-party organizations cited in the April 30, 2018, notice of proposed rulemaking are not cited in the preamble of the final rule.
The purpose of this action is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. These requirements build upon open data initiatives in the federal government and scientific community and advance the EPA's mission and commitment to the public by prioritizing transparency of the underlying dose-response data in pivotal science for the most impactful of EPA's assessments and regulatory actions.
The EPA expects to identify pivotal science, and the consideration afforded to pivotal science, in proposed significant regulatory actions and external review drafts of influential scientific information, which will allow the subject matter experts, if they so choose, to independently validate the pivotal science and provide comment to the EPA. The EPA believes that this approach will allow the public to more effectively comment, engage, and hold the EPA accountable during the future development of specific significant regulatory actions and influential scientific information.
The articles and third-party organizations cited in the April 30, 2018, notice of proposed rulemaking are not cited in the final rule. The preamble of the final rule provides discussion (with citations) of how the scientific community has embraced greater data transparency.
17.2.2 Administrative Conference of the United States' Study
Comment: Commenters (6125, 6133, 6137, 6912, 6916) specifically refer to the study for the Administrative Conference of the United States (ACUS) as not supporting the EPA's proposal.
Commenters (6125, 6133, 6192) report that the author of the ACUS' Science in the Administrative Process Project and the Bipartisan Policy Center's Science for Policy Project, Wendy Wagner, a law professor at the University of Texas, addressed EPA's use of her study as follows:
      I really don't know what the problem is that they think they're fixing," she said, adding that many of her co-authors "would laugh and hoot" at some of the scientific ideas expressed in the rule.
      They don't adopt any of our recommendations, and they go in a direction that's completely opposite, completely different," she told me after reading the rule. "They don't adopt any of the recommendations of any of the sources they cite. I'm not sure why they cited them.
For example, the commenters state that the ACUS Science in the Administrative Process Project, which EPA listed, recommends that agencies take a flexible approach to data disclosure: "To the extent practicable and in compliance with applicable legal restrictions, privileges, protections, and authorities, agencies should seek to provide disclosure of data underlying scientific research[.]" 78 Fed. Reg. 41,352, 41,358 (July 10, 2013). This recommendation thus states only that agencies should "seek to provide disclosure" of data "to the extent practicable" and acknowledges that many legitimate barriers to disclosure of data may exist. Id. Furthermore, ACUS contemplates scenarios where "data are not subject to legal or other protections" but where "the data's owners nonetheless will not provide such access." Id. In these cases, ACUS does not recommend that agencies deprive themselves of data. Rather, ACUS writes, "agencies should note [that the data's owner did not provide access] and explain why they used the results if they chose to do so." Id. Thus, the ACUS's policy in no way supports exclusion of studies simply because the underlying data is not publicly available.
According to the commenters, this recommendation thus states only that agencies should "seek to provide disclosure" of data "to the extent practicable" and acknowledges that many legitimate barriers to disclosure of data may exist. Id. Furthermore, ACUS contemplates scenarios where "data are not subject to legal or other protections" but where "the data's owners nonetheless will not provide such access." Id. In these cases, ACUS does not recommend that agencies deprive themselves of data. Rather, ACUS writes, "agencies should note [that the data's owner did not provide access] and explain why they used the results if they chose to do so." Id. Thus, the ACUS's policy in no way supports exclusion of studies simply because the underlying data is not https://intranet.ord.epa.gov/sites/default/files/2020-01/Creating%20a%20Dataset%20in%20ScienceHub%20for%20EPA%20Public%20Access.pdf publicly available.
Commenters (6137, 6916) state that the EPA contends that its proposed requirement to make data available is supported by the ACUS' Science in the Administrative Process Project. The commenters argue that the ACUS' policy does not support the exclusion of studies simply because the underlying data is not publicly available. The commenters assert that ACUS' policy supports the use of studies even when the underlying data cannot be made publicly available, including in cases where there are factors other than legal restrictions that limit the ability to publicly disclose data.
Response: The final rule does not reference the Administrative Conference of the United States (ACUS) Science in the Administrative Process Project or the Bipartisan Policy Center's Science for Policy Project.
With regard to concerns for cases that underlying data cannot be made publicly available, the Agency supports the use of restricted or tiered access for data and models that have confidential business information (CBI), proprietary data, or personally identifiable information (PII). Restricted or tiered access in this rule means that the underlying data or models are be available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
17.2.3 National Academies of Science (NAS) Reports
Comment: Commenters (4831, NAS, et al.) and other commenters (4591, 5172, 6046, 6924) express that EPA's contention that the National Academies of Science (NAS) Report(s) [Innovations in Federal Statistics: Combining Data While Protecting Privacy (NASEM, 2017b), and Federal Statistics, Multiple Data Sources, and Privacy Protections: Next Steps (NASEM, 2017c).] support the proposal is misleading. The commenters contend that there are important differences between the type of data that is the subject of the proposed rule and the data that is used strictly for statistical purposes that is the subject of the reports.
Commenter (4831) explains that the EPA's proposed rule references the NAS reports to identify current approaches for protecting confidentiality while providing data for statistical purposes, such as those used by the Federal Statistical Research Data Centers. The commenter states that the reports consider the kinds of data that are typically collected and acquired under pledges of confidentiality for exclusively statistical purposes  -  pledges that are backed by strong statutory protections, with criminal penalties for violations.
According to the commenters (4831), there are several differences in the confidential microdata collected from individuals and businesses by federal statistical agencies through surveys, versus data and results from the kinds of studies that are within the scope of the EPA proposed rule. The commenters assert that these differences have important implications about making data publicly accessible. The commenters state that what works well in the federal statistical environment may not translate effectively to EPA, where stakeholders might be strongly motivated to discount study results that run counter to their regulatory preferences.
In addition, the commenters (4831) assert that the EPA's proposed rule ignores the inherent risks involved in data disclosure, the everchanging risk landscape, and the efforts needed to mitigate those risks  -  all of which are discussed in the cited National Academies reports. For example, the commenters provide that the security of data held by federal agencies is exposed to new and evolving threats. In addition to cybersecurity concerns, the commenters state that computer scientists and cryptographers have demonstrated that statistical analyses of data sets that generate highly precise results  -  such as geographic specificity or other characteristics that identify respondents  -  may result in privacy breaches (NASEM, 2017b; NASEM 2017c). The commenters report that this presents a new challenge that federal statistical agencies are just beginning to address.
Commenter (6924) asserts that the rule is not consistent with the principles of open science, inappropriately codifies how science should be conducted, and codifies science-policy decisions in direct conflict with consensus reports from the NAS. The commenter reports that the EPA cites NAS studies and deliberations on the very important issue of data sharing and transparency as supporting the proposal, but the commenter asserts that the NAS reports contain conclusions that are in contradiction to the rule, such as:
 Clear statements that unfettered public access to data for any purpose, as proposed in the rule, is inappropriate, risky and unjustified. For example their reports state: "Since unrestricted access can cause harm to individuals and also conflicts directly with respect for individual autonomy, it is not an appropriate policy," and "The panel concludes that no one way is optimal for all data users or all purposes," and "The actual data collected for statistical purposes from households, individuals, business establishments, and other organizations through censuses and surveys under a pledge of confidentiality are never made available to users."
 Acknowledgement that all parties' interests in and purposes for performing an analysis and/ or validation of data would not be beneficial, as assumed in the rule, and thus protections are required. For example: "If not carefully considered, a reanalysis may be used as a tool to delay action. It also requires the participation of the original investigators," and "[The panel recommends] restriction of access to public-use data to those who agree to abide by the confidentiality protections governing such data and the institution of meaningful penalties for willful misuse of those data." This recommendation is meant to address potential use of data for non-research purposes, such as by marketers and companies for advertising.
 Assertion that simple de-identification of private information is not enough to protect an individual's data, which is counter to statements in the rule: "Statistical disclosure involves using data available outside the survey to breach the protection thought to have been afforded a survey data set by various data deletion and masking techniques."
Commenter (4591) reports that the recommendations from one report cited in the proposed rule of the National Academies of Science, Engineering, and Medicine, "Optimizing the Nation's Investment in Academic Research," were incorporated into the 21st Century Cures Act and discusses the potential burdens of data sharing policies, but it does not suggest that a policy such as the one being proposed by the EPA would be beneficial.
Commenters (5172, 6046) report that the proposed regulation cites a panel report published by the National Academies: "Expanding Access to Research Data: Reconciling Risks and Opportunities." According to the commenters, rather than providing support for the proposal, the National Academy Report conflicts with the proposal. The commenters explain:
 The executive summary of the report makes an important statement: "Research using detailed confidential data is needed ... for well- informed policy making ..." EPA's proposal conflicts with this statement; the proposal would deny the agency the use of studies that rely on confidential data.
 None of the report's 19 recommendations call for the government to stop relying on studies that involve confidential data.
Response: EPA supports making data and model publicly accessible to allow for reanalysis for independent validation. EPA is also committed to protecting CBI and PII according to all existing rules and regulations. The Agency supports the use of restricted access for data and models that have CBI, proprietary data, or PII. Restricted access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The EPA would like to emphasize that this rule does not impose requirements on any entity outside of EPA. This is a rule of internal procedures and does not direct or require any outside entity or EPA to establish data sharing mechanisms. Further, the rulemaking does not require the EPA to collect, store, or publicly disseminate data and models underlying pivotal science.
The studies noted in these comments above and/or referenced in the 2018 proposed rule are not cited in the final rule or its preamble.
17.2.4 National Research Council Reports
Comment: Commenter (6916) contends that two National Research Council Reports EPA cites also fail to support requiring release to the public of "data and models underlying scientific studies that are pivotal to the regulatory action." According to the commenter, far from supporting the concept of allowing reliance on studies only where the data are available to the public, these reports suggest "exploiting the research potential of microdata and maintaining acceptable levels of confidentiality" and advocate "nuanced case-by-case judgment[s]" for releasing underlying data. Nor, according to the commenter, do the articles the Agency cites on the reproducibility of studies support its Proposal, and EPA does not explain why it thinks they do. In fact, the commenter asserts that at least one of the cited articles, by Steven Goodman, is entirely irrelevant - it does not demonstrate that there is a problem with the reliability of science in support of Agency decision making, nor does it attempt to. Rather, the commenter states it discusses the confusion in the lexicon about reliability and attempts to more properly define the term "reproducibility." The commenter asserts that in no way does it support eliminating consideration of studies because the underlying data is not publicly available.
Response: EPA shares with the National Research Council, support for making data and models publicly accessible to allow reanalysis for independent validation. The Agency supports the use of restricted access for data and models that have CBI, proprietary data, or PII. Restricted access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information. See Section III.A.1 of the preamble to the final rule for a full discussion of how the scientific community has embraced greater data transparency and numerous cited examples that support data transparency.
17.2.5 Health Effects Institute (HEI)
Comment: Commenter (5015) agrees with the continuing need to enhance transparency and data access, but notes that these issues are not new, and have been addressed now for over 15 years by administrations from both parties and by the scientific community:
 This has included Guidelines for the Information Quality Act adopted by the Office of Information and Regulatory Affairs (OIRA) in 2002, numerous actions by the scientific community and journals to enhance access, and most recently the requirements for enhanced data access across the Federal Government promulgated by the Office of Science and Technology Policy (OSTP) in February 2013
 The commenter would strongly urge EPA to review the progress already made under these several major initiatives, and to carefully consider whether or not there are additional efforts that could further enhance transparency, before proceeding with a final rule.
Commenter (6912) states that the president of the Health Effects Institute noted that because it is costly and difficult to protect patient privacy, the kind of data reanalysis the rule promotes should only be done sparingly. According to the commenter, while the Health Effects Institute has a policy not to take public positions on regulatory questions, its president stated that the Institute's own data policy, which EPA claims supports the proposed rule, is "not quite what 's being proposed here.",
Response: As discussed in the preamble to the final rule, the EPA's Open Government Plan 5.0 also details the EPA's progress in implementing the tenets of the numerous data transparency initiatives in the federal government prior to 2018, including the Office of Management and Budget (OMB) M-10-06, the Office of Science and Technology Policy Memorandum of February 22, 2013, and OMB M-13-13. The journal articles and their underlying data resulting from intramural and extramural research funded by EPA are made available to the public.
EPA is committed to protecting CBI and PII according to all existing rules and regulations. The Agency supports the use of restricted access for data and models that have CBI, proprietary data, or PII. Restricted access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information. The EPA would like to emphasize that this rule does not impose requirements on any entity outside of EPA. This is a rule of internal procedures and does not direct or require any outside entity or EPA to establish data sharing mechanisms. Further, the rulemaking does not require the EPA to collect, store, or publicly disseminate data and models underlying pivotal science.
It should be noted that this final rule does not require reanalysis of a study's underlying data. Lewandowsky et al. (2020) evaluated the cost-effectiveness of reanalysis studies under various scenarios and concluded that reanalysis studies are most cost-effective when they are focused on studies of the greatest interest to the scientific community (in this study, the number of citations was a surrogate for interest). This finding is consistent with results in other studies that found and encouraged narrowing the focus of attempted reanalysis studies to those studies of greater significance (see preamble of final rule). In this final rule, rather than considering all studies that support the assessment of the relationship of a dose or exposure of a pollutant, contaminant, or substance to the magnitude of a predicted health or environmental impact as "pivotal science," the EPA is balancing transparency and feasibility by focusing on those studies that inform the quantitative relationship between the dose or exposure of a pollutant, contaminant, or substance and an effect.
17.2.6 Center for Open Science
Comment: Commenter (6137) provides that the Center for Open Science does not advocate for excluding science merely because the underlying data is not public, but rather counsels flexibility in determining what level of transparency is appropriate for a given study. The commenter explains that its Transparency and Openness Promotion (TOP) Guidelines for journals "recognize[] that not all of the standards are applicable to all journals or all disciplines. Therefore, the commenter explains that rather than advocating for a single set of guidelines, the TOP Committee defined three levels for each standard." These levels "provide flexibility for adoption depending on disciplinary variation." The commenter further explains that the Center for Open Science has invited journals to "suggest revisions that improve the guidelines or make them more flexible or adaptable for the needs of particular subdisciplines." As with other organizations upon which EPA relies, it too has adopted a flexible approach to transparency and thus provides no support for EPA's rigid exclusion.
Commenter (6912) states that the executive director of the Center for Open Science, whose policies were cited as supporting the proposed rule, stated that it was "20 percent off the rails."
Commenter (1589) states that the proposed rule cites open science advocates in footnote 10 and changes made in scientific journals in footnote 11, without understanding them or discussing their recommendations and conclusions.
Commenter (6360) states that, while all EPA applications of science should be covered by this rulemaking, they recommend that EPA draw from (or adopt) existing transparency screening tools developed by consensus in the scientific community to further define the scope and applicability of the rule's requirements.
Specifically, the commenter (6360) states that, many scientific institutions have developed requirements that scientists must follow to publish their claims in participating journals. The commenter states that there are several examples of such systems that have been developed with minimum levels of transparency to ensure a claim can be objective and reproducible.
The commenter (6360) points out that one important, open source system, the TOP guidelines, has adopted by over 5,000 major scientific organizations and journals. The commenter states that the TOP guidelines include eight standards, each with three levels of increasing stringency. The commenter states that Journals select which of the eight transparency standards they wish to adopt for their journal and select a level of implementation for each standard. For example, the commenter states that the scientific journal Science adopted most of the Level III TOP standards effective January 1, 2017. According to the commenter, claims made in a journal requiring Level III disclosure are more likely to be recognized as scientific facts than those published to Level I standards and the TOP Level II and Level III requirements mirror somewhat the applicability of the proposed standards in EPA's rulemaking. The commenter recommends that EPA require Level III transparency for all significant regulatory actions, and that all other EPA policy decisions have at least Level II standards for transparency to the public. The commenter states that as EPA takes comment on a proposed decision and notes that the underlying judgements in the applied science meet Level II criteria, the public could evaluate the claims and provide valuable feedback to EPA regarding the appropriate rigor of its data transparency procedures.
The commenter (6360) asserts that the TOP criteria require researchers to post data, analytic methods, and research materials to a trusted depository. The commenter recommends that for EPA applications of science, this depository should be the public docket of a proposed rule or the accompanying administrative record associated with a proposed decision. However, for sensitive information, the commenter recommends that EPA adopt the depository standards in common use by other federal agencies, journals, and EPA itself.
Response: EPA shares with the third-party organizations, support for making data publicly accessible to allow reanalysis for independent validation.
The Agency agrees that it is important not to categorically exclude any study because the underlying dose-response data at the stage required by this rulemaking may not be available for independent validation. EPA acknowledges and agrees with commenters that there may be pivotal science where the underlying data are not publicly available or available through restricted access. The final rule is limited to dose-response data and, as no studies are categorically excluded from consideration, the EPA will continue to rely on the full body of the highest quality, most relevant studies available in determining the potential for hazard due to exposure to a pollutant, contaminant, or substance. The EPA will also consider the degree to which pivotal science is available and will give greater consideration to pivotal science for which the underlying dose-response data are available, as described in the rule.
The Agency supports the use of restricted access for data and models that have CBI, proprietary data, or PII. Restricted access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Comment: Commenter (6362) suggests that EPA revise its statement that the proposed rule "takes into consideration the policies or recommendations of third-party organizations who [sic] advocated for open science." The commenter states that the recommendations referenced by EPA actually emanate from a survey of the members of three professional organizations whose memberships represent repositories of knowledge and experience in regulatory assessment. As such, the commenter requests that reference 10 in EPA's proposal be revised to also cite the publication, Expert Opinion on Regulatory Risk Assessment, A Survey by the Center for Media and Public Affairs (CMPA) and Center for Health and Risk Communication (CHRC) at George Mason University (December 6, 2013).
Response: Thank you for the recommendation. The final rule does not cite these references.
17.2.7 Risk Assessment Specialty Section of the Society of Toxicology and Dose-Response Specialty Group (DRSG) of the Society for Risk Analysis (SRA)
Comment: Commenters (1945, 1949, 6137) dispute that the EPA cites "policies and recommendations from . . . members of the Risk Assessment Specialty Section of the Society of Toxicology, the Dose Response Section of the Society for Risk Analysis, and the International Society for Regulatory Toxicology and Pharmacology[.]" 83 Fed. Reg. at 18,770, n.10. The commenters assert that responses to an online poll by individuals do not represent endorsement of Society of Toxicology (SOT), Dose-Response Specialty Group (DRSG) of the Society for Risk Analysis (SRA).
Commenter (6137) asserts that, in reality, these "policies and recommendations" are merely responses to an online questionnaire by a limited number of members of the three societies. The commenter recommends that the EPA not rely on a limited set of responses to an online poll, especially when the Agency does not possess information about conflicts of interest among respondents.
Commenter (1945, SOT) objects to Footnote 10. The commenter argues that the footnote ascribes support for the proposed rule from "members of the Risk Assessment Specialty Section of the Society of Toxicology." The commenter states that this is incorrect. The commenter, who represents the SOT, asserts that while our individual members are free to support or oppose issues as they see fit, it is not appropriate to infer that this constitutes an endorsement from either the Specialty Section or the SOT as a whole. The commenter respectfully requests that any and all references to "members of the Risk Assessment Specialty Section of the Society of Toxicology" be removed from the Final Rule.
Commenter (1949), the current Chair of the DRSG of the SRA, reports that the DRSG has never adopted any "policies or recommendations" on this or any other topic (see http://www.sra.org/drsg). According to the commenter, although "members of" the DRSG may have made statements on this topic in their personal or professional capacities, views of individual members should not be misconstrued as representing the view of the DRSG or of the SRA. Moreover, the commenter explains that the DRSG has a broad membership (including employees of the Environmental Protection Agency) that may have diverse views on this topic.
Commenter (1949) requests the following:
 In the Final Rule and any accompanying preamble text, EPA removes any and all references to the "Dose-Response Section [sic] of the Society for Risk Analysis."
 In the comment response prepared for the rule, EPA clarify the record that "third party organizations" whose policies and recommendations were considered do not include the Society for Risk Analysis or the Dose-Response Specialty Section.
Response: The final rule does not include references to the members of the Risk Assessment Specialty Section of the Society of Toxicology, the Dose Response Section of the Society for Risk Analysis, and the International Society for Regulatory Toxicology and Pharmacology.
17.2.8 Bipartisan Policy Center Report
Comment: The Bipartisan Policy Center (BPC) (0670, 6126) and other commenters (6133, 6137, 6166, 6880, 6915, 6916, 9227) assert that the proposed rule is not consistent with the BPC Report. The commenter suggests that EPA follow the procedures the BPC took to developing the report and consider the recommendations in the report which are relevant to EPA.
Commenter (0670, BPC)) reports that the EPA's press release for the proposed rule states that "Strengthening Transparency in Regulatory Science" is "consistent with ... recommendations from" the BPC Report. The commenter further reports that the "Background" section of the Federal Register notice for the proposed rule states: "The proposed rule takes into consideration the policies or recommendations of third-party organizations who advocated for open science" and cites the BPC Report, among others. The commenter states that they want to be clear that the proposed rule is not consistent with the BPC Report in substance or intent. Other commenters (6133, 6166, 6915, 9225, 9227) referred to the BPC comment stating that EPA had mischaracterized the BPC Report. One of these commenters (6915) states that EPA's false assertion of consistency with the policies and positions of leading science groups misleads the public and inhibits their informed participation
Several commenters (0670, 6166, 9225, 9227, 6125, 6137) assert that while the Science for Policy Project panel encouraged greater transparency and access to data, the report never suggested excluding studies from consideration in developing regulations if data from those studies were not publicly available. The commenters assert that the BPC notably concluded in its comments on this proposal that the proposal misrepresented its conclusions:
      While the Science for Policy Project panel encouraged greater transparency and access to data:
      The report never suggested excluding studies from consideration in developing regulation if data from those studies were not publicly available. Indeed, the panel's overarching recommendation for assembling the "best available science" reads: "Agencies and their scientific advisory committees should cast a wide net (emphasis added) in reviewing studies relevant to regulatory policy, and should make their methods for filtering and evaluating those studies more transparent."
Commenter (6880) states that the BPC's Science for Policy Project states "[s]tudies used in the formulation of regulation should be subject to data access requirements equivalent to those under the Data Access Act (Shelby Amendment) and its implementing circular regardless of who funded the study." The commenter asserts that the implementing circular is OMB Circular A-110 (https://www.gpo.gov/fdsys/pkg/FR-1999-10-08/pdf/99-26264.pdf), which directs federal agencies to use and share research data but specifically notes that "research data" do not include trade secrets and "[p]ersonnel and medical information and similar information the disclosure of which would constitute a clearly unwarranted invasion of personal privacy." (https://bipartisanpolicy.org/wpcontent/uploads/sites/default/files/BPC%20Science%20Report%20fnl.pdf.)
Commenter (6916) states that the BPC Report cited by the Agency recommends that decisionmakers should "cast a wide net" and "base policy on a thorough review of all relevant research and provisions of the relevant statutes," giving peer reviewed papers "great weight," but nowhere promotes the idea that sensitive or confidential raw data should be publicly released as suggested by EPA's proposed rule. According to the commenter, to the contrary, the paper recommends reliance on existing authorities and guidance, such as the Data Quality Act and the OMB Guidelines.
Commenter (6103) references the fact that the think tanks whose research was used to justify this new rule have said that they do not support the proposal. As an example, the commenter states that the BPC, whose research was used to support the supposed need for transparency, has stated in their comments on this proposal that the substance and intent of their research is not consistent with the administration's new policy. The commenter states that the BPC report from 2009 cited by EPA in the proposed rule supports looking at a wide array of data and making the method of evaluation more transparent, which does not coincide with the 'transparency' the EPA proposed rule seeks to promote. The commenter asserts that the EPA should use the best available science in the most effective way to truly fulfill its mission of protecting human health and the environment.
Response: The final rule does not reference the Bipartisan Policy Center's Science for Policy Project for policies and recommendations from third party organizations that were taken in consideration. The Bipartisan Policy Center is mentioned once, but only in noting that the Center was mentioned by a public commenter.
The Agency agrees that it is important not to categorically exclude any study because the underlying dose-response data at the stage required by this rulemaking may not be available for independent validation. EPA acknowledges and agrees with commenters that there may be pivotal science where the underlying data are not publicly available or available through restricted access. The final rule is limited to dose-response data and, as no studies are categorically excluded from consideration, the EPA will continue to rely on the full body of the highest quality, most relevant studies available in determining the potential for hazard due to exposure to a pollutant, contaminant, or substance. The EPA will also consider the degree to which pivotal science is available and will give greater consideration to pivotal science for which the underlying dose-response data are available, as described in the rule.
The Agency supports the use of restricted access for data and models that have CBI, proprietary data, or PII. Restricted access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
17.2.9 EPA's Proposal is inconsistent with policies adopted by major scientific journals
Comment: Commenters (4877, 4596, 6103, 6109, 6125, 6133, 6137, 6144, 6153, 6166, 6168, 6351, 6358, 6891, 6912, 6937, 8275, 9224, 9227) state that EPA's claim that the proposed rule is informed by the policies recently adopted by some major scientific journals (EPA cites Proceedings of the National Academy of Sciences, PLoS ONE, Science, and Nature) is incorrect and that the EPA's proposal is inconsistent with policies adopted by major scientific journals.
Commenter (9227) notes that while scientific journals and other institutions have encouraged making data and models publicly available, there is widespread recognition in the scientific community that doing so is often legitimately constrained due to legal and ethical protections on the confidentiality and privacy of data, or because the data is unavailable. Moreover, the commenter states that no scientist or scientific organization supports the Proposal's approach of excluding research for which the underlying data cannot be disclosed. According to the commenter, none of the materials EPA cites support such an extreme approach. To the contrary, the commenter states that the scientific community recognizes that the quality of a study is not determined by whether the underlying data is publicly available and has long utilized a variety of tools for ensuring the integrity and rigor of research findings. For all these reasons, the commenter notes that numerous representatives of the scientific community -- including editors of the very scientific journals whose policies EPA cites to in the Proposal, the American Association for the Advancement of Science, members of the Science Advisory Board (SAB), and other scientists cited to by EPA -- have already voiced serious concerns about the Proposal.
Commenter (6133) contends that these various journal policies are more flexible in their terms for data sharing and nuanced in their practical approaches than what EPA fundamentally misrepresents in the Proposal and that an examination of these sources indicates, in fact, that the language of the Proposal is not consistent with best practices and is unworkable in practice.
Commenter (6137) asserts that EPA provides no information as to how these journals' policies supposedly informed the data access guidelines and policies.
Commenter (4877) states that the data sharing policies that these journals use explicitly recognize the wide variety of workflows in use across scientific disciplines and provides for data sharing at different levels of openness. The commenter notes that they expressly state that not in every case can all data be shared, and in some cases, it is permissible for data to not be shared at all for legitimate and stated reasons. According to the commenter, as these journals  -  and the research community as a whole  -  well knows, this does not mean that the data underpinning these studies cannot be assessed. Commenters (6153, 6358, 6874) assert that reviewers can judge the merits of studies relying on non-public data, because "scientists are trained in assessing research publications by judging the articulation and logic of the research design, the clarity of the description of the methods used for data collection and analysis, and appropriate citation of previous results."
Commenter (6109) contends that EPA is proposing to dismiss from consideration research that would be admissible to these major scientific journals based upon an incorrect interpretation of their data-sharing policy and a disregard for the rigors of peer review.
Several commenters (4596, 6103, 6109, 6125, 6137, 6144, 6153, 6166, 6168, 6351, 6358, 6891, 6912, 6937, 8275, 9224, 9227) refer and/or support a statement submitted by the editors-in-chief of Science, Nature, PLOS, Proceedings of the National Academy of Sciences , and Cell issued in response to this proposed rule stating that this policy will exclude important studies from consideration in the rule-making process, thus adversely impacting the decision-making process. They note that these organizations concluded in their issued joint statement on the proposal:
      [I]t does not strengthen policies based on scientific evidence to limit the scientific evidence that can inform them; rather, it is paramount that the full suite of relevant science vetted through peer review, which includes ever more rigorous features, inform the landscape of decision making. Excluding relevant studies simply because they do not meet rigid transparency standards will adversely affect decision-making processes.
(http://science.sciencemag.org/content/early/2018/04/30/science.aau0116)
Commenter (6125) states that, clearly, the joint statement indicates that their policies on publication offer no support for the premise that studies are unreliable unless their underlying data is made publicly available.
Commenter (6144) argues that the EPA is incorrect in claiming that they are following accepted practice on open science that major science journals and the NAS have adopted. The commenter states that the open data initiatives adopted by these journals acknowledge that Transparency and Openness of Promotion (TOP) standards "recognize the array of workflows across scientific fields and make the case for data sharing at different levels of stringency; in not every case can all data be fully shared." The commenter states that these standards are in accordance with OSTP's 2013 memo Increasing Access to the Results of Federally Funded Scientific Research, which notes the importance of making data available in realizing the goal of increased transparency in the federal government, while protecting confidentiality, personal privacy, CBI, and "preserving the balance between the relative value of long-term preservation and access and the associated cost and administrative burden."
Commenter (6163) states that the journal policies and the TOP standards do not inhibit the use of research if the underlying data is not able to be published, such as sensitive PII and CBI.
Commenter (6164) asserts that the proposed policy wrongly claims to parallel TOP guidelines adopted by peer-reviewed scientific journals like those managed by the commenter (American Geophysical Union (AGU)). According to the commenter, unlike EPA's proposal, the TOP standards encourage open data while recognizing that each scientific discipline has its own constraints, as expressed by editors-in-chief of peer-reviewed publications cited in the proposed rule. The commenter states that, at times, underlying scientific information cannot be made public, particularly when it includes personal identifiable information that ethically and/or legally cannot be shared. The commenter states that even in these cases, the merits of a scientific study can still be evaluated by reviewers during the peer-review process and by scientists who are trained to assess scientific conclusions.
The commenter (6168) states that, at the very least, transparency requires careful and explicit discussion of the data, assumptions and boundary conditions, sources of error, accuracy, precision, and statistical significance. The commenter states that scientific journals also require introductory sections to orient the reader to the purpose of the study and the evidence from which the study is designed, methods sections to precisely define and describe the study design, and discussion sections to articulate results, their limitations, and their implications. Scientific journals also require extensive contextualization, and to take their data transparency requirements out of context is disingenuous and damaging. According to the commenter, providing raw data without context can cloak scientific research as much as providing no data; providing raw data encourages not transparency but even greater manipulation, which can be used to produce false uncertainty.
Commenter (6133) asserts that the proposed rule is not, as it claims, "consistent with requirements for many scientific journals." (EPA cites Taylor & Francis, Elsevier, PLOS and Springer Nature) Specifically, the commenter points out that the Taylor and Frances journal policy for data transparency is much more nuanced than EPA claims and offers a range of options for data submission, demonstrating the need for flexibility and discipline-specific concerns with respect to the public sharing of sensitive data. Additionally, the commenter states that the Springer Nature Research Data Policy is similarly flexible, describing requirements across a spectrum for four types of underlying research data. The commenter states that for only one of four types of research data is data sharing required as a condition for publication. The commenter states that the frequently asked questions document for the Springer Nature Data Policy notes that "[t]he policies apply to all research that support publications but reasonable restrictions on data availability are permitted to protect human privacy, biosafety or respect reasonable terms of use for data obtained under license from third parties." The proposed rule's categorical exclusion and prohibition are thus, according to the commenter, flatly inconsistent with the Springer Nature Research Data policy cited in the proposed rule.
The commenter (6133) also states that the Springer Nature polices began during the first quarter of 2016 and did not apply retroactively, as the Proposal would.
Commenter (6133) asserts Elsevier's corresponding policy is optional for authors, and states that the journal: "will . . . [e]ncourage and support researchers to share research data where appropriate and at the earliest opportunity, for example by enhancing our submission processes to make this easier." The commenter reports that a frequently asked questions page further explaining this policy says that the "policy is clear in that we encourage and support authors to share their research data rather than mandating them to do so and provide tools and services to enable them to do this effectively. Where there is community support for (often discipline-specific) mandates regarding data deposit, submission and sharing, some of our journals may reflect this with their own mandatory data sharing policies." The commenter states that this same supporting frequently asked questions resource from Elsevier says that Elsevier "respect[s] authors who need to keep research data under embargo." The proposed rule, by contrast, according to the commenter, does not allow researchers to keep their research data under embargo. Nor does the proposed rule offer such discipline-specific flexibility and, as a result, is neither practically workable nor consistent with the policies of the world's leading scientific journals.
Commenter (6133) explains that the Elsevier policy does not apply strict data release requirements to include publicly accessible information. It says that "[r]esearch data should be made available free of charge to all researchers wherever possible and with minimal reuse restrictions." It further states that "[r]esearchers should remain in control of how and when their research data is accessed and used, and should be recognized and valued for the investments they make in creating their research data and making it available." The commenter states that, under the proposed rule, researchers retain no such control over their data; the proposed rule ignores these harmful ramifications.
Commenter (6133) contends that the PLOS Data Availability Policy notes that, for studies involving human participants, "data must be handled so as to not compromise study participants' privacy." The PLOS Policy itself links to the National Institutes of Health Data Sharing Workbook, which states that: