While ACUS recommends agencies increase transparency of how they rely on scientific information and strive to make data underlying scientific information publicly available, nowhere do they suggest that agencies should not consider or rely on studies where underlying data and models cannot be made publicly available, or that these circumstances make scientific information less valid. They instead suggest that information be made publicly available for assessment and reproducibility purposes "[c]onsistent with the limitations in the Information Quality Act (IQA) guidelines issued by the Office of Management and Budget and its own IQA guidelines." They acknowledge valid limitations such as legal protections for privacy, trade secrets, and CBI. Thus, they recommend data be made public only "[t]o the extent practicable and permitted by law and applicable policies." Unlike the Proposal, the recommendation acknowledges that agencies may still use information where underlying data cannot be publicly disclosed, and suggest agencies "note that fact and explain why they used the results if they chose to do so." It thus provides a much more nuanced policy recommendation than that outlined in the Proposal -- which suggests EPA either find a way to make underlying data and models public, despite the numerous potential obstacles and concerns in doing so, or completely disregard the research study.
II. National Academies Improving Access to and Confidentiality of Research Data
Rather than containing any particular recommendations or policy proposals, this report discusses a number of issues pertaining to data disclosure and privacy protection, the tradeoffs "between increasing data access on the one hand and improving data security and confidentiality on the other," and "alternative approaches to limiting disclosure risk while facilitating data access the benefits and limitation of various approaches to these issues." Thus, rather than calling on agencies to rely only on scientific studies where the underlying data and models are made public, the report in fact discusses challenges and obstacles to achieving greater data disclosure, for which the Proposal provides no substantive or meaningful explanation.
The report discusses why exercising caution with respect to disclosing confidential personal information is so important, because if such information is exposed it could lead to
being arrested for a crime, being denied eligibility for welfare or Medicaid, being charged with tax evasion, losing a job or an election, failing to qualify for a mortgage, or having trouble getting into college. Disclosure of a history of alcoholism, mental illness, venereal disease, or illegitimacy can result in embarrassment and loss of reputation. Less directly, research results based on personal data can cause harm by affecting perceptions about a group to which a person belongs.
The report reveals very legitimate reasons why researchers and study participants would be reluctant to allow underlying data to be made publicly available -- and these reasons in no way compromise the validity of the scientific conclusions based upon this data.
The report also discusses the nuances of selecting methods to protect privacy while making underlying data publicly available. For example, while EPA casually makes claims that controlled access is an example of a solution in place across federal agencies -- this report points out the drawbacks of such an approach:
The use of restricted access arrangements, which has been deemed necessary to provide adequate protection for confidential information about individuals and businesses, results in increased costs to conduct research. Custodians of the data files need additional resources to process applications, operate inspection systems, staff research data centers, and inspect outputs to ensure that disclosure does not occur. Researchers require resources to prepare applications for access, to provide appropriate physical security for the data, or to visit a secure site.
The report also discusses the difficulty of funding such centers -- noting that while the costs are currently covered by a combination of federal agency budgets and user fees, including grants from the NSF and National Institute on Aging, federal funding may no longer be able to support such efforts. EPA's cursory mention to use of restricted access facilities as a potential solution to the concerns implicated by the Proposal fail to mention or address any of these challenges.
III. National Academies Expanding Access to Research Data: Reconciling Risks and Opportunities
EPA's Proposal in no way takes into consideration the recommendations of the National Academies report Expanding Access to Research Data: Reconciling Risks and Opportunities. This report considers competing approaches to increase use of research data while protecting confidentiality, and concludes that "no one way is optimal for all data users or all purposes" and, importantly, that "the nation's statistical and research agencies must provide both unrestricted access to anonymized public-use files and restricted access to detailed, individually identifiable confidential data for researchers under carefully specified conditions." In other words, the report finds that making data publicly available without restriction while respecting confidentiality concerns is not currently feasible or compatible with the missions of federal agencies.
Furthermore, the report mainly concerns itself with how agencies might increase access to data in their control and possession to allow for more research in social issues and provide a better basis for more informed policy decisions -- it does not discuss whether federal agencies should make data publicly available in order to allow for independent validation of scientific research they rely on for regulatory purposes and thus cannot be a basis for the Proposal. While the report discusses that one of the benefits of data sharing is that it allows for "verification, refutation, or refinement of original results," nowhere does the report suggest that agencies should rely only on research studies that make data publicly available or that such verification is necessary to validate a research study. Indeed, it details a discussion on this topic that presents competing views on requirements to make research data available to the public to allow for replication. John Bailar raised concerns that researchers would be deterred from doing certain kinds of work if they feared it would be subject to "hostile scrutiny" and that competitors could seize data for their interests. Others disagreed with this position. However, EPA failed to engage any of these considerations or at all justify its decision to implement a policy that could have severe negative implications. None of the researchers stated agencies should disregard the study if underlying data could not be made public.
The "recommendations" made by the report do not endorse EPA's proposal. The report provides 15 recommendations in Chapter 5. Recommendations 1-4 concern documentation and data access and call on agencies to better document how the data they make available is used; to use a variety of modes to provide access to data they produce or fund using a combination of restricted access to confidential data and unrestricted access to appropriately altered public-use data; to support research to guide more efficient allocation of resources among different data access modes; and to involve users in planning modes of access to their data.
In this Proposal, EPA does nothing to better document use of data that it makes public, has only called for a requirement to make research data and models "publicly available" rather than recognizing that a variety of modes and levels of access may be necessary, and does nothing to support more research into methods of making data more widely available without compromising confidentiality -- indeed blithely assuming that such means are already available and sufficient -- and also has not indicated that there has been any widespread call for EPA to make such data available or pointed to any comments of users of this data in this process.
Recommendations 5-8 concern public use data and call on agencies to support research on techniques to provide useful innovative public-use data that minimizes the risk of disclosure; streamlined procedures to allow researchers access to public-use microdata through existing and new data archives; a warning on all public-use data that they are provided for statistical purposes only and that any attempt to identify an individual is a violation, and requiring users to attest to having read the warning; and restricting access to public-use data to those who agree to abide by confidentiality protections, subject to meaningful penalties.
EPA's proposal once again ignores these recommendations that call for greater research and a measured approach to making data more widely available. The Proposal provides no ideas or methods or support for research that would help strengthen confidentiality protections while making data more available.
Recommendations 9-13 concern research data centers, remote access, and licensing agreements and call on the Census Bureau to (1) broaden the interpretation of the criteria for assessing the benefits of access to data; (2) maintain the continuous review cycle; and (3) take account of prior scientific review of research proposals by established peer review processes when awarding access to research data centers; for more research on cost effective means of providing secure access to confidential data by remote access; increasing use of licensing agreements for access to confidential data; working with data users to develop flexible, consistent standards for licensing agreements and implementation procedures for access to confidential data; and including auditing procedures and legal penalties in licensing agreements for willful misuse of confidential data.
EPA's proposal does not increase any research into use of remote data centers or licensing agreements, simply making passing references to these modes as potential solutions with no discussion or explanation -- and ignoring the recommendations here suggesting that more work is needed to realize their potential.
Recommendations 14-15 concern maintaining the public's trust and call on agencies to give certain basic information about confidentiality and data access to everyone asked to participate in statistical surveys; and to support continuing research on the views of data providers and the public about research benefits and risks.
EPA's proposal does not involve anything that increases the public knowledge about confidentiality protections or their views on research benefits and risks.
Recommendations 16-19 concern training, monitoring, and education to complement other protections on data. They call on data collection agencies to provide employees with continually updated written guidelines on confidentiality protection and training in confidentiality practices and data management and to institute procedures for monitoring violations of confidentiality protections practices and confidentiality breaches. They also call on educational and professional organizations to provide training in ethical issues for all those involved in the design, collection, distribution, and use of data obtained under pledges of confidentiality and for the development of strong codes of ethical conduct that reflect the need to protection confidentiality.
EPA's proposal also contains no provisions on increasing training, monitoring, or education, within the agency or among researchers to allow for more careful handling of confidential data.
Thus, EPA's Proposal completely ignores the careful research and thinking the National Academies and researchers have done on what is needed from federal agencies in order to make data more publicly available, and how to do so in a responsible manner. It does not implement any of the recommendations in the report, and in no way builds upon this work.
IV. National Academies Access to Research Data in the 21st Century: An Ongoing Dialogue Among Interested Parties: Report of Workshop
EPA cites to the National Academies' Access to Research Data in the 21st Century: An Ongoing Dialogue Among Interested Parties: Report of Workshop as one for which it took into consideration "policies or recommendations," despite the fact that this report comes with the explicit limitation that:
The goal of the workshop was not to reach conclusions or recommendations; nor could it address other pressing issues beyond the regulatory process, such as protection of intellectual property, the influence of broader access on scientific competition, the potential for increased administrative burdens and changes in the research process, and the challenge of providing data access in an increasingly electronic world.
Thus, this report stresses the many unanswered, challenging policy questions that must be addressed as agencies contemplate how to make data publicly available. These are the questions EPA should have addressed in its Proposal but did not.
The Report offers a look into the scientific review process that also calls into question the underlying assumption in EPA's proposal -- that making data publicly available is necessary to ensure the validity of a scientific finding. The report notes that scientific claims "are not `binary'" they instead "fall in the category of being uncertain to various degrees." The reliability of a particular scientific finding can be assessed using various mechanisms, starting with an examination of the strength of the design, methods, and statistical results. Then "one asks whether there is consistency within the data (pertaining to mechanisms of effect or related outcomes) and with other studies and scientific theories." Finally, "the robustness of the findings is evaluated through the use of different analytical approaches."
The report describes how studies may be validated through a range of approaches. While it notes that in some cases it is possible to exactly replicate the original study, this is not always the case, especially in large epidemiological studies where "repeating a study is seldom either possible or desirable." Then "replication" can take a variety of forms, not all of which require access to underlying data, including:
 Additional analyses done on the data set by the original or collaborating Investigators;
 New results generated from older data sets;
 New studies addressing the same hypothesis;
 Independent analysis of the same data set by different people;
 Monitoring of the results of actions taken on the basis of the findings.
Another form of replication the report describes is meta-analysis, which is a systematic strategy for comprehensively describing and summarizing a body of research evidence from two or more studies. The goal is to produce a quantitative synthesis of the evidence presented in multiple studies that relate to a research question. In a typical meta-analysis, all the data used have been published in the public domain and are easy to inspect and analyze.
The report specifically mentions the Harvard Six Cities Study as an example of a study where data could not be made publicly available, but which was verified to allow the agency to justifiably rely on it to set important air standards. Thus, unlike the Proposal the report acknowledges the many different pathways that exist to for researchers to assess other studies, and does not suggest that allowing the general public access to underlying data and models is necessary.
One of the panels of the workshop discussed the Shelby Amendment, and public access to data underlying agency regulation. A bench scientist expressed concerns that, though the idea of sharing data was a good idea, because any person could request information for any reason, this mechanism could be used to harass scientists whose work was found objectionable. A representative of NIH similarly stated that while sharing data with other researchers was good scientific practice, allowing for indiscriminate public access to data serves "little purpose for those without the skills to reanalyze it." Additionally, access through FOIA does not allow for limitations to be put on the use of the data, which is typically available in other data-sharing modes. A representative from EPA raised issues including:
The Shelby Amendment. . . raises several questions for the EPA about rulemaking as a legal and deliberative process. At what point should the agency disclose what type of regulation is going to be considered or issued? The timing of the release can influence its reception. Should the agency use contracts to support the research needed for regulations? Contracting, as opposed to grants that support more flexible work, might narrow the type of information the agency receives and could possibly limit the scope of the science underlying the regulation.
These questions and concerns are highly relevant to the Proposal as well, yet EPA provides no indication that it has given them any consideration.
Finally, a representative from NRDC pointed to other mechanisms that are already in place to ensure agencies rely on high quality data. For example, under the APA, agencies must respond to any comments that raise questions about a scientific studies design, performance, or conclusion. Courts can determine whether an agency was reasonable in its decision to refuse to accept the findings of a study because it could not access underlying data or refuses a request from a study participant. EPA does not explain why these existing mechanisms are not sufficient to ensure the integrity of the science it relies on.
V. The HEI
In the original Federal Register notice, EPA provided no specificity as to which Health Effects policy EPA was referring to or why it supported the Proposal. Such a vague and unspecified reference does not meet the notice requirements of the APA and other statutes and makes it impossible to respond.
VI. COS
In the original Federal Register notice, EPA provided no specificity as to which COS policy EPA was referring to or why it supported the Proposal. Such a vague and unspecified reference does not meet the notice requirements of the APA and other statutes and makes it impossible to respond.
VII. Members of the Risk Assessment Specialty Section of the Society of Toxicology (SOT), the Dose Response Section of the Society for Risk Analysis (SRA), and the International Society for Regulatory Toxicology and Pharmacology (ISRTP)
In the original Federal Register notice, EPA provided no specificity as to which policy of the Members of the Risk Assessment Specialty Section of the SOT, the Dose Response Section of the SRA, and the ISRTP EPA was referring to or why it supported the Proposal. Such a vague and unspecified reference does not meet the notice requirements of the APA and other statutes and makes it impossible to respond.
VIII. Bipartisan Policy Center's Science for Policy Project
In the original Federal Register notice, EPA provided no specificity as to which Bipartisan Policy Center's Science for Policy Project policy EPA was referring to or why it supported the Proposal. Such a vague and unspecified reference does not meet the notice requirements of the APA and other statutes and makes it impossible to respond.
Response: Regarding the original proposal, see response to Comment 2157-149.
Comment [8750-1908]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 11 of the proposal:
      For example, see related policies from the Proceedings of the National Academy of Sciences, PLOS ONE, Science, and Nature.
The commenter states that the EPA claims that the Proposal takes into consideration policies adopted by scientific journals, but does not specify which "related policies" from these journals. While some of these journals have adopted certain policies encouraging or requiring researchers to share underlying data for the studies they publish, they all allow for exceptions when data cannot be released for compelling reasons, such as confidentiality protections.
Furthermore, the editors of these journals have issued a joint statement opposing the Proposal and noting that their policies do not endorse such an approach by EPA. They note that some data sets cannot be shared publicly, and that there are still other methods available to verify scientific findings. The statement also strongly condemns the notion of excluding scientific information from consideration when underlying data cannot be made publicly available:
It does not strengthen policies based on scientific evidence to limit the scientific evidence that can inform them; rather, it is paramount that the full suite of relevant science vetted through peer review, which includes ever more rigorous features, inform the landscape of decision making. Excluding relevant studies simply because they do not meet rigid transparency standards will adversely affect decision-making processes.
Thus, EPA cannot claim that the Proposal is in any way supported by the data sharing policies of these scientific journals.
Response: See response to Comment 4829-2022.
Comment [8750-1911]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 12 of the proposal:
      See: https://www.nature.com/articles/s41562- 016-0021; http://journals.plos.org/plosmedicine/ article?id=10.1371/journal.pmed.0020124; http://science.sciencemag.org/content/343/6168/229.long; https://www.economist.com/news/leaders/21588069-scientific-research-has-changedworld-now-it-needs-change-itself-how-science-goes-wrong.; http://stm.sciencemag.org/content/8/341/341ps12.full.
The commenter states that the EPA claims that the Proposal is informed by the policies of scientific journals in response to the "replication crisis." EPA provides no explanation or evidence to support the fact that such a "crisis" is occurring or that EPA's Proposal would do anything to address the crisis. The sources EPA cites for this proposition speak to a concern about scientific studies being reproducible or replicable due to a number of different conditions related to poor scientific practices. While some of the articles speak about making data more available as an ideal to aspire to, none of them support the idea that a research study whose underlying data has not been made publicly available should, for that reason alone, be considered invalid. Further, many of these articles speak to how current scientific norms do not result in underlying data being available, which is a huge barrier to EPA's Proposal that EPA does not at all address. Specific commenter provided by the commenter include the following:
I. Marcus R. Munafó et. al, A Manifesto for Reproducible Science, 1 Nature Human Behavior 1 (2017)
Far from suggesting that agencies rely only on scientific studies if the underlying data is made public, or even that making underlying data public is necessary to ensure validity of scientific conclusions, the article discusses at a high level a number of systemic and cultural challenges to reproducible science. By ignoring the nuances of this article and presenting it without any explanation as support for its Proposal, EPA runs into the problem the article specifically cautions against, warning: "Some solutions may be ineffective or even harmful to the efficiency and reliability of science, even if conceptually they appear sensible."
This article does not endorse the existence of a "replication crisis" and in fact says, "[w]hether `crisis' is the appropriate term to describe the current state or trajectory of science is debatable." Instead it notes a very different problem than the one EPA appears to target with the Proposal. It points broadly to an issue of there being "substantial room for improvement with regard to research practices to maximize the efficiency of the research community's use of the public's financial investment in research."
This article makes clear that open data requirements are just one of many solutions and steps to take towards increasing efficiency of use of resources and robustness of scientific findings -- and never suggests that a lack of publicly available underlying data should automatically disqualify a research finding from consideration. It discusses a number of other improvements including protecting against cognitive biases through blinding, improving methodological training, implementing methodological support, encouraging collaboration and team science, promoting study pre-registration, improving quality of reporting, diversifying peer review, and changing incentives to promote efficient and effective research instead of just innovative outcomes.
While the article recognizes transparency as a "scientific ideal" it notes many challenges that currently exist to achieving this ideal, which EPA does not at all address. The article notes, "In reality, science often lacks openness: many published articles are not available to people without a personal or institutional subscription, and most data, materials and code supporting research outcomes are not made accessible, for example, in a public repository." It further finds "substantial barriers to meeting these ideals, including vested financial interests (particularly in scholarly publishing) and few incentives for researchers to pursue open practices." Nowhere does the article suggest that the many scientific studies for which data is not available due to prevailing scientific norms and practices be completely discarded. These challenges suggest that many studies EPA wishes to rely on may not be able to meet the rigid requirements of EPA's proposal severely restricting the science EPA can use, degrading the quality of its decision-making.
Marcus R. Munafó, lead author on this paper, has since published a piece specifically dismissing science policy approaches that overemphasize the importance of replication. It states that the overemphasis on replicability is detrimental to science -- that "[i]f a study is skewed and replications recapitulate that approach, findings will be consistently incorrect or biased." Instead, the author suggests that "an essential protection against flawed ideas is triangulation" or "the strategic use of multiple approaches to address one question." This involves looking at a broad base of different scientific studies and does not require underlying data to be made publicly available, not individual studies based on whether or not they can be replicated. By excluding scientific studies from EPA's consideration, the Proposal overemphasizes the value of replication to the detriment of being able to evaluate a study in the context of many other studies examining the same issue through a variety of methods. The Proposal may well lead to reliance on less robust science and is thus arbitrary.
II. John P.A. Ioannidis, Why Most Published Research is False, 2 PLoS Medicine 0696 (2005)
The article suggests "the high rate of non-replication (lack of confirmation) of research discoveries is a consequence of the convenient, yet ill-founded strategy of claiming conclusive research findings solely on the basis of a single study assessed by formal statistical significance, typically for a p-value less than 0.05." It looks at a number of different contributors to false positive findings and discusses solutions to this problem. Importantly, it stresses the need to focus on large studies, consider the totality of the evidence, and improve understanding of pre-study odds. These solutions each involve considering more evidence and more scientific studies to contextualize any one given study. Nowhere does the article suggests requiring underlying data be made public or fewer studies be considered. EPA's proposal contrarily emphasizes data disclosure above all other practices for ensuring scientific integrity -- and will result in fewer studies being considered to shed light on the scientific truth.
The author of this article has specifically criticized EPA's Proposal, saying that, if it is finalized, "science will be practically eliminated from all decision-making processes" and "[r]egulation would then depend uniquely on opinion and whim." The author highlights the inherent problem in EPA's Proposal, that "most of the raw data from past studies are not publicly available" and that indeed "[i]n a random sample of the biomedical literature (2000 - 2014) none of 268 papers shared all of their raw data. . . [and] [o]nly one shared a full research protocol." EPA has not addressed this major issue that suggests the Proposal would bar EPA from relying on massive amounts of scientific research. The article notes that reproducibility issues vary across the disciplines and that in many areas in which EPA operates, a solid and large foundation of scientific research has produced credible and widely-affirmed findings, including "in fields such as air pollution and climate change." Even in these other fields, however, it firmly states that "simply ignoring science that has not yet attained such standards, is a nightmare."
III. Marcia McNutt, Reproducibility, 343 Science 229 (2014)
EPA cites an announcement by Science that, in response to reports "that a troubling proportion of peer-reviewed preclinical studies are not reproducible," Science is adopting new policies requiring authors making submissions to the journal to disclose "whether there was a pre-experimental plan for data handling (such as how to deal with outliers), whether they conducted a sample size estimation to ensure a sufficient signal-to-noise ratio, whether samples were treated randomly, and whether the experimenter was blind to the conduct of the experiment." While the article considers steps to increase reproducibility of science, it notes that data availability is not a necessary or sufficient step to ensure credibility of research findings, and that "ultimate responsibility lies with authors to be completely open with their methods, all of their findings, and the possible pitfalls that could invalidate their conclusions." EPA's Proposal ignores the ability to assess studies through these other important indicators to assure their validity.
VI. How Science Goes Wrong, Economist (Oct. 21, 2013)
This article opposes the view that verification of a study depends solely on the underlying data being made publicly available. While it identifies that much scientific research is unable to be replicated, the solution it proposes include tightening standards, particularly in statistics, registering research protocols in advance and monitoring them, and: "[w]here possible, trial data also should be open for other researchers to inspect and test." Thus, even to the extent it discusses data availability, it suggests data should be open for other researchers, as opposed to the public, and recognizes this may not always be possible.
VII. Steve N. Goodman, What does research reproducibility mean?, 8 Science Translational Medicine 1 (2016)
Rather than saying anything about agencies relying only on scientific studies where underlying data is made public, this article discusses the importance of clearly defining key terms in the discussion about scientific reproducibility, noting that there is a lack of standardized definitions of terms such as "reproducibility, replicability, reliability, robustness, and generalizability." This raises a key issue of vagueness in EPA's proposal -- EPA does not provide definition for key terms such as "independently validate" or "reproducible" and confusing mentions a "replication crisis" while citing to articles that speak to a "reproducibility crisis."
While providing definitions for these various terms, the article notes that there terms all represent various methods of attempting to verify studies to ensure "scientific claims based on scientific results are true" and cautions against "treating reproducibility as an end in itself --  rather than as an imperfect surrogate for scientific truth." Instead, it promoted the view of looking across studies to "assess their cumulative evidential weight." EPA Proposal thus directly contradicts the suggestions of this article.
Response: See the final rule preamble at Section III.A and Chapter 2 of this document.
Comment [8750-1912]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 13 of the proposal:
      EPA has not consistently followed previous EPA policy (e.g, EPA's Scientific Integrity Guidance, referenced above) that encouraged the use of non-proprietary data and models.
The commenter states that while EPA in a footnote suggests that EPA has not consistently followed EPA's Scientific Integrity Policy encouraging the use of non-proprietary data and models, it misses the fact that EPA's policy was not written as an absolute standard, but was intended to be a flexible one. The policy states only that "the use of non-proprietary data and models are encouraged, when feasible, to increase transparency." EPA must thus explain and justify its deviation from its prior flexible approach that the Proposal now imposes.
Response: See the final rule preamble at Section III.A and Chapter 2 of this document for an explanation of the change in policy and the need for greater transparency.
Comment [8750-1913]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 14 of the proposal:
      https://www.whitehouse.gov/wp-content/uploads/2017/11/2005-M-05-03- Issuance-of-OMBs-Final-Information-Quality-Bulletin-for-Peer-Review-December-16- 2004.pdf
The commenter states that the Proposal appears to issue a requirement for independent peer review of all pivotal regulatory science used to justify regulatory decisions, consistent with the requirements of the OMB Final Information Quality Bulletin for Peer Review. EPA cites to OMB's Final Information Quality Bulletin for Peer Review, explaining existing peer review requirements that nowhere does EPA suggest are not already being complied with.
According to the commenter, there is some vagueness as to whether the Proposal maintains, expands, or narrows these already existing requirements. OMB's bulletin underwent a rigorous stakeholder process including response to comments on multiple drafts from stakeholders, a federal agency workshop at NAS, outreach to major scientific organizations and societies, a formal interagency review. EPA's Proposal has not gone through nearly the same level of review, or as our comments detail, even met the minimum legal requirements for consultation and review. OMB's guidance further provides that agencies should consider the "tradeoffs between depth of peer review and timeliness" This includes considering a benefit cost framework for peer review that takes into account "the direct costs of the peer review activity and those stemming from potential delay in government and private actions that can result from peer review." As our comments detail, EPA has not provided any meaningful benefit-cost analysis of the Proposal. Thus, it would be improper and in conflict with OMB's guidance for EPA to be expanding the peer review requirements through this Proposal.
Response: See the final rule preamble at Section III.H and Chapter 13 of this document.
Comment [8750-1914]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 15 of the proposal:
      February 22, 2002 (67 FR 8453) OMB's Guidelines Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information (2002) https://www.federalregister.gov/documents/2002/02/22/R2-59/guidelines-for-ensuring-andmaximizing-the-quality-objectivity-utility-and-integrity-of-information.
The commenter states that as discussed on their comments regarding Proposal Federal Regiser notice footnote 6, EPA's attempt to align its proposal with OMB's guidelines is misguided.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document.
Comment [8750-1916]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 16 of the proposal:
      See examples from the U.S. Department of Health and Human Services, National Institute of Standards and Technology, U.S. Department of Education, and the U.S. Census Bureau.
The commenter states that in the original Proposal EPA provided no specific "examples" and this vague cite provided very little direction about what EPA was referencing here -- making it impossible to review these examples or respond to them.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document.
Comment [8750-1917]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 17 of the proposal:
      https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/deidentification/index.html.
The commenter states that the EPA states that other agencies have tools to de-identify information private information but fails to recognize that these methods are not transferable to EPA's context. EPA links to guidance on de-identification requirements under HIPAA. This guidance provides two methods for de-identifying data: (1) expert determination method, where an expert determines that, after application of statistical and scientific principals and methods, the risk is very small that the information alone or with other available information could be used to identify the subject; and (2) the safe harbor method, requiring that a number of identifiers are removed. The first method requires case-by-case work and EPA has provided no information regarding how EPA could implement it or how much it might cost and thus the feasibility of requiring researchers or EPA to de-identify data this way is questionable. The second method requires removal of much information useful for research that may be necessary to be able to independently validate the research, so it is unclear that it would satisfy the Proposal's demands. Furthermore, the safe harbor method has been shown to provide potentially insufficient privacy protections.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document.
Comment [8750-1918]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 18 of the proposal:
      https://www.nap.edu/catalog/11434/expanding-access-to-research-datareconciling-risks-and-opportunities.
The commenter states that in this footnote, EPA cites to a report by the National Academies for the proposition that "The National Academies have noted that simple data masking, coding, and de-identification techniques have been developed over the last half century. . . " This incorrectly makes it seem as though the National Academies have identified simple techniques to de-identify data for public release without compromising personal piracy. A full review of the report reveals the opposite is true, that The National Academies in fact recognize that complex, evolving, and yet undeveloped techniques are needed to resolve these concerns. It offers recommendations that are intended to improve upon existing techniques, indicating that this area is under constant change and many advances are left to be made. Further, the report notes this improvement requires "strong partnership between the research community and statistical and research agencies in the design of innovative research on disclosure avoidance techniques and data access modalities and in the implementation of the advances that result from such research." The Proposal takes no steps towards advancing design of new techniques or providing resources to undertake all that needs to be done to make the Proposal remotely feasible.
Further, the Report notes that a changing landscape is making it increasingly difficult to apply past techniques to sufficiently protect data from identification, saying: "Initially, relatively simple data masking techniques, such as top coding income amounts. . . were used to generate restricted data products [,] [d]uring the last decade the increasing risks of confidentiality breaches have led researchers to develop increasingly sophisticated methodologies for restricted data products." They state, "more research is clearly needed to assess the relative ability of different masking methods, and of synthetic data, to reduce the risk of disclosure while preserving data utility." EPA does not acknowledge these newly emerging concerns.
The National Academies recognize the current limitations of producing restricted data that sufficiently limits identifiability to allow it to be made publicly available in a useful form. They note that "well-informed policy making" requires "[r]esearch using detailed confidential data" that cannot be made public -- which the Proposal fails to acknowledge to the detriment of the quality of EPA's policy decisions. Just because certain information cannot be made public for legitimate reasons does not mean the government should refuse to use it to inform policy. And much of the data useful for environmental and health research is particularly sensitive -- the report notes there is increased vulnerability in "[d]ata with geographic detail, such as census block data" and longitudinal data obtained in panel surveys, which is often salient in environmental research. In the meantime, the National Academies state that more work is needed to allow "[h]igh-quality public-use files" that still assure "the inferential validity of the data while safeguarding their confidentiality."
They also point to broader implications of not implementing sufficient privacy protections that EPA does not consider at all may result from the Proposal. The quality of data collected is likely to suffer as "[i]t is essential that respondents believe they can provide accurate, complete information without any fear that the information will be disclosed inappropriately." Essentially, the report leaves as an open question "decisions about how much disclosure risk is acceptable in order to achieve the benefits of greater access to research data involve weighing the potential harm posed by disclosure against the benefits potentially foregone." Thus, EPA wrongfully points to this report as supporting the notion that simple techniques exist to address privacy concerns. The report recommends only more research to reduce risks and increase data utility along with consultation with data users and providers about these issues -- which the Proposal does not implement and thus the report does not support the Proposal.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document
Comment [8750-1919]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 19 of the proposal:
      https://www.cep.gov/content/dam/cep/report/cep-final-report.pdf; https://www.nap.edu/catalog/24652/innovations-in-federal-statistics-combining-datasources-while-protecting-privacy; https://www.nap.edu/catalog/24893/federal-statisticsmultipledata-sources-and-privacy-protection-next-steps.
The commenter states that the EPA claims that "the National Academies and the Bipartisan Commission on Evidence Based Policy have discussed the challenges and opportunities for facilitating to secure access to confidential data for non-government analysts." The proposal does not explain how these examples are relevant, as there is no indication that secure access to underlying data would meet the requirements of making underlying data "publicly available." Further, even if it were relevant, a review of the sources cited reveal that they do discuss many challenges in this space -- which the Proposal does not at all address -- and provide no support for the Proposal. The commenter specific comments follow:
I. Commission on Evidence-Based Policymaking, The Promise of Evidence-Based Policymaking (2017)
This report centers on how to enhance infrastructure to increase the access and use of data between federal agencies to support government policy-making, rather than increase public access to data to non-governmental analysts for purposes of independently validating regulatory science. Further, its focus is to help efforts to make more data available for government purposes to better inform policies. The Proposal on the other hand seeks to make data available to validate individual studies while ultimately making less data available for EPA to consider as it creates policies.
To the extent the report does speak to making more data publicly available, it envisions an entirely new framework to provide adequate privacy protections. Chapter Three of the report discusses increasing threats to privacy as "the amount of information about individuals that is publicly available has grown and the technology that can permit unauthorized re-identification has improved." It notes that forming solutions to this problem while preserving the quality of data is difficult, and that a challenge is "ensuring that enhanced statistical disclosure methods do not change the data in ways that increase the difficulty of reproducing research results." It thus specifically notes that protecting confidentiality can be in tension with allowing data to be used for reproducibility purposes.
The report recommends: (1) amending federal statutes to require Federal departments to conduct a comprehensive risk assessment on de-identified confidential data intended for public release and release de-identified confidential data subject to the Privacy Act and Confidential Information Protection and Statistical Efficiency Act (CIPSEA) only after a disclosure review board approves the release and publicly provides the risk assessment and a description of steps taken to mitigate risk; (2) federal departments to adopt state-of-the-art database, cryptography, privacy-preserving, and privacy-enhancing technologies for confidential data used for evidence building; (3) federal departments assign a senior official the responsibility for coordinating access to and stewardship of the department's data resources; (4) new legislation ensuring that data acquired under a pledge of confidentiality are kept confidential and used exclusively for statistical purposes. The Proposal does not discuss or contribute to any of these efforts.
Chapter Four recognizes that some data cannot be made publicly available without sacrificing the utility of the evidence and thus sets forth recommendations for creating a new National Secure Database Service to allow researchers to access "detailed data that cannot be made publicly available, and only for exclusively statistical purposes." This report thus implicitly recognizes the value of using confidential data to "securely generate evidence about government policies and programs." While transparency is a crucial goal, using data that cannot be made publicly available can help inform government policies in important ways.
The Report details the many obstacles to making data publicly available, and ultimately concludes that much more work is needed in this area, none of which is being furthered by EPA's Proposal.
II. NAS, Innovations in Federal Statistics: Combining Data Sources While Protecting Privacy (2017)
This report provides recommendations to increase sharing and use of data by the federal government and between agencies. It places maintaining privacy and confidentiality at the forefront. The report provides a discussion of the benefits and challenges to allowing external researchers to access data held by government agencies. This assumes that agency has access to data in the first place -- which may not be the case with the studies EPA wishes to rely on that would be barred by its Proposal.
The report notes multiple risks to privacy and confidentiality from data breaches, identity theft, and the threat from the ability to combine multiple data sources to re-identify anonymized data as more and more data is made publicly available. The solutions that the report proposes to minimize these risks include: data minimization, restricted data, restricted access (including licensing agreements, FSRDCs, nongovernment data enclaves). The Proposal does not allow for data minimization since it is aimed at making public complete underlying data that is likely to involve salient personally identifiable information for an unlimited amount of time. Data restriction involves "removing explicit identifiers and applying a variety of statistical disclosure limitation methods to the dataset to reduce the risk of disclosure." However, because these techniques "decrease the precision of the variables in the dataset and. . . introduce errors" it is unclear that they would preserve data for independent validation while also sufficiently protecting privacy. Restricted access involves using "administrative procedures and technology to restrict who can access the dataset and what kinds of analyses can be done with the data to reduce the risk of disclosure." This specifically limits access to data from the general public, which seemingly would not meet the requirements of EPA's proposal. Thus, EPA has not addressed how it would meet any of the challenges raised in this document.
III. NAS, Federal Statistics, Multiple Data Sources, and Privacy Protection: Next Steps (2017)
This report is not directly relevant as it discusses ways to combine diverse data sources from government and private sector sources and the privacy issues that arise from combining multiple data sets. The purpose of the report is to help "federal statistical agencies examine and evaluate data from alternative sources and then combine them as appropriate to provide the country with more timely, actionable, and useful information for policy makers, businesses, and individuals." EPA's proposal will in fact restrict the information that EPA can use.
The report notes that the "privacy status of data is dynamic over time, that datasets that are not individually identifiable today may in the future become individually identifiable" with the availability of new techniques and auxiliary data. It notes that as data sets are linked, these privacy threats increase. The Proposal does not discuss or address threats to privacy from data linkages.
The panel highlighted a number of threats to privacy and data security, including from security threats and inferential disclosure, and concluded "there is awareness of weaknesses of current statistical disclosure limitation methods, but the feasibility for federal statistical agencies of implementing new technologies, such as differential privacy, has not been clearly demonstrated." Finally, they state:
      Overall, much work, interaction, and collaboration will be needed across the various disciplines and stakeholders as agencies seek to move forward to provide stronger privacy protection for the data they either collect from respondents or acquire access to from other administrative and private-sector sources for statistical purposes. It will be critical for there to be robust discussions of the implications of this approach for all stakeholders and these discussions will need to be informed by concrete examples to help everyone understand how use of these technologies will affect them.
The report notes that in order to provide greater access to data much more research and resources are needed. The Proposal identifies no such resources or processes needed to develop needed methods and techniques to allow for greater data disclosure.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document. EPA notes that this final rule does not require EPA to categorically exclude any studies. See response to Comment 4829-2022.
Comment [8750-1920]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 20 of the proposal:
      For example, see policies or recommendations of publishers Taylor & Francis, Elsevier, PLOS, and Springer Nature.
The commenter states that the EPA cites to "policies or recommendation" of several journals that require data be deposited in public data repositories as an example of the Proposal's requirement of data availability. EPA provided only a list of journals with no reference to any specific policies making it difficult to respond fully to this statement.
Each of these journals, however, has exceptions to its data availability requirements when there are valid reasons preventing authors from making their data publicly available via a public data repository. Further, the editors of these journals released a joint statement that explains why their policies with regards to data availability should not be used to support a policy by a federal agency that would in fact restrict the scientific studies it could rely on. Given the vastly different contexts and aims of federal agencies and scientific journals when it comes to making data publicly available, journal policies should not inform EPA's direction. None of these journals claims that lack of data availability in itself calls into question the validity of a scientific conclusion based on that data -- and thus these policies do not support the Proposal.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document.
Comment [8750-1921]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 21 of the proposal:
      For example: https://osp.od.nih.gov/scientific-sharing/requesting-access-tocontrolled-access-data-maintained-in-nih-designated-data-repositories-e-g-dbgap/; https://www.census.gov/fsrdc.
The commenter states that as examples of controlled access to data in federal research data centers, EPA cites to the NIH's policy for requesting access to controlled-access data maintained in NIH-designated data repositories and the U.S. Census Bureau's website on FSRDC, secure facilities providing authorized access to restricted-use microdata for statistical purposes only. NIH requires researches to be a tenure-track professor, senior scientist, or equivalent and go through required procedures prior to gaining access. The U.S. Census Bureau requires researchers to obtain Census Bureau Special Sworn Status, which requires passing a moderate risk background check and swearing to protect respondent confidentiality for life, with significant financial and legal penalties under Title 13 and Title 26 for failure to do so.
It is unclear how these policies are informing EPA's proposal. EPA's proposal would require data to be made "publicly available," and these forms of restricted access specifically do not make data publicly available. They require significant resources and infrastructure and careful thought about who will be permitted to access such data and under what conditions --  none of which EPA has provided any discussion of in the Proposal.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document.
Comment [8750-1922]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 22 of the proposal:
      These recommendations are consistent with those of Lutter and Zorn (2016). https:// www.mercatus.org/system/files/Mercatus-Lutter-Public-Access-Data-v3.pdf.we re.
The commenter states that the EPA cites to a working paper by Randall Lutter and David Zorn as supporting the proposition that "EPA should collaborate with other federal agencies to identify strategies to protect confidential and private information in any circumstance in which it is making information publicly available. These strategies should be cost-effective and may also include: Requiring applications for access; restricting access to data for the purposes of replication, validation, and sensitivity evaluation; establishing physical controls on data storage; online training for researchers; and nondisclosure agreements."
Lutter and Zorn reference these strategies as ones agencies could use to minimize the risks to personally identifiable information when agencies make data publicly available. However, EPA's proposed regulations do not discuss or propose implementation of any of these strategies. The Proposal would result in a rule that mandates only that data be made "publicly available" without any possibility for more restricted release. As the comments discuss, EPA has further not consulted with other federal agencies on this Proposal.
Lutter and Zorn additionally do not argue that agencies should immediately disregard studies where data cannot be made publicly available, and provide alternative procedures agencies should utilize in those cases when still relying on studies. In a separate statement on the HONEST Act, which contains similar requirements as the Proposal, Lutter and Zorn stated that the legislation "should also allow agencies to regulate in instances where they do not possess data." While these additional procedures they recommend agencies follow could still be overly burdensome and barriers to EPA promulgating important safeguards, it is important to note that even they see the dangers in a rule that would force the agency to disregard studies when underlying data could not be made public.
Response: The final rule does not require EPA or any entity outside of the agency to make data publicly available. EPA notes that the supplemental proposal underwent interagency review.
Comment [8750-1923]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 23 of the proposal:
      https://www.nap.edu/catalog/11434/ expanding-access-to-research-datareconciling-risks-and-opportunities.
The commenter states that the Proposal claims "[t]he benefits EPA ensuring that dose response data and models underlying pivotal regulatory science are publicly available in a manner sufficient for independent validation are that it will improve the data and scientific quality of the Agency's actions and facilitate expanded data sharing and exploration of key data sets." EPA cites to a National Academies report. This report does speak to many benefits of making data available to researchers, including helping to maintain and improve data quality; promoting new research and exploration of new questions using existing data; and allowing for verification, refutation, or refinement of original results.
However, the report simply considers the benefits of making data publicly available in a broad sense, it does not consider the issue in the Proposal -- which is that new data is not necessarily being made publicly available that was not before, and at the same time EPA's consideration of scientific research is being limited. Thus, it does not consider the costs to government policy-making that come from EPA's refusing to consider scientific research where underlying data is not publicly available. Since it is questionable whether the Proposal will result in any new data being made available to the public, and certain that it will result in EPA's ignoring valid scientific findings, it is unlikely that this Proposal will "improve the data and scientific quality of the Agency's actions" as EPA claims.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document. EPA notes that the final rule does not require EPA or any entity outside of the agency to make data publicly available.
Comment [8750-1924]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 24 of the proposal:
      https://www.mercatus.org/system/files/Mercatus-Lutter-Public-Access-Datav3.pdf.
The commenter states that the EPA cites to a paper by Randall Lutter and David Zorn for its analysis that ``an increase in existing net benefits from greater reproducibility, which, if it occurred, would cover the costs of obtaining the data and making the data available." However, there are important limitation to this analysis that seriously call this conclusion into question.
First, the statement that EPA cites to is taken out of context. The entire sentence is: "More specifically, we can calculate an increase in existing net benefits from greater reproducibility, which, if it occurred, would cover the costs of obtaining the data and making the data available." This statement is not a conclusion that the benefits of making publicly available data underlying research that federal agencies use to promulgate significant public policies would outweigh the costs. It is describing the figure that Lutter and Zorn go on to calculate -- the threshold level of increase in net benefits required by this policy to equal the costs of implementation. They find that "an improvement in net benefits of 0.02 to 2.08 percent would imply that the net benefits of requiring data access are positive." They themselves note that this estimate "fall[s] short of proving that the benefits outweigh the associated costs."
Their analysis itself is suspect because it differs greatly from the cost estimate provided by the CBO for H.R. 1430, Honest and Open New EPA Science Treatment Act of 2017. The CBO estimated that, if the agency were to choose to rely only on studies that met the Act's requirements from the outset, implementing this legislation would cost about $5 million from 2018-2022. They assumed it would cost $10,000 per study to make data available to enable use of studies. They estimated costs of at least $100 million per year if EPA were to continue to rely on as many studies to support its actions as it has done in recent years. An older cost estimate from CBO on a prior version of the HONEST Act estimated that it would cost "about $250 million a year for the next few years." This assumed that EPA would spend from $10,000 to $30,000 per study to make the data available and that EPA would reduce the number of studies it relies on by about one-half.
Zutter and Lorn calculated an alternative amount for the costs to EPA of this legislation. They find that "the total cost to the EPA for data collection and public accessibility would be $2,558 per study, or about 26 percent of the $10,000 per study cost estimated by CBO." They used estimates that EPA reported under the Paperwork Reduction Act for time that entities in the chemical industry would need to spend to comply with EPA's Health and Safety Data Reporting Rule (40 C.F.R. 716). While they purport that the requirements of that rule are similar to the activities that EPA would undertake to comply with the HONEST Act and similar legislation, they provide no further basis for this. Given the great discrepancy between their and CBO's estimates, it is unclear that their estimate sufficiently accounts for the numerous costs associated with EPA locating underlying research data not currently in its possession and upgrading it to enable it to be made publicly available.
They also rely on questionable assumptions in their calculation. They assume that "given modern technology, by the time research has been published, almost all relevant underlying data and computer code and models will be in electronic format" so time spend photocopying studies will be reduced. This does not consider that EPA may want to rely on older studies where all relevant information is not available in electronic, easily accessible formats. They provide unsupported estimates for activities that EPA would need to undertake to comply with HONEST Act-like legislation that has no corresponding requirement in EPA's Health and Safety Data Reporting Rule -- such as estimating 10 hours for EPA to format unformatted data for public access.
They additionally produce their own estimate for the number of studies that EPA relies on each year, looking at materials posted in dockets on regulations.gov and coming to a total of 18,000 pieces of scientific research per year. CBO estimated 50,000 scientific studies per year. Assuming that EPA continued to rely on all 18,000 studies per year, Zutter and Lorn came to total implementation costs of about $46 million per year, far below the estimate by CBO assuming EPA still relied on at least half of the studies it does currently. Thus, one should view this cost estimate with suspicion, and there is no reason it should be relied on over CBO's cost estimates and does not suffice for EPA providing its own cost benefit analysis.
Response: See Section III.J of the final rule preamble.
Comment [8750-1926]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 10 of the proposal:
 Administrative Conference of the United States' Science in the Administrative Process Project: https://www.acus.gov/research-projects/science-administrativeprocess
 Improving Access to and Confidentiality of Research Data: https://www.nap.edu/read/9958
 Expanding Access to Research Data: https://www.nap.edu/catalog/11434/expandingaccess-to-research-data-reconciling-risks-and-opportunities
 Access to Research Data in the 21st Century: https://www.nap.edu/catalog/10302/access-to-research-data-in-the-21st-century-anongoing
 Health Effects Institute: https://www.healtheffects.org/system/files/AppendixD-dataaccess_3.pdf
 Center for Open Science: https://osf.io/x2w9h/?_ga=2.15543670.1160736397.1518527893- 776332106.1518527893
 Members of the Risk Assessment Specialty Section of the Society of Toxicology, the Dose Response Section of the Society for Risk Analysis, and the International Society for Regulatory Toxicology and Pharmacology: http://www.isrtp.org/GMU%20WEBINAR_DEC_2013/GMU%20Study%20Docum ent4.pdf
 Bipartisan Policy Center's Science for Policy Project: http://bipartisanpolicy.org/wpcontent/uploads/sites/default/files/BPC%20Science%20Report%20fnl.pdf
Specific comments provided include the following:
I. Health Effects Institute
The EPA provides a link to the HEI Policy on The Provision of Access to Data Underlying HEI funded Studies. This policy is "to provide access expeditiously to data for studies that it has funded and to provide that data in a manner that facilitates review and verification of the work but also protects the confidentiality of any volunteers who may have participated in the study and respects the intellectual interests of the original investigator of the work." It is written to be consistent with OMB Circular A-110, which requires agencies to respond to FOIA requests for data underlying federally supported research used to develop federal agency actions with the force and effect of law. EPA already has policies in place to make public the data underlying research that it funds, and already must comply with OMB Circular A-110, thus, it is unclear how this Proposal builds upon this policy.
Furthermore, the policy specifically excludes "personal and medical information and similar information that is personally identifiable, and the disclosure of which would constitute a clearly unwarranted invasion of personal privacy, such as information that could be used to identify a particular person in a research study" and requires the requestor to pay reasonable costs. In this manner, it further deviates from the Proposal.
II. Center for Open Science
The EPA links to the COS's 2017-2020 Strategic Plan. While the strategic plan outlines COS's own mission to "increase openness, integrity, and reproducibility of scholarly research" and to meets its goal of creating "a future scholarly community in which the process, content, and outcomes of research are openly accessible by default" nothing in this strategic plan suggests anything like EPA's Proposal. It does not discuss barring use of studies or ensuring access to underlying data -- and thus is completely irrelevant to the Proposal.
III. Members of the Risk Assessment Specialty Section of the Society of Toxicology, the Dose Response Section of the Society for Risk Analysis, and the International Society for Regulatory Toxicology and Pharmacology
EPA links to a survey conducted by the Center for Media and Public Affairs and Center for Health and Risk Communication at George Mason University. They surveyed members of the Risk Assessment Specialty Section of the SOT, the Dose Response Section of the SRA, and the ISRTP. However, the survey thus does not represent any official recommendation or policy position from these professional organizations and represent only the views of the members who chose to participate in the survey.
Thus, while the survey found 69 % of those surveyed "regard it as "very important" for assessors to have access to underlying raw data for the most critical studies in order to independently analyze their results," this should be viewed in the rightful context. The survey did not ask whether agencies should continue to rely on scientific studies where the underlying data cannot be made public or independently analyzed. The survey question further appears to have only asked whether researchers assessing studies should have access to underlying data to independently analyze results, not whether underlying data should be made publicly available.
Further, the Dose Response Section of the SRA has since submitted a comment to EPA that states this footnote and the claim that EPA makes that the Proposal took into consideration these recommendations and policies is "inaccurate" and that "the `Dose-Response Section [sic] of the Society for Risk Analysis' has never adopted any `policies or recommendations' on this or any other topic." They have asked that EPA remove all references to the organization and make clear in the comment response for this rule that "'third party Organizations' whose policies and recommendations were considered do not include the Society for Risk Analysis or the Dose-Response Specialty Section."
The SOT similarly have said this survey does not constitute support from the Specialty Section or the SOT as a whole, and requesting "that any and all references to "members of the Risk Assessment Specialty Section of the Society of Toxicology" be removed from the Final Rule." They also specifically comment that "invalidating data solely on the basis of public availability is inappropriate."
IV. Bipartisan Policy Center's Science for Policy Project
The EPA provides a hyperlink to the Final Report of the Science for Policy Project Improving the Use of Science in Regulatory Policy. This report makes a number of recommendations, none of which endorse the Proposal. In relevant part, Recommendation Three suggests "Agencies and their scientific advisory committees should cast a wide net in reviewing studies relevant to regulatory policy, and should make their methods for filtering and evaluating those studies more transparent." They urge agencies to increase availability of data and information on research studies and subject all studies relied on in the formulation of regulation to be subject to the requirements of the Shelby Amendment and OMB Circular A-110 regardless of who funded the study. Importantly, those requirements contain important exception for confidentiality and privacy concerns -- and thus do not support the Proposal. This recommendation is also aimed at increasing use of science in regulatory policy and does not suggest that agencies not rely on studies where those data access requirements cannot be met because of other concerns. It also highlights that the use of CBI to prevent access to data appears to be overused and urges agencies to make procedures more stringent to allow only for legitimate claims of CBI -- which EPA does not address in its Proposal. Recommendation Four states: "The federal government, universities, scientific journals and scientists themselves can help improve the use of science in the regulatory process by strengthening peer review, expanding the information available about scientific studies, and setting and enforcing clear standards governing conflict of interest." As part of this recommendation, the report "Federal agencies, universities and journals should encourage or require on-line publication of the methods and data underlying published scientific studies."(citation omitted) However, it once again does not say that agencies should not consider research studies where this is not possible due to privacy or other compelling reasons.
Wendy Wagner, who served on the panel that produced the recommendations has stated: "They don't adopt any of our recommendations, and they go in a direction that's completely opposite, completely different. . . . They don't adopt any of the recommendations of any of the sources they cite. I'm not sure why they cited them."
Response: See Section III.A of the final rule preamble and Chapter 2 of this document.
Comment [8750-1927]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 11 of the proposal:
 Proceedings of the National Academy of Sciences: http://www.pnas.org/page/authors/journal-policies#xi
 PLOS ONE: http://journals.plos.org/plosone/s/data-availability
 Science: http://www.sciencemag.org/authors/science-journals-editorial-policies
 Nature: http://www.nature.com/authors/policies/data/data-availability-statementsdata-citations.pdf
The commenter states that while EPA links to journal policies that encourage or require, in some instances, sharing data, they contain exceptions when privacy would be compromised. The editors of these journals issued a joint statement opposing the Proposal. They note that some data sets cannot be shared publicly, and that there are still other methods available to verify scientific findings. The statement also strongly condemns the notion of excluding scientific information from consideration when underlying data cannot be made publicly available:
It does not strengthen policies based on scientific evidence to limit the scientific evidence that can inform them; rather, it is paramount that the full suite of relevant science vetted through peer review, which includes ever more rigorous features, inform the landscape of decision making. Excluding relevant studies simply because they do not meet rigid transparency standards will adversely affect decision-making processes.
Thus, journal policies encouraging the sharing of underlying data do not support a proposal by a regulatory agency to exclude from consideration studies when the underlying data is not publicly available.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document. EPA notes that this final rule does not require EPA to categorically exclude any studies. See response to Comment 4829-2022.
Comment [8750-1931]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 16 of the proposal:
 U.S. Department of Health and Human Services: https://www.hhs.gov/hipaa/forprofessionals/privacy/special-topics/de-identification/index.html
 National Institute of Standards and Technology: https://nvlpubs.nist.gov/nistpubs/ir/2015/NIST.IR.8053.pdf
 U.S. Department of Education: https://studentprivacy.ed.gov/sites/default/files/resource_document/file/data_deident ification_terms.pdf
 U.S. Census Bureau: https://www.census.gov/about/adrm/linkage/technicaldocumentation/processing-de-identification.html
The commenter states that the EPA suggests the examples linked to could address concerns about privacy and confidentiality arising from the Proposal. However, the cited sources provide no assurance that the Proposal could be implemented to expand disclosure of personal data without serious risks to privacy. The commenter provides the following specific comments:
I. U.S. Department of Health and Human Services
EPA first points to guidance on de-identification requirements under HIPAA. This guidance provides two methods for de-identifying data: (1) expert determination method, where an expert determines that, after application of statistical and scientific principals and methods, the risk is very small that the information alone or with other available information could be used to identify the subject; and (2) the safe harbor method, requiring that a number of identifiers are removed. The first method requires case-by-case work and EPA has provided no information regarding how EPA could implement it or how much it might cost and thus the feasibility of requiring researchers or EPA to de-identify data this way is questionable. The second method requires removal of much information useful for research that may be necessary to be able to independently validate the research, so it is unclear that it would satisfy the Proposal's demands. Furthermore, the safe harbor method has been shown to provide potentially insufficient privacy protections.
II. NIST
EPA links to a NIST document entitled De-Identification of Personal Information as a potential solution to address concerns about confidentiality and privacy. This document discusses different techniques and issues with de-identification of personal information. However, the document does not discuss de-identification of personal information specifically for the purposes of making research data publicly available for independently validating scientific studies. The document instead notes that:
The purpose of de-identifying data is to allow some uses of the de-identified data while providing for some privacy protection by shielding the identity of the data subjects. These two goals are antagonistic, in that there is a trade-off between the amount of de-identification and the utility of the resulting data. However, de-identification opens up new uses for the data that were previously prohibited due to privacy concerns. It is thus the role of the data controller, standards bodies, regulators, lawmakers, and courts to determine the appropriate level of security, and thereby the acceptable trade-off between de-identification and utility.
EPA completely fails to note this obstacle, that as data is stripped of identifiable material it also loses utility to researchers. EPA cites to broad privacy protection techniques without explaining whether they could be applied to protect privacy while still allowing enough utility in the data set to allow for independent validation as required by the Proposal.
The document notes many of the challenges to protecting privacy including that: "de-identification approaches based on suppressing or generalizing specific fields in a database cannot provide absolute privacy guarantees, because there is always a chance that the remaining data can be re-identified using an auxiliary dataset." The harms of data linkages and increasing difficulty to preserve privacy as more and more information about individuals is made available is another challenge that EPA has not addressed.
III. U.S. ED
EPA links to a document of the Privacy Technical Assistance Center, Data Deidentification: An Overview of Basic Terms, which provides a high-level overview of key terms and practices to help educational agencies and institutions comply with the FERPA. EPA has not explained why the requirements of FERPA are applicable here. This document is concerned with data disclosure that occurs "when schools, districts, or states publish reports on student achievement or share students' data with external researchers" not to make information publicly available for independent validation. Thus it's unclear that methods used to de-identify but preserve data for those purposes would be adequate in this context.
For example, one of the methods that the U.S. ED uses for disclosure avoidance for tabular data is to not release information for any cell that has a size below some minimum, which essentially means not disclosing information where there are small numbers in a certain cell. This could obviously lead to a loss of information that would prevent a deidentified data set from being used to independently validate research findings.
IV. U.S. Census Bureau
EPA provides a link to a website titled Data Ingest and Linkage that details the U.S. Census Bureau's approach to linking data across many records held by the Bureau, permitting more detailed information to be linked back to one individual to allow for analysis and research. The website links to a working paper that describes the method by which the Bureau assigns a unique person identifier to records it holds that enables it to link records together to create the final file. It is totally unclear how this process on linking together records is a solution that EPA could implement to protect privacy of individuals when disclosing data as it concerns how to identify data to specific people -- not how to make data available while protecting their privacy.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document.
Comment [8750-1936]: Commenter (9227) states that the EPA has provided no sources or authorities that support or provide a reasoned basis for the proposed rule and that many of the sources raise key implementation concerns that EPA fails at all to address -- rendering the proposal arbitrary and capricious.
As an example, the commenter cites footnote 20 of the proposal:
 Taylor & Francis: https://authorservices.taylorandfrancis.com/data-repositories/
 Elsevier: https://www.elsevier.com/authors/author-services/research-data
 PLOS: http://journals.plos.org/plosone/s/data-availability
 Springer Nature: https://www.springernature.com/gp/authors/research-datapolicy/repositories
The commenter states that the EPA cites to "policies or recommendation" of several journals that require data be deposited in public data repositories as an example of the Proposal's requirement of data availability. While these journals have policies that encourage authors to deposit data in public data repositories, they all have important exceptions in cases where this is not feasible or ethical.
The hyperlink for Taylor & Francis links to a page that provides information about how to find public data repositories to submit date to in order to comply with journal sharing policies. However, Taylor & Francis' basic data sharing policy "which applies across many of [their] journals" does not require data be submitted to a public data repository, but "encourages authors to share and make data open where this does not violate protection of human subjects or other valid subject privacy concerns." Thus, this policy is flexible and allows exceptions for when privacy concerns are at stake.
The hyperlink for Elsevier links to a page providing general information about data sharing. While the web page notes that researchers "are increasingly encouraged, or even mandated, to make. . . research data available, accessible, discoverable and usable," it also provides important qualifications. It notes, "there are times when the data is simply not available to post or there are good reasons why it shouldn't be shared." In these cases, authors are encouraged to provide a data statement explaining why the data cannot be shared.
The hyperlink for PLOS links to a page describing PLOS's data availability policies. It explains, "PLOS journals require authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception." The policy recommends deposition of the data into a public repository, however, it recognizes that there are instances when this may not be ethical or legal, for instance because the "underlying data pose privacy or legal concerns e.g., where data might reveal the identity or location of participants." In these instances, it allows an exception to this policy.
The hyperlink for Springer Nature links to a page listing recommended repositories. While Springer Nature's data policies support data sharing via public data repositories, it notes, "reasonable restrictions on data availability are permitted to protect human privacy, biosafety or respect reasonable terms of use for data obtained under license from third parties."
Response: See Section III.A of the final rule preamble and Chapter 2 of this document.
Comment [8750-2620]: Commenter (9227) states that the EPA has failed to explain why it has singled out dose response studies to be excluded if their underlying data and models are not publicly available but has not similarly targeted any other types of studies commonly used by EPA.
EPA also has proposed to target the requirements for public availability specifically to the data and modeling underlying one specific subset of scientific research -- dose response studies. EPA has provided no explanation or justification for targeting dose response studies in particular or for not including other types of studies or scientific information. EPA has not suggested that these studies are inherently less reliable than other studies, that they more commonly fail to publicly disclose data and modeling information, that replication is more necessary for these studies than others, or any other conceivable reason. Absent any explanation from the agency, it is impossible to comment on the factual predicates for EPA's proposed decision, or the reasonableness of EPA's justification, except to state that it appears completely arbitrary in the absence of any rationale. See, e.g., Transactive Corp., v. United States, 91 F.3d 232, 237 (D.C. Cir. 1996) ("A long line of precedent has established that an agency action is arbitrary when the agency offered insufficient reasons for treating similar situations differently.").
Response: See Sections III.A, III.B and III.E of the final rule preamble. EPA notes that this final rule does not require EPA to categorically exclude any studies. See response to Comment 4829-2022.
Comment [8750-2896]: Commenter (8227) states that the EPA has failed to explain why it has singled out dose response studies to be excluded if their underlying data and models are not publicly available, but has not similarly targeted any other types of studies commonly used by EPA.
EPA also has proposed to target the requirements for public availability specifically to the data and modeling underlying one specific subset of scientific research -- dose response studies. EPA has provided no explanation or justification for targeting dose response studies in particular or for not including other types of studies or scientific information. EPA has not suggested that these studies are inherently less reliable than other studies, that they more commonly fail to publicly disclose data and modeling information, that replication is more necessary for these studies than others, or any other conceivable reason. Absent any explanation from the agency, it is impossible to comment on the factual predicates for EPA's proposed decision, or the reasonableness of EPA's justification, except to state that it appears completely arbitrary in the absence of any rationale. See, e.g., Transactive Corp., v. United States, 91 F.3d 232, 237 (D.C. Cir. 1996) ("A long line of precedent has established that an agency action is arbitrary when the agency offered insufficient reasons for treating similar situations differently.").
Response: EPA notes that this final rule does not require EPA to categorically exclude any studies. See response to Comment 4829-2022.
Comment [8750-3567]: Commenter (9227) asserts that the EPA arbitrarily fails to offer a reasoned explanation for its departure from existing policies that broadly require the agency to consider all available scientific information when undertaking rulemakings.
In addition to the statutes that require EPA to use the best available science when making regulatory decisions, a number of EPA's own policies embed this requirement as well. By arbitrarily limiting the science EPA considers when making regulatory decisions, the Proposal contravenes these policies, injuring the scientific integrity of EPA's actions. As discussed in more detail in Section II.E [of the commenter's letter] because EPA is changing course from established policy, EPA must fully acknowledge and justify its decision, which it has failed to do in the Proposal.
EPA's own existing Scientific Integrity Policy states:
      To support a culture of scientific integrity within the Agency, this policy. . . [r]ecognizes . . . policy makers within the Agency weigh the best available science, along with additional factors such as practicality, economics, and societal impact, when making policy decisions.
The Proposal conflicts with this policy by restricting what may be the best available science on a given topic from EPA's consideration solely because the underlying data cannot be made public. As described above, public availability of data is neither necessary nor sufficient to ensure that studies constitute "best available science." The Proposal does not acknowledge this departure from the agency's Scientific Integrity Policy, much less explain why such a departure is reasonable.
Likewise, the Proposal is in tension with EPA's Information Quality Guidelines, developed in response to OMB guidelines issued under Section 515(a) of the Treasury and General Government Appropriations Act for Fiscal Year 2001, which require EPA to ensure the objectivity of influential scientific information it disseminates by using "the best available science and supporting studies conducted in accordance with sound and objective scientific practices." EPA considers information to be disseminated when EPA prepares and distributes information to support an Agency decision or regulation or when EPA distributes information in a way that suggests EPA agrees with it, that it supports EPA's viewpoint, or if in the distribution EPA proposes to use it to support or formulate a regulation or agency decision. Thus, the Proposal conflicts with the Guidelines by restricting scientific studies that EPA may use to support regulations, which may cause it to disseminate other information to support its regulations that is not based on the best available science.
EPA's Peer Review Handbook similarly acknowledges that "EPA strives to ensure that the scientific and technical bases of its decisions meet two important criteria: (1) they are based upon the best current knowledge from science, engineering, and other domains of technical expertise; and (2) they are credible." EPA's Science Policy Council Handbook on Risk Characterization also requires reasonableness in the agency's risk assessments, which is achieved when "the characterization is based on the best available scientific information." These policies clearly impact EPA's regulatory actions, and thus will be impacted by the Proposal. Yet EPA completely fails to analyze the impact the Proposal will have on its ability to comply with these policies and fails to explain why it is changing course or justify its decision to do so. Indeed, the Proposal fails to even acknowledge that the agency is changing positions.
Response: See Section III.A of the final preamble and Chapter 2 of this document. EPA notes that this final rule does not require EPA to categorically exclude any studies. See response to Comment 4829-2022.
Comment [9066-1950]: Commenter (6188) asserts that the Proposed Rule is arbitrary and capricious, as written and as it would be applied. See 5 U.S.C. § 706(2)(A) (an agency's action can be set aside if it is "arbitrary, capricious, an abuse of discretion, or otherwise not in accordance with law"). For example, the Proposed Rule applies only to EPA regulatory actions, which often regulate industry to protect human health and the environment, and omits from its ambit Agency adjudications, which often involve private companies seeking regulatory exemptions or permissions from EPA and are more likely to implicate industry-provided studies that are specific to the particular adjudication in question. The NPRM offers no explanation for this distinction, but documents obtained through the FOIA suggest that EPA introduced the distinction mid-way through the rulemaking process, when the Agency realized the implications of its Proposed Rule for pesticide registration proceedings, many of which rely on industry-generated and proprietary data, which cannot be publicly disclosed. Similarly, while the Proposed Rule suggests that EPA "should be guided by this policy to the maximum extent practicable during ongoing regulatory action," it also provides that the Administrator -- and the Administrator alone -- would be empowered to offer case-by-case exemptions in myriad instances. See 83 Fed. Reg. 18771. As discussed further in the their comment letter, they suggest that this would likely lead to inconsistent application of the Proposed Rule, suggesting its implementation would be arbitrary at best.
Implementation of the Proposed Rule could also impair future Agency rulemaking actions by forcing EPA to ignore key comments and regulatory record evidence in violation of the APA. See Motor Vehicle Mfrs. Ass'n of the United States, Inc. v. State Farm Mut. Automobile Ins. Co., 463 U.S. 29, 43 (1983) (an agency must "examine the relevant data" and acts arbitrarily and capriciously when it "entirely fail[s] to consider an important aspect of the problem"). For example, if a commenter were to cite studies that are based upon data that is not entirely publicly available, the Agency would not be able to consider those studies -- or fully evaluate the comment -- as part of the rulemaking action, regardless of the scientific validity of the comment. In other words, the Proposed Rule would expose the Agency to challenges not only to arbitrary and capricious application of the rule itself, but also to the arbitrariness of future rules.
Simply put, the commenter contends that the Proposed Rule fails to satisfy the basic requirements of the APA and denies the public a meaningful opportunity to engage in this rulemaking process and in future rulemaking processes.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document. Also, see general response regarding compliance with the APA. Regarding the Administrator exemption, see response to comment 6112-3062.
As discussed in the final rule preamble at Section III.A, EPA will issue statute-specific regulations following this procedural rule, some of which will be substantive and therefore subject to public notice and comment. Any implementation issues will be addressed as appropriate in subsequent rulemakings.
Comment [9217-3571]: Commenter (9225) states that substantively, the rule violates numerous public health and environmental provisions contained in these laws, as well as requirements to use the best available science or to consider all available information, while procedurally, it violates the APA and a number of other laws that set forth specific procedures EPA must follow during its rulemaking process.
Response: See response to Comment 4829-2022.
Comment [9298-4772]: Commenter (8771) argues that the proposal is arbitrary and capricious for the three reasons stated above: (1) failing to interpret one's data via a model EPA likes is in no way a reason to ignore the data themselves -- so this part of the proposal must be stricken; (2) requirements for transparency must be applied to cost and valuation studies as well as risk ones, or else the proposal is arbitrary; and (3) "reducing reliance on defaults" is tantamount to "ignoring good science," which would be contrary to law.
Response: See Section III.D of the final rule preamble. EPA notes that this final rule does not require EPA to categorically exclude any studies. See response to Comment 4829-2022.
Comment [6891-1526]: Commenter (6912) states that the proposed rule is illegal because it is arbitrary and capricious, and it is arbitrary and capricious in at least three ways. First, the proposed rule grants the Administrator broad discretion to determine whether its terms apply, effectively making it a rule establishing the whim of the Administrator. Particularly in an agency so captured by industry, this cannot be tolerable in a nation of laws. Second, the proposed rule is not supported by any independent authority. Third, the proposed rule is not the product of reasoned decision-making; it is a deliverable, pre-ordained by interested parties laden with bias.
Separately, and in addition, the proposed rule is illegal because it is the result of an effective delegation of rulemaking authority to private interests.
The commenter provides that the APA permits courts to set aside agency actions found to be "arbitrary, capricious, an abuse of discretion, or otherwise not in accordance with the law." In determining whether an agency action was "arbitrary and capricious, the courts look to several factors, whether: "(1) the agency 'relied on factors which Congress has not intended it to consider,' (2) the agency 'failed to consider an important aspect of the problem,' (3) the agency explained its decision in a way 'that runs counter to the evidence,' or (4) the action 'is so implausible that it could not be ascribed to a difference in view or the product of agency expertise. "'
Courts have also held that a rule is arbitrary and capricious if the promulgating agency did not "genuinely engage in reasoned decision making" or if it did not "articulate a satisfactory explanation for its action including a 'rational connection between the facts found and the choice made."'
Response: See response to Comments 2157-149, 4829-2022, 6112-3062, and 6163-1114.
Comment [6168-4100]: Commenter (6178) provides the NPRM proposes to establish a new EPA regulation that will `ensure the data and models underlying the science pivotal to what the Agency deems `significant regulatory actions' is publicly available in a manner sufficient for validation and analysis', hereinafter referred to as the `Transparency Rule' or `Rule'. Specifically, the commenter raises the following concerns with the Transparency Rule proposed by the NPRM:
      1. The Agency cites specific sections of statutes it administers as the basis of the Agency's authority to adopt the Transparency Rule; specifically: the CAA sections 103, 301(a), 42 U.S.C. 7403, 7601(a); CWA sections 104, 501, 33 U.S.C. 1254, 1361; SDWA sections 1442, 1450(a)(1), 42 U.S.C. 300j - 1, 300j - 9(a)(1); RCRA sections 2002(a)(1), 7009, 42 U.S.C. 6912(a)(1), 6979; CERCLA (as delegated to the Administrator via Executive Order 12580) sections 115, 311, 42 U.S.C. 9616, 9660; EPCRA section 328, 42 U.S.C. 11048; FIFRA sections 25(a)(1), 136r(a), 7 U.S.C. 136r(a), 136w; and the TSCA, as amended, section 10, 15 U.S.C. 2609 (collectively the `Acts'). The Agency rightly cites acts of Congress in an attempt to support the NPRM as such authority is necessary for the Agency to adopt regulations pursuant to the above listed Acts.
      2. However, Congress, in each of the Acts' sections cited, never instructs the Agency to consider transparency to the public of supporting data and/or models of the scientific research when deciding to rely on such research as support for regulations passed pursuant to any of the Acts. Consequently, the Transparency Rule requires the EPA to rely on factors which Congress has not intended it to consider when promulgating regulations pursuant to the Acts, rendering the Transparency Rule arbitrary and capricious under the APA.
      3. Further, not only does the Transparency Rule lack sufficient Congressional intent to avoid being deemed arbitrary and capricious, implementing the Rule would actually contradict Congressional instruction to conduct scientific research with a focus on `promotion' and `coordination' among itself and other institutions that produce scientific research material to the objectives of the Acts Congress requires the EPA to enforce. This provides additional cause for finding the Transparency Rule arbitrary and capricious under the APA.
Response: See response to Comments 4829-2022 and 6163-1114.
Comment [4837-3057]: Commenter (4848) states that the Administrative procedure requires that EPA consider data submitted by the public in evaluating regulations. Let's be clear. Scientific studies submitted have always been of uneven quality. EPA has processes in place (including use of a SAB, testimony and written and oral public notice and comment) using both internal and external peer review to evaluate data. Depending on context, some studies are given greater weight than others. Some studies are disregarded entirely.
It is inappropriate, and likely unlawful under the APA and the express language of numerous environmental statutes including the SDWA and the TSCA, for EPA to categorically eliminate certain types of studies, and hence certain types of data, without considering context.
Response: The final rule does not require EPA to categorically exclude any studies. See response to Comment 4829-2022.
Comment [1929-3901]: Commenter (1943) states that for the reasons stated below, The Minerals, Metals, and Materials Society (TMS) finds the proposal to be arbitrary and limiting, both of which are antithetical to the principles of scientific exploration. We strongly believe that our federal government's regulatory rulemaking process is better served by consideration of all available data and information, especially that which has been carefully reviewed and objectively approved by the scientific community. On behalf of all TMS members, we urge the EPA to reconsider its proposal and cease any development of a final rule to limit the science that underlies the Agency's regulatory process.
Response: The final rule does not require EPA to categorically exclude any studies. See response to Comment 4829-2022.
Comment [6852-2780; 6128-2235]: Commenters (6874, 6153) state that the EPA's process for promulgating this rule represents a gross deviation from the agency's typical action development process. EPA failed to perform an appropriate regulatory impact analysis, despite indications that implementation costs could be substantial. EPA has not received sufficient input from the OIRA, which conducted a highly truncated review and does not appear to have met with outside groups during that process, or from its SAB, which states that the rule "deals with a myriad of scientific issues for which the Agency should seek expert advice from the Science Advisory Board." Taken together, these deviations from typical process leave EPA unable to consider its proposal accurately, and constitute an arbitrary and capricious approach to rulemaking.
Response: See response to Comment 2157-149. EPA's Action Development Process is not legally binding and is a purely internal, collaborative process. In any event, the supplemental proposal underwent OMB and interagency review and was thoroughly reviewed by the SAB. The final rule discusses the SAB's findings and the SAB's report is available in the rulemaking docket.
Comment [6163-1155]: Commenter (6133) notes that their comments discuss the failure of statutory authorities cited by EPA to provide any legal support or authorization whatsoever for the Proposal and its approaches. The commenter notes that the Proposal also cites various executive orders, memoranda, reports, guidelines and the like with the suggestion or implication that these materials somehow provide support for the Proposal. They do not, and thus the Proposal violates the law. See, e.g., Public Citizen Health Research Group v. Tyson, 796 F.2d 1479 (D.C. Cir. 1986) (reversing and remanding agency decision to carry out last-minute directive by White House OMB without any apparent justification in the administrative record). First, of course, EPA's proposed rulemakings must be authorized by federal statutes. Executive orders provide no legal authority for agency rulemakings. Nor may executive orders contradict or alter legal responsibilities an agency has under federal statutes or justify arbitrary and capricious agency action. Equally obvious, memoranda, reports, guidelines, and the like provide no legal authority for agency rulemakings, nor may they justify arbitrary and capricious agency action. See, e.g., Medellin v. Texas, 552 U.S. 491, 524 (2008) ("The President's authority to act, as with the exercise of any governmental power, `must stem either from an act of Congress or from the Constitution itself.'" (citation omitted)); Chrysler Corp. v. Brown, 441 U.S. 281, 302 (1979) ("The legislative power of the United States is vested in the Congress, and the exercise of quasi-legislative authority by governmental departments and agencies must be rooted in a grant of such power by the Congress and subject to limitations which that body imposes."). Second, an agency's proposed rulemaking may not be at odds with federal statutes, may not be creatures of the agency's imagination or policy preferences, and may not be otherwise arbitrary, capricious, or inconsistent with law. The Proposal fails on all of these scores.
Response: See response to Comment 6163-1114.
Comment [6163-2477]: Commenter (6133) states that when an agency reverses course, it must "provide reasoned explanation for its action." FCC v. Fox Television Stations, Inc., 556 U.S. 502, 515 (2009);Motor Vehicle Mfrs. Ass'n of U.S., v. State Farm Mut. Auto. Ins. Co., 463 U.S. 29, 43 (1983). And when that reversal "rests upon factual findings that contradict those which underlay [the agency's] prior policy," a "more detailed justification" is needed. Fox, 556 U.S. at 515. Indeed, "an agency's decision to change course may be arbitrary and capricious if the agency ignores or countermands its earlier factual findings without reasoned explanation for doing so." Id. at 537 (Kennedy, J., concurring).
As the Supreme Court explained in its 2016 Encino Motorcars decision, an agency must supply "good reasons" for a policy revision, cannot leave "unexplained inconsistency," and must address "serious reliance interests." Encino Motorcars, LLC v. Navarro, 136 S. Ct. 2117, 2126 (2016). In Encino, the DOL reversed its decades-long practice of treating service advisors at automobile dealerships as exempt from the Fair Labor Standards Act's overtime provisions, offering minimal explanation for the policy change. Id. at 2123. The Court overturned the rule, holding that the Department had not met its obligation to offer a "reasoned explanation," especially given the decades of reliance on the policy. Id. at 2126. It was not enough that the Department included conclusory statements declaring its new policy to be a reasonable interpretation of the statute because the Department failed to provide any good reasons for the new policy. Id. at 2127. As explained by the Court, "[t]his lack of reasoned explication for a regulation that is inconsistent with the Department's longstanding earlier position results in a rule that cannot carry the force of law." Id.
In Organized Village of Kake v. United States Department of Agriculture, the USDA, relying on a detailed factual record, decided not to exempt the Tongass National Forest from a rule that would limit road construction and timber harvesting in national forests, explaining that the benefits would outweigh the potential economic loss. 795 F.3d 956, 959 - 61, 967 - 68 (9th Cir. 2015) (en banc). Just two years later, on "precisely the same record," the agency issued a new decision reversing course. Id. at 968. The court concluded that the "absence of a reasoned explanation for disregarding previous factual findings violate[d] the APA." Id. at 969. The court also recognized that "[e]lections have policy consequences," but even when reversing a policy after an election, "an agency may not simply discard prior factual findings without a reasoned explanation." Id. at 968.
EPA previously routinely used and considered science and studies for which the underlying data was not publicly available in regulatory actions. As explained above, EPA has not identified even one example in which EPA has observed the policies underlying the Proposal, and our research has likewise uncovered no such instance. The Proposal essentially admits as much, stating:
      Historically, EPA has not consistently observed the policies underlying this Proposal, and courts have at times upheld EPA's use non-public data in support of its regulatory actions. See Coalition of Battery Recyclers Ass'n v. EPA, 604 F.3d 613, 623 (D.C. Cir. 2010); American Trucking Ass'ns v. EPA, 283 F.3d 355, 372 (D.C. Cir. 2002).
83 Fed. Reg. at 18,769, n.3. The Proposal then goes on to say that "EPA is proposing to exercise its discretionary authority to establish a policy that would preclude it from using such data in future regulatory actions." Id. The Proposal's categorical exclusion of non-publicly available "dose response data" is also departure from EPA's previous practice, as described in the 2002 EPA Guidelines, of weighing all relevant information.
In short, according to the commenter, EPA provide no basis for changing course on this issue, especially when EPA has enshrined the previous policy in agency guidelines and litigation. EPA's failure to explain this change in course violates the law.
Response: See Section III.A of the final rule preamble and Chapter 2 of this document.
<IP_ADDRESS>.2 Was the Proposed Rule Issued in Compliance with the APA?
Comment [6099-566]: Commenter (6109) notes that the EPA states that the proposed rule is "consistent with requirements in the APA to ensure public participation in the rulemaking process." However, this proposed rule lacks justification or a practical explanation of how EPA will implement the rule and thereby fails to meet the legal threshold of seriousness defined by the APA, § 307 of the CAA, and court decisions that require an agency to give public notice about the who, what, when, where, why and how of its plans when it wants to create or change a regulation.
Response: See response to Comment 2157-149.
As discussed in the final rule preamble at Section III.A, EPA will issue statute-specific regulations following this procedural rule. Forthcoming substantive rulemakings will be subject to public notice and comment and judicially reviewable, and any implementation issues will be addressed as appropriate.
Comment [6112-1622]: Commenter (6137) provides that the APA establishes rulemaking procedures and judicial review requirements for agency rules that EPA has failed to follow here. 5 U.S.C. §§ 553, 706. These requirements  -  which EPA entirely and unlawfully ignores  -  are central both to ensure an opportunity for the affected public to comment, and to ensure an adequate record for judicial review. This includes:
 Providing a meaningful opportunity for notice-and-comment;
 Ensuring that the docket contains all documents on which EPA relies;
 Providing additional information regarding what the public is being asked to comment on, and
 Adding details that are missing from the proposal, before taking further comment.
According to the commenter, the EPA has failed to comply with these mandates, rendering this rulemaking unlawful under the APA, for several reasons.
First, in issuing this Proposed Rule, EPA violated the APA by failing to place in the administrative record all of the documents on which it purports to rely. The preamble to the Proposed Rule cites more than thirty specific documents on which EPA purportedly relies. See, e.g., 83 Fed. Reg. at 18,769-72 & nn.1-2, 4-24. Of the specifically cited and relied upon documents, only about 12 are included in the docket, available at www.regulations.gov, Docket ID: EPA-HQ-OA-2018-0259, as of the date of publication of the Proposed Rule. In addition, EPA provides no support for its proposed conclusion that "EPA believes the benefits of this proposed rule justify the costs." 83 Fed. Reg. at 18,772. Nothing in the record demonstrates what costs or benefits EPA considered to reach this determination, or whether it evaluated at all the harm to public health and to privacy that this Proposed Rule would cause.
Absent the ability to review such documents, the public is deprived of adequate notice or an ability to provide informed comments regarding the Proposal and EPA's rationale for issuing the Proposed Rule. EPA's failure to put the documents on which it relies into the record violates notice-and-comment under the APA and other statutes that supplement such notice requirements. See, e.g., Am. Radio Relay League v. FCC, 524 F.3d 227 (D.C. Cir. 2008) (agency may not cherry-pick documents on which it relies for public review; it is a violation of public notice and comment to refuse to put the documents in full on which the agency relied into the record).
Second, failing to place all of the documents on which EPA relies into the record also violates the judicial review provision of the APA by making it impossible for commenters to provide adequate comments or a complete record for judicial review of any final action EPA takes. 5 U.S.C. § 706 ("In making the foregoing determinations, the court shall review the whole record or those parts of it cited by a party, and due account shall be taken of the rule of prejudicial error." (emphasis added)); Am. Radio Relay League, 524 F.3d at 242-43 (Tatel, J., concurring) (reiterating importance of the requirement for agency to place unredacted studies into the docket because not doing so "undermines this court's ability to perform the review function APA section 706 demands"); see also Walter O. Boswell Mem'l Hosp. v. Heckler, 749 F.2d 788, 792 (D.C. Cir. 1984) (citing Citizens to Preserve Overton Park, Inc. v. Volpe, 401 U.S. 402, 420 (1971)) ("[R]eview is to be based on the full administrative record that was before the Secretary at the time he made his decision." (emphasis in original)).
Not only is EPA's failure to include in the regulatory docket all of the documents upon which it relies a violation of the APA, but it likewise contravenes procedural requirements in Title 28 of the U.S. Code that govern judicial review of agency action. See, e.g., 28 U.S.C. § 2112(b) ("The record to be filed in the court of appeals . . . shall consist of the order sought to be reviewed or enforced, the findings or report upon which it is based, and the pleadings, evidence, and proceedings before the agency . . . concerned." (emphasis added)); Fed. R. App. P. 16 ("The record on review or enforcement of an agency order consists of . . . any findings or report on which it is based." (emphasis added)); see also 28 U.S.C. §§ 2071-77 (providing federal courts with authority to establish binding rules of procedure and evidence). Thus, EPA's slipshod creation of the regulatory docket for the Proposed Rule cannot withstand scrutiny under either the APA or the judicial review provisions for Title 28.
Third, EPA has violated the APA (and all statutes that track its notice-and-comment requirements) by failing to provide sufficient specificity regarding the Proposed Rule. Under the APA, notice is only sufficient "`if it affords interested parties a reasonable opportunity to participate in the rulemaking process,' and if the parties have not been `deprived of the opportunity to present relevant information by lack of notice that the issue was there.'" WJG Tel. Co., Inc. v. FCC, 675 F.2d 386, 389 (D.C. Cir. 1982) (citations omitted); see Fla. Power & Light Co. v. Nuclear Regulatory Comm'n, 846 F.2d 765, 771 (D.C. Cir. 1988). The Proposed Rule is plagued by a lack of detail as to how EPA will interpret the broad and vague terms it creates, how it will use the Rule in connection with particular rulemakings, and how it will implement the new "independent peer review" and "exemption" provisions, and these shortcomings undermine the sufficiency of the notice provided by the Proposal. For example:
 The new definitions of "dose response data and models," "pivotal regulatory science," "regulatory decisions," "regulatory science," and "research data," in § 30.2 are extremely broad and impermissibly vague. EPA provides no specific examples of what these terms mean or how they will actually be implemented under the long list of statutes implicated by this rule.
:: Sections 30.1, 30.2, 30.3  -  discussing the purpose of the Proposed Rule, the applicable definitions, and how the Proposed Rule will apply  -  provide broad requirements for EPA without giving any information regarding how or when EPA will implement them in connection with any given proposed or final rule.
 EPA states in § 30.5 that the Proposal will require EPA to "ensure that dose response data and models underlying pivotal regulatory science are publicly available in a manner sufficient for independent validation . . ." Yet EPA provides no indication of what this means, how it will so "ensure," or what it will do if it is not possible to ensure this, among other shortcomings.
 Section 30.6 addresses the "additional requirements" related to the use of dose response data and models underlying "pivotal regulatory science," but it is devoid of details necessary to assure meaningful public review and comment on how, when, or in what way EPA will implement this. It also cites no specific science and gives no indication of how or why EPA is attempting to redefine scientific information on dose response differently than it has done in the past to allow for informed consideration of why EPA is even attempting to address this issue.
 Sections 30.7 and 30.9 describe the "role" of "independent peer review" in the Proposal and permissible "exemptions" under the Rule, yet these descriptions fail to provide the process that will be used to inform the public of when a given document or study impacted by this rule does or does not qualify for an exemption. There is no indication of whether EPA will provide notice or comment, or any steps allowing for public participation, when deciding whether or not to exclude a given study from consideration.
 The preamble to the Proposed Rule is replete with broad (and unsubstantiated) statements regarding limits on EPA's ability to consider and use science in rulemakings, as well as environmental issues EPA envisions being impacted by the Proposed Rule  -  for example, the NAAQs  -  but EPA fails to provide any specific information to inform public notice and comment on the purported impact of Proposed Rule. Commenters have done their best in view of this to provide comment on all of the likely harm that EPA's vague and general statements would cause if fully implemented in regulatory language, but they are severely prejudiced due to EPA's refusal to provide specific notice as required by the APA.
In sum, the commenter contends that the EPA has failed to satisfy the requirements for specific public notice that can assure meaningful comment, falling far short of the fundamental threshold requirements for notice and comment. This shortcoming is fatal to its ability to finalize this Proposed Rule.
In addition, the commenter notes that it is especially problematic that EPA's Proposal states that EPA will apply the Proposed Rule to "regulatory decisions" which it defines as "final regulations determined to be `significant regulatory actions' by [OMB]." 83 Fed. Reg. at 18,773 (proposed § 30.2). Under this definition, EPA can exclude and ignore science in future rulemakings without specifying where or how it will do so and without providing any public notice or comment in a future rulemaking that would be impacted by this Rule. EPA's attempt to limit the rulemaking process for undefined other rules in this manner is a violation of the APA for all of the reasons described above.
Finally, the commenter states that the EPA's APA violations would cause significant harm to Commenters and the public if the Rule were finalized without correcting these problems. The inability to review and attempt to understand the documents on which EPA relies and thus to comment meaningfully and receive effective judicial review cause severe prejudice to the affected public. See 5 U.S.C. § 706(2). Even if EPA were to add documents to the docket at a later date, that would not cure this fatal flaw. Given the sweeping scope of this Proposed Rule  -  which would restrict the consideration and use of important health science in rulemakings which, in turn, would likely lead to a weakening of air, water, waste, chemical, pesticide and other protections  -  the failure to publish the documents for review for an adequate time period has undermined the public's ability to comment meaningfully, and impermissibly prevented the affected public from being able to seek and receive effective judicial review based on the record.
Response: See response to Comment 2157-149. As discussed in the final rule preamble at Section III.A, EPA will issue statute-specific regulations following this procedural rule. Forthcoming substantive rulemakings will be subject to public notice and comment and judicially reviewable, and any implementation issues will be addressed as appropriate.
Comment [6163-1115]: Commenter (6133) states that, as EPA is aware, when an agency drafts a proposed rule pursuant to congressionally delegated authority, the exercise of that authority is governed by the informal rulemaking procedures outlined in the APA, 5 U.S.C. § 553.5. EPA is required to provide the public with adequate notice of a proposed rule, followed by a meaningful opportunity to comment on the rule's content. 5 U.S.C. § 553 (b)-(c).
The requirement under § 553 to provide the public with adequate notice of a proposed rule is generally achieved through the publication of a notice of proposed rulemaking in the Federal Register, and the APA requires that the notice of proposed rulemaking include "(1) the time, place, and nature of public rulemaking proceedings; (2) reference to the legal authority under which the rule is proposed; and (3) either the terms or substance of the proposed rule or a description of the subjects and issues involved." 5 U.S.C. § 553(b)1-3. Generally speaking, the notice requirement of § 553 is satisfied when the agency "affords interested persons a reasonable and meaningful opportunity to participate in the rulemaking process." Forester v. Consumer Prod. Safety Comm'n, 559 F.2d 774, 787 (D.C. Cir. 1977).
The commenter adds that they believe that the Proposal fails to reference any other legal authority to support its adoption. The agency claims its Proposal is "consistent with" Administrative Procedure Act provisions to ensure public participation in the rulemaking process, 83 Fed. Reg. at 18,769/2, but this faint "consistent with" falls far short of any legal authority for the Proposal, or even any claim of such authority. The APA provides no authority for the Proposal and, tellingly, EPA does not and cannot identify any authority therein. Even were this "consistent with" claim an attempt by EPA to claim any legal authority for the Proposal, the throw-away statement fails to provide sufficient notice for the public to comment on the proposed rule or any asserted legal authority in the APA.
Finally, the commenter states that the Proposal's solicitation of comment -- "on whether additional or alternative sources of authority are appropriate bases for this proposed regulation" -- does not and cannot itself provide any justification for EPA finalizing a rule based on additional or alternative sources of legal authority. This fails to provide sufficient notice for the public to comment on the proposed rule or any other possible legal authorities. For all these reasons, the commenter contends that the EPA lacks any basis to finalize a rule invoking any other legal authorities to support its adoption.
Response: See response to Comment 6163-1114.