Commenter (8771) states that "reducing reliance on defaults" sounds reasonable for a brief second, but why on earth would an Agency want to reduce reliance on sound science? The commenter states that defaults should be discarded when they are wrong, either across-the-board or correct in general but wrong for a specific chemical or application (NAS 1994; Finkel 2003; NAS 2008). The commenter asserts that it is arbitrary for EPA to penalize research that relies on reasonable and time-tested models merely because they are parsimonious and have been used as "defaults." According to the commenter, replacing evidence-based "default" models and assumptions with half-baked exculpatory theories is a recipe for the erosion of science -- and has no place in this proposal. The commenter states that defaults are often simple and were arrived at decades ago -- so were Newton's laws and Maxwell`s equations --  but that has no bearing on whether they are also theoretically correct, logical, and predict risk better than fancier models that look more sophisticated but may be completely unwarranted.
Commenter (6924) supports updating established defaults to account for newer science demonstrating that EPA's typical safety factor of 10 is insufficient to account for variability due to life stage, genetics, underlying disease status, and external stressors that may be due to poverty or other difficult life conditions.
Response: EPA has not finalized proposed 40 CFR 30.6 so there are no requirements specific to dose-response models.
Do Not Support the Use of Default Assumptions
Comment: Commenter (2426) supports the proposed rule language relating to consideration of dose response models and uncertainty, rather than relying on default assumptions.
Commenter (6911) concurs that this proposal should include EPA's actions on exposure-response relationships and model selection. The commenter asserts that, despite the fact that significant progress has been made in the U.S. and internationally to define scientific frameworks and evaluate toxicity datasets to determine biologically plausible modes of action and relevance to humans, the EPA, in practice, has embraced conservative default assumptions, and any movement to modify these approaches has been slow.
Response: EPA has not finalized proposed 40 CFR 30.6 so there are no requirements specific to dose-response models.
<IP_ADDRESS>.8 Other Comments on Models
Comment: Commenter (4541) asserts that the EPA's new policy of scientific integrity and transparency should be applied to computer climate models that currently prevail in EPA's funded, published, and cited climate science. The commenter contends that continued use of default models, without consideration of alternatives or model uncertainty, creates a false scientific justification for EPA actions, policies, and regulatory burdens. The commenter states that arbitrary choices made by EPA officials have resulted in assumptions about model accuracy that have been proven incorrect, unable to recreate past climate history or project future climate changes. According to the commenter, all EPA-adopted models have predicted a much higher rate of temperature sensitivity to CO2 than has been recorded by temperature observations. The commenter adds that such direct proof of model inadequacies cannot be ignored and must be addressed before policy and regulatory actions are proposed.
Response: This rule does not apply to models. See the preamble of the final rule which explains the decision to limit the scope of the rule to dose-response data.
Comment: Commenter (9224) agrees that any data, methodology, or models produced by EPA itself should be transparent and available to the public in a reproducible manner.
Response: EPA agrees. The purpose of this rule is to increase transparency of dose-response data underlying pivotal science.
Comment: Commenter (6891) states that, when appropriate, EPA should apply transparency principles to data and models that EPA has considered but rejected, particularly when EPA relies on data or models that do not meet stated acceptance criteria for their use. The commenter states that adopting a policy of explaining acceptance and rejection of data or models would be well-supported by EPA's long-held policies of transparency, validity, and public participation. The commenter suggests that EPA make rejected data and models available for public review and evaluation. According to the commenter, by allowing comments on data and models that EPA evaluated but excluded, EPA will further its goals and provide a check on its own assumptions and analyses.
Response: This rule applies narrowly to pivotal science. Applying this rule to all information considered by EPA, which may include thousands of studies, would be unfeasible.
Comment: Commenter (8281-PH20) states that the proposed rule's requirements for specific types of defaults, test methods, dose response models, and/or analysis are not supported by current science.
Response: This rule does not apply to models or set requirements for test methods of the use of defaults.
Comment: Commenter (6373) is concerned that §30.6 imposes unsustainable burdens on EPA that will inhibit it from timely considering scientific data. The commenter suggests that the requirements of §30.6 will quickly overwhelm EPA staff and result in a backlog of relevant scientific studies.
Response: EPA has not finalized proposed 40 CFR 30.6. See the preamble of the final rule which explains the decision not to finalize this proposed provision.
<IP_ADDRESS> Publicly Available in a Manner Sufficient for Independent Validation
<IP_ADDRESS>.1 Support/Opposition to Requirement
Comment: Several comments were received that generally supported or opposed EPA's proposal that data and models underlying scientific studies pivotal to EPA regulations be "publicly available in a manner sufficient for independent validation."
Commenters (0555, 1974, 6129, 6148, 6175, 6176, 6177, 6179, 6366, 6449, 6867, 8281-PH83) express support for EPA's proposal that the data and models underlying scientific studies pivotal to EPA regulations be publicly available in a manner sufficient for independent validation. Reasons provided for their support include:
 Commenter (6176) believes that public access to underlying data will benefit the EPA. The commenter expresses that the public participation in the rulemaking process helps to inform and shape the decisions made by agencies, and EPA staff who may have access to underlying data and methodology will benefit from hearing different views on this data and methodology, including views from other scientists. The contends that even the "experts" could use the public's feedback and the expertise of other researchers and scientists.
 Commenter (6867) asserts that promises to provide other researchers with access to data, methodology, or computational code are not an adequate substitute for public availability. According to the commenters' experience, such promises often go unfulfilled (example provided). Significantly, the commenter states that many uncredentialed members of the public are capable of reviewing and identifying errors in data, methodology, and computational code (provides examples).
 Commenter (6449) states that increased data and model availability will facilitate and expedite corrections, for example, where changes were made in a model or to data and accompanying text was not updated. The commenter adds that data and model availability will also allow interested parties to determine if alternative data and model choices would substantially change results.
 Commenter (6175) believes that increased transparency around the assumptions and methods employed in underlying scientific studies will enhance meaningful review and public comment on proposed regulatory actions and decisions. Similarly, commenter (6366) expresses that a more-informed stakeholder community, which has access to such data, will be better suited to participate in the notice and comment process.
 Commenter (1974) asserts that the Proposal would result in the American public having greater confidence in the policies advance it its (EPA's) name. According to a recent National Academy of Sciences report titled "The Irreproducibility Crisis in Modern Science," the commenter states that the EPA is contrasted unfavorably with agencies like the NIH for lax science standards: "We recommend that other government agencies, especially the Environmental Protection Agency and the Department of Energy, adopt the NIH's new standards." (https://www.nas.org/storage/app/media/Reports/Irreproducibility%20Crisis%20Report/NAS_irreproducibilityReport.pdf, page 50)
 Commenter (6129) notes that their experience with EPA's physiologically based pharmacokinetic (PBPK) model for trichloroethylene (TCE) indicates that replication of this model poses difficulties for even sophisticated users, based on the level of documentation that is currently available to the public in the IRIS Toxicological Review for TCE. Thus, the commenter encourages the EPA to develop a system by which the same models that EPA uses in its risk assessments can also be used by the public for independent validation efforts.
 Commenter (6148) notes that, in some cases, concerns have been raised that data have been "cherry-picked" to achieve a pre-determined outcome. According to the commenter, unless data is publicly available for independent assessment, these concerns are likely to continue. The commenter states that data transparency not only allows interested parties to know what data was used in an analysis, but also what data has been omitted. The commenter contends that the most practical means to alleviate such concerns is to enable interested parties the ability to review and comment on the raw data and resulting analysis that form the basis of EPA decisions.
 Commenter (6177) asserts that the EPA regulations based on dose response data and models to predict health or environmental impacts that are opaque end up creating unnecessary costs and competitive disadvantages for U.S. companies. The issue for many within industry has been the inability to understand the Agency's underlying analyses and conclusions, as data sets and models often are incomplete or otherwise unavailable for review. According to the commenter, the proposed rule would provide access to the information and data needed for those parties not part of EPA's SAB or peer-review panels to comment more thoroughly on proposed environmental regulations.
 Commenter (6179) asserts that the Transparency Proposal, when finalized, can enhance public input into the rulemaking process, resulting in better air regulations. The commenter adds that the Transparency Proposal will lead to increases in replicability and verification in the scientific process, thereby allowing the testing of critical methodological assumptions and mitigating biases in key studies that inform significant regulations. The commenter believes that the Proposal recognizes that transparency can be more than simply maximizing disclosure, but rather a means by which studies are contextualized to enhance public participation in the notice-and-comment process critical to effective rulemaking.
 Commenter (0555) asserts that the EPA's proposal to clearly identify available studies and make them available for public review is not only important for ensuring decisions are supported by the best information, but also consistent with policies on scientific integrity espoused by previous administrations (cites the Office of Management and Budget's (OMB's) 2002 Information Quality Guidelines, and a 2009 Memorandum issued by President Obama that encouraged transparency) and scholarly journals (cites Science and Nature).
 Commenter (0555) states that the selection of the model used to estimate responses to different exposures to contaminants can have significant impacts on estimated regulatory benefits and notes that, in 2007, the Office of Information and Regulatory Affairs (OIRA) and the Office of Science and Technology Policy (OSTP) observed in a memorandum to agency heads on risk assessment that a "high degree of transparency with respect to data, assumptions, and methods will increase the credibility of the risk analysis, and will allow interested individuals, internal and external to the agency, to understand better the technical basis of the analysis."
Response: The EPA agrees with commenters that the opportunity to independently validate the pivotal science that the EPA relies upon is important in furthering scientific understanding and the Agency's mission. The EPA also believes that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will improve the opportunity for the public to access the EPA's scientific analyses and resulting regulatory actions in a way that is beneficial to the scientific process, the Agency's mission, and the public's health and safety.

Through this rulemaking, the EPA is pursuing an incremental approach to maximizing transparency in the pivotal science that it relies upon to ensure that implementation is done in a thoughtful and deliberate way that focuses on the EPA's most impactful actions, minimizes unintended consequences, and informs future transparency requirements. As described in the preamble of the final rule, the EPA is focusing on the underlying dose-response data for this rulemaking because of the influence these data have on particularly impactful decisions at the Agency.

Comment: Commenters (0044, 0671, 0980, 1973, 2337, 5017, 6133, 6135, 6168, 6772, 6868, 6939, 9227) express opposition for EPA's proposal that the data and models underlying scientific studies pivotal to EPA regulations be publicly available in a manner sufficient for independent validation. Reasons provided for their opposition include:
 Commenter (0671) contends that researchers are always free to confirm a study by performing their own study, and they often do, and doing so is a much better use of limited scientific resources than reanalyzing old data. The decision to make data public (and which data to make public) should stay between the researcher, the IRB, and journal editors. The best available science should be adjudicated via the peer review process, not by a applying a restrictive and unreasonable data access standard.
 Commenter (6868) notes that federal courts have already considered requests for revealing scientific data used by EPA. The commenter states that the court found these requests lacking in merit. According to the commenter, in the 2002 American Trucking Associations v Environmental Protection Agency heard by the D.C. District Court of Appeals, the court ruled that the CAA imposes no obligation to obtain and make public the data underlying certain "key studies" and similarly rejected arguments that a general requirement be imposed for EPA to obtain and publicize the data underlying published studies on which the agency relies. The court further stated, "requiring agencies to obtain and publicize the data underlying all studies on which they rely would be impractical and unnecessary." The commenter asserts that the court's findings call into question EPA's justification for the proposed rule.
 Commenter (0980) asserts that requiring all data and methodology to be published is completely impractical. The commenter contends that published studies will not be updated to reflect this new requirement, and new studies will only meet these standards if the authors have this requirement in mind. According to the commenter, scientists do not publish their data reduction pipelines, not because they are hiding something, but because there are standard procedures, or because they are easily reproducible by other competent scientists. The commenter states that the use of scientific studies by regulatory agencies needs to align with how scientific studies are used by others in the scientific community. According to the commenter, if a scientist wants to see more data or ask a question about some methodology, they email the author of a study.
 Commenter (1973) asserts that making data and models publicly available in a manner sufficient for independent validation is not necessary to assure people that the studies have been reasonably conducted because the study protocols, recruitment criteria, measurement techniques, and statistical modeling methods including lists of the adjustment variables have all been made publicly available, and peer reviewed. The commenter adds that tables of descriptive statistics are also already public. Moreover, the commenter asserts that the peer review process of journals means that a group of experts unrelated to the study have already commented on, and requested changes in, methods that they find wanting. Moreover, the commenter states that the scientific community has already adopted several protocols and guidelines specifically designed to improve reporting and evaluation of studies and improve the quality and transparency of the interpretation of findings.
 Commenter (6133) asserts that neither the Proposal nor docket contains any factual, scientific, technical, logical, or legal support for the suggestion that science and data that are "publicly available in a manner sufficient for independent validation" are necessary elements for the "validity," "reliability," or "transparency" of scientific information. The commenter adds that the EPA provides no basis for its assumption that science or studies for which data are publicly available yield more valid or reliable results than the best available, peer-reviewed, independent, credible science, for which the underlying data are not publicly available. Similarly, the commenter notes that the Proposal arbitrarily fails to address, much less explain, why prior EPA regulatory actions that relied upon studies, data, or other information did not reflect the "best available science" or why they were otherwise unreliable, despite failing to meet the Proposal's standards.
 Commenters (2337, 6914, 6939, 9227) assert that there are many reasons (legal/ethical/practical) why data cannot, or cannot easily, be made public, including data no longer being available (older studies), the size of the study and volume and format of the underlying data, and privacy and individual/business confidentiality concerns. Commenter (9227) asserts that the agency's failure to consider or examine any of the legitimate reasons for not making data publicly available is arbitrary and capricious.
 Commenter (6135) opposes asking scientific researchers to make publicly available administrative claims datasets (e.g., hospital admission and emergency department records, Medicare records, etc.) or other population datasets (e.g., National Health and Nutritional Examination Survey, NIH-AARP Diet and Health Study, etc.) which would not only violate data use and IRB agreements but is wholly unnecessary given that these records are already openly available to any individual or group that adheres to data privacy requirements.
 Commenter (6939) asserts that the Proposal would necessitate the sharing of data protected under health privacy laws and otherwise barred from disclosure and that its main impact would be to undermine science in regulatory decision-making by making it impossible for EPA to consider the best available science when regulating by prohibiting, except in limited circumstances, the consideration of relevant science from the decision-making process if the underlying data is not publicly available. The commenter notes that, in the absence of substantial new resources, efforts to replicate studies and prepare data for public release would reduce the resources available for new research and delay the regulatory process.
 Commenter (9227) notes that, with respect to studies that have already been completed, there are formidable barriers to public release of underlying data and models, particularly older studies where data is no longer available for a multitude of reasons.
 Commenter (6168) states that requiring full data availability and broad application of default models encourages analyses aiming to sow false uncertainty and put off protective regulation. According to the commenter, in addition to eliminating evidence from the regulatory process, the proposed measures on data and model availability facilitate not transparency but the culture of science denialism, which fits with the anti-protection agendas of fossil fuel and chemical industry lobbyists. The commenter contends that scientific uncertainty is used not only to deny risks, but to make potential benefits of regulation uncertain and to give weight to the costs; such cost-benefit analysis justifies failure to regulate. In particular, the commenter notes that external "validation" -- a central component of this proposed rule -- can instead take the form of "deconstruction of evidence" through "data dredging".
 Commenter (9227) notes that there are legal and ethical requirements that restrict making public the data underlying studies, including rules to shield private personal information, requirements to maintain CBI, situations where obtaining the necessary permissions to release data are logistically difficult or impossible (e.g., historic studies), and situations in which researchers have made significant investments in developing datasets that they intend to continue to work with for future studies. According to the commenter, not all these barriers can be overcome, nor can they be overcome in every case. The commenter notes that, while there are ways to potentially address some of them, they can be extremely costly and burdensome, and/or may harm the prospects for further research. Accordingly, the commenter asserts that, while the scientific community has made efforts to make more data publicly available, to the best of our knowledge all of the policies adopted by government and academic journals recognize that data is not, and need not be, publicly available to evaluate their quality. The agency's failure to consider or examine any of these legitimate reasons for not making data publicly available is arbitrary and capricious.
 Commenter (9227) notes that, even with respect to prospective application of EPA's proposal, providing for public release of underlying data and models is costly and resource intensive, creating a serious disincentive for researchers to meet EPA's proposed requirements. The commenter provides that investigators willing to make their study underlying data publicly available would still face the logistical hurdle of making the data and models available in a manner sufficient for independent validation by the public. In addition to the cost of thoughtful and effective deidentification or redaction of sensitive information, the commenter notes that the proposed text would likely require researchers to prepare annotated manuals including precise detail as to what variables were collected, how information was collected, and the rationale for each step taken. According to the commenter, some manuals alone run into hundreds of pages. The commenter provides that one press account noted the example of publicly available datasets from the National Center for Health Statistics, which can come with 100-page manuals; where researchers would need to hire additional staff to meet such requirements.
 Commenter (5017) notes that some studies cover populations that are so limited in size or specialized in their characteristics that those data should not be posted on the web for all the world to see.
Response: Based on a consideration of public comments received in response to the 2018 proposed rule and the 2020 SNPRM, the EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. As described in the preamble to the final rule, the EPA is not categorically excluding any studies from consideration when promulgating significant regulatory actions or developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information.
As described in the preamble to the final rule, the transparency provisions of the final rule apply to dose-response data underlying pivotal science. Under this rule, the EPA will give greater consideration to pivotal science for which the underlying dose-response data are either publicly available in a manner sufficient for independent validation or, in the case of PII, CBI, or proprietary data, available through restricted access that affords privacy in a manner sufficient for independent validation. In cases where dose-response data underlying pivotal science are not available in a manner sufficient for independent validation, the EPA may still use the pivotal science after either giving it lesser consideration or receiving an exemption from the requirements of this rule under 40 CFR 30.7.
The requirements included in the final rule provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical considerations in order to ensure the EPA maintains a strong scientific basis for its decision-making. The incremental progress made possible by the final rule provides an important step towards prioritizing transparency in particularly impactful EPA rules and assessments.
<IP_ADDRESS>.2 Making Data and Models Available Does Not Make Them Understandable to the Public
Comment: Commenters (2169, 6046, 6168) assert that the Proposal to make data and models underlying scientific studies pivotal to EPA regulations be "publicly available in a manner sufficient for independent validation" does not make the studies understandable to the public.
Commenter (6168) states that the requirements of this rule fail to support significant public participation, because data and models are meaningless without explanation. According to the commenter, simply making them available does nothing to make them understandable or usable to the public. The commenter notes that there is already a huge gap at EPA between public availability of information and meaningful accessibility and transparency. The commenter states that this is true not only regarding data availability but also when analysis is made public without adequate explanation. The commenter notes, as an example, the extensive analyses underlying the NAAQS, which regulate the six criteria air pollutants. For each of these pollutants, the commenter states that the EPA synthesizes all relevant scientific information into an Integrated Science Assessment and a Risk or Exposure Assessment and then develops a Policy Assessment based on these reviews and recommendations from an advisory council. The commenter states that these thorough assessments are massive documents -- together running into the thousands of pages for each of the six pollutants -- that are publicly available. The commenter provides that the webpage for each pollutant, however, contains shockingly little information to help users understand and use this highly technical information. For example, according to the commenter, the page on particulate matter contains a single sentence describing "What Scientific and Technical Information Supports Review of the Standards" and a handful of sentences explaining the health effects of particulate matter. The commenter asserts that the EPA provides nothing between this handful of sentences and the thousands of pages in the assessments that would make this sufficient for public education and engagement. The commenter contends that this form of "transparency" falsely equates availability with accessibility. The commenter concludes that, if making analyses available without context fails at transparency and public participation, then making raw data and dose-response models available without context is an even greater failure.
Commenter (6046) notes that very few members of the general public have the statistical expertise or time to properly evaluate epi or pesticide study data and very few probably have any interest. Thus, the commenter asserts that disallowing the use of non-publicized data would not impact the general public. The commenter provides that some scientific researchers may be able to and have interest in evaluating such data. The commenter provides that currently, if EPA researchers use non-publicized data, researchers can request the raw data from the study author. According to the commenter, if the researcher cannot release the data to the requester due to confidential reasons, there is frequently sufficient summary statistics and study methodology reported by EPA that a researcher can replicate and evaluate the study results (e.g. using Monte Carlo methods).
Commenter (2169) provides that releases of information/models necessary to allow the "public to understand" is not achievable. The commenter notes that public educational attainments are not at a level needed to insure understanding. Alternatively, the commenter asserts that the releases necessary to allow competent scientists, with access to statistical and other tools of scientific assessment, are possible. According to the commenter, the problem presented by the proposed standard of understanding is that simple public confusion becomes a basis for setting aside a regulation. The commenter asks: How is the public confusion to be ascertained? By public surveys? By assessing comments received from the public?
Response: In response to these comments, the EPA has clarified the purpose of the rulemaking in the preamble of the final rule. The purpose of this rulemaking is to increase transparency by codifying internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information. The EPA believes that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will improve the opportunity for the public to access the EPA's scientific analyses and resulting regulatory actions in a way that is beneficial to the scientific process, the Agency's mission, and the public's health and safety.
Comment: Commenter (6046) notes that the proposed rule will not meet its stated goal of better informing the public. According to the commenter, by disallowing EPA researchers and analysts the use of study data which are not published, the non-published study data, regardless of their quality and applicability, will not be used or discussed by EPA, thus hiding their existence from the public. Thus, the commenter contends that the public would be less informed about the spectrum of data. Further, the commenter adds that disallowing study data would mean that a thorough critical weight of evidence evaluation cannot be conducted.
Response: Based on a consideration of these comments and other, similar comments received in response to the 2018 proposed rule and the 2020 SNPRM, the EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. As described in the preamble to the final rule, the EPA is not categorically excluding any studies from consideration when promulgating significant regulatory actions or developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information.
As described in the preamble to the final rule, the transparency provisions of the final rule apply to dose-response data underlying pivotal science. Under this rule, the EPA will give greater consideration to pivotal science for which the underlying dose-response data are either publicly available in a manner sufficient for independent validation or, in the case of PII, CBI, or proprietary data, available through restricted access that affords privacy in a manner sufficient for independent validation. In cases where dose-response data underlying pivotal science are not available in a manner sufficient for independent validation, the EPA may still use the pivotal science after either giving it lesser consideration or receiving an exemption from the requirements of this rule under 40 CFR 30.7.
Implementation of this rule will more effectively share pivotal science for external consideration and, thus, increase the opportunity for independent subject matter experts to validate pivotal science should they choose to do so. As the dose-response data are better understood through independent reanalysis, the public will, if they so choose, be able to more effectively comment, engage, and hold the EPA accountable during the development of future significant regulatory actions and influential scientific information.
<IP_ADDRESS>.3 What Degree of Public Availability is Sufficient
Proposal Unclear Regarding the Level of Access and Disclosure/Explanation is Needed to be Publicly Available
Comment: Commenters (2426, 6133, 6194, 6362, 6875, 6915) express that the Proposal is not clear regarding what EPA considers to be sufficient public availability and the level of access and disclosure/explanation to the public that is intended. Commenter (6362) suggests that, another important aspect relevant to "public availability" is the level of data refinement EPA will require.
Commenter (6362) expresses that the EPA does not define what it means by its use of the term, "publicly available." The commenter explains that there is more than one definition of the term currently in use by federal agencies. Commenter (2426) states that, perhaps one of the more frequently used terms that has caused the most confusion is "publicly available." The commenter provides that, outside the context of this proposed rule, "publicly available" generally means that information is available without restriction to anyone who wants it. Commenter (6194) explains that the "public" referenced in the Proposal could be scientists trained in the relevant field or members of the general public. Commenter (6875) notes that a significant number of journals require a subscription or payment in order to read their articles. The commenter asks: Does EPA consider these studies publicly available? Commenter (6362) adds, that if EPA intends to determine this on a case-by-case basis, that also be made clear.
Commenter (2426) notes that, while making the level of accessibility to information available without restriction to anyone who wants it is acceptable for most information, the commenter notes that it becomes problematic for sensitive personal information, personal health information, or CBI. In order to meet existing privacy and confidentiality laws that protect this type of information, the commenter states that increased accessibility tends to come with reduced refinement (i.e., less detailed information) in the data.
Commenter (6133) asserts that, although the phrase, "in a manner sufficient for independent validation," is repeated frequently throughout the Proposal, and is integral to its very operation, the phrase is not defined in the proposed definitional section (§30.2). The commenter notes that, later in proposed regulatory text, the Proposal does specify that "[i]nformation is considered `publicly available in a manner sufficient for independent validation' when it includes the information necessary for the public to understand, assess, and replicate findings." 83 Fed. Reg. at 18,773 - 74 (proposed §30.5). The commenter adds that proposed §30.5 goes on to list categories of information that "may" be included in this concept. The commenter notes that the explanation provided by proposed §30.5 is a non-definition; it provides no additional clarification. The commenter asks: How much information is sufficient for the public to understand, assess and replicate findings? Can this standard sometimes be met by releasing methodology but not raw data?
Commenter (6133) expresses that the proposal fails to explain how the term, "in a manner sufficient for independent validation," and the proposed §30.5 definition will increase transparency in science or why it is necessary to ensure that EPA will consider the best available science. To the contrary, the commenter contends that the Proposal's approach would preclude EPA from considering the best available science that is relevant to EPA's responsibilities. The commenter adds that the EPA also fails to explain why data underlying peer-reviewed studies must be publicly available "in a manner sufficient for independent validation" when independent researchers can verify science without making the underlying data, which is often confidential, publicly available.
Commenter (6194) states that the Proposal provides that, "where necessary, data would be made available subject to access and use restrictions." 83 Fed. Reg. at 18,773-74. However, the commenter states that what this process would look like is not addressed in the Proposal or discussed at all in the preamble. According to the commenter, this leaves significant open questions, such as who will pay for and implement these restrictions and who will decide which people can access the data and under what standards.
Commenter (6915) states that the "information necessary for the public to understand, assess, and replicate findings" examples provided in the Proposal include "(a) Data (where necessary, data would be made available subject to access and use restrictions)[;] (b) Associated protocols necessary to understand, assess, and extend conclusions; (c) Computer codes and models involved in the creation and analysis of such information; (d) Recorded factual materials; and (e) Detailed descriptions of how to access and use such information." 83 Fed. Reg. at 18,774. The commenter states that this could be interpreted to require maintenance of data down to the most basic level, verging on the absurd, and could impose unreasonable and impractical requirements that go well beyond those already included in current protocols. For example, the commenter notes that it could require maintenance of records that are not routinely archived by academic research labs, such as printouts of data from all calibration curves and analyses from instruments that measure clinical parameters in blood or other similar endpoints in animal and human studies, or photos of each individual organ as it is evaluated for gross pathology in toxicology studies. According to the commenter, even if such data are maintained for a period after a study is completed, it is not feasible for such records to be maintained indefinitely by research laboratories, which would then make the study that the data supports unavailable for use in future regulations. According to the commenter, for toxicology studies, such data availability requirements would result in favoring studies performed under Good Laboratory Practice (GLP) protocols, which typically retain more raw data than research studies. The commenter provides, however, that GLP studies may not evaluate the most sensitive and relevant toxicological effects of the chemical being studied and are not inherently of higher quality than studies conducted under other protocols.
Commenter (0671) cites the Proposal preamble as stating "[t]he benefits of EPA ensuring that dose response data and models underlying pivotal regulatory science are publicly available in a manner sufficient for independent validation are that it will improve the data and scientific quality of the Agency's actions and facilitate expanded data sharing and exploration of key data sets; this is consistent with the conclusions of the National Academies." The commenter states that nothing in that report suggests the EPA should require publication of PII on human subjects. The commenter adds that there is also a problem here with the definition of "data... underlying pivotal regulatory science". The commenter asks: Does this mean the raw data, or the data used to make the final figures and tables in a paper or the data as it exists at some processing step in between? If the raw data, how raw? Some raw genomic data is so large it can't be efficiently stored. The commenter contends that there is no one unique thing called "the data" for a study.
Response: In response to these comments and other, similar comments, the EPA issued a supplemental notice of proposed rulemaking (SNPRM) in 2020 that included definitions for key terms in the 2018 proposal and clarified definitions included in the 2018 proposal. The EPA also proposed a modified approach to the public availability provisions for data and models that would underly significant regulatory decisions and an alternate approach.
Data Availability Usability Certification
Comment: Commenter (0556) states that the EPA needs to specify the requirements of proper data availability as needed for usability. According to the commenter, this might be a simple checklist of questions, or perhaps something more complex. The commenter provides that there is already considerable discussion and action within the open science community, as to just what availability for the purpose of replication requires. The commenter recommends that EPA draw on this existing work and expertise. The commenter notes that the basic requirements are not that mysterious: (1) The data and codes need to be available and properly documented. (2) The procedures followed, and decisions made along the way need to be properly explained. (There is already relevant literature on this topic in some fields, especially biomedical.) (3) Research that cannot be certified is excluded.
Commenter (0556) suggests that the EPA might need to certify that each piece of research it wants to use is properly available for attempted replication. The commenter provides that they believe that the best way to do this is for EPA to contact the researchers to determine that the research is properly available. According to the commenter, another possibility is to establish a practice of self-certification among those researchers who desire that their research be usable by EPA. The commenter adds that certification of EPA usability might also be done by third parties, such as scholarly societies. The commenter states that such certification could even become a condition of funding with some funders.
Response: As described in the preamble to the final rule, scientific information is considered available in a manner sufficient for independent validation when it includes the information necessary to understand, assess, and reanalyze findings. The EPA has identified the information that may need to be available for dose-response data to be considered publicly available in a manner sufficient for independent validation, including:
 Data (data would be made available subject to access and use restrictions);
 Associated protocols necessary to understand, assess, and extend conclusions;
 Computer codes and models involved in the creation and analysis of such information;
 Recorded factual materials; and
 Detailed descriptions of how to access and use such information.
As described in the final rule, the EPA intends to clearly identify the studies considered pivotal in the documentation at the proposed rule stage for significant regulatory actions and when influential scientific information is disseminated for peer review. The Agency will also identify the pivotal science that was given lesser consideration and provide a description of why lesser consideration was given.
The EPA would like to emphasize that this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedure and does not direct or require any outside entity to make dose-response data available or determine that dose-response data are available. Further, the final rule does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Necessary Associated Protocols for Making Data and Models Available
Comment: Commenter (2426) recommends that the EPA strongly consider implementing a formal request and approval process for qualified researchers to obtain access to data that is protected by confidentiality laws, rather than aiming for all data to have unrestricted access. The commenter contends that this process would still allow identifiable or refined de-identified data to be re-evaluated by experts external to the EPA or the original study authors, while protecting the privacy of the study participants or businesses who created the model or dataset.
Commenter (6362) states that the National Academies of Science, Engineering, and Medicine (NAS) held a workshop in 2016 to discuss obstacles for sharing data. The commenter provides that the NAS defined several key terms to ensure clarity at the workshop. The commenter recommends that the EPA consider adopting a similar lexicon to increase the clarity of its regulation and refers the EPA to Table 1 of Appendix B of their comment letter for these terms (Table 1. Definitions in NAS Principles and obstacles for sharing data from environmental health research: Workshop summary). In addition, the commenter provides that the NAS Report suggests a "cleaned dataset" would be acceptable to use for all routine analyses and verification. (See Table 2 in Appendix B of their comment letter (Table 2. Data flow from NAS Report)). The commenter recommends that the EPA establish clear standards on the acceptability of "cleaned datasets." The commenter contends that this will help to standardize data reporting and formatting. It will also prevent over- and under-reporting.
Response: Based on a consideration of these comments and other, similar comments received in response to the 2018 proposed rule and the 2020 SNPRM, the EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. Under this rule, the EPA will give greater consideration to pivotal science for which the underlying dose-response data are either publicly available in a manner sufficient for independent validation or, in the case of PII, CBI, or proprietary data, available through restricted access that affords privacy in a manner sufficient for independent validation. In cases where dose-response data underlying pivotal science are not available in a manner sufficient for independent validation, the EPA may still use the pivotal science after either giving it lesser consideration or receiving an exemption from the requirements of this rule under 40 CFR 30.7.
In the 2020 SNPRM, the EPA presented the different stages of data described in the National Academies of Science, Engineering, and Medicine (NAS) 2016 workshop reported cited by the commenter and proposed a definition for data consistent with cleaned-up data (i.e., the set of recorded factual material commonly accepted in the scientific community as necessary to validate research findings in which obvious errors, such as keystroke or coding errors, have been removed and that is capable of being analyzed by either the original researcher or an independent party).
Computer Codes and Models
Comment: Commenter (6175) that they support the broader proposed requirements to describe and document any modeling assumptions and methods used; evaluate default assumptions; explain the scientific basis for each model assumption used; conduct sensitivity analyses of the modeled results to alternative assumptions; and give consideration to high-quality studies that explore other factors, including confounding variables. The commenter recommends that the EPA also evaluate the effect of variability and uncertainty on results, giving preference to studies that evaluate a range of models.
Commenter (6175) asserts that, while scientific modeling plays an important role in informing policy and regulatory decisions, models need to have a foundation in actual real-world data. The commenter notes that often there can be a significant gap in modeling assumptions and what is really occurring on the ground. The commenter states that disclosure of the underlying data, assumptions and methodology will allow affected stakeholders to provide input that will improve the credibility and reproducibility of modeled outcomes.
Response: As explained in the preamble to the final rule, the EPA is pursuing an incremental approach to maximizing transparency in the science that it relies upon to ensure that implementation is done in a thoughtful and deliberate way that focuses on the EPA's most impactful actions, minimizes unintended consequences, and informs future transparency requirements. Thus, the EPA is focusing on the underlying dose-response data for this rulemaking because of the influence these data have on particularly impactful decisions at the Agency.
While the scope of the final rule is focused on dose-response data underlying pivotal science, the final rule requires the EPA to describe critical assumptions and methods used in the Agency's dose-response assessments and to characterize the variability and uncertainty of such assessments.
Access to Data
Comment: Commenter (6194) states that the Proposal provides that, "where necessary, data would be made available subject to access and use restrictions." 83 Fed. Reg. at 18,773-74. However, the commenter states that what this process would look like is not addressed in the Proposal or discussed at all in the preamble. According to the commenter, this leaves significant open questions, such as who will pay for and implement these restrictions and who will decide which people can access the data and under what standards.
Response: Based on a consideration of these and other similar comments, the EPA proposed, through a 2020 SNPRM, a modified version of the 2018 proposed rulemaking regulatory text at 40 CFR 30.5. In the 2020 SNPRM, the EPA proposed Agency consideration of studies where there is tiered access to data and models that have CBI, proprietary data, or PII that cannot be sufficiently de-identified to protect the data subjects. For all other studies, data and models should be publicly available if the studies are to be used as pivotal regulatory science or pivotal science. In the 2020 SNPRM, the EPA also proposed an alternative. Under alternative proposed 40 CFR 30.5, when promulgating significant regulatory decisions or finalizing influential scientific information, the Agency would, other things equal, give greater consideration to studies where the underlying data and models are available in a manner sufficient for independent validation either because the information is publicly available or available through tiered access when the data include CBI, proprietary data, or PII and appropriate techniques have been used to reduce the risk of re-identification.
Based on a consideration of public comments received in response to the 2018 proposed rule and the 2020 SNPRM, the EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. The EPA would like to emphasize that this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedure and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (6362) provides that several agencies and organizations, in addition to NIH, have successfully addressed the issue of data access while maintaining confidentiality that should be considered by EPA. For example:
 The existing rule requiring federally funded research to be made available to other researchers. This standard could be adopted and applied to third-party funded researchers.
 Health care claims and related data are now being made available to researchers in de-identified form by some health insurance companies, such as Optum, which offers a "proprietary research database of health care and administrative data that links patient, physician, and treatment attributes from millions of geographically diverse individuals in the U.S." Optum appears to have developed methods and procedures to appropriately address confidentiality concerns.
 Medicare claims data are already available to researchers in de-identified form. Algorithms and methods developed by the Center for Medical Services should be examined by the EPA.
 Several professional societies have guidance on the protection of health data and de-identification, such as the Institute of Electrical and Electronic Engineers and the International Association of Privacy Professionals.
Response: Based on a consideration of these and other similar comments, the EPA proposed, through a 2020 SNPRM, a modified version of the 2018 proposed rulemaking regulatory text at 40 CFR 30.5. In the 2020 SNPRM, the EPA proposed Agency consideration of studies where there is tiered access to data and models that have CBI, proprietary data, or PII that cannot be sufficiently de-identified to protect the data subjects. For all other studies, data and models should be publicly available if the studies are to be used as pivotal regulatory science or pivotal science. In the 2020 SNPRM, the EPA also proposed an alternative. Under alternative proposed 40 CFR 30.5, when promulgating significant regulatory decisions or finalizing influential scientific information, the Agency would, other things equal, give greater consideration to studies where the underlying data and models are available in a manner sufficient for independent validation either because the information is publicly available or available through tiered access when the data include CBI, proprietary data, or PII and appropriate techniques have been used to reduce the risk of re-identification.
Based on a consideration of public comments received in response to the 2018 proposed rule and the 2020 SNPRM, the EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. The EPA would like to emphasize that this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedure and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (2171) supports open data, and that their state is a national leader in scientific and regulatory transparency. The commenter notes that their agencies are at the forefront of making environmental and health surveillance data publicly available, providing technical assistance for using that data, and engaging partners across communities and research institutions around effective data dissemination and utilization. The commenter adds that their agencies host multiple platforms for accessing high-quality health surveillance and environmental monitoring data, while protecting privacy and providing essential risk communication and prevention strategies. According to the commenter, detailed data are similarly available for research uses, under the approval and guidance of state IRBs.
Response: Based on a consideration of public comments received in response to the 2018 proposed rule and the 2020 SNPRM, the EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. The EPA would like to emphasize that this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedure and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (6726) states that one way to ensure transparency in the regulatory science aimed at solving a problem affecting our civilization is to fund two or more independent studies of the problem if questions arise about a single peer-reviewed study. According to the commenter, such questions do not often arise, but if they do it is an excellent opportunity for the authors of the original study to submit additions and corrections for peer review, as well as for others to independently study the problem. The commenter notes that everybody makes mistakes, but scientists and engineers are trained to recognize, avoid, and correct systematic errors in their experiments. The commenter provides that a peer-reviewed study usually contains a list of assumptions and approximations and an analysis of random and systematic errors to aid others in repeating the experiment and analyzing their own data.
Response: The EPA agrees with commenters that the opportunity to independently validate the pivotal science that the EPA relies upon is important in furthering scientific understanding and the Agency's mission. Through this rulemaking, the EPA is pursuing an incremental approach to maximizing transparency in the science that it relies upon to ensure that implementation is done in a thoughtful and deliberate way that focuses on the EPA's most impactful actions, minimizes unintended consequences, and informs future transparency requirements.
As described in the preamble of the final rule, the EPA is focusing on the underlying dose-response data for this rulemaking because of the influence these data have on particularly impactful decisions at the Agency. Where underlying dose-response data in pivotal science are available, subject matter experts could independently reanalyze the data to affirm original research conclusions, check for errors, test alternative assumptions, and better understand and evaluate the implications of the uncertainty. Such independent reanalyses will subsequently enable the EPA to make stronger, data-driven decisions in future rulemakings or in revisions to existing rules or influential scientific information.
Comment: Commenter (5419) states that the proposed rule mentions data research centers that facilitate secondary data usage by the public. The commenter suggests that, if the Centers for Medicare & Medicaid Services (CMS) data are considered to fall in this category, they would encourage EPA to work with CMS to streamline and simplify the data acquisition process and make the data available to the public free of charge. Otherwise, according to the commenter, the cost and security requirements of obtaining these data may place an undue burden on stakeholders seeking to replicate analyses, since the use of this data for a project they conducted were acquired at a significant cost (> $100,000) and a comprehensive data management plan that outlined rigorous security protocols was required. The commenter asserts that, clearly, these requirements would make replication of the work untenable for many researchers. The commenter expresses concerns that the challenges of making these data available in an equitable fashion may result in the data and insights from this study being removed from consideration in standard‐setting activities for PM2.5, ozone, and NO2.
Response: The EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. The EPA would like to emphasize that this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedure and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Commitment to Development of Permanent Long-Term Storage
Comment: Commenter (0045) asserts that a foundation for open and repeatable science is the availability of original data sets. The commenter provides that the availability of original data is often retained in government repositories, both in the USA and internationally. The commenter suggests that it should be a basic principle of the proposed rule that the U.S. government commit to the permanent long term storage of all data collected under government funding and the establishing of open access to such data, with the proper protections for data which may be subject to legal controls such as commercial confidentiality or patient HIPAA data. The commenter recommends that effective access be implemented to permit peer review, reproducible tests, or new work, based upon enabling persons to be authorized as broadly as possible, and best practices of anonymizing data. The commenter suggests that the USA work with governments around the world to obtain similar safeguarding of fundamental data which may be of universal benefits, with reciprocal access arrangements. The commenter adds that where EPA receives data, studies, analysis, and projections from sources beyond government funded science (e.g., industry, private citizens, or foreign entities), such data, studies, and projections should be subject to the same regime of durable storage, access for peer review, independent reproducibility, and safeguards, which are accorded to government funded sources and studies.
Response: The EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. The EPA would like to emphasize that this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedure and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA to collect, store, or publicly disseminate dose-response data underlying pivotal science.
<IP_ADDRESS>.4 "Independent Validation"
Independent Validation is Not Described/Defined
Comment: Commenters (6168, 6185, 6449, 6955) assert that "independent validation" is not described/defined. Commenter (6373) notes that the impacts of the proposed Rule are difficult to predict unless EPA better defines what data the Rule applied to, and terms such as "independent validation."
Commenter (6955) states that validation usually requires a comparison to new data; it is rarely used in scientific studies to mean a check to see that data leads to the stated results using the stated methods and/or software. The commenter notes that new studies can validate a model or numerical finding, but reproducing the analysis only validates the math. The commenter adds that neither can sensitivity analysis validate a study. The commenter provides that the meaning of the phrase, "sufficient for independent validation" is unclear and likely to raise great confusion, as different stakeholders define it differently. What would invalidate a study? We are not told. Furthermore, the commenter adds that no details are given as to what would constitute sufficiency. The commenter contends that, by not explicitly identifying its definition of validation, the agency has skipped over the statistical and regulatory implications of post-hoc analyses, losing the public comments that would otherwise have been forthcoming.
Commenter (6185) states that the approach for "independent validation" is not described. The commenter asks: Would EPA's staff scientists conduct these validations, or would EPA contract this out to third parties? How would EPA determine that third parties have the necessary qualifications and resources to perform these validations? According to the commenter, without these listed information elements, commenters are left to guess at the scope and potential impact of EPA's proposal as it applies to "independent validation." For example, the commenter provides that a recent study that is likely to be considered "pivotal regulatory science" for the current particulate matter (PM) NAAQS review is by Di, et al., "Air pollution and mortality in the Medicare population." The commenter notes that this study of chronic effects, along with a complementary study on acute effects, uses 460,310,521 person years of follow-up and has billions of data points. According to the commenter, the computational resources necessary to replicate or validate the analysis are available only at large institutions like the Harvard-MIT Data Center. The commenter asks: How does EPA's proposal on independent validation apply to a study like this?
Commenter (6168) states that, while the proposed rule emphasizes that data and models should be available in a manner sufficient for independent validation, the rule fails to clearly define independent validation. As a result, the commenter suggests that it is unclear if the rule applies only to replication of analysis or also to replication of entire studies. According to the commenter, by emphasizing availability of data and models, the rule seems to be narrowly focused on replicating analysis, particularly as related to dose-response relationships. Yet, the commenter notes that wording such as "using scientific information that can be independently validated" [FR 18770] can be interpreted expansively so that the rule can appear to require the ability to replicate entire studies, as was the intent of the failed HONEST Act. The commenter asserts that there are several problems with this. First, the commenter provides that if the intention is to require the ability to replicate entire studies, then this proposed rule is misleading; as written, it seems to be only about replicating analysis. As a result, the commenter states that this proposed rule is a stealth-- not at all transparent-- way to eliminate the use of a wide swath of studies that are currently used for regulatory decision-making. For example, the commenter suggests that studies based on tragic accidents involving toxic releases provide important public health knowledge, but cannot be replicated (for more on this, see EDGI's white paper). Second and more broadly, the commenter provides that the lack of clarity on this matter would allow the EPA to pick and choose, in an arbitrary manner, which studies to include or exclude in risk analysis. The commenter contends that this misuse of power would easily influence the findings of risk analysis.
Similarly, commenter (6449) states that the definition of "independent validation" is not clear. The commenter adds that it is not clear if the intent is to reproduce study conclusions using underlying data, replicate entire studies, or validate the use of data extracted from published studies in a dose response model. The commenter also notes that it is not clear whether this means all the information required to replicate the entire study should be included or whether this is limited to data once generated. The commenter requests that EPA present a clear definition.
Response: In response to these comments and other similar comments, the EPA issued a supplemental notice of proposed rulemaking (SNPRM) in 2020 that included definitions for key terms in the 2018 proposal, such as "independent validation," and clarified definitions included in the 2018 proposal.
Based on comments received on the 2018 proposed rule and the 2020 SNPRM, the EPA finalized the definition of "independent validation" as "the reanalysis of study dose-response data by subject matter experts who have not contributed to the development of the study to evaluate whether results similar to those reported in the study are produced."
Comment: Commenters (6114, 6116, 6131) state that the proposed rule states that "... EPA should ensure that the data underlying those (studies) are publicly available in a manner sufficient for independent validation," but the proposal does not say by whom. The commenters assert that scientists who are familiar with these types of studies are also familiar with the protocols under which these studies are carried out and peer reviewed, ensuring that an independent, critical assessment is performed. The commenters provide that scientists of this caliber and experience understand the assumptions and limitations that have been made during the study and are qualified to make judgments of whether the studies are valid or not. The commenters contend that, if the EPA seeks "independent validation"' by those who are not familiar with these protocols or with how these studies are normally conducted, that type of validation may be meaningless.
Response: In response to these comments and other similar comments, the EPA issued a supplemental notice of proposed rulemaking (SNPRM) in 2020 that included definitions for key terms in the 2018 proposal, such as "independent validation," and clarified definitions included in the 2018 proposal.
Based on comments received on the 2018 proposed rule and the 2020 SNPRM, the EPA finalized the definition of "independent validation" as "the reanalysis of study dose-response data by subject matter experts who have not contributed to the development of the study to evaluate whether results similar to those reported in the study are produced."
Independent Validation of Studies/Data Submitted to EPA by the Public
Comment: Commenter (6133) asserts that it is possible (even likely) that studies or data submitted by the public during comment periods would need to be independently assessed/validated before consideration by EPA. According to the commenter, against the backdrop of EPA rulemakings with public comment periods and open rulemaking time periods and the voluminous amounts of data that would need to be de-identified, shared, and re-analyzed, it would be impossible to achieve independent verification of relevant dose-response information.
Response: The EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. The EPA would like to emphasize that this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedure and does not direct or require any outside entity or the EPA to independently validate dose-response data underlying pivotal science.
Independent Validation by Inspection
Comment: Commenter (0044) states that, if the Agency is so concerned about transparency and validity of studies, they may well copy the Food and Drug Administration (FDA) approach to validating data. The commenter provides that, in human research and marketing applications, manufacturers must submit all clinical (i.e. human) data, even for studies that don't meet the statutory requirement of "adequate and well-controlled." The commenter notes that all studies and data are used in the evaluation and decision-making processes. The commenter adds that data are validated by inspection. The commenter further adds that the FDA has an inspectorate that inspects not only clinical data (especially so-called pivotal data), but also preclinical data and manufacturing data. According to the commenter, the results of these inspections are used to determine the quality and validity of the data and thereby the studies utilizing these data.
Response: The EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. The EPA would like to emphasize that this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedure and does not direct or require any outside entity or the EPA to independently validate dose-response data underlying pivotal science.
Independent Verification Where Laws and Regulations Prohibit the Public Release of Data
Comment: Commenter (6831) states that, throughout their professional career, they dealt with laws and regulations that prohibited the public release of data that could identify an individual or individual business. The commenter provides that they appreciate the need for independent verifications, but there should never be a restriction on the science used to support regulations, especially those involving human or environmental health. The commenter asserts that there are work arounds that can achieve independent verifications, as needed: release of aggregate data, for example, or individuals needing access can agree to and sign confidentiality agreements to access the data, still with the same restrictions on them for publishing individual data. The commenter states that the EPA must rely on relevant science and work with researchers for independent verification of data/results as needed. According to the commenter, the proposed rule is NOT needed and should be abandoned.
Response: Based on a consideration of these and other similar comments, the EPA proposed, through a 2020 SNPRM, a modified version of the 2018 proposed rulemaking regulatory text at 40 CFR 30.5. In the 2020 SNPRM, the EPA proposed Agency consideration of studies where there is tiered access to data and models that have CBI, proprietary data, or PII that cannot be sufficiently de-identified to protect the data subjects. For all other studies, data and models should be publicly available if the studies are to be used as pivotal regulatory science or pivotal science. In the 2020 SNPRM, the EPA also proposed an alternative. Under alternative proposed 40 CFR 30.5, when promulgating significant regulatory decisions or finalizing influential scientific information, the Agency would, other things equal, give greater consideration to studies where the underlying data and models are available in a manner sufficient for independent validation either because the information is publicly available or available through tiered access when the data include CBI, proprietary data, or PII and appropriate techniques have been used to reduce the risk of re-identification.
Based on a consideration of public comments received in response to the 2018 proposed rule and the 2020 SNPRM, the EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. As described in the preamble to the final rule, the EPA is not categorically excluding any studies from consideration when promulgating significant regulatory actions or developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information.
As described in the preamble to the final rule, the transparency provisions of the final rule apply to dose-response data underlying pivotal science. Under this rule, the EPA will give greater consideration to pivotal science for which the underlying dose-response data are either publicly available in a manner sufficient for independent validation or, in the case of PII, CBI, or proprietary data, available through restricted access that affords privacy in a manner sufficient for independent validation. In cases where dose-response data underlying pivotal science are not available in a manner sufficient for independent validation, the EPA may still use the pivotal science after either giving it lesser consideration or receiving an exemption from the requirements of this rule under 40 CFR 30.7.
The requirements included in the final rule provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical considerations in order to ensure the EPA maintains a strong scientific basis for its decision-making. The incremental progress made possible by the final rule provides an important step towards prioritizing transparency in particularly impactful EPA rules and assessments.
No Requirement that Data Be Independently Validated or Replicated Before EPA Can Consider It
Comment: Commenter (6133) states that there is nothing in the proposed regulatory text or preambular language that requires information, science or data to be independently validated or replicated before EPA may consider it. The commenter contends that the EPA does not base the Proposal upon any requirement or expectation that the information, science or data be shown to be accurate, trustworthy, reliable or correct before EPA may consider it. The commenter asserts that this portion of the Proposal reveals EPA's unlawful agenda to be one concerned with prohibiting EPA from considering relevant, peer-reviewed, quality science, not one concerned with actual replication or validation. The commenter adds that the Proposal's condition that science and information be "publicly available in a manner sufficient for independent validation and replication" is revealed to be mere smokescreen for an EPA enterprise to censor the best available science that would support adoption of more protective health and environmental safeguards.
Response: The EPA believes that the opportunity to independently validate the pivotal science that the EPA relies upon is important in furthering scientific understanding and the Agency's mission. Through this rulemaking, the EPA is pursuing an incremental approach to maximizing transparency in the science that it relies upon to ensure that implementation is done in a thoughtful and deliberate way that focuses on the EPA's most impactful actions, minimizes unintended consequences, and informs future transparency requirements.
As described in the preamble to the final rule, the EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. The EPA is not categorically excluding any studies from consideration when promulgating significant regulatory actions or developing influential scientific information. Rather, the Agency will continue to evaluate the quality of all relevant studies, consistent with the intended use of the information.
The requirements included in the final rule provide a workable framework for evaluating pivotal science in the context of the availability of its underlying dose-response data, while balancing important technical considerations in order to ensure the EPA maintains a strong scientific basis for its decision-making. The incremental progress made possible by the final rule provides an important step towards prioritizing transparency in particularly impactful EPA rules and assessments.
<IP_ADDRESS>.5 Assertion that EPA's Proposal is Consistent with the Conclusions of the National Academies (NAS)
Comment: Commenter (6133) cites the proposal preamble as stating that "[t]he benefits of EPA ensuring that dose response data and models underlying pivotal regulatory science are publicly available in a manner sufficient for independent validation are that it will improve the data and scientific quality of the Agency's actions and facilitate expanded data sharing and exploration of key data sets; this is consistent with the conclusions of the National Academies." 83 Fed. Reg. at 18,772 (footnote omitted). The commenter provides that the last statement links to a 120 page document titled "Expanding Access to Research Data Reconciling Risks and Opportunities," by the Panel on Data Access for Research Purposes, Committee on National Statistics, Division of Behavioral and Social Sciences and Education of the National Research Council of the National Academies Press. (https://www.nap.edu/catalog/11434/expanding-access-toresearch-data-reconciling-risks-and-opportunities). The commenter provides that the Proposal does not suggest that its plan to preclude the use of scientific studies from regulatory decision making is consistent or supported by the National Academies. Rather, the commenter notes that the Proposal generally states that benefits of data availability the Proposal seeks is consistent with conclusions of the National Academies. The commenter further states that the Proposal does not say what the conclusions of the National Academies are or how they support the Proposal. According to the commenter, the charge to the Panel in the cited document was "to assess competing approaches to promoting exploitation of the research potential of microdata -- particularly linked longitudinal microdata -- while preserving respondent confidentiality." According to the commenter, the Panel was asked to consider the tradeoffs between the benefits and risks of data access and to make recommendations about "how microdata should optimally (from a societal standpoint) be made available to researchers." The commenter explains that the Panel offered various recommendations, focused on agencies that have data-collection responsibilities providing data to researchers. The commenter adds that this is a different context than EPA's proposal to preclude the consideration scientific studies when undertaking its statutorily required decision making to protect human health and the environment. The commenter asserts that the EPA's general citation to this 120-page document for consistent conclusions does not support the Proposal.
Response: The EPA agrees that this was an incorrect citation and has removed the reference to the document titled "Expanding Access to Research Data Reconciling Risks and Opportunities."
<IP_ADDRESS>.6 Other
EPA Should Work with Entities Where Scientific Data are not Publicly Available in a Manner Sufficient for Independent Evaluation
Comment: Commenter (6362) recommends that where data are not available in a manner sufficient for independent evaluation, EPA attempt to work with data owners to reach an agreement to make the information available to the public to the greatest extent practicable without jeopardizing the privacy, confidentiality, or the proprietary interests that deserve protection. In circumstances where there is significant difficulty making data available in a meaningful way, the commenter suggests that the EPA consider contracting with external experts in the scientific discipline at issue, have them sign confidentiality agreements, analyze the data, and prepare a confidential report with a non-confidential summary for EPA to share publicly.
Response: The EPA is finalizing internal procedural requirements for how the Agency will consider the availability of underlying dose-response data of pivotal science when promulgating a significant regulatory action or developing influential scientific information that relies on dose-response data. The EPA would like to emphasize that this final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedure and does not direct or require any outside entity or the EPA to independently validate dose-response data underlying pivotal science.
Proposal Not as Bad as Either Side Perceives
Comment: Commenter (0844) provides that the George Washington University Director of Regulatory Studies Susan Dudley wrote in a May 21, 2017 Forbes.com article, "Critics and Supporters of EPA's Transparent Science Proposal: File Comments On It, But Read It First," that the EPA transparency proposal "is not as dramatic as either side [of the political divide] would lead one to believe, however. Essentially, it requires that the studies, models, and data underlying significant regulatory decisions are `publicly available in a manner sufficient for independent validation.' Independent validation is the essence of the scientific method, which depends on falsifiable hypotheses, data gathering, replication, dissent, and challenge."
Response: The EPA agrees that the opportunity to independently validate the pivotal science that the EPA relies upon is important in furthering scientific understanding and the Agency's mission. The EPA also believes that codifying internal procedures aimed at prioritizing transparency in significant regulatory actions and influential scientific information into regulation will improve the opportunity for the public to access the EPA's scientific analyses and resulting regulatory actions in a way that is beneficial to the scientific process, the Agency's mission, and the public's health and safety.
Through this rulemaking, the EPA is pursuing an incremental approach to maximizing transparency in the pivotal science that it relies upon to ensure that implementation is done in a thoughtful and deliberate way that focuses on the EPA's most impactful actions, minimizes unintended consequences, and informs future transparency requirements. As described in the preamble of the final rule, the EPA is focusing on the underlying dose-response data for this rulemaking because of the influence these data have on particularly impactful decisions at the Agency.
Comment: Commenter (11393) expresses concern regarding the rule's potential impacts on health risk assessments. The commenter specifically expresses opposition to language that removes or dilutes the linear no-threshold (LNT) dose response in the evaluation or establishment of radiation health risk assessments, rule-making or establishment of radioactive recommended action levels or benchmarks (influential scientific information) made available to the public.
Response: After considering comments on both the 2018 proposed rule and the 2020 SNPRM, the EPA has determined that this rule should apply to dose-response data rather than dose-response data and models. Therefore, the rule does not include discussion of any particular dose-response model. Given the specificity of 40 CFR 30.6 to dose-response data and models, and in particular dose-response models, the EPA is not finalizing 40 CFR 30.6. The EPA is adapting one provision of 40 CFR 30.6 as a factor in 40 CFR 30.5 in determining the consideration to afford pivotal science for which the dose-response data are not available for independent validation. Specifically, the EPA is finalizing as a factor in 40 CFR 30.5 the consideration that the EPA would give to high quality studies that explore a broad class of parametric dose-response models, non-parametric models that incorporate fewer assumptions, various threshold models, and models that investigate factors that might account for spatial heterogeneity.
Comment: Commenter (11492) indicates that the EPA continues to assign greater value to dose-response studies, which by definition reduce the use/value of epidemiological studies. Section 30.6 of the supplemental proposal has been revised but continues to give "explicit consideration to ... parametric dose-response or concentration-response models." As the commenter suggests they stated in its 2018 comments, the EPA itself admits that, "there is frequently a lack of dose-response data available for human subjects. When data are available, they often cover only a portion of the possible range of the dose-response relationship." The commenter states that this gives more weight to toxicological studies as opposed to epidemiological studies, which form the basis of many human health exposure studies. The commenter asserts that the EPA must continue to use epidemiological studies; to do otherwise is to compromise the health of all Americans. Commenter (12403) concurs, indicating that the SNPRM's identification of specific dose-response modeling techniques that would be given "explicit consideration" is inappropriate and ignores the improvements and refinements that are the hallmark of good science.
Response: Although the EPA is prioritizing transparency in pivotal science, the Agency also recognizes that there will be instances where the underlying dose-response data of pivotal science is unavailable for independent validation. In order to ensure that the Agency maintains a strong scientific basis for its decision-making, a study with underlying dose-response data that are not available should be considered as long as other significant technical considerations can provide some level of certainty or confirmation of a study's conclusions, importance, and applicability, even in the absence of maximum transparency. Title 40 CFR 30.5(d) provides additional factors that the Agency shall use when determining the consideration to afford pivotal science with data that is unavailable for independent validation. The EPA could also give pivotal science with underlying dose-response data that are unavailable for independent validation full consideration upon receiving an exemption from the requirements of this rule from the Administrator, as detailed under 40 CFR 30.7.
The EPA disagrees that the rule implies a bias against the use of epidemiology studies or data. The final rule includes sufficient flexibility in terms of the allowance for restricted access, considerations in 40 CFR 30.5, the Administrator's exemption in 40 CFR 30.7, and the EPA's incremental approach to maximizing transparency by first focusing on dose-response data to minimize such potential unintended consequences. The EPA will continue to consider and evaluate all relevant and appropriate science, including epidemiology studies, in its significant regulatory actions and influential scientific information.
In addition, after considering comments on both the 2018 proposed rule and the 2020 SNPRM, the EPA has determined that this rule should apply to dose-response data rather than dose-response data and models. Therefore, the rule does not include discussion of any particular dose-response model.
Comment: In support of existing model availability, commenter (11894) provides the specific example of the trichloroethylene (TCE) case study. The commenter notes that the TCE assessment relied on approximately 15 models. The commenter notes that this is likely to be the norm with the same ones being used repeatedly. The commenter adds that most of the models EPA employs in the Toxic Substances Control Act (TSCA) program have been developed by the Agency and are publicly-available. The commenter provides that access can be gained through a number of EPA webpages and/or searches on the internet, and others are described in the peer-reviewed literature, access to which is controlled by the authors.
Response: After considering comments on both the 2018 proposed rule and the 2020 SNPRM, the EPA has determined that this rule should apply to dose-response data rather than dose-response data and models. Therefore, the rule does not include discussion of any particular dose-response model.
Comment: Commenter (12465) states that the EPA relies upon many types of data and models to support its regulatory decisions and it is unclear that the current proposal has adequately considered the numerous types. The commenter expresses concern that the treatment of existing data or models under the proposals disincentivize the generation of new data under the new requirements while reducing the importance of consideration of existing data, regardless of the scientific validity or merit of that data.
The commenter (12465) provides the example of the Westat Survey. The commenter notes that the report entitled "Household Solvent Products: A National Usage Survey" (or the Westat Survey) is a survey of consumer usage for a wide variety of products to determine various attributes and inform exposure assessments. According to the commenter, while the report was completed in 1987, it remains in use today informing the EPA's Consumer Exposure Model and the Westat Survey informed several of the first 10 chemicals currently being evaluated under reformed TSCA. The commenter states that it is unclear in the proposal how the Westat Survey would be treated, let alone the multitude of exposure models that rely upon it.
Response: As further described in the preamble to the final rule and the overarching response to this chapter, the EPA is not finalizing the definition of "model." The rule only applies to dose-response data and will not impact the Agency's use or reliance on models, including exposure models. Since the Westat Survey is a consumer behavior survey for assessing exposure, it would also not be applicable to this rule.
The EPA disagrees that the rule disincentivize the generation of data. The final rule includes sufficient flexibility in terms of the allowance for restricted access, considerations in 40 CFR 30.5, the Administrator's exemption in 40 CFR 30.7, and the EPA's incremental approach to maximizing transparency by first focusing on dose-response data to minimize such potential unintended consequences. The EPA will continue to consider and evaluate all relevant and appropriate science in its significant regulatory actions and influential scientific information.
<IP_ADDRESS> Applicability of Requirements to Other Types of Data and Information
<IP_ADDRESS>.1 Arbitrary Rationale for Limiting Applicability to Dose Response Data and Models
Comment: Commenter (9227) questions why EPA has singled out dose response studies to be excluded if their underlying data and models are not publicly available and has not similarly targeted other types of studies commonly used by EPA. According to the commenter, EPA has provided no justification for targeting dose response studies and not other types of studies or scientific information. The commenter asserts that, absent any explanation from the Agency, it is impossible to comment on the factual predicates for EPA's proposed decision, or the reasonableness of EPA's justification, except to state that it appears to be completely arbitrary in the absence of any rationale. See, e.g., Transactive Corp. v. United States, 91 F.3d 232, 237 (D.C. Cir. 1996) ("A long line of precedent has established that an agency action is arbitrary when the agency offered insufficient reasons for treating similar situations differently.").
Response: In this final rule, the EPA will be considering the availability of dose-response data underlying pivotal science, not dose-response data and models. The EPA is focusing on the underlying dose-response data for this rulemaking because of the influence these data have on particularly impactful decisions at the Agency. Risk assessments and regulations that target emissions and risk reduction of one or more pollutants, contaminants, or substances are integral to the Agency's mission and the underlying dose-response data that inform the quantitative value used to evaluate and mitigate potential risk are critical to understanding the assessment or regulatory action. In addition, the data underlying the dose-response assessment are more distinct than the broad range of data informing an entire risk assessment. Therefore, the EPA is concentrating its current efforts to increase transparency on a well-defined step in the quantitative assessment of risk supporting specific Agency actions. This final rule provides an important step in furthering the progress already being made toward maximizing transparency and will provide important insight for developing future statute-specific requirements.
<IP_ADDRESS>.2 Support Applicability to Other Data and Models
Clarification on Whether or How Proposed Rule Would Apply to Cost-Benefit Determinations
Comment: Commenter (6915) states that the EPA seeks comment as to "whether the disclosure requirements . . . should be expanded to cover other types of data and information, such as, for example, economic and environmental impact data and models that are designed to predict the costs, benefits, market impacts and/or environmental impacts of specific regulatory interventions." 83 Fed. Reg. at 18,772 (emphasis added). However, the commenter adds that the EPA also states that the "pivotal regulatory science" to which the Proposed Rule would already apply includes "studies, models, and analyses that drive the magnitude of the benefit-cost calculation." Id. at 18,770. According to the commenter, it is thus unclear whether and how EPA intends the Proposed Rule to apply to the cost-benefit determinations that it performs.
Response: The EPA acknowledges the lack of clarity in the proposed rule and emphasizes that this final rule only applies to does-response data. Rather than having this final rule apply to all the studies that support the assessment of the relationship of a dose or exposure of a pollutant, contaminant, or substance to the magnitude of a predicted health or environmental impact, the EPA is focusing on those studies that describe the quantitative relationship between the dose or exposure of a pollutant, contaminant, or substance and an effect. Specifically, the scope of dose-response data in this final rule is those studies consisting of the data integral to characterizing dose-response relationships. These dose-response studies that constitute pivotal science may be used to identify candidate PODs that are then employed in the benefit-cost analysis, but not all studies informing the benefit-cost calculation would be considered pivotal science.
General Support to Applying the Rule to Other Data and Models
Comment: Commenters (0844, 4541, 5185, 6141, 6150, 6161, 6182, 6360, 6375, 6867, 6913, 6915, 6917, 6921, 8771) suggest that the Agency uses other data and models (in addition to dose response data and models) to support regulatory decisions that would benefit from an improved regulatory framework that enhances transparency with respect to underlying pivotal data and assumptions.
Commenters (0844, 4541, 6150, 6182, 6375, 6921) state that the Proposal disclosure requirements applicable to dose response data and models should be expanded to cover other types of data, such as economic and environmental impact data and models that are designed to predict the costs, benefits, market impacts and/or environmental effects of specific regulatory interventions on complex economic or environmental systems. Commenter (0844) believes it is imperative that this type of information should be applicable to EPA's proposed transparency requirements, considering the enormous economic effect EPA regulations can have on the U.S. economy.
Response: The EPA has considered applying this rule to all data and models and proposed to do so in the 2020 supplemental notice of proposed rulemaking. In this final rule, however, rather than considering all studies that support the assessment of the relationship of a dose or exposure of a pollutant, contaminant, or substance to the magnitude of a predicted health or environmental impact as "pivotal science," the EPA is balancing transparency and feasibility by focusing on those studies that inform the quantitative relationship between the dose or exposure of a pollutant, contaminant, or substance and an effect. Thus, "pivotal science" includes only those studies that are integral to characterizing dose-response relationships (e.g., identifying candidate PODs). These are the studies that drive the requirements or quantitative analyses of EPA significant regulatory actions or influential scientific information. In the context of this rule, studies informing the dose-response modeling (e.g., benchmark response selection), studies identifying data for toxicokinetic adjustments, or studies informing the selection of uncertainty factors do not drive the requirements or quantitative analyses of EPA significant regulatory actions or influential scientific information. Future statute-specific rulemakings may interpret "pivotal science" more broadly.
Support Application to Economic and Environmental Data
Comment: Commenter (6921) notes that economic and environmental data are traditionally more available and generally present less of a transparency problem. However, the commenter states that, to the extent that data and models used to estimate economic or environmental impacts are not generally available, they support including them in this rulemaking if it can be done without delaying the promulgation and implementation of the rule.
Commenter (6141) notes that when the Agency employs models, such as those used in the Social Cost of Carbon (SCC) calculation (especially with respect to climate damages), to support its regulatory decisions, the EPA should at a minimum provide documentation that facilitates peer review of its analysis as well as the assumptions used to support that analysis. According to the commenter, data that models rely upon to project damages may suffer from the same peer review problems as with dose response data, leading to false or dubious conclusions that are then carried forward into regulatory requirements. The commenter adds that, in other cases, the underlying data may be out of date and unreliable for generalizations and decision-making purposes. The commenter requests that the EPA ensure that modeled projections are appropriately characterized. As an example, the commenter notes that mathematical modelling often gives the impression of precision and tempered complexity, in spite of significant and potentially problematic uncertainties involved in the calculations. According to the commenter, not disclosing the uncertainties, or performing robust sensitivity analyses on them, may obfuscate shortcomings of the modelled results. The commenter suggests that an improved regulatory framework that enhances transparency and deliberation with respect to underlying pivotal data and assumptions will improve the regulatory process by increasing confidence that the rules accurately reflect and appropriately address risks to human health and the environment.
Commenter (5185) states that EPA's benefit monetization is highly subjective and controversial and that they hoped that EPA concurrently examines transparency associated with the development of regulatory cost-benefit analyses. The commenter provides that data compiled by the U.S. Chamber found that, between 2000 and 2016, EPA issued 62 rules claiming a total of $923 billion in regulatory benefits. Incredibly, according to the commenter, $898 billion of these benefits -- or 97.2 percent  -- were monetized based on non-public data associated with fine particles (PM2.5). The commenter states that these benefits comprise nearly 80 percent of all regulatory benefits across the entire federal government. The commenter notes that, even though the vast majority of these rules were not intended to address PM2.5, and even though the vast majority of their corresponding claimed benefits came from areas of the country that are already deemed safe and in compliance with EPA's PM2.5 standard, the Agency repeatedly touted these figures to build public support for its regulations. The commenter contends that federal rules that impact millions of people and billions of dollars should be held to a higher standard.
Commenter (8771) asserts that the economists' analysis of the costs of regulation and the values of longevity and other benefits underlying EPA regulations are flawed, opaque, and in need of reanalysis. The commenter believes that every criticism leveled in this proposal against risk science (epidemiology, toxicology, exposure assessment) ought to be first applied to regulatory economics. The commenter notes that the EPA defines "pivotal regulatory science" as results that determine the final regulatory standard, and that costs are as "pivotal" or more "pivotal" as estimates of risk since EPA uses informal "bright lines" to reject regulatory controls if they are "too expensive" or "economically infeasible". The commenter states that regulatory cost estimates are notoriously biased high for reasons of self-interest but also due to inevitable factors such as economies of scale and technological learning. The commenter adds that cost estimates are often surrounded by more uncertainty than that surrounding risk estimates but are rarely if ever presented with error bars. According to the commenter, nothing could be more uncertain than a prediction whose very sign is incorrect (e.g., a prediction of job losses when in fact jobs will be created, or vice versa; a prediction of dire costs when in truth there will be cost savings) -- and yet, the commenter complains that not a word in the Proposal requires any additional transparency, disclosure, reanalysis, or sensitivity analysis be done by economists or by the industry or Agency consultants who prepare such estimates.
Commenter (8771) offers two economic study examples that they believe illustrate the need of repair:
      First, in a new paper, Branden Johnson and I [the commenter] analyzed the data reported from more than 1,000 estimates of the VSL (value of a statistical life) derived from hundreds of stated-preference studies worldwide. Only 42 percent of the estimates were presented with any information about the standard deviations or ranges of the individual VSL values given by subjects questioned in the study --  so in the majority of cases, EPA has had to derive a value for the VSL (certainly the single most "pivotal" quantity in all of risk regulation!) with no information other than the average VSL among dozens or hundreds of persons surveyed. No one can reanalyze most of these studies to see what higher or lower values of the VSL are also completely compatible with the data, because the economists fail to provide even a summary of the variability in the individual data.
      Perhaps the most well-known "study" of the aggregate costs of regulation is the (series of) reports from Mark and Nicole Crain suggesting that regulations "cost" the U.S. nearly $2 trillion per year. This work is of no real value, since it relies on comparing the GDP [gross domestic product] growth rates of different countries to a subjective index of "regulatory quality," which of course has no relationship to regulatory burden. And I hope needless to say, even a robust correlation between GDP and actual burden is not useful for estimating "cost." But a recent article details how a researcher asked the Crains for their data and data sources so he could see if their results, such as they were, would hold up after controlling for important differences among nations. The requester received no information: "Crain and Crain have refused to publish or share the data supporting their 2014 study, despite numerous requests from this author. In so doing they follow the course they adopted with their 2010 study, in which they refused to divulge their data even to the Government Accountability Office".
The commenter (8771) concludes that EPA must require all significant economic analyses to comply with whatever strictures remain in the final rule (assuming it is not withdrawn); otherwise, the commenter believes that the final rule cannot withstand judicial review, because the essence of arbitrariness is putting hurdles in front of only half of each analysis and ignoring the other half.
Commenters (6161, 6360, 6867) support EPA expanding the Proposed Rule to encompass all science. Commenter (6360) specifically points out that the EPA commonly employs fate-and-transport models, route of exposure models, cost estimation models, statistical techniques to project analytic data beyond method detection limits, and numerous other specific applications of science that involve judgement by EPA and by the application developers. According to the commenter, they see no logical principle to require transparency for one application and not another. As an example, the commenter notes that a fate and transport model could have as much economic impact as a dose-response model. The commenter notes that the importance of this principle is highlighted by EPA's current proceedings within the U.S. Court of Appeals for the D.C. Circuit where the judges are unable to follow EPA's statistical methodology for calculating the best performing sources for a MACT standard. The commenter provides that this case is instructive since it shows (1) the lack of transparency in other scientific applications besides dose-response models; (2) the cost in social resources when EPA fails to be transparent; and, (3) the risks that the lack of transparency may delay measures to protect human health and the environment. Commenter (6867) asserts that the replication crisis and public confidence rationales that underlie the Proposed Rule suggest that data, methodology, and computational code should be disclosed for all science relied upon to support Agency actions. The commenter recommends, as a first step, the phrase "dose response data and models" should be replaced with "data and models, including dose response data and models" throughout the Proposed Rule.
Commenter (6913) provides that the Proposed Rule places emphasis on dose-response data and models that provide the foundation for "pivotal regulatory science". The commenter expresses that this implies that the transparency rule may only apply to human health effects under the environmental statutes. According to the commenter, EPA's criteria, whether developed for protection of human health, aquatic health, agriculture, livestock health, or some other reason, has the same enforceable impact on industry, states, and the public. According to the commenter, there is no reason that a standard of scientific transparency should not apply to all EPA actions, including criteria developed for any environmental media (water, air, soil, etc.) and protection of all endpoints (human health, aquatic life, etc.). The commenter asserts that the EPA must implement the same transparency standard towards all their criteria derivation actions, regardless of whether they are protective of human health or specific aspects of the environment.
Commenter (6366) asserts that recent EPA action on the retrospective review of the Lead; Renovation, Repair and Painting (RRP) Program under Section 610 of the Regulatory Flexibility Act (EPA-HQ-OPPT-2016-0126) reinforces the need for the proposal. The commenter expresses that they have repeatedly called on EPA to perform an updated economic analysis of the program. According to the commenter, in finalizing the 610 review, EPA claimed a new economic analysis had been completed, yet the data provided cannot be reproduced and is insufficient to fully evaluate the decisions EPA made. The commenter notes that this is extremely disappointing given that the "new" information had a direct bearing on the Agency's findings which concluded that despite the lack of an accurate lead paint test kit, the rule's requirements should remain unchanged. The commenter states that, since the data used to conduct EPA's analysis has not been transparently presented in the associated docket, stakeholders have been unable to fully evaluate the report's underlying economic claims or their resulting impact on the review's conclusion. The commenter contends that since data is not transparently presented, it creates continuing challenges for stakeholders as they seek to work with EPA to ensure the program is effective and robust and that all impacts on small entities have accurately been taken into consideration.
Response: The EPA has considered applying this rule to all data and models and proposed to do so in the 2020 supplemental notice of proposed rulemaking. In this final rule, however, rather than considering all studies that support the assessment of the relationship of a dose or exposure of a pollutant, contaminant, or substance to the magnitude of a predicted health or environmental impact as "pivotal science," the EPA is balancing transparency and feasibility by focusing on those studies that inform the quantitative relationship between the dose or exposure of a pollutant, contaminant, or substance and an effect. Thus, "pivotal science" includes only those studies that are integral to characterizing dose-response relationships (e.g., identifying candidate PODs). These are the studies that drive the requirements or quantitative analyses of EPA significant regulatory actions or influential scientific information. In the context of this rule, studies informing the dose-response modeling (e.g., benchmark response selection), studies identifying data for toxicokinetic adjustments, or studies informing the selection of uncertainty factors do not drive the requirements or quantitative analyses of EPA significant regulatory actions or influential scientific information. Future statute-specific rulemakings may interpret "pivotal science" more broadly.
The EPA is addressing the manner in which it develops cost-benefit analyses for its regulations in other rulemakings, starting with rulemakings under the CAA, and as such the issue of what economic data the EPA considers in its cost-benefit analyses is outside the scope of this rulemaking. Pivotal science relying on dose-response data may be used to identify candidate PODs, which are then employed in the benefit-cost analysis for a significant regulatory action, but not all studies informing the benefit-cost calculation would be considered pivotal science.
Support Application to Pesticide Registrant Data
Comment: Commenters (0196 (mass campaign representing thousands of commenters), 0320, 6046) support EPA increasing availability of pesticide registrant data.
Commenter (6046) notes that pesticide registrants submit toxicology study data as well as the identity of inert ingredients to EPA in order to register a pesticide product. The commenter provides that these data are considered CBI and are thus not publicly available. According to the commenter, the public must rely on EPA scientists to appropriately evaluate the study data and the toxicology of the inert ingredients, just as the public does when EPA uses unpublished study data for other regulatory decisions. The commenter states that non-publicly available pesticide registrant toxicology data are routinely evaluated by EPA scientists. The commenter states that there are often many unpublished pesticide registrant studies for each active ingredient pesticide. The commenter asserts that not requiring that pesticide registrant study data used by EPA be publicly available, when there is a need for confidentiality for both pesticide and non-pesticide data at the same time as requiring dose response data and models be publicly available illustrates a bias and if EPA is interested in transparency to the public, then methods to make pesticide registrant data available to the public should be evaluated and put into practice. The commenter notes that pesticide registrant data meets the definition of pivotal regulatory scientific data. The commenter recommends that the EPA take the lead to identify means to make inert ingredient hazard information available to the public. The commenter adds that the current list of approved inert ingredients is a start but does not provide consumers with sufficient information to determine if they may, for example, have an asthmatic allergic reaction to un-disclosed inert ingredients.
Commenters (0196, mass campaign representing thousands of commenters) note that, under the proposed "transparency" plan, the public will still lack access to key data about the effects and efficacy of commercial poisons approved for sale and application in their communities and homes. The commenters state that the proposed policy declares that it will "help ensure that EPA is pursuing its mission of public health and the environment in a manner that the public can trust and understand," yet it applies only to a very limited set of studies used to support certain EPA regulations. According to the commenters, the pesticide registration and review processes are particularly lacking in transparency, opportunity for public review, and access to data. The commenters add that, because pesticides are toxic chemicals broadcast into the environment, nowhere is transparency more important than pesticide registration.
Commenters (0196, mass campaign representing thousands of commenters) state that, because the pesticide manufacturer produces the underlying data for these EPA approvals and controls access to them, no transparency will be provided in pesticide registration:
 The public does not have access to the underlying data provided by the manufacturer to justify registering a new pesticide for commercial distribution;
 Industry will not have to provide the data used to assess health and environmental effects and farmworker impacts to set allowable dietary and non-dietary exposure limits; and
 Neither the public nor EPA has access to the underlying efficacy data, except for very limited purposes.
The commenters assert that, to ensure full review, it is critical that the public and independent scientific community have complete access to safety testing data before pesticide products enter the marketplace. According to the commenters, if EPA applies the proposed policy to pesticides, the failure to allow consideration of independent study results without disclosing confidential patient information will cripple the validity of EPA safety reviews.
Commenters (0196, mass campaign representing thousands of commenters) contend that EPA pesticide regulation is based on secret science and is particularly lacking in transparency. The commenters provide that nonpublic data submitted by the registrant are used by EPA to assess health and environmental effects of the pesticide and impacts on farmworkers, and to set allowable human exposures through dietary and nondietary routes  - all without any opportunity for public review. The commenters assert that full disclosure of known and unknown adverse effects is needed in the registration process and on the label. The commenters assert that conditional registration is missing crucial data and allows market entry for a product in the absence of certain data normally required for registration and that efficacy data on pesticide products are lacking. According to the commenters, (1) the public do not have access to, and EPA does not review, manufacturer data on pesticide efficacy, even though the statutory registration standard requires weighing the risks of pesticides against their benefits; (2) secret ("inert") ingredients" in pesticide products are not disclosed, although they have been classified as hazardous under a variety of environmental laws; and (3) only active ingredients, not formulations, are tested, so data on the human health and environmental effects of the actual product on the market are entirely lacking.
Commenter (0320) provides that, to ensure full review, it is critical that the public and independent scientific community have complete access to safety testing data before pesticide products enter the marketplace. The commenter contends that, if EPA applies the proposed policy to pesticides, the failure to allow consideration of independent study results without disclosing confidential patient information will cripple the validity of EPA safety reviews.
Response: Other commenters have noted that the 2018 proposed rule appeared to conflict with FIFRA pesticide registration requirements and associated implementing regulations, which require registrants to submit data and information to the EPA to enable the Agency to make its unreasonable adverse effects determinations. These commenters argued that, under the 2018 proposed rule, the EPA would not be able to consider these data, which are often claimed as CBI, when evaluating the pesticide registrations because the data could not legally be made publicly available. In this final rule, the EPA has codified in 40 CFR 30.3 that in the event the procedures outlined in this part conflict with statutes the EPA administers, or their implementing regulations, the statutes and regulations will control. This would include FIFRA and the Agency's pesticide registration activities. This final rule does not require any dose-response data to be made publicly available in a manner that would violate law, and rather protects privacy, confidentiality, CBI, and is sensitive to national security.
Support Application to EPA Integrated Planning Model (IPM)
Comment: Commenter (6917) provides that the EPA Integrated Planning Model (IPM) is frequently used in rulemaking to estimate cost impacts of regulations on the electric utility industry. The commenter notes that the IPM data output might be considered to be in the class of "pivotal regulatory science" since the model output data is "... critical to the calculation of a final regulatory standard or level, or to the quantified costs, benefits, risks and other impacts on which a final regulation is based." The commenter recommends that the Agency's use of IPM be made more transparent. The commenter notes that, although inputs into the model may be listed in one or more documents located somewhere on the rulemaking webpage or in the docket, finding all the input data is like looking for a needle in a haystack. In addition, according to the commenter, even if the input data can be found, it's not always clear how the model processes the data to produce the output. As a result, the commenter contends that, if the model were to produce erroneous output results, it may not be detected or the reason for a detected error may not be determined.
As an example, commenter (6917) states that, in year 2014 in the proposal for the Emission Guidelines for Existing Stationary Sources: Electric Utility Generating Units (Clean Power Plan or CPP), the supporting IPM output data erroneously indicated the retirement of Cleco's Brame Energy Center Unit 2 by year 2016. However, the commenter notes that, as written in the 2014 Cleco comments on the CPP proposal, Cleco's own predictive modeling at the time indicated there would be normal 2016 operation. In addition, the commenter states that, at the time of the commenting on the proposed CPP, the reason for the erroneous IPM output regarding Unit 2 could not be determined. The commenter states that, as a result, confidence in the IPM model results for the CPP proposal was reduced. The commenter recommends that, to increase transparency in the application of the IPM, clear information should be shared with the regulated community regarding the data-input to the IPM model and how the model's output is arrived at.
Response: Rather than pursuing the expansive scope of pivotal science outlined in the 2018 proposed rule, which may include IPM data, the EPA is pursuing an incremental approach to maximizing transparency in the science that it relies upon to ensure that implementation is done in a thoughtful and deliberate way that focuses on the EPA's most impactful actions, minimizes unintended consequences, and informs future transparency requirements. The EPA is focusing on the underlying dose-response data for this rulemaking because of the influence these data have on particularly impactful decisions at the Agency. Risk assessments and regulations that target emissions and risk reduction of one or more pollutants, contaminants, or substances are integral to the Agency's mission and the underlying dose-response data that inform the quantitative value used to evaluate and mitigate potential risk are critical to understanding the assessment or regulatory action. In addition, the data underlying the dose-response assessment are more distinct than the broad range of data informing an entire risk assessment. Therefore, the EPA is concentrating its current efforts to increase transparency on a well-defined step in the quantitative assessment of risk supporting specific Agency actions. This final rule provides an important step in furthering the progress already being made toward maximizing transparency and will provide important insight for developing future statute-specific requirements.
Support Application to Other Data and Models in a Subsequent Rulemaking
Comment: Commenter (6151) suggests, in response to EPA's request for input as to whether the transparency rule should apply to other data, such as economic analyses or environmental impact models, that EPA initially focus on data related to pivotal scientific regulatory decisions as outlined in the notice.
Commenter (6375) notes that, although they support expanding the Proposed Rule to cover other types of data, they understand that this is a focused rulemaking effort and that the current rulemaking may not be the appropriate mechanism to address other types of data and information. The commenter encourages that, if EPA decides not to address other types of data and information in this rulemaking, EPA address in a subsequent rulemaking.
Response: The EPA agrees with the commenters, and notes that this final rule provides an important step in furthering the progress already being made toward maximizing transparency and will provide important insight for developing future requirements in statute-specific rulemakings.
Support Application to Economic and Environmental Data and Models but Protect CBI and PII
Comment: Commenter (1011) suggests that EPA should provide access to economic and environmental impact data and models that are designed to predict the costs, benefits, market impacts and/or environmental effects of specific regulatory interventions on complex economic or environmental systems, so long as CBI, private personal information, and other potentially sensitive data sources are not made public in a manner that could violate confidentiality (i.e., EPA should aim to make aggregated data available). The commenter adds, however, that EPA must be cognizant of the increased burden any such requirement would place on researchers, and the agency should avoid making extensive or repetitive requests for information and explanation.