3. Exposure Assessment Sub-Committee. The role of the exposure assessment Sub- committee is limited to determining that the methodology and reliability used to determine the exposure of the cohort is adequate and appropriate. The Sub- committee would provide commentary on whether or not exposure was supported by analytical chemical data, by oral history or otherwise. Also, if the actual persons exposed provided oral history by direct conversation, survey or by telephone. The committee will determine if indirect exposure information was provided by a relative (or friend or coworker) is reliable.
      4. Statistical Evaluation Sub-Committee. In the initial review of the study, the sole role of the statistical evaluation sub-committee is limited to determining if the statistical methods were/were not appropriate and adequately conducted to support the conclusion of the report. This committee would not redo the statistics at this time since the original data would be needed. Input from Sub-committees 1 (ethics), 2(3ndpoint evaluation) and 3(exposure assessment) that clarifies all issues that are the responsibility of their respective disciplines would be needed before statistical reanalysis could be conducted. Thus, receipt of the original data in a manner that satisfies Sub-committees 1, 2 and 3 and well as any requests by the Statistical Evaluation Sub-committee before any statistical reanalysis would be conducted.
      5. Analytical Chemistry Assessment Sub-Committee. The role of the analytical chemistry committee is limited to determining if the analytical techniques/ methodology were appropriate and adequate for the study. In some cases where no analytical chemistry data were generated, the committee will comment on the need to have included such data and/or if quantitative analysis of exposure data was even possible.
      6. Animal Toxicity and Structure Activity Relationship (SAR) Assessment Sub- Committee. In most cases with chemicals, the Agency should already have a battery of standard animal toxicity studies that were required for the registration of that chemical. There may also be studies in the open literature attempting to define the toxicity of the chemical. Thus, the person(s) responsible for evaluating the animal toxicity studies would be a member of this Sub-committee. In addition, this Sub- committee will have individuals that can address the SAR issues in relation to the possibility that the chemical could affect the endpoint in question.
      It should be noted here that a Record of Review (ROR) signed by at least two qualified toxicologist is needed for any publication supporting a mechanism of action for the chemical.
      If the endpoint occurs in animals or has SAR correlates, such information can support the epidemiological findings. If not, it does not automatically dismiss the epidemiological findings since the effect on the endpoint may be unique to humans.
      7. Council Chairperson, Council meetings and resulting product: ROESR.
      The Council Chairperson should not be in the same program that has responsibility for regulating the chemical or environmental condition. That is, if the chemical is a pesticide, the Chairperson cannot be from the Office of Pesticide Programs (OPP). Representatives of OPP may still be on the Council if an epidemiological study with a pesticide is being reviewed.
      At the initial Council meeting, the Council will discuss the conclusions of each the six Sub-committees. In some cases, the Council may determine that the association between exposure and the endpoint is otherwise strong enough to support regulatory action to protect the public health and no additional information is needed from the study authors.

During the initial meeting the Council can overrule a request for additional data made by any of the Sub-Committees. However, clear justification for overruling a Sub-Committee's decisions must be provided.
      In other cases, the Council may determine that there is a definite need to require additional data supporting the study methodologies and conclusions are required. The Council would then notify the authors articulating of the need for the specific additional information.
      The Council will also determine if the authors of the epidemiological report should be requested to attend a closed meeting with the Council and Sub-committee members. The purpose of the meeting will be an opportunity for the study authors to address any concerns that the Sub-committees have. This will include problems with providing any additional information that the Sub-committees requested. If the study authors refuse to attend a requested meeting, the Council will determine if the study should be rejected outright.
      The Council Chairperson (or secretary of the Council) will prepare the ROESR product that contains the decision with supporting justification. The ROESR product would include the report of the Council with the Chairperson and Council members' signatures. Separate attachments for the signed Sub-Committee reports and the original epidemiological report (i.e. a publication) will also be attached. An executive summary of any meeting(s) with the study authors will also be appended.
      Purpose. The purpose of this procedure is to assure that responsible persons for each of the critical Sub-disciplines independently reviewed the study in terms of their expertise. The Council Chairperson and members of the Council and the sub-discipline committees own their decisions and are responsible for defending them. If industry or any concerned public interest groups object to the conclusions, they can address the conclusions presented in the ROESR and/or the sub-disciplines. Therefore, the decisions in the resulting ROESR should be transparent.
      The ROESR is not a final conclusion (no documents in EPA should be considered final). It is very likely that a Science Advisory Panel will ultimately review the ROESR. Also, as new information is generated, the ROESR can be updated and the recommendations for use (or otherwise) can be adjusted as appropriate.
Response: EPA will continue conducting reviews according to the existing peer review policies and practices, including any programmatic specific processes.
<IP_ADDRESS> Oversight of Data Storage and Sharing
Comment: Commenter (2426) suggests that, depending upon the identity and role of the governing entity [in determining exemptions], the same body or a separate third-party entity should oversee the storage of the data and grant access to qualified researchers. The commenter states that the independence of the entity that would grant access to the data also ensures that the EPA is not put in the paradox of collecting data to ensure data transparency and then possibly having to reject requested access to that data.
Response: The final rule does not require the EPA to disclose or host data, but to determine if dose-response data are available and to give greater consideration to those studies for which such data are available.
Comment: Commenter (6117) suggests that the EPA focus on creating a data sharing system for long term human subject studies where privacy concerns exist. The commenter suggests that, in order to make data available to external researchers for scientific purposes, the EPA follow previous efforts to facilitate the sharing of data. The commenter recommends:
 An independent intermediary group should be created and should partner with the EPA to oversee data requests. By allowing an independent group to have full jurisdiction to make decisions regarding data access, potential political motivations can be avoided.
 The independent group should ensure that data are made available for scientific research that is aimed at advancing knowledge. Data requestors should provide basic information about the principal investigators, all study personnel and funders, and submit a detailed proposal outlining specific aims and specifying study methodology. All data requests should undergo external review, to facilitate feedback from independent experts in the field to verify scientific merit. To ensure transparency, the independent group should make all of the data requests publicly available, along with the reasons for granting or denying data requests.
 All requestors should be required to sign a Data Use Agreement (DUA), which states that access to the data will be used to enhance knowledge and that all findings will be made publicly available through publications and meetings.
 The independent group should ensure that the scope of the analyses is limited to the specific aims set out in the proposal and that additional objectives are outlined in new submissions.
Response: The final rule does not require the EPA to disclose or host data, but to determine if dose-response data are available and to give greater consideration to those studies for which such data are available.
The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying data or models are publicly available or available through restricted access. Restricted or tiered access in this rule means that the underlying data or models are be available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
14.1.2 Incorporating Stronger Data and Model Access Requirements in Grants and Cooperative Agreements
<IP_ADDRESS> Support Stronger Data and Model Access Requirements/Cooperative Agreement and Grant Requirements
<IP_ADDRESS>.1 Encourage and Ensure that More Data and Models be Publicly Available
Comment: Commenter (6918) supports efforts EPA can reasonably undertake to encourage and ensure, as a general matter, that more data and models underlying EPA decisions are available for the public to use and independently verify, while respecting lawful restrictions on data access. The commenter states that EPA currently underutilizes certain tools it has. The commenter states that, as an agency of the federal government, EPA currently possesses the authority to obtain data produced by federal grants. 2 C.F.R. § 200.315(d)(1) ("The Federal Government has the right to: (1) Obtain, reproduce, publish, or otherwise use the data produced under a Federal award. . . ."). The commenter further states that EPA also has the authority to authorize others to use such data. Id. § 200.315(d)(2) ("The Federal Government has the right to: ... (2) Authorize others to receive, reproduce, publish, or otherwise use such data for Federal purposes."). The commenter states that current regulations recognize the importance of public access to data generated by federal grants and that research data produced from federal grants are subject to the Freedom of Information Act (FOIA) in a specific way. The commenter notes that when a federal agency uses research findings produced from federal grants "in developing an agency action that has the force and effect of law," if a FOIA request is made for the research data relating to these findings, the agency that issued the grant "must request, and the [grantee] must provide, . . . the research data so that they can be made available to the public" through the FOIA process. Id. § 200.315(e)(1). The commenter states that these regulations also recognize that release of certain research data may be inappropriate, specifically, the definition of "research data" for FOIA purposes exempts the following from requirements for public release:
      (i) Trade secrets, commercial information, materials necessary to be held confidential by a researcher until they are published, or similar information which is protected under law; and
(ii) Personnel and medical information and similar information the disclosure of which would constitute a clearly unwarranted invasion of personal privacy, such as information that could be used to identify a particular person in a research study.
Response: EPA has implemented the public access requirements for grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." Grantees and contractors are required to make their data and models underlying any peer-reviewed journal articles resulting from EPA-funded research accessible to the public. This is requirement went into effect for any funding granted as of September 26, 2019. EPA's Environmental Dataset Gateway provides metadata records stating where those data will be available. PII and CBI are still protected under the public access requirements.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
<IP_ADDRESS>.2 National Academy of Sciences (NAS) Guidance
Comment: Commenter (3971) notes that the NAS has issued guidance on this matter and suggests that transparency, as well as privacy can be achieved if carefully executed. Commenter (6356) states that, with regard to Grants and Cooperative agreements, EPA should stipulate that methodologies and technologies designed to provide protected access to sensitive information on personal identities should be based on criteria adopted by the National Academies for "masking" personal identity, or similar, policies for masking personal information.
Response: The public access requirements for grants and cooperative agreements requires that data undergo quality review before they are made publicly accessible to safeguard against the release of personally identifiable or proprietary data.
<IP_ADDRESS>.3 Data Sharing Provisions
Comment: Commenters (6102, 6150, 6179, 6360, 6362, 6375, 6918) suggest EPA establish new contracting and grant requirements with data sharing provisions.
Commenters (6102, 6150, 6179) suggest that EPA can establish new contracting requirements to include data and model access requirements.
Commenters (6102, 6150) suggest that EPA establish new contracting requirements to include data sharing provisions, create new interagency data sharing agreements, implement protocols to de-identify data with personal information, and ensure protected data sharing mechanisms where complete de-identification is not possible. Commenter (6102) states that the increased transparency will spur new research and increase the value of federal research investments. The commenter provides that data sharing will allow testing of new ideas on existing data sets. The commenter also recommends that new causal analytical techniques be used to test validity of existing data thereby increasing public confidence in results.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." Grantees and contractors are required to make their data and models underlying any peer-reviewed journal articles resulting from EPA-funded research accessible to the public. This is requirement went into effect for any funding granted as of September 26, 2019. EPA's Environmental Dataset Gateway provides metadata records stating where those data will be available. PII and CBI are still protected under the public access requirements.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
<IP_ADDRESS>.4 Study Design Requirements
Comment: Commenter (6179) suggests EPA could incorporate "transparency by design" requirements into its research grants by conditioning funding on accounting for transparency throughout studies' lifecycles. The commenter provides that, rather than employing after-the-fact de-identification protocols, it would be far easier to achieve the proposal's replicability and verification goals if transparency were incorporated into studies from the outset. The commenter suggests that EPA go beyond the proposal to develop methods that encourage transparency early in studies' development. The commenter states that EPA should borrow from the well-known privacy policy, "privacy by design," which calls for privacy to be taken into account throughout an entire developmental process. The commenter states that EPA could incorporate "transparency by design" requirements into its research grants by conditioning funding on accounting for transparency throughout studies' lifecycles. The commenter suggests that EPA could conduct further research on "transparency by design" protocols, such as anonymous coding methods, to be incorporated in these requirements. According to the commenter, such forward-looking planning can most efficiently accomplish the proposal's goal of better informing public comments in significant EPA rulemakings.
Commenter (6911) encourages EPA in the Final Rule to create "Transparency by Design" in its research requirements. Commenter states that this would be much more helpful that employing after-the-fact de-identification protocols. Commenter states that such a policy would encourage transparency early in study or a series of studies development.
Similarly, commenter (6918) states that, when EPA carries out its own studies or other data generating activities, EPA has the ability to take care, at that time, to avoid using or generating data and models with access issues, can design data-masking procedures into the process, or can undertake other reasonable actions to enhance the public's ability to use and independently verify the underlying data and models at a later point, should the study or data be used to justify regulatory action. The commenter states that EPA's role as the generator of the data should allow for increased access as a general principle. According to the commenter, EPA should be held to a higher standard when it has the opportunity from the start to require that the study design include appropriate measures to allow for transparency without violating relevant legal and policy limits on data access.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." EPA researchers, grantees and contractors are required to make their data and models underlying any peer-reviewed journal articles resulting from EPA-funded research accessible to the public. This is requirement went into effect for any funding granted as of September 26, 2019. EPA's Environmental Dataset Gateway provides metadata records stating where those data will be available. As part of the grant applications, grantees are required to develop a scientific data management plan (SDMP), which includes information on where the data underlying any journal publications resulting from the project will be made publicly accessible. Contractors will also need to develop SDMPs for any projects that could result in journal articles.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
<IP_ADDRESS>.5 Require Level II or Level III Transparency and Openness Promotion (TOP) Guideline Requirements
Comment: Commenter (6360) recommends EPA require Level II or Level III Transparency and Openness Promotion (TOP) Guideline Requirements standards for all its research to be consistent with the best and the current standards of science, regardless of whether the researchers are EPA employees, other federal agency employees, or private researchers employed by companies, universities, or other organizations. According to the commenter, by adopting these standards in EPA's research enterprise, EPA's application of the funded science is ready to support policy judgements.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." EPA researchers, grantees and contractors are required to make peer-reviewed journal articles and the underlying data and models resulting from EPA-funded research accessible to the public.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
<IP_ADDRESS>.6 Require Open-Access/Public Availability
Comment: Commenters (0671, 6362) suggest that EPA implement requirements that all EPA-funded data, models and results be open-access.
Commenter (0671) states that EPA can increase public access to EPA-funded data by requiring publications based on EPA-funded data to be open-access (or open-access after 12 months).
Commenter (6362) suggests that EPA can implement requirements that all models and results developed under EPA Cooperative Agreements and Grants be open access and not proprietary.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." EPA researchers, grantees and contractors will be required to make peer-reviewed journal articles and the underlying data and models resulting from EPA-funded research accessible to the public. All peer-reviewed journal articles that were funded by EPA are posted on NIH's PubMed Central within one year of publication. The data and models underlying those journal articles are to be made publicly accessible.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
Comment: Commenter (6375) recommends that EPA clearly stipulate in cooperative agreements and grant terms that data used by researchers receiving funding from EPA must be made publicly available to the extent possible after consultation with EPA. The commenter also suggests that EPA require as part of the cooperative agreement or grant that the data be placed into EPA's designated system to allow access to the underlying data and models.
Commenter (6449) states that, if the models and data are generated from government funds, EPA should require that they be publicly available on EPA's website or by request to EPA. Commenter states that, in some cases where compelling reasons are given and publicly available, a time delay might be justified, which should also be identified.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." Grantees and contractors are required to make their data and models underlying any peer-reviewed journal articles resulting from EPA-funded research accessible to the public. This is requirement went into effect for any funding granted as of September 26, 2019. EPA's Environmental Dataset Gateway provides metadata records stating where those data will be available. For intramural research, the data underlying journal publications are posted on EPA's ScienceHub and are available to the public via data.gov.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
<IP_ADDRESS>.7 Data Management Plan
Comment: Commenter (6362) recommends that EPA require all grant proposal applicants to include as part of any grant proposal a data management plan, similar to those required by the NIH. The commenter suggests that EPA may elect to exclude from these requirements grants/agreements of some specified annual amount, but that annual amount should be reasonable and ensure that the vast majority of models and results developed under grants/agreements is shared.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." Grantees and contractors will be required to make their data and models underlying any peer-reviewed journal articles resulting from EPA-funded research accessible to the public. This is requirement went into effect for grants as of September 26, 2019. As a part of grant application, applicants are required to provide a scientific data management plan (SDPM) for their proposed project. Part of the SDMP requires applicants to specify how they will make their scientific research results publicly accessible.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
<IP_ADDRESS>.8 Common Data Templates and Digital Platforms
Comment: Commenter (6362) states that EPA should develop common data templates and digital platforms for the most common types of research studies to be used by entities subject to Cooperative Agreements and Grants to facilitate public use and validation.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." Grantees and contractors are required to make their data and models underlying any peer-reviewed journal articles resulting from EPA-funded research accessible to the public. This is requirement went into effect for any funding granted as of September 26, 2019. EPA's Environmental Dataset Gateway provides metadata records stating where those data will be available.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
<IP_ADDRESS>.9 Anonymize or Mask Data to the Extent Necessary to Facilitate Data Sharing
Comment: Commenter (6918) states that when EPA funds research, there are opportunities to maximize both transparency and privacy. The commenter recommends that, to facilitate increased access without forcing grant or award recipients to bear unforeseen costs to achieve it, EPA consider adding a new condition to its grants and awards that requires recipients of federal funding through grants or agreements to "anonymize" or mask the data, to the extent necessary to facilitate the sharing of data with EPA and others. The commenter asserts that EPA's role as the funder allows EPA to require the grantee to consider transparency values at the beginning of the process. The commenter notes that recognition from the outset that EPA may request the underlying data will encourage the grantee to put into place, at the start of the study, appropriate data-masking or anonymizing procedures. According to the commenter, this will efficiently allow for increased data access without running afoul of legal limits on data access or policy reasons for limiting access.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." Grantees and contractors will be required to make their data and models underlying any peer-reviewed journal articles resulting from EPA-funded research accessible to the public. This is requirement went into effect for any funding granted as of September 26, 2019. EPA Order 1000.17B "Policy for Increasing Access to Results of EPA-Funded Extramural Scientific Research" states, ". . . data should undergo quality review before they are made publicly accessible to safeguard against the release of personally identifiable or proprietary data."
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
<IP_ADDRESS> Do Not Support Stronger Cooperative Agreements and Grants
Comment: Commenters (0671, 1011, 4595, 5181, 6895) do not support establishment of stronger contracting requirements with data sharing provisions.
Commenter (6895) asserts that the issue of incorporating stronger data and model access into the terms and conditions of cooperative agreements and grants has been previously and adequately addressed in the OMB Final Information Quality Bulletin for Peer Review (70 FR 2664). This document states that pivotal research must include information so that agency reviewers can "ensure that scientific uncertainties are clearly identified and characterized." The commenter states that this is the task of peer reviews. The commenter notes that there is no need to make anything else available if scientific uncertainties can be quantified. (See, discussion above related to EPA's Peer Review Handbook.)
Commenter (0671) states that EPA should not incorporate stronger data and model access requirements into the terms and conditions of cooperative agreements and grants. Commenter states that such requirements are already very strong and strengthening them will add to the regulatory burden on scientists doing research of potential regulatory import without improving the reproducibility of science underlying regulation. Commenter states that, if incorporating these requirements prevents the EPA from entering into cooperative agreements or funding grants involving human subjects then the EPA will have prevented itself from using the best available science, or, even worse, prevented the highest quality science from being done in the first place, which is counter to the text and intent of the CAA and its amendments.
Commenter (1011) states that, when EPA funds research activities through grants or cooperative agreements, the agency should seek to minimize administrative burdens on researchers.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." Grantees and contractors will be required to make their data and models underlying any peer-reviewed journal articles resulting from EPA-funded research accessible to the public. This is requirement went into effect for any funding granted as of September 26, 2019 through EPA Order 1000.17B "Policy for Increasing Access to Results of EPA-Funded Extramural Scientific Research."
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
Comment: Commenter (4595) states that it would be premature to introduce additional changes to the terms of research grants until the full impacts of any proposed EPA transparency policy are understood. Commenter (5181) recommends that the sufficiency of existing approaches be fully considered before making substantive changes. Commenter (5181) also recommends that adequate safeguards be considered in any proposed rules to protect personal information of relevant parties.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." Grantees and contractors will be required to make their data and models underlying any peer-reviewed journal articles resulting from EPA-funded research accessible to the public. This is requirement went into effect for any funding granted as of September 26, 2019. EPA Order 1000.17B "Policy for Increasing Access to Results of EPA-Funded Extramural Scientific Research" states, ". . . data should undergo quality review before they are made publicly accessible to safeguard against the release of personally identifiable or proprietary data."
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
14.1.3 Building Upon Other Federal Agencies' Policies Regarding Grantee and Cooperator Requirements
Comment: Commenter (9227) expresses concern that, while EPA claims that the proposal builds on the "experience of other federal agencies in this space," EPA simply lists other federal agencies without referring to any policies, documents or actions by those agencies, except for one particular Census Bureau database that allows federal Census data to be shared securely. The commenter states that an uncited statement that other federal agencies have "experience in this space" is far too vague to allow meaningful comment by the public on EPA's rationale for its action, much less provide any support or rationale for the proposed policy.
Commenter (9227) states that the Census Bureau database cited in the Proposal is an example of how an agency can provide secure access to its own data, but it does nothing to explain or justify EPA's proposal to exclude third party studies with nonpublic data from consideration in rulemaking. The commenter provides that the U.S. Census Bureau operates the Federal Statistical Research Data Centers (FSRDCs), which are secure facilities providing authorized access to restricted-use microdata for statistical purposes only. The commenter states that to gain access, researchers must obtain Census Bureau Special Sworn Status -- passing a moderate risk background check and swearing to protect respondent confidentiality for life. The commenter explains that this approach meets the U.S. Census Bureau's needs by allowing access to confidential information only to researchers whose proposals meet certain criteria, who go through a vetting process, and who agree to protect the information. According to the commenter, this is a structure designed to protect data collected by the government, not third parties, and there are substantial costs to this approach, which are borne by the Census Bureau. The commenter contends that it is clearly not directly transferable to the context of the proposal. The commenter asserts that it is also unclear whether such a structure, even if it were practical (which it is not), would be sufficient to satisfy EPA's requirement to make data and models "publicly available."
Response: The Census Bureau, the NIH and the National Science Foundation are not cited in the final rule.
Comment: Commenter (1012) states that the National Science Foundation (NSF) has long required data sharing as a condition of funding. The commenter provides that, since 2007, the NIH has required public access to publications for all of its funded research after 12 months. The commenter notes that an interagency committee set up by OSTP to develop and coordinate federal policies for long term data stewardship and dissemination of federally funded research has seen at least 17 agencies phase in such data policies. The commenter notes that all of the Agency plans exempt data that is legally protected from disclosure.
Commenter (0556) states that the Federal Government already has rules about sharing data that is developed in federally-funded research. The commenter notes that in some ways this proposed rule looks like a natural extension of the existing requirement for a Data Management Plan. The commenter states that these rules are part of what is called "Public Access," a program which began in 2013 and requires every federal science agency to have a Public Access Plan mandating that research data be shared. The commenter states that EPA's proposed rules extend the Public Access Program from research that is federally-funded to research that is federally-used.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." EPA requires all journal articles resulting from Agency-funded research be made available to the public within 12 months of publication. Also, data underlying those journal articles are to be made publicly available while still protecting CBI and PII according to existing requirements.
Under 40 CFR Part 30, the EPA will give greater consideration to pivotal science for which the underlying dose-response data are either publicly available in a manner sufficient for independent validation or, in the case of PII, CBI, or proprietary data, available through restricted access that affords privacy in a manner sufficient for independent. This is for pivotal science regardless of whether or not it is funded by EPA.
This final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedures and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
14.1.4 Platform for Increasing Public Access to EPA-Funded Data
<IP_ADDRESS> Do Not Support Development of a Platform for Increasing Public Access to EPA-Funded Data
Comment: Commenters (0671, 1012, 6895) do not support development of a platform for increasing public access to EPA-funded data.
Commenter (0671) states that if EPA is considering purchasing or developing a software platform that would magically encompass any and all data used in scientific studies potentially used in rulemaking, the EPA should not waste taxpayer money on what they believe would be "a colossal boondoggle."
Commenter (1012) states that, with a reduced number of EPA-funded research studies conducted, a platform to increase access to the data collected is likely to be irrelevant. The commenter states that, if this rule is to be implemented as written, it is likely that fewer scientists or research institutions will want to participate in EPA-funded research if they are concerned that the personal health information they collect may have to be publicly-disclosed. The commenter notes that scientists and researchers may also find it difficult to recruit participants for long-term, large-scale studies that are needed to understand air and water quality issues if the participants are concerned that their medical records may be made publicly-available.
Commenter (6895) contends that a platform to house data from different studies is not necessary. The commenter states that corresponding authors are required to respond to public requests and should be responsible for requests for data necessary to replicate study results. According to the commenter, EPA's request for comment on possible exceptions to the proposed rule, specifically exceptions that would be made on a case-by-case basis, underscores that even EPA has doubts as to the necessity and validity of the proposed rule.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." Researchers are required to make their data and models underlying any peer-reviewed journal articles resulting from EPA-funded research accessible to the public. For intramural research, the data underlying journal publications are posted on EPA's ScienceHub and are available to the public via data.gov. The public access requirements under the 2013 OSTP memorandum only concern EPA-funded research and was prospective upon implementation.
Under 40 CFR Part 30, the EPA will give greater consideration to pivotal science for which the underlying dose-response data are either publicly available in a manner sufficient for independent validation or, in the case of PII, CBI, or proprietary data, available through restricted access that affords privacy in a manner sufficient for independent. This is for pivotal science regardless of whether or not it is funded by EPA.
This final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedure and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
Comment: Commenter (1012) states that OMB Circular A-110 limits the kind of data produced under a federally funded award that is required to be made public, in order to strike a balance between preserving the integrity of the research process and the need for public access to data. The commenter provides that this is accomplished by permitting such access through the procedures established under FOIA, and including appropriate exemptions such as "personnel, medical, or similar files whose release would constitute an unwarranted invasion of privacy." The commenter states that much of the data from the previously cited air pollution studies would fall under this exemption. The commenter is concerned that this proposed rule could potentially contradict these protections by requiring sensitive data that would have previously been exempted to be made "publicly available in a manner sufficient for independent validation." The commenter states that this requirement stipulated in the proposed rule is particularly concerning as "publicly available" is further defined as "information necessary for the public to understand, assess, and replicate findings."
Commenter (2427) states that there are already various means for public access to studies that EPA uses, and in some cases their underlying data, without the release of confidential information, including the FOIA, which provides an avenue to request raw data, including a process ensuring that sensitive data is protected.
Response: The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
<IP_ADDRESS> Support Development of a Platform for Increasing Public Access to EPA-Funded Data
Comment: Commenters (1941, 6126) suggest that EPA should work to set up data-housing capabilities and protocols at a Research Data Center (RDC). Commenter (1941) states that an RDC or another suitable data-storage/analysis site could facilitate the receipt and centralized management of pivotal study data in a manner that can streamline third-party access to data, while also providing safeguards and Agency oversight to protect confidentiality concerns.
Commenter (6126) suggests EPA establish a strong culture and practice of protecting such data with a sufficiently secure data infrastructure. The commenter asserts that EPA's data infrastructure and use of authorities to protect such data appear to be limited. The commenter recommends that EPA explore the availability of the FSRDCs as an option for making confidential data available for approved research projects and statistical activities. The commenter states that the FSRDCs have existing protocols about the de-identification of data and disclosure review processes on which EPA could reasonably build. The commenter provides that FSRDCs may also be useful for EPA to make sensitive, de-identified data available in a secure manner that reduces the risk of re-identification.
Response: EPA has implemented the public access requirements for EPA researchers, grantees and contractors as required by the February 22, 2013, OSTP memorandum "Increasing Access to the Results of Federally Funded Scientific Research" and the 2016 EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research." Researchers are required to make their data and models underlying any peer-reviewed journal articles resulting from EPA-funded research accessible to the public. For intramural research, the data underlying journal publications are posted on EPA's ScienceHub and are available to the public via data.gov. Also, data underlying those journal articles are to be made publicly available while still protecting CBI and PII according to existing requirements. The public access requirements under the 2013 OSTP memorandum only concern EPA-funded research and was prospective upon implementation.
The EPA "Plan to Increase Access to Results of EPA-Funded Scientific Research" requires the data underlying a journal article resulting from research funded by EPA be made publicly accessible. However, in the final rule, the EPA is not identifying a specific step in a multi-step analysis as the stage of data that would be sufficient for independent validation through reanalysis because this would be overly prescriptive and not informative. The step at which the final clean data set will be generated will vary from study to study. The level of detail required would be that needed for a separate party to reanalyze the study. The appropriate step is where the data are ready to be analyzed to extract relevant information.
<IP_ADDRESS> Consider Various Existing Data Management Platforms/Requirements
Comment: Commenters (0556, 2171, 2426, 2479, 6375, 6449, 6908) suggest EPA consider the use of various existing data management platforms/requirements. Commenter (2426) notes that other organizations already have infrastructure and policies in place to handle sensitive information, and it makes sense to use these existing capabilities, rather than attempting to re-create them at the EPA. Commenter (6375) states that EPA should study existing systems and adopt, adapt or set up new online data repositories similar to those that already exist that allow public access to data on which agency actions rely that do not contain personal or confidential information.
Commenter (6375) notes that various government agencies, such as the Centers for Disease Control and Prevention (CDC), the Centers for Medicare & Medicaid Services (CMS) and the U.S. Census Bureau maintain online databases and data repositories to allow public data access or restricted access to identifiable data. The commenter provides that CDC and CMS both provide restricted and secured access to health data that contain potentially identifiable and privacy information, such as name, residential address, vital status, birth date, and disease diagnosis. The commenter notes that EPA could also implement additional background screening processes and security measures for access to data that may contain identifiable or proprietary information, similar to those employed by the CDC and CMS for access to research identifiable data.
Commenter (6449) encourages EPA to cooperate with other federal agencies to generate common data access, management and storage standards. Commenter also encourages EPA to participate with the PubChem and the National Toxicology Program databases. Commenter states that the same types of legally binding documents that EPA needed to sign to obtain such data, e.g., non-disclosure agreements, would be expected to be sufficient for other parties. The commenter states that EPA could consider requiring data acquisition and analysis by parties independent of the stakeholder. The commenter suggests, for example, if CBI were requested by an industrial competitor, EPA might require that a non-affiliated consulting firm or academic be the entity that signs the NDA and receives the data for analysis.
Commenter (2479) states that policies vary based on the type of research and the confidential information researchers gather, but the NIH maintains several online databases, like the National Cancer Institute Data Catalog, that qualified researchers can access for free. Commenter (6144) asserts that the NIH system is not publicly accessible and requires significant resources to manage and share with researchers.
Commenter (2171) states that Minnesota agencies are at the forefront of making environmental and health surveillance data available, providing technical assistance for using data, and engaging partners across communities and research institutions around effective dissemination and data utilization. The commenter notes that their agencies host multiple platforms for accessing high-quality health surveillance and environmental monitoring data, while protecting privacy and providing essential risk communication and prevention strategies.
Commenter (2479) states that many scientific journals, like Science, already require study authors to publish their data in online databases, but at Science, for example, the policy allows exceptions for studies that contain sensitive details like medical information.
Commenter (6908) states that Bayer is uniquely positioned to provide information on the efforts to make data transparent. The commenter notes that Bayer has submitted nearly 50,000 volumes of research to the agency in support of regulatory actions related to their products and nearly the full set is currently available to almost everyone upon demand. The commenter provides that beginning in 2016 Bayer made the decision to make the safety data underlying its products available to interested parties and began the process of developing a platform to accomplish this. The commenter states that in late 2017 Bayer's transparency platform was launched and may accessed at https://www.cropscience-transparency.bayer.com/.
Response: The EPA is finalizing the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (9224) suggests EPA should encourage authors to submit their data to public repositories as well as considering giving preference, when possible, to studies where the data is publicly available in this manner.
Response: This rule exclusively pertains to the internal practices of the EPA and does not regulate the conduct or determine the rights or obligations of any entity outside the federal government.
<IP_ADDRESS> Issues Related to the Development of a Data Sharing Platform
Comment: Commenters (6144, 6191, 6909, 6915) express concern that the proposed rule does not address numerous issues related to the development of a platform. The commenters assert that ensuring all this extra information is received, stored, and made publicly accessible would be no small undertaking. Commenter (6144) inquires where the raw data collected will reside, how will the resources to manage the data be obtained, and how will the agency deal with legally and ethically protecting data that is confidential or sensitive.
Commenter (6191) states that guidelines would need to be established as to how long the data would be held and a repository would need to provide mechanisms for file format migration that ensures that data remain un-corrupted and readable. According to the commenter, aggregating sensitive health data onto one platform would unnecessarily increase risks to human subjects. The commenter asserts that data breaches at the Office of Personnel Management (OPM) (https://www.opm.gov/cybersecurity/cybersecurity-incidents/ ) and other commercial organizations (e.g., Equifax, Yahoo) demonstrate that no platform can ever be completely secure.
Commenter (6909) states that, even where privacy issues do not exist, insufficient infrastructure can present a substantial barrier to uniform public sharing of data. The commenter asserts that this is a reason why, in seeking to increase data sharing and improve transparency in science, EPA should take a measured and nuanced approach, consulting closely with the relevant stakeholders rather than swiftly imposing a bright line rule. According to the commenter, existing infrastructure is often still insufficient for hosting and sharing large data sets. The commenter notes that, for example, mechanisms for aggregating information about associated research projects together (rather than having them distributed across numerous repositories), and for properly and usefully identifying and tracking related resources  -  all of which are central to improving openness and reproducibility of studies  -  are often lacking. The commenter provides that some groups are developing such services and integrations, but these capabilities need significant improvement. The commenter states that, in addition, sharing of data increasingly means processing large amounts of data stored not in one location but rather on a distributed database. The commenter states that existing technologies are sometimes unable to process such large-distributed datasets efficiently.
Response: The final rule does not impose requirements on any entity outside of the EPA. This is a rule of internal procedure and does not direct or require any outside entity or the EPA to establish data sharing mechanisms. Further, the final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
14.1.5 Methodologies and Technologies Designed to Provide Protected Access to Data
<IP_ADDRESS> Consultation with and Consideration of Methodologies and Technologies Designed to Provide Protected Access to Data Already Developed and Used by Agencies and Organizations
Comment: Commenter (6915) expresses concern that, while the proposal says EPA "should collaborate" with other agencies ("to identify strategies to protect confidential and private information in any circumstance in which it is making information publicly available"), there is no timeframe for this process, no explanation of what will happen until such strategies are formed, and no indication of what these strategies will be.
Response: The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science. The final rule does not mention collaborating with other agencies to identify strategies to protect confidential and private information.
Comment: Commenters (0039, 4537) recommend that EPA, as it determines how best to implement the methodologies and technologies of a "reproducibility architecture"' to facilitate the adoption of a best available science/best available reproducible science (BARS) standard, consult with representatives or the Center for Open Science, the Meta-Research Innovation Center at Stanford (METRICS), and the Laura and John Arnold Foundation Research Integrity Initiative. The commenters also recommend that the EPA consult with the American Statistical Association about how to institute standard procedures that will ensure that all scientific research used or funded by the EPA is conducted according to the highest standards of statistical practice.
Commenter (5185) encourages work to protect and de-identify sensitive information to allow for its continued use in regulatory decision-making. The commenter suggests that coordination with other federal agencies could assist; for example, the Department of Health and Human Services (DHSS) and National Institute of Standards and Technology (NIST) have developed tools and guidance to de-identifying PII. According to the commenter, drawing on the expertise and experience of other agencies across the federal government through active and ongoing coordination will improve EPA's ability to make this effort a success. Commenter suggests that, such an effort would be undertaken in concert with the OMB Office of Information and Regulatory Affairs.
Commenters (3971, 6126, 6375) suggest EPA should seek inter-agency advice and technical support on this matter. Commenter (6126) suggests that, regarding EPA's inquiries about nondisclosure avoidance protocols, EPA should consult with the Federal Committee on Statistical Methodology and the federal statistical agencies that have developed much of the research and practice regarding de-identification approaches for confidential data. Commenter (6375) states that multiple government agencies provide restricted and secured access to data that contain privacy or confidential information.
Commenter (6921) states that for sensitive data sets that are otherwise protected due to personal information or CBI, EPA should work with other federal agencies to develop: (1) de-identification protocols that will allow an increasing number of data sets to be fully released to the public; and (2) protected data sharing platforms that will allow the release of more sensitive data sets to researchers subject to confidentiality agreements.
Commenter (6362) states that several agencies and organizations, in addition to NIH, have successfully addressed the issue of data access while maintaining confidentiality that should be considered by EPA. For example:
 The existing rule requiring federally-funded research to be made available to other researchers. This standard could be adopted and applied to third-party funded researchers.
 Health care claims and related data are now being made available to researchers in de-identified form by some health insurance companies, such as Optum, which offers a "proprietary research database of health care and administrative data that links patient, physician, and treatment attributes from millions of geographically diverse individuals in the U.S." Optum appears to have developed methods and procedures to appropriately address confidentiality concerns.
 Medicare claims data are already available to researchers in de-identified form. Algorithms and methods developed by the Center for Medical Services should be examined by the EPA.
 Several professional societies have guidance on the protection of health data and de-identification, such as the Institute of Electrical and Electronic Engineers and the International Association of Privacy Professionals.
Commenter (9224) encourages EPA to consult with entities, including the journals that EPA references, that have implemented similar efforts in order to better inform the methods and protocols that should be in place for a rule of this nature.
Response: The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science. The final rule does not mention collaborating with other agencies to identify strategies to protect confidential and private information.
<IP_ADDRESS> Methodology and Technology Limits to Protection of Data
<IP_ADDRESS>.1 De-identification Methods and Technologies are Limited and Can Potentially Lead to Re-identification
Comment: Commenters (0569, 2746, 5173, 6046, 6110, 6126, 6132, 6135, 6357, 6446, 6868, 6924, 6937, 9227) caution that there are limits as to how much an individual's identity can be protected based on commonly used de-identification methods and technologies. Therefore, they do not believe the proposed rule can be effectively implemented.
Commenters (0569, 6446) assert that researchers could redact and anonymize personal health data but there are limits as to how much an individual's identity can be protected, especially in small communities where basic identifiers may be enough to recognize an individual.
Commenters (5173, 6110, 6446, 6924) assert that, while the proposal references simple data masking, coding, and de-identification techniques, these methods will not sufficiently protect patient identity in studies such as those in which personal health information is integral to the study. Commenter (6868) suggests that sharing data on animal research or controlled exposure of healthy people can be relatively straight-forward but sharing medical data about real people with diseases at work and at home is much more challenging. The commenter states that full de-identification of the study participants in studies that use dates and addresses to estimate pollution exposure would be impossible.
Commenters (5173, 6126, 6132, 6153, 6191, 6924, 9227) cite several studies to demonstrate that de-identified and anonymous data can be re-identified.
Commenter (5173) asserts that the need to protect sensitive information would make compliance with the proposed rule impossible but discarding such studies from the regulatory decision-making process would exclude relevant and important science, thereby compromising public health. Commenters (6126, 9227) assert that, as the Commission on Evidence-Based Policymaking described in its final report, simply removing direct identifiers is often insufficient to minimize the risk of re-identification. Commenter (6132) states that, in a time of "big data", it is all too easy to crack any de-identification process, especially when lots of publicly available spatial environmental data are matched to people in the study. Commenter (6153) states that, although public release of data collected on individuals has relied on anonymization to protect privacy, researchers have demonstrated the potential of "reidentification" techniques, which combine seemingly innocuous data from multiple publicly available sources, to identify individuals.,  Commenter (6191) references two studies and concludes that de-identified and anonymous data can be re-identified., Commenter (6924) states that, according to an NAS Workshop report re-identification of respondents may be increasingly possible because of high-speed computers, external data files containing names and addresses or other direct identifiers as well as information about a variety of individual characteristics, and sophisticated software for matching survey and other files. Commenter (6924) states that re-identification is not hypothetical; study participants, including children, have been identified from data sets which were thought to be sufficiently masked.
Commenter (9227) cites additional studies that show the reality and scope of the re-identification threat.,.. The commenter states that the Commission on Evidence-Based Policymaking, which EPA also cites in the proposal, also stresses the dangers of re-identification of data that has been stripped of direct identifiers: "No existing statistical disclosure limitation method. . . is able to completely eliminate the risk of re-identification," despite increasingly complex techniques that have been developed since the 1970s. The commenter provides that the Commission also notes the threat posed by the "cumulative amount of information available about individuals and businesses that could be used for reidentification," with the threat increasing as available information grows and technology to allow re-identification improves.
Commenter (9227) states that while EPA suggests de-identification and redaction of sensitive information can be used to protect privacy when study data is made public, EPA ignores the real concerns, based in empirical evidence, about reidentification of individuals through cross linking with existing public datasets and the ensuing breach of privacy. Commenter states that experts have observed that even the disclosure of redacted or "de-identified" data sets has become more fraught as public health studies have become more rigorous, because these studies are relying upon greater quantities of ever more granular personal information.
Commenter (2746) states that the proposed rule does not address or define the required data set granularity that must be made publicly available; that is, would data presented in aggregate be sufficient for independent validation or would only raw data be sufficient? The commenter states that depending upon the data set, inference and other methods can often be used to deanonymize data, which would result in compromised privacy for study subjects.
Commenter (9227) asserts that, while EPA states that "[o]ther federal agencies have developed tools and methods to de-identify private information," the Agency then cites to only one source, which does not address the concerns raised here. The commenter states that this guidance provides two methods for de-identifying data: (1) expert determination method, where an expert determines that, after application of statistical and scientific principals and methods, the risk is very small that the information alone or with other available information could be used to identify the subject; and (2) the safe harbor method, requiring that a number of identifiers are removed. The commenter notes that the first method requires case-by-case work, and EPA has provided no information regarding how EPA or others could potentially implement it or how much it might cost. The commenter states that there is no indication of how broadly this technique might be applicable to adequately de-identify data. i.e., EPA must provide its views on whether this technique is likely to be applicable to the majority of studies relevant to EPA with non-public data, some studies, or only a handful. The commenter provides that the second method requires removal of much information that may be necessary to be able to reanalyze or reproduce the research results, so it is unclear whether it would satisfy EPA's requirements in the proposal. According to the commenter, even the safe harbor method has been shown to provide potentially insufficient privacy protections due to the mosaic effect.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying data are publicly available or available through restricted access. Restricted or tiered access in this rule means that the underlying data are be available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The rulemaking does not require the EPA to collect, store, or publicly disseminate data underlying pivotal science.
Comment: Commenter (9227) takes issue with the suggestion in the proposal that the National Academies are of the opinion that simple data masking, coding, and de-identification techniques have been developed over the last half century so that data can easily be modified to address privacy concerns. The commenter notes that the National Academies in fact recognizes that complex, evolving, and yet undeveloped techniques are needed to resolve these concerns, that there are limitations of producing restricted data that sufficiently limit identifiability to allow it to be made publicly available in useful form, and that more work is needed to allow high quality public-use files that still assure inferential validity of the data while safeguarding confidentiality.,,, The commenter adds that the National Academies note, "data that are most useful to legitimate researchers typically have characteristics that pose substantial risk of disclosure" and there is increased vulnerability in "[d]ata with geographic detail, such as census block data" and longitudinal data obtained in panel surveys, which is often salient in environmental research.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (9227) states that ongoing developments in data analytics make data de-identification more difficult to conduct and less likely to adequately protect privacy and confidentiality. The commenter expresses concern that EPA fails to mention, let alone address, the increasing risk of re-identification through data analysis using multiple data sets. According to the commenter, the so-called "mosaic effect" makes even very limited, redacted releases of data to the public a threat to the privacy of study subjects. The commenter provides that OMB has recognized the threat to privacy from the mosaic effect, which it describes as "when the information in an individual dataset, in isolation, may not pose a risk of identifying an individual (or threatening some other important interest such as security), but when combined with other available information, could pose such risk." The commenter notes that OMB specifically highlighted the complicated nature of this threat and the need for agencies to address it carefully, particularly as they may not possess the needed expertise.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenters (6191, 6357, 6446, 9227) express concern that EPA fails to recognize the risks to privacy that remain even if these methods are employed. Commenter (6191) states that government systems and public sector organizations with highly personal data have been breached and there is no reason to believe any system could be completely secure. Commenter (6357) states that making certain types of data public carries substantial privacy risks and may not always be necessary or sufficient to achieve transparency. Commenter (6357) states that, with advancing computational methods and increasing types of data collected on individuals in multiple domains, re-identification of de-identified human subjects becomes more likely without evolving protections. Commenter (6446) states that the fact that the public would be able to de-anonymize much epidemiological data, after de-identification processes have been applied, would almost certainly have a chilling effect on voluntary public participation in important research.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
<IP_ADDRESS>.2 De-Identification Methods and Technologies Could Potentially Make Datasets Unusable
Comment: Commenters (6135, 6924, 9227) suggest that another limitation of de-identification efforts is that they may make data sets unusable for reanalysis purposes.
Commenter (6135) asserts that it is disingenuous that the proposed rule suggests that health datasets can be sufficiently de-identified in the public domain in a way that would simultaneously allow for reanalysis while also preventing the illegal disclosure of protected information. Since there is no way to provide de-identified data in the public domain that would allow for secondary reanalysis of both the exposure assessment and health analysis in studies assessing the health impacts of air pollution, the commenter recommends EPA should instead continue to focus on assessing results across multiple studies to evaluate the veracity and general applicability of scientific findings.
Commenter (6924) states that often, identifying information is a metric key to a study's data analysis and conclusions, such as when home addresses are used as a basis to estimate exposure measures for pesticides or other pollutants. The commenter states that, in this case, no meaningful analysis could be conducted with de-identified data where home addresses have been removed to ensure participant privacy. According to the commenter, for these types of studies, data de-identification and masking do not provide a viable solution that would both allow re-analysis and protect participant privacy.
Commenter (9227) states that work by other experts in this area suggests that de-identification can be carried out and help protect privacy, but it may produce datasets that have lost vital information needed for specific analyses. The commenter states that the Health Insurance Portability and Accountability Act (HIPAA) guidelines document states: "Of course, de-identification leads to information loss which may limit the usefulness of the resulting health information." According to the commenter, such results limit the utility of de-identified data sets and would not meet the requirements of the proposed rule which state that "EPA will ensure that the data and models underlying the science is publicly available in a manner sufficient for validation and analysis."
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Due to limits in the ability to protect data, commenters (6110, 6446, 9227) express concern that EPA may not be able to rely on the best science. Commenter (6110) expresses concern that the proposed rule has oversimplified the issue at hand, which will lead to difficulties in implementing this rule while maintaining EPA's commitment to using the best available science in its regulatory actions. Commenter (6446) states that the inability to de-anonymize much epidemiological data would also prevent EPA from relying on the "best available science" in carrying out its work.
Commenter (9227) states that some researchers might respond to the rule by choosing to work only on public administrative datasets, but this would harm rather than strengthen science quality by curtailing scientific inquiry. The commenter states that the effects of EPA's proposed approach would cause some researchers to choose not to pursue research with human subjects, stifling scientific discovery, while others would forgo compliance with EPA's regulatory requirements and have their research ignored by EPA. The commenter contends that, as a result, EPA's rule would both discourage the development of best available science as well as EPA's use of it.
Response: The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
<IP_ADDRESS>.3 Concerns About Access to Confidential or Private Information Being Addressed by Common Solutions
Comment: Commenters (6110, 6114, 6046, 6448, 6937, 9227) do not agree with the statement in the proposal (which includes footnote 16) that "EPA believes that concerns about access to confidential or private information can ... be addressed through the application of solutions commonly in use across some parts of the Federal Government."
Commenter (9227) notes that EPA claims that there are examples of common solutions from the Department of Health and Human Services (DHHS), the NIST, the Department of Education, and the Census Bureau. Commenter states that, apart from a reference to DHHS guidance on data de-identification, EPA does not actually identify or cite to any specific examples from these agencies in the proposed rule itself, making it impossible to discern what examples EPA believes exist or to meaningfully comment upon the degree to which such examples, if they exist, might suggest that these issues are manageable. The commenter states that the additional hyperlinks added to the docket on May 25, 2018, weeks into the comment period, also link to examples that provide no further assurance that this proposal can be implemented without implicating privacy concerns, and the vaguely referenced other agencies' "solutions" are unlikely to be of much help.
Commenters (6110, 9227) assert that the proposed rule addresses privacy issues only in a vague sense and does not state what those "solutions" would be. Commenters (6114, 6448) assert that this statement is not supported by any information in the proposal. Commenter (6046) states that, with respect to footnote 16 in the proposal, confidential data are not easily accessible and to make them so would require resources and planning not provided for or outlined in the proposed rule. Commenter (6937) states that the examples cited in footnote16 are difficult to transfer because of differences in the nature of the data involved, the sample sizes (and thus the degree to which anonymity can be guaranteed through aggregation), and the nature of incentives and preexisting trust between the parties that access the data. Commenter (6937) asserts that, while the proposal claims that science using protected/private health data will still influence public policy (because viable protections are available), this is not true.
Commenter (9227) states that the Proposal ignores or assumes away real and significant issues, suggesting that existing mechanisms and techniques can be used to protect privacy and confidentiality while making underlying research data publicly available. According to the commenter, the evidence (including several of the sources that EPA cites) indicates that the potential mechanisms alluded to by EPA would only have the potential to address some of the barriers, have serious limitations even for those, and are actually becoming less effective as it becomes easier to combine and manipulate public data sets. The commenter provides that EPA's own SAB voiced concerns that EPA was discounting the challenges to making even limited releases of data, saying:
      The proposed rule oversimplifies the argument that "concerns about access to confidential or private information can, in many cases, be addressed through the application of solutions commonly in use across some parts of the Federal government." For studies already completed or underway, the participation of human subjects is undertaken according to terms approved by the cognizant IRB [Institutional Review Board]. These terms can vary from study to study. In some cases, the data cannot be released simply by redacting portions of it. For example, data may have been collected with an assurance to the participating individuals that their data would be kept confidential.
Commenter (9227) states that the "solutions" EPA might have in mind do not address the issues raised by the Proposal because no other agency has tried to implement a requirement such as the one EPA proposes. The commenter states that other agencies provide guidance and techniques to protect privacy during data collection and disclosure to allow more use of data collected by the government, not to mandate that data collected by academic or industry researchers be publicly available for purposes of replicating analyses. For example, the commenter provides that the Department of Education has shared techniques for institutions to provide data on students and schools to meet reporting requirements without compromising privacy. The commenter states that they recognize that each technique "requires some loss of information." According to the commenter, while de-identified information may still be useful, e.g., to show overall school progress, in the context of the Education Department, it is not clear these techniques are transferable to other contexts.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (9227) states that EPA links to a document of the Privacy Technical Assistance Center, Data De-identification: An Overview of Basic Terms, which provides a high-level overview of key terms and practices to help educational agencies and institutions comply with the Family Educational Rights and Privacy Act (FERPA). The commenter provides that this document addresses concerns with data disclosure that occurs "when schools, districts, or states publish reports on student achievement or share students' data with external researchers" and not to make underlying data publicly available for independent validation, thus, it is unclear that methods used to de-identify but preserve data for those purposes would be adequate in this context.
Response: The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Comment: Commenter (9227) states that EPA links to a NIST document entitled De-Identification of Personal Information by Simson L. Garfinkel (NISTIR 8053), which discusses de-identification, but not in the context of making research data publicly available for independently validating scientific studies. The commenter states that the document instead notes that "that there is a trade-off between the amount of de-identification and the utility of the resulting data" and that "[i]t is thus the role of the data controller, standards bodies, regulators, lawmakers and courts to determine the appropriate level of security, and thereby the acceptable trade-off between de-identification and utility." The commenter adds that it further notes that "de-identification approaches based on suppressing or generalizing specific fields in a database cannot provide absolute privacy guarantees, because there is always a chance that the remaining data can be re-identified using an auxiliary dataset."
Response: The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Comment: Commenter (9227) states that EPA's reference to the U.S. Census Bureau is similarly unhelpful. The commenter notes that EPA provides a link to a website titled Data Ingest and Linkage that details the U.S. Census Bureau's approach to linking data across many records they hold. The commenter provides that the Website links to a working paper that describes the method by which the Census assigns a unique person identifier that enables it to link records together to create the final file. According to the commenter, it is totally unclear how this process on linking together records is a solution that EPA could implement to protect privacy of individuals when disclosing data as it concerns how to identify data with specific people -- not protecting privacy.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (9227) states that, while other agencies are clearly grappling with the issue of how to make government-collected data available, they have also highlighted the many challenges in protecting privacy and confidentiality while doing so -- such as the ability for de-identified data to be re-identified --  and these agencies accept that there is more work to be done before these concerns are fully addressed. The commenter provides that the letter filed in this docket by the Presidents of the National Academies of Science, Engineering and Medicine underscores these difficulties, specifically noting the National Academies' previous work finding that "statistical analyses of data sets that generate highly precise results -- such as geographic specificity or other characteristics that identify respondents -- may result in privacy breaches . . . The commenter states that this presents a new challenge that federal statistical agencies are just beginning to address." The commenter contends that EPA does not even acknowledge, much less try to address, these gaps in agencies' abilities to protect sensitive data.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (9227) states that EPA cursorily mentions a range of options for facilitating secure access to confidential data, including: "[r]equiring applications for access; restricting access to data for the purposes of replication, validation, and sensitivity evaluation; establishing physical controls on data storage; online training for researchers; and nondisclosure agreements." The commenter expresses concern that EPA does not indicate whether it would deem providing access with these types of controls in place sufficient to meet EPA's proposed requirement "publicly available in a manner sufficient for independent validation."
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted or tiered access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (6875) states that, while it is reassuring that EPA believes that concerns about access to confidential information or private information issues can be addressed by common solutions, it is important that the agency outline a clear protocol that is transparent and clear. The commenter recommends that EPA solicit public comment on this protocol before implementing it agency-wide.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science. Further details regarding implementation of the rule will be developed before the publication in the Federal Register when the final rule is effective.
<IP_ADDRESS>.4 Legal, Ethical, and Practical Barriers to Making Data Publicly Available
Comment: Commenter (6915) states that the generally accepted professional practice for the collection of human data requires IRB review and informed consent from the individuals from whom the data are collected. Although the proposal states that "concerns about access to confidential or private information can, in many cases, be addressed through the application of solutions commonly in use across some parts of the Federal government," 83 Fed. Reg. at 18,770, the commenter asserts that this will not be possible for many studies because IRBs dictate the specific terms of this informed consent, including that the conditions of collection and analysis of human data be specified before initiation of the study. The commenter notes that these a priori conditions include the types of analyses that will be performed, how the data will be used, and whether and how the data can be shared. In general, the commenter asserts that a priori conditions preclude sharing raw data with entities not included in the original IRB approval and performing analyses not specified in the original IRB approval, even if portions of the data are redacted. Furthermore, the commenter adds that clinical data collected from physicians, hospitals, clinics, etc., may also be subject to restrictions under HIPAA, over and above IRB restrictions.
Commenter (9227) states that there are legal and ethical requirements that restrict making public the data underlying studies, including rules to shield private personal information, requirements to maintain CBI, situations where obtaining the necessary permissions to release data are logistically difficult or impossible, and situations in which researchers have made significant investments in developing datasets that they intend to continue to work with for future studies. The commenter states that not all of these barriers can be overcome, nor can they be overcome in every case. The commenter contends that, while there are ways potentially to address some of them, they can be extremely costly and burdensome, and/or may harm the prospects for further research.
Commenter (9227) states that, with respect to human subjects, there are strong legal and ethical privacy and confidentiality protections, which researchers are bound to respect. The commenter notes that, in some cases, researchers would be subject to civil or criminal penalties for violations. According to the commenter, in some cases, de-identification techniques are simply not applicable or still leave significant risk of breach of privacy. The commenter states that, for studies that had received IRB approval prior to finalization of this proposed rule, there may be no practical opportunity to make the data publicly available. The commenter contends that even for new studies going forward, it may be extremely difficult, require additional (often unavailable) funding for elaborate protective measures, or simply impossible to obtain IRB approval for protocols that would allow the data to be made publicly available.
Commenter (9227) states that, with respect to studies that have already been completed, there are additional formidable barriers to public release of underlying data and models. The commenter notes that, with older studies, simply finding the data sets and determining ownership may be expensive or impossible. According to the commenter, for older studies with human subjects, obtaining consent to release of data may be practically impossible, and the data may have been collected in ways that would make protecting privacy with release difficult or impossible.
Commenter (9227) states that, for some studies, administrative issues related to the data could be the most difficult barrier to overcome in providing for public release. The commenter states that larger and more costly studies are often performed by groups of researchers within a university, across multiple institutions, or across multiple individual companies. The commenter provides that over time, the data itself may become lost or misplaced, or it may become unclear who actually owns and controls access to the data. The commenter adds that academics move among institutions, companies merge and spin off, and the initial agreements were not always clear in the first instance. According to the commenter, obtaining consent from multiple institutional players takes extensive time and resources, at minimum, and simply may no longer be possible in some instances. The commenter contends that these problems are exacerbated with respect to human subject studies. Additionally, the commenter states that researchers are legally and ethically obliged either to protect the privacy of the individual study subjects or attain each subject's consent to share data. The commenter provides that this can be impractical for older studies and virtually impossible for larger studies, and extremely burdensome. For example, the commenter states that the ACS CPSII extended analysis by Krewski in 2009, which is central to PM2.5 national ambient air quality standards (NAAQS), was initiated in 1979 and encompassed data from 500,000 study participants who lived in 116 metropolitan areas. The commenter asserts that, for these types of situations, tracking down participants (or where the participants have passed away, their family members) to get consent is simply not realistically possible.
Commenter (9227) states that studies conducted on behalf of industry or with industry cooperation may contain CBI, the release of which could jeopardize a company's competitiveness. The commenter provides that, in some instances, researchers cannot make their data sets public without losing much of the value to the researcher of these laboriously and meticulously collected sets of information.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
<IP_ADDRESS> Support the Development of Methods and Technologies
Comment: Commenters (5165, 6141, 6150) support the development of methods and tools for making underlying confidential and private information publicly available while ensuring protection of that information. Commenter (6141) recommends that tools and methods include the following: requiring applications for access; restricting access to data for purposes of replication, validation, and sensitivity evaluation; establishing physical control on data storage; requiring the use of nondisclosure agreements.
Commenter (6877) states that a "one-size-fits-all" approach for access to regulatory data does not work and that a framework for providing varied levels of access to regulatory research data is essential. The commenter states that, as has been learned from confidential data centers providing access to data in a secure environment for some statistical agencies, such access is administratively complex, takes considerable time to implement, and is costly to both agencies and the scientists who use it. The commenter supports the use of such access to data across the federal government, but recommends it be implemented and developed independent of this proposed rule.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (6918) suggests that, for data that EPA neither generates nor funds, additional measures may be necessary to reflect EPA's values and achieve good governance. The commenter supports efforts, by EPA or others, to develop data-masking measures that lawfully and responsibly increase availability of data and models. The commenter provides that, as EPA acknowledges in the Proposal, there are currently available tools that EPA underutilizes in its rulemakings. The commenter states that there may be new or modified methods that may be appropriate and should be further investigated. The commenter suggests that EPA recognize that, in certain limited cases, the most appropriate approach may be to make the data and related information available to third-party, neutral investigators.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
<IP_ADDRESS> Do Not Support the Development of Methods and Technologies
Comment: Commenter (0671) does not trust any methodology and asks if EPA would retain legal liability for a data breach.
Commenter (1006) states that there is no good way to provide transparent data while still protecting the identity of those in the study. The commenter states that there are plenty of studies that show a person's identity can by revealed with a set of a few data points. The commenter asserts that it seems counter intuitive that by making the data public, it helps make better science.
Commenter (1012) states that the methodology and technology used to provide access to the underlying data would be irrelevant if the overarching policy precludes the use of certain data sets that would provide the most comprehensive information.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
<IP_ADDRESS> Recommended Methodologies and Technologies to Provide Protected Access to Data
Commenters request that EPA make efforts to make underlying data available and make several recommendations for methodologies and/or technologies to provide protected access to data.
<IP_ADDRESS>.1 General
Comment: Commenter (6155) agrees with EPA's goal of making data, as well as the underlying assumptions and options used in the modeling of data, both transparent and available. Commenter (6935) states that in any final rule, EPA also should balance strict adherence to new transparency requirements against the need for regulatory certainty and efficiency.
Response: The purpose of this rule is to codify internal procedural requirements for how the EPA will consider the availability of the underlying dose-response data that it relies upon to promulgate significant regulatory actions and develop influential scientific information.
Comment: Commenter (6375) recommends that EPA clearly define data processing/scrambling/analytical measures that are deemed sufficient for protection of privacy and confidentiality of study participants, protection of proprietary data and CBI, and other compelling interests. The commenter suggests that EPA, for example, could consider whether certain analytical processes can be compartmentalized to allow further de-identification of the participants and better protection of the privacy information.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (5185) states that additional guidance explaining how agency officials should advance transparency goals while ensuring protection of sensitive information under varied circumstances would be beneficial.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (4534) states that data released to the public should be published in a way that is easy for members of industry to obtain and understand. The commenter suggests EPA establish standards for how to publish data so users are able to easily understand how EPA arrived at their conclusions.
Response: This rule exclusively pertains to the internal practices of the EPA and does not regulate the conduct or determine the rights or obligations of any entity outside the federal government. This rule does not require any researcher or other outside entity to provide data to the EPA. Further, the final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (6155) supports a process for protecting data if release of such data may impair privacy, confidentiality, or national security. The commenter states that this can be accomplished in part by doing the following:
      (a) clearly detailing an objective and consistent process that would determine when release of data underlying pivotal studies would be deemed inappropriate in order to protect privacy, confidentiality, and national security, and
      (b) specifying conditions under which EPA could rely on studies for which the underlying data cannot appropriately be released, or released in ways that will protect privacy, confidentiality, or national security.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Comment: Commenter (6362) states that EPA might consider an approach followed under the Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA) where Data Evaluation Records of studies are made publicly available, but not full studies.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Comment: Commenter (0560) recommends that EPA's policy and regulations address each of the issues detailed by Noble Laureate Richard Feynman in "Cargo Cult Science" (1974). According to the commenter, the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or another.
Response: The rule does not categorically exclude studies, even studies where the underlying dose-response data are not available for independent validation, and therefore any impact on researchers who are developing science and deciding whether to voluntarily make the underlying dose-response data available is incidental. Instead, it governs internal agency procedures for determining the consideration to afford various studies according to factors that include data availability.
<IP_ADDRESS>.2 Privacy Data
Protection of Health Data
Comment: Commenter (6362) recommends that EPA ensure that it implements its final rule in a manner that enables it to use confidential health records that may exist with certain kinds of studies, such as long-term air pollution and workplace exposure studies that involve confidential health records.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
The final rule does not require the EPA or any other entity to collect, store, or publicly disseminate dose-response data underlying pivotal science.
Comment: Commenter (6362) states that, when protecting data while also promoting data access, NIH guidelines should be consulted. The commenter believes many of these guidelines could be applied in EPA's implementation of this proposed policy under each of the statutory programs EPA administers to ensure the guidelines adopted suit the specific needs of each statute.
Response: The EPA has finalized the approach that gives greater consideration to pivotal science whose underlying dose-response data are publicly available or available through restricted access. Restricted or tiered access in this final rule means that the underlying dose-response data are available through a data sharing mechanism, such as through an agreement with the originating author or institution, access to a refined or redacted dataset that anonymizes the more sensitive portions of the analyzable dataset, a restricted access data repository or secure data enclave, or some other mechanism (e.g., Data Use Agreements) that allows a qualified subject matter expert access to enough data to support independent validation while still protecting sensitive information.
Comment: Commenter (2479) recommends that, when researchers disclose their datasets, they should not black out personal health information; instead, they should group the results and the databases should then be accessed through a strict ID and password system.
Response: This rule does not regulate researchers or anyone outside of the EPA.