Well, hello everyone and welcome to the sixth meeting of the committee, the National Academy's committee charged with developing a long term strategy for lotus radiation research in the United States. Next slide please. A few administrative comments. When you're not speaking, please mute your audio. And just to remind everyone that the question and answer session after each of our presentations is really for Academy members and the staff members and if you do have a question, please raise your hand and I'll call on your name to begin the discussion. We are not going to be entertaining questions from the audience through the chat function today, but we will have a time beginning at four o'clock or public comment. And for those of you who are not committee members and wish to speak we hope that you'll limit your comments to about five minutes to allow time for everyone to make their presentations. Public comments are welcome during this period or anytime and so if you have things that you would like to bring to the attention of the committee. We will email them to a run your custody at the email address that's indicated on the slide here, and just to remind everyone the meeting is being recorded and will be available for viewing later time. Next slide please. Just to remind everybody that we hope to keep the information gathering sessions in informative informal and civil and so we appreciate your cooperation and making that happen. Next slide. I'm Joe Gray on the privilege to chair the committee. And again, our task is to develop a long term strategy for Lotus radiation research in the United States. Next slide. The distinguished committee comprised of individuals from with a variety of areas of expertise. They're indicated on this slide. And if you would like to find out more detail about their various backgrounds. Those are bio sketches are available at the URL that's listed at the bottom of the slide here. Next slide. The committee that is taking on this task was selected by the National Academy, and they were really picked to have the expertise needed to accomplish the goals and I'm sure that the Academy and the committee have reviewed conflicts of interest, and I think that. So far, we have dealt with those as appropriate screening for conflicts of interest continue throughout the life cycle cycle of the project and so for the present committee the academies have judged that the committee is free of conflict of interest. We do have the appropriate range of expertise for the task, and if they have a balance of perspectives so that we can carry out the charge objectively and credit. Next slide. We received our charge through the consolidation appropriation act or 2021, which really directed the Secretary of Energy to carry out a low dose research program, both low dose and low dose rate, really to understand to increase the scientific understanding of the uncertainties associated with the effects of exposure to radiation and to improve risk assessment and risk management methods with respect to radiation. And in particular, the Secretary was directed to enter into an agreement with the National Academy of Sciences to develop a long term strategic and prioritized research agenda, and that is what this committee is charged with doing. So that's where our charter came from. Next slide. This has evolved into a statement of seven tasks that the committee is addressing, defining the health and safety issues identifying current scientific challenges, assessing the status of current lotus radiation research, recommending long term strategic and prioritized research agenda, defining the essential components of the program, addressing aspects of coordination, and identifying potential monetary and health related impacts to many different interested communities. We are well along in that process and the exercise today really is part of our information gathering exercise we've already heard from over 70 presenters, and we'll end up varying from about 80 such individuals before the committee concludes its work. Next slide please. We are on an aggressive schedule. The enterprise was started in March of this year. The committee was selected in the May, June timeframe and we began work in July, and we're well along in the process of discussing the task and gathering information and then beginning to draft a final report. We hope that the final report will be available in February for review by the academies and the defendant committee, and that the report will be released sometime in April of 2022. It's worth noting that when the report is released, there will be a 15 day public comment period where anyone is welcome to read and comment on the report, and that information will be made available with the final report when it's released. Next slide. The report review process is actually quite rigorous. It's a hallmark of the national academies and it really distinguishes the academies for other organizations offering advice. The report is intended to be independent of the sponsoring agencies, and it will be reviewed after it's prepared by our committee by an external and diverse group of experts, and the reviewers are going to be asked to determine whether or not the evidence and the reports that are presented in the report are fully responsive to the charge and are properly supported. The names of the affiliations of the participants in the review process are made public when the report is released, but their comments individually will remain And I think it's very important for everybody to appreciate the fact that the sponsors of the study do not have an opportunity to see the report during the process or otherwise influence its content. So it is done with their support, but not with their guidance. Next slide please. So there may be comments that many of you will have or suggestions about the other things that we should be thinking about. So the best way to bring that information to our attention is by contacting Ronnie Acosta whose email address and phone number are listed on this slide. And so with that, I think I've made the introductory comments and we now need to turn to the information gathering part of the program and our first exercise is to hear from a panel of three speakers who will talk to us about lessons that they've learned in research from other areas, particularly in the area of air pollution but not limited to that. We have three panelists, Daniel Kruski from the University of Ottawa, John Samet from the Colorado School of Public Health and Menechi from the Harvard DHH Chan School of Public Health. They will each present remarks for about 15 minutes and at the end of that we'll have a discussion of all three presentations. So burning questions ask as you go along but otherwise we'll hold the discussions to the end. So with that, let me turn the podium over to Daniel and ask you to present your remarks. We really appreciate your taking the time to join us today. I have unmuted myself, thank you. Let me begin just by thanking Dr. Gray and Dr. Costi for the opportunity to offer some of my own perspectives on development of a long term strategy for low-dose radiation research in the United States. So what I'll do is talk to you briefly about some advances in risk science that may be of general relevance to your charge. I want to bring in some new results on key characteristics of human carcinogens, particularly radiation agents. I'll talk a little bit about some of our epidemiological work on ionizing radiation in medical contexts, occupational and environmental exposures. A project we did on radiation hermesis, looking at over 800 animal experiments on that and finishing up with a couple of comments on the importance of evidence integration and value of information. So some of the advances in risk science that can be helpful to us if we go back to the NRC report in 2007, which I had the privilege of chairing toxicity testing in the 21st century, lay out a bold vision for how to apply new toxicity testing approaches to better inform risk issues of chemical and radiological agents that was followed by, that became known as TT21C. So I'm going to name the 2012 follow-up report on exposure science ES21C, exposure science in the 21st century, which looked at new ways to characterize exposure. And the most recent contribution that Dr. Samet shared on how far we've come since 2007 and how can we use the best available science to do the best job in risk assessment. A parallel activity that I worked on with the US Environmental Protection Agency was a project to develop a framework for the next generation of risk science. This was a large project with extensive consultation. The two main outputs that I would point the committee to is our 2014 paper in environmental health perspectives and I'll show you the paradigm that we developed on my next slide for that. And then there's a follow-up more detailed report, which is the synopsis of the entire EPA report on this initiative. The framework that we developed for the next generation of risk science is shown on the left panel here. It incorporates all of the new science that we had been talking about over the last decade. It brings in a advanced risk assessment methodologies, the 2009 NRC report on science and decisions being a prototype for that. We also took a population health approach, looking not just at one agent and one outcome, i.e. radiation and lung cancer, for example. But all of the other determinants of that outcome and how they might interact with the agent of interest. This is a population health approach that we pioneered at the Ocloctin Center at the University of Ottawa. And then multiple interventions of a regulatory and non-regulatory nature to manage risk is also part of the population health paradigm. I want to draw the committee's attention to the problem formulation step. This has become a key component of modern risk assessment paradigms. And also to value of information, which I'll talk about in my last slide, second last slide, what value is additional information that we might gather to meet the charge at hand. This is a framework that I've adapted from a nice paper by Dr. Mel Anderson published in Altex a couple of years ago on how to use the new scientific approaches, so-called new approach methodologies. It's kind of tax-automized into four levels, different applications. So this could be useful background information for determining if any of these approaches would be relevant for the program that's being developed at the present time. My next topic is key characteristics of human carcinogens. It's a very nice publication by the International Research Agency for Research on Cancer. I am excited to be publication number 165, which has a lot of detail on what we've learned on mechanisms of human cancer over the last 50 years. So I've built on the 10 key characteristics of human carcinogens articulated by Marcus Smith. What I did in my group at the University of Ottawa was evaluate these 10 key characteristics by going to fundamental biological events, topological indicators that would represent these key characteristics for every one of the group one carcinogens, agents that IARC has determined to be clearly causes of human cancer. And we've shown the distribution of key characteristics in this slide for various types of carcinogen. So radiation is shown in the bottom left. These are the 10 key characteristics. And 100% of the radiation agents are showing multiple key characteristics such as genotoxicity and altered DNA repair rates. And the pattern of key characteristics for radiation agents is notably different from that for chemical agents shown in the bottom right. And the pattern for chemicals coincidentally is somewhat similar to that for pharmaceuticals which are also chemicals. So these key characteristics could inform dose related assessments that the committee would be interested in. A couple of brief words on exposure to ionizing radiation. This won't be new to the committee, but I want to make a couple of points. Medical exposures, particularly CT scans and environmental radon constitute a very high portion of our total annual exposure to radiation. And I'm wondering whether these two sources of exposure radon and medical diagnostic procedures, particularly CT, could be rich sources of data for epidemiological risk assessment. I'll point the committee's attention to a recent paper which does use medical records to document an increased risk of leukemia and brain tumors associated with pediatric CT scans. And I'd also like to generalize this theme that the use of electronic health records can really be a very useful way of identifying risks in real, under real-world conditions, large populations at low exposures. We've had great success in using a database assembled by CERN Corporation, which now contains electronic medical records on over 100 million patients in the US going back in some cases, 10 or 15 years. So this is a paper that we just published this year on a pharmacovigilance issue, looking at quintalones and acute liver failure, but the same sort of analysis could be done with these big databases and others as well, looking at radiological exposures used in medicine. I want to highlight the importance of radiation dose registers in radiation risk assessment. We have a paper which talks about this in general going back some time when I used to work very actively in this area. We now have done two analyses of the National Dose Register of Canada. The first one on the top slide is on cancer incidence. The second one is on cancer, sorry, the first one is on cancer mortality, the second one is cancer incidence. We now have almost 800 or 900,000 people with measured exposures to radiation in occupational environments comprising well over 80 different job categories dating as far back as the 1950s. If you can link that individualized exposure data with individualized outcome data, which we did, first to the Canadian mortality database at the national level and second to the Canadian cancer incidence database at the national level. We were able to establish associations between occupational radiation exposure and a number of different types of cancer. And with the large sample sizes that you have here, the measured exposures, the well-determined outcomes, this can really be a fruitful line of epidemiological investigation for risk assessment purposes. I also want to mention some work that we've done on the National Dose Registry using biologically based radiation risk models as opposed to empirical or statistical models. So we took the simple two-stage problem expansion model of carcinogenesis and fit that model to data from the National Dose Register. It's summarized in this publication. And the thing that I want to draw attention to is this is a nice way to try and understand dose rate effects because we were able to compare long-term low-level exposures in occupational environments that was recorded in the National Dose Registry of Canada. With higher and almost instantaneous exposures experienced by the atomic bomb survivors and show that this two-stage model that we fit to the data from the Community National Dose Registry actually showed a compatibility between these two data sources after adjusting for dose rate effects within the context of the biologically based model. I've also spent some time wondering about radiation or mesis. I've heard lots of presentations on this, listened to different perspectives, and I thought, well, why don't we take a look at just what the empirical data have to say on this issue. What we did that is we assembled a database of all of the animal experiments exposed to radiation, involving radiation exposure that we could find. And we ended up with about 800 experiments with different types of radiation in different species by different routes of exposure. So it's a pretty big database that was a labor of love to put this together over a period of years. So we went through and analyzed the data and asked the question in a meta-analysis, do we see U-shaped dose response curves with a decrease in risk at low doses followed by an increase in risk at moderate to high doses, more often than we would expect by chance. And it was, well, maybe, maybe not. There was actually little empirical evidence that was convincing. But at the same time, because the number of data points in the very low dose region wasn't as great as it would have liked, there is a question of whether we got enough power to really address that question empirically inadequate matter. The other example I wanted to turn to where we've had some successes with low-dose radiation exposures is residential radon. And we were aware of radon a long time ago and did one of the very first large-scale case control studies on radon in homes and lung cancer in Winnipeg, Canada, which is the city with the highest average radon exposures, which is why we chose it. Large case control study, 1,500 subjects, extensive exposure monitoring. Every house that every subject lived in throughout their entire lives, cases and controls, was measured with one-year integrated CR-39 alpatrack detectors. At the end of the day, we found no clear association. We put that data, however, with six other studies from the United States and a combined analysis, and we actually were able to tease out a significant increase in lung cancer risk at residential exposure levels ranging from 1 to 200 per cubic meter, with the combined data showing the power of pooling data from multiple sources. And I can talk to you afterwards if you like about why we didn't pick this up in the first study that we did in Winnipeg, why I think that's the case. And one other point about this slide is if we look at all of the data, and then we look at a subset of the data where we had our best exposure measurements, more complete exposure measurements, we get an increase in the signal. And the risk estimate goes from about 11% excess risk per 100 per cubic meter to about 18%. And you'll see this is a theme that adjusting for exposure measurement error can actually sharpen estimates of risk in these kinds of studies. So a lot of questions I was asked to talk about is lessons learned from other endeavors such as air pollution epidemiology. We worked a number of years ago on a reanalysis of the famous Harvard Six City study and a similar study based on the American Cancer Society cohort, which has a million people followed for 25 years now and what we did was we developed smooth spatial representations of air pollution surfaces across the United States and use those ecological exposure measures to actually estimate the risk in the ACS cohort where we had individual data and we could adjust for individual covariates like tobacco smoking and others. And we were able to identify a clear risk with these ecological exposure metrics associated with air pollution. I have a summary in this short note in Nijm which shows that no matter when we looked at the data which exposure metrics that we used as we got to more and more refined exposure surfaces, we're still getting very consistent estimates of the risk of mortality due to all causes, lung cancer, cardiovascular disease. So this approach showed really consistent estimates across different spatial modeling techniques and different data sets that we used in our analysis over the years. Well, we actually tried this with the ACS cohort with Radon. We took county level measurements of Radon from the US Environmental Protection Agency and married that with the ACS cohort where we had the individual data. And this paper shows that we were actually able to identify the same signal. This is the Turner paper in the bottom here, a 15% increase in relative risk shown in that study which is very consistent with the pooled North American data, a similar pooling in Europe done by Sarah Darby with over 20 case control studies. So you're saying and with the underground liners that were analyzed in Bear 6 as well. So you're seeing a consistency across diverse sources, some of which had only ecological measures of Radon exposure as in the study by Michelle Turner shown here. I also show two studies which were able to adjust in some way for exposure measurement error. I showed you the North American pooling where the risk went up with adjustment for exposure measurement error. And a similar thing happened in the European pooling and the adjusted risk estimates are very compatible between those two data sets. Time check, Rania, how am I doing for time? I think you have another two minutes because you started a bit later.  Okay, so I'll be really quick here. Extrapolation of the minor data which we did in Bear 6 down to residential exposure levels shows very good compatibility. And I'm actually rated my last two slides so my timing is not too bad. The first point I want to make is evidence integration is a big theme now in risk assessment. We're going to integrate evidence from multiple evidence streams and also from multiple data sets within those evidence streams to come up with the best possible overall estimate of risk. This is emphasized in a paper we have in press and all text, and it was also emphasized in the 2014 NRC review of the US EPA iris process. Before I give you my bottom line on some suggestions for your consideration is some recent work that we've just completed on a framework for value of information. One paper is out in risk analysis. We have another one that's under review. And these two papers combined give you a very detailed framework for looking at if I were to go out and collect this additional data. How much additional information would it buy me with respect to achieving my risk assessment objectives and this might be a useful way to approach what collecting data under a proposed Lotus radiation research program might actually contribute to reducing the uncertainties and radiation risk estimates. The last slide is for four takeaways. From my perspective, epidemiology continues to be a very important source of information for Lotus radiation risk assessment. We live in a world of big data where both radiation dose registers and big databases of electronic medical records can be exploited to our advantage. The two fundamental advances in biology and toxicology have led to these a wide array of new approach methodologies. The Anderson paper I cited at the outset is a nice place to learn about those. My final point is evidence integration across multiple evidence streams and data pooling within and across evidence streams can really lead to some informative conclusions that you might not get by looking at individual data sets. And with that, I'm finished. Thank you very much. That was really helpful. We appreciate it. We'll come back to the Q&A's at the end of the presentation. But with that, let's move on to John Samet. John, thanks for joining us. We appreciate your taking the time. Okay, and thanks everybody and actually looking at who is in the audience I see no need to give the talk since most of you have lived through much of what I'm about to say. But here goes. And so I'm going to talk a bit about air pollution and radon and Dan and I and Francesca have been partners and a lot of work on these topics. So this is the starting point and I really think this is critical and thinking about research is to define what the question actually is. You know, having lived through these questions and sort of seen the science policy interface. I think it's really important from the outside to say well what is the question that's going to be addressed because it's critical for design it's critical thinking about sample size needs the consequences of measurement error and more. So here's some basic questions that are familiar to all I think Dan already got at this question of shape when he talked about hormesis, of course, you know how low our risks documented. Is there a hazard a quote low dose levels, and so on so I think up front it's important to define what the actual question is. And I think the, that I'm going to emphasize throughout this question of low dose, often relates to some inherent policy decision. And that begins to shape. What is the science need to address the policy question and what is the related degree of certainty that's needed. This is from a book from Robert Proctor from long ago that I still like that sort of gets at this question of the policy implications different forms of those response relationships, particularly as we go to lower lower levels that may have very different implications. And for example, the Clean Air Act as originally written around the criteria pollutants reads as though there was an expectation that in fact thresholds would be found that would be quite convenient for setting levels for the national ambient air quality standard so these different curves have very different implications. So very often our studies may be directed at addressing really to two things what is the shape of the curve, and what is the relationship between dose or concentration or exposure, and risk at some levels of policy policy interest shown here. Now, also for impact analysis this becomes very important if we're calculating the potential benefits of a lowering of exposures or dose than what curves are used has important implications. So with that, let me turn to the example of air pollution. First, here's the policy mandate for the criteria pollutants at least that is posed by the Clean Air Act with a heavy burden placed on the administrator of EPA to set a standard the primary ambient air quality standards, and the main minute and maintenance of which in the judgment administrator, based on such criteria criteria means evidence, and allowing an adequate margin of safety there's that policy message is requested to protect the public health. Carol Browner in Time Magazine, when she promulgated the 1997 PM 2.5 standard the fine particle standard that was science based, but controversial and again came with very important policy implications. So over time, in fact, driven by science, the level of our standards has dropped substantially the indicators have changed from total suspended particulates down to far more refined measure. And again, I'm thinking about looking at the impact of these changes what curve we think we're on makes an important difference, but also makes an important difference in thinking about, say, if there were a further reduction of this is the annual national ambient air quality standard what the benefits would be clearly depend on which curve might be the appropriate one. Now, here is the London fog of 1952. You're well aware of this wake up call event that happened that killed thousands and thousands of people and you can see the death number of deaths, the levels of sulfur oxide and smoke you can turn that into particles in your mind you know 100 fold higher than what we might see in the, in the US. And he sophisticated analysis to decide that there's probably a relationship between air pollution and deaths, and there are in fact 10,000 excess deaths roughly that's the number still debated. There's 47 data points on the slide. And little data can be powerful. Of course, especially when it's like a hammer, hitting public health is in this as in this case and of course, now we're talking about billions of data points and some of the larger data sets to try and tease out these signals. So the challenge here and this goes back to work. When I was at Hopkins with Francesca, the minute to you who you'll hear from next and Scott ziger, as we began to try and sort out the question of, how did we find the daily mortality signature imprint of air pollution in the face of noise that came with the black the annual time of death rates, which is quite substantial in the variation with temperature to pull out the smaller signal of particulate air pollution. And this is the challenge that remains in the air pollution world that we've been addressing with ever larger data sets and more sophisticated methods. Okay, this paper was published 40, 40 years ago. It was one of the early time series studies of morbidity, looking at air pollution in Stephenville, Ohio, a very industrial city, and time series of hospitalizations and of emergency room visits. The levels of particles were very high at the time these are 24 means values 24 hours up to 700 which would be unimaginable these days, hopefully high levels of sulfur dioxide, and using at the time very methods would be very considered very primitive. This is the discussion approach that I took, and the data set was handed to me after being collected in Stephenville on cards. So again, little data and a signal was found. Now, over time, the questions became. Were a mortality air pollution relationship, what was it as the national ambient air quality standards had been lowered, had there been a had that relationship remained. There was a very large review published in the American Journal of Epidemiology. In the eight, when particle levels were still quite high that said the air pollution mortality relationship would never be useful. Again, probably one of the most incorrect statements written about the air pollution health relationship. So here's our team that did the so called national morbidity mortality and air pollution studies Scott Zieger and and Francesca we are perched on a very hot roof of a parking garage in Baltimore when this picture was taken. But what we did in this and Dan was later part of putting together time series studies internationally was to say, we don't need to take just one city at a time. This was over 20 years ago that data data management capabilities and analytic capabilities would allow for taking essentially all the data from the larger cities, which is what we did in this study we would have our best shot at finding the finding the signal. The, this is from one of the important papers published from that study this is the Bayesian pooled estimates of the effect of particulate matter on daily mortality, with an adjustment for different pollutants essentially showing that the PM effect was robust to consideration of other pollutants and as I recall data here perhaps pooled from about 17 million individuals which seem like a lot at the time but I think in this presentation you'll hear about much larger studies but as as the levels have dropped fortunately we have used larger larger studies to look for signals with this approach we were able to look for heterogeneity across the country this slide showing the estimates for different cities and regions. The question of what may be happening at lower and lower levels of exposure has become important as the distribution of exposure has shifted downwards in some countries but not all as you can see here. The Health Effects Institute has been funding a series of studies who call low level studies that are brought together very large population based administrative data sets like Medicare various European data sets and in Canada various studies. I think importantly you can see that these span a level of particulate matter PM 2.5 that is quite, quite low and these studies again are continuing the show significant associations at these lower levels. Kind of information drove the recently released WHO air quality guidelines, which for a number of pollutants like PM 2.5 were reduced from the values within the 2005 guidelines and this was based on extensive reviews, on the synthesis and meta analysis and looking at where the, where the effects were seen statistically significant in the various studies. And they continue to do a lot of work on air pollution using epidemiological approaches. This is a simple crude PubMed search of papers on particulate matter and mortality so there's lots to pull out there. And these studies are being done now around around the world providing a better picture so again there's an opportunity in air pollution to put multiple data sets together, as done here for example in support of the WHO. I'm sorry this is the EPA slide this is EPA data from the 2018 integrated science assessment. Very similar effort done to support the WHO air quality guidelines and again with the finding of generally significant associations in some very large studies at what would now be considered low but our contemporary levels of exposure. The lessons from going low, I will say low is ever lower. The demand for certainty has increased and that again relates back to the policy implications. There's work on subgroups of interest of course looking at those who are most susceptible and vulnerable. The research go lower, bigger study populations, many more studies pooling of different sorts and enhanced analytical methods and I think a willingness of the research community to participate in these pooling exercises. I'm just going to move on and say a few words about Radon because I think it's a useful contrast and then already alluded to some of the issues here in particular the major policy question has been, what are the risks of indoor Radon and what guideline conditions should be used for mitigation this is the home of Stanley watchers identified in the mid 80s. It had Radon levels as high as those in underground uranium mines and the question became then what were the risk for occupants of homes in general since we quickly learned not surprising that Radon was present in homes in general. So the policy question was how much should we worry about Radon and at the time and certainly into the 90s this topic was very controversial. The challenge was that in looking at the distribution, it was log normal so that some people were at very high risk out in the tail of the distribution, the population exposure in general was being driven by the lower end so there was a need for some certainty about risks. So Radon like the London fog there were these there were dramatic events. This is a paper I published in 1984 essentially showing that in the Navajo uranium miners who were largely non smokers. The lung cancer was essentially all caused by work in uranium mines and this is a picture just showing the distribution of our cases who were clustered around the sites where the historically been underground mining the relative risk here was infinite. A comment about how these risks have been looked at the biological effects of ionizing radiation reports, then alluded to beer six have addressed this beer for published in 1988 led the. And working group within the beer for committee, and we developed with particularly expertise from Don Pearson, Jay Lubin age and time dependent risk models that use data for beer for from four different studies. The next step was the pooling effort of the studies of underground miners exposed to Radon, this is our pooling group, and this led to a monograph published by the National Cancer Institute, and then the beer sex six effort where Dan and his colleagues had a lead role in much of the modeling beer six I think importantly adopted a linear no threshold model on a strong mechanistic basis. And again here I'll just emphasize the importance of integration of mechanisms and thinking about them at the lower doses in the case of rate on the energy that an alpha particle in parts to the respiratory epithelium is a property of nature it's a decay it's radioactive decay, and it's not dependent on what the inhaled concentration may be so there was a strong synergy, if you will tie in between what is mechanistically and the modeling. And again, you saw something like this restricting the analysis to the lowest levels of exposure in the miners. We found this relationship roughly linear extending down the low doses. Some picked out this point said aha, hormesis but as you can see there's wide confidence intervals. The next step in this story is led by David Richardson a committee member the pooled underground miners analysis project that is underway and that now gives another more powerful look, particularly with the addition of the German cohort to the studies because these are all uranium miners, and don't include the others. I make one other note. When we did beer for the indoor rate on studies were just emerging. And we realized Jay Lubin clay one Claire Weinberg and I wrote a paper saying essentially that none of the individual studies had enough power, and that pooling would be needed so we actually supported a set of workshops to bring the investigators together. So the data sets could be harmonized and support we hoped ultimately a global pool but the North American pooling that Dan mentioned, and the European pool led by Sarah Darby. Low doses a loaded term if you will. And it's one that is used when there are often substantial policy implications that finding intense scrutiny is likely for the findings related to risk at low doses and issues of uncertainty measurement error and its consequences, residual for epidemiological studies, often become critical and often highlighted by those who are critical. I think mechanistic understanding is important around plausibility and I think useful counters are right on where essentially there's a single agent alpha decays in the air pollution where there are a complex mix of pollutants breathed in PM 2.5 itself is a very complicated and highly varying a heterogeneous entity. People talk about mechanisms, inflammation and more but with rate on the mechanistic basis is far simpler and it's, I think far easier to support particular form of the dose response curve at lower doses. So with that, I will end. Thank you. Thank you, John. Very helpful. As always, we appreciate it and we'll come back with questions in a bit. Let's move on to Francesca at this point for the third of our presentations and then we'll have some Q&A. Francesca, thanks for joining us. Sorry, but by this time I think I should know how to use zoom. Still, right? Okay, so thank you again for the opportunity. And I think, you know, John and then give it a perfect background actually what I'm going to take. So my, what I'm going to talk my point of view would be a little bit more into the data science and statistical issues of estimating exposure response function. And especially when the focus is into the low dose. So these are really the questions that I wanted to try to address in my 23 minutes that were provided to me. And so really, really focusing with respect to the challenge of the opportunity of the methodological approach and especially in this new era of data science. So so these are really the questions that you guys, the committee provided to me. So I want to start, which generally don't, but this time I'm going to start with an equation. And I'm going to walk you through about what are some of the key challenges when we are trying to estimate an exposure response function in any field, whether it is radon radiation air pollution, anything that you can think about right. And so most of the most, you know, I would say the traditional statistical approaches for estimating an exposure response function, basically assume that you have a function f of an exposure, let's say, x and you know we model that other linearly. Similarly, if we assume that as john say that the only interest that we have is whether or not there is a risk, or non linearly right if we want to see whether or not there is a threshold or whether we want to submit a shape. And then we add the several terms linearly or non linearly to adjust for confounding right where they that could be the confounder and I will say that that is what probably, you know, there are probably thousands of different formulation of this equation, but I would say that that's kind of the traditional way of an estimating exposure response function. And so I wanted to point out some really very important challenges that actually we're just starting to address in the literature. And these are really important challenges because if we get any of these four points wrong. So what estimates of whether or not there is an effect on low dose will be wrong. So first of all, the confounding adjustment that we assume in many of our analysis might not be additive and there are many examples where the dis element of the exposure response function this part might be just around our model could be wrong. We might specify F, but our exposure response function in a way that we might not be able to detect the threshold because we assume that is smooth, which is what is done in most of the analysis. There could be different confounders at the high level of exposure and a low level of exposure and this is actually something that we have seen in the inner pollution data and I will guess that something that you might be seeing most of the situation so when you're trying to estimate whether or not there is a risk of low dose. It's possible that the confounding and the type of confounder might be different when you're looking at the estimation at the high dose. And then this is something that then has been mentioning and you know this is something we're a little bit more familiar that could be error in the exposure and the confounders although it's not completely clear still how we account for that.  And then there was a commentary that I wrote we call it Ziggler where we really talked about causality is basically what's what's causality. Okay, so it's really a better way for estimating the relationship between a change in exposure and outcome that is less sensitive to model And so let me give you clearly a very simple example in the context of a cartoon right so we have actually let's consider for a moment the six city study right you have stew and bill Ohio which has a high level of exposure and portrait which is a low level of exposure. And so when you're trying to compare these two, these two cities, these two cities are different right so stew and bill hypothetically have a larger number of people that have a low education, and the large number of people that are smoking that have a higher level of exposure. Right, so this is this very hypothetical so what we do is when we want to just for confounding, we just add into the model a linear term for education, and we add into the model linear term for smoking. When these two confounding adjustment as often it is are no linear, our estimates will be wrong. And so the cause of inference open the door, then instead of adjusting for confounding linearly, we can do matching. And how does it work. Well, it's very simple, especially if you have a ton of data. So for example, let's say that you take Jane, Jane lives in swing below Ohio, Jane has a low education, and she's no smoker. Look, we have the ability to measure the life expectancy of Jane. And so what we need to do is we need to estimate what the life expectancy of Jane would have been if Jane would have lived in Portage, and how you do that. It's actually pretty simple you go to Portage, and you find all of the twin sister of Jane, you find all of the women that have the same age as Jane. They are also low education as Jane, and they're no smoker on Jane. They, you estimate the life expectancy and that is what we call the counterfactual. So, technically, and, and conceptually it's not that difficult, but by doing exact matching. It's an opportunity to estimate an exposure response function and to adjust for confounding that it's not relying to model assumption, and that can make a huge difference, like giving the right answer the wrong answer. The issue of matching the cause of inference in the context of continuous exposure now is becoming almost like a mainstream and by the way there is a national Academy of Science committee which I'm part of, which is also assessing causality from a multi disciplinary evidence set in the national under quality standards so the field of cause of inference has been around and out for 40 years. And I think with the richness of data and expertise is probably, again, it's not the perfect, the perfect world, but it will allow us to not rely more on very strict model assumption as we did before. We have done that and we have used cause of inference methodology for estimating the exposure response function in our course that is for the Medicare participant we have 67 million people from 2000 2016 we have daily and one kilometer to one kilometer grid estimates of exposure to PM 2.5. And we actually this paper, if you're interested in digging into it, we estimate the causal exposure response function between fine particular matter and mortality in a totally specific way using matching so we'll be very, you know, no less not sensitive to model assumption. And by the way, there is also the code and the statistical package that is fully fully available. We don't have much time to go on the detail of the studies basically it's a big extension of what, you know, was the national mortality morbidity air pollution study because we follow individuals for all the continental United States for 16 years. And we consider several potential confounding. And so, going back to the equation that show you instead of adjusting for confounding by adding these terms into linear model we do exact matching and we do exact matching using the generalized propensity score. So I think it's really important about a causal inference methodology and I want to spend a little bit of time on this plot is that you can actually visually assess whether or not you have properly adjust for confounding. So what this plot in in in the left to shows shows so they read dots are the absolute correlation between your continuous exposure PM 2.5 or radiation it could be, and each of the potential confounder right so if the correlation is above 0.1. It means that the exposure is correlated with a confounder so there is confounding. So then what we do we do this exact matching idea that I show you. And then we create a new data set where remember Jane is to unveil Ohio is matched with many genes, they are important Oregon they're the same as Jane except they have a different exposure. And then match data set in this match data set, we break down the correlation between the confounders and the spojores so that means that you have if you know effectively and in no parametric way, adjusted for confounding so these are the shape of the exposure response function that we estimate in the context of the Medicare and PM 2.5 study. So we have two shapes of because one is used with a computational or computational efficient approach, then, then the other. So, both online is exact matching a non parametric estimation of exposure response function I think has a lot of important features, because it is special in the context of big data to really not rely on string model assumption for confounding adjustment. The other thing that I want to point out is that often one of the goal is to identify a threshold or identify a change point and there are now new data approaches we have a paper that we're going to publish soon that allow you to from an inferential standpoint to to really identify the change point and quantified how much uncertainty you have when there is a change point. This is a toy example this is just a toy example this is simulated data where the true exposure response function is the blue curve right so we have some hypothetical and the true change point in the exposure response function. In developing new methodology that then will be able to estimate the exposure response function and the confidence involved but what is important is that we can calculate the derivative of the exposure response function, which is at this panel here, and the uncertainty around the derivative does not cover zero, it means that there are change points. So, there is actually interesting that often and for 40 years we're talking about change point or threshold the detection, only recently we are really at the point to where we can quantify where and how much evidence we have on the threshold and so we have done that we haven't published this this is really literally work we are doing now for the Medicare study where we have estimated is non parametric exposure response function, we now have calculated the derivative of the exposure response function and you see that the derivative of the exposure response function and the confidence band doesn't touch zero around 12 microgram per cubic meter really identifying now with uncertainty and with statistical significance where there could be two change point in the exposure response function and so that's important because instead of visually look at exposure response function and see oh I think there is a threshold that is actually an inferential procedure that allow you to do that. The other thing which I'm not going to go into detail but we have published this work is that when you have this hypothetical exposure response function, remember and just keep in mind that you might have a different amount of confounding and different type of confounders when you are when you are looking at the low dose versus the high dose. And so do not assume or give for granted that you have the same mechanism of confounding bias at the different levels of exposure we had the paper I give you all the reference where we actually have identified these in the context of a pollution study and so I will guess that that's a common problem in other situation. So to to wrap up, what are some data science opportunity and challenges that low dose radiation research couldn't explore and how. So I think there are a lot of opportunities in terms of non parametric estimation of a causal exposure response function and again the world calls earlier means our ability to adjust for measure confounding bias that does not rely on strict modeling assumption, which are often incorrect. Now I didn't have a chance to tell you but there is all it is a really important field that when I say I'm going to match Jane with her twin sister. You can match Jane with women similar to Jane except for the exposure by actually using machine learning methods, and the estimated propensity score and again these are being shown, and there is a stensibly church in the data science community that are seemingly effective. And then you can assess covariance balance you can really visually see whether or not, when you are estimating the causal effect of a continuous exposure you have indeed eliminated measure confounding bias. There are still a lot of challenges I think that the issue of how we propagate the exposure error in the summation of exposure response function. It's not straightforward. I know there is a lot of statistical literature on this, but honestly, I think again, it rely on a lot of strict assumption and so we are doing more work on this, that it's really more reflecting the real world of data science and not strict modeling assumption or strictly mathematical assumption that everyone gives, you know, considered that are correct one. The confounders could be different at low level versus high level of exposure. Just want to mention that we have we develop in additional methodological innovation packages, software packages and so when I tell you about estimating a causal exposure response function non parametrically. We actually have submitted on cram a in a package that can be used, of course for any, any field that any application. So, these are some of the references I think if you wanted to get a sense of the key ideas of causal inference. I think it's a great starting point. The work with Georgia Papa Papa the gorgeous is the work where we deal with different confounder at different level, the work with both both Georgia Papa the gorgeous and and a job will work PhD student in my lab and Joe developed this approach for estimating non parametrically and exposure response function, and then also with boy ran as the way of quantifying evidence statistical significant evidence of a threshold by looking at the derivative of an exposure response function. Thank you for your attention. Thank you very much. Excellent presentation we appreciate it. I think we've come to the Q amp a part of the session so let me open the floor to my committee members to see if we have questions. And please raise your hand. Interesting. Well, one of the things I'll go ahead and then I'll come back to the question. Thanks. Thank you everyone for the really really helpful presentations. Thank you Francesca I wanted to pick on you perhaps not surprisingly. One thing I struggle with, you know when I hear talks about causal inference and machine learning and, you know, measurement error methodology is, how do we put all of things together into one coherent approach to do the analysis that we want to do.  How, how do we do it in practice. So yeah I mean it's it's hard. So first of all let me say that I think that we are now at the point where we can estimate non parametrically and exposure response function that has a causal interpretation, and we can quantify evidence of a threshold. And we are there and we have three publication from my lab that and the software package as well so that you can take the software package and apply right. I think how we propagate exposure error in this framework, I think that it's it's still it's it's complicated I don't think we are there yet. I think we will be able to overcome the issue probably in another year or two. I don't think we are yet have a framework where you take all of the challenge exposure error, and then propagate it but I do think that it eventually we will get there and I think what I want to encourage you to committee is not, you know, to be like to open your eye or what's happening, and not to stick with what we have been doing 20 years ago because I don't think that what we're doing now. Again, it's not perfect but I do think that we're making a ton of progress. In machine learning, it's extremely effective when you want to do matching, because if you want to find another person that is identical to Jane somewhere else, and you want to match with respect to all of the characteristics. You can take machine learning to estimate the score that will make them as similar as possible and matching that score and that can be also extremely, extremely effective. It's not the solution for everything, but I do think that we're making a ton of progress I think the exposure error though it's not completely integrated into the framework. Yeah, thanks, thanks. Dan, do you want to comment on that. Just very briefly, I had some slides that I took out of my presentation because they were based on some work on non ionizing radiation, but I will send Rania a paper we have in the American Journal of epidemiology a few years ago on multiple bias modeling in epidemiological studies where we can adjust simultaneously for systematic and random errors if you have some way of calibrating those that might speak partially to Ben's question. Thank you. Simon question. Yeah, thank you. So I think this is mainly directed at john and Dan, and forgive me it's a bit of a naive question so I'm not an epidemiologist. But I'm struck by the importance that time series and geographical studies seem to have had in the evolution field in particular. I was just wondering whether you thought there's any significant opportunities in the low dose radiation area in those sort of approaches to graphical and time series still. Well, I'll go first and I don't know if Dan has a different opinion but you know I actually, what are the famous episodes and rate on was in fact an effort to have ecological studies and there's a physicist at Pittsburgh is a named on the many Bernie and assisted that in fact it was a negative relationship between rate on and lung cancer, based on his ecological county level analysis which turned out to be quite confounded by smoking and certainly radiation there been other efforts to look at other areas with higher natural background radiation cancer risk but I think those are, those are not been fruitful exercises and people often turn of course to available data Simon and find things that are probably just driven by biases so I think on the low dose questions, you know my answer would be no I think the kinds of studies that Francesca is talking about and some of the methods of exposure estimation were sort of at one point sort of more had some ecological or population level and individual level characteristics but as things have been refined I think we've been able to do a better and better job of geographically pinpointing the exposure estimates so you know that that that area has changed but then please comment the Bernie Cohen story was quite a powerful one for a long time and rate on So I'll make two and a half points Simon in response to your question. First, I'm going to suggest that there are bigger opportunities for you guys, which are based on large data sets we have access to big data like we've never had before electronic health records individualized lifetime radiation exposure metrics from various dose registries. Those give terrific data and medical exposures those give us terrific data at the individual level with individual confounders. So if I'm doing low dose radiation risk research. I'm probably looking there first I think we've had some great advances with spatial and temporal methods in the air pollution area. But I don't see huge payoffs and trying to exploit those further might my half point was I wanted to have a little bit of fun in 30 seconds the Cohen negative associations I do have it in one of my slides in my presentation. If I can put it on the screen just for 10 seconds while I'm speaking. That would be okay. So the, the, the Cohen hypothesis was county level rate on showing a decrease in lung cancer risk and he offered a member john a $5,000 prize for anybody who could show why he was wrong. He won that prize but didn't collect in our 2011 paper because we were able to adjust county level rate on Michelle Turner's 2011 paper for individual tobacco smoking and completely eradicated this confounding. Unfortunately Jerry Puskin won the prize, I think a year earlier with a similar argument but not based on the same powerful data so I'm crying on your shoulder that I didn't have a chance to to collect the the Cohen prize for showing why he was misleading. Thank you very much I'm very glad they asked the question now. Thank you. Yes, hi. It is good to see three, three old friends, some of them going back to the 80s in case of john summit. The comment and then a question. The comment is that Francesca has given a for me at least one of the most understandable talks to me, but I just want to highlight that the computational complexity are doing some of these analysis is quite, quite, quite formidable. And one of the things that the committee has been discussing is what kind of infrastructure might be needed for for research on low gross radiation. And we have not talked to much about computational issues but but although Francesca I think has now has simpler programs algorithms for doing these, but but there's something to keep in mind. The question actually is is not as well not a scientific science question but to the three of you who are very experienced in doing air pollution research and many other areas of research. So the task that the committee is given in how to organize research on low dose radiation. And you all know the DOE story. It's long period of struggle with communities, anyone with scientists, issues of mistrust and what have you. How do we do you have any insights on how a research program along on low dose radiation maybe organize that can be as effective in furthering the field of low dose radiation as a pollution has gone. Maybe this is too big a question but I'll make one comment and that is that I think one success I'll point to is the committee that I chaired on research priorities for airborne particulate matter which you know well and went on from 98 to 2004. And I think what that committee did that proved useful was to highlight exactly what the uncertainties were that the research agenda needed to address. And it, it, it offered a framework that sort of the research could be hung on and the funding needs could be hung on so it actually and I think in practice it proved useful to EPA in the scientific community, or generally as you know and to he I so that that's the one comment I will offer that I think in that instance. It proved that that kind of framework proved successful and it went beyond sort of listing, you know, priorities and saying here's the top 10 or whatever so. Dan. I'll offer one as well and it's it's kind of organizational and it's kind of kind of scientific at the same time. We've had great successes as you heard in data pooling you know the pooling of the residential rate on case control studies the pooling of the underground minor cohorts. The pooling indeed of cohort studies internationally on air pollution and those pooling exercises have all offered tremendous insight but you may not realize it but those are difficult things to do you have to get people to work together you have to coordinate multiple institutions and countries, you know this because you've been successful at this at he I received. So, getting people together to kind of form a consortium and contribute data to address a common research objective would be my one point that I would offer the response to your question. Thank you Francesca. I will quickly respond to the shade about the computational infrastructure, I think that this is why it's so important for us to have the opportunity and in some way also thanks to the GI to hire a software engineer. And then when I say we have a package. It means that everyone that has access to, you know, good computational capability that every academic center should have can now run the analysis, not because we develop the methodology but because we had the ability to hire software engineer to work with us and develop the software package so in the same way you use a SAS macro to run if an old linear regression model now you run another macro that can allow you to do that it is more computational responsive but not something that any academic center has, you know, has the capacity. Thank you. Question. Yeah, so one of our tasks is to talk about some of the research gaps and I feel like they're really nice parallels I really appreciate all of your talks between radiation and an air pollution. And we wonder about the idea that you get to a certain point where the potential health impacts from a given dose when that dose is relatively low. Health impacts are perhaps quite small, particularly in comparison to the potential for confounding effect modification dose error and so I'm wondering if you can just reflect on on your intuition about the data as you try and move down to those lower exposure ranges and the relative magnitude of the health impacts of of the dose itself as opposed to potential confounders or effect modifiers. So, you know, I think this is a great question and I think that's why it's important to number one have access to as much as data as you can possibly have and to have potentially a lot of information on the low dose. Right, because if you have a lot of information or a lot of people that are exposed a low dose, then now we have I think the capacity and then analytical tool to disentangle the confounder from the exposure a low dose so we don't know that right actually in our pollution research with the PM 2.5 with the three projects there's been one project in the US and PI and one project in Canada and one project in Europe, all three of us in the context of exposure response relationship are finding a steeper slope, a low level than a high level. So, I don't know what are the research gaps in the context of low dose radiation by seems that that is one, and I think that's something that should be addressed but it's not that is impossible to address, especially if you have data. Thank you. John, question. Not your meter john. Those are great presentations so very stimulating. I had two questions for the group. There is this continued measurement of effects of PM 2.5 even though everybody knows that PM 2.5 is this complicated mixture. I forget how was described by one of you you've got sulfates you've got nitrates you've got direct carbon containing particles. And from the policy point of view these things come from different sources so you would really like to have these risk estimates from the different sources. What's not clear to me is why can't you use the same tools you've been talking about to develop the component specific effect estimates rather than continually studying to PM 2.5 as a combination. That's the first question and the second question which is ties this more to our committee's charges. And we're struggling with how to bring the radiation biology information together with epidemiological statistical information. And to be quite honest with you as I listened to your presentation. Most of the advances you described to me just seemed like better, better epidemiology and better biostatistics. I mean, is there some contribution here from from biological evidence to the what we're understanding with PM 2.5 and how does that get brought into the picture. Thank you. You know I may make a few quick comments and others will want to weigh in on the PM components john and this is one of the strengths of the research framework in the 1998 committee. We highlighted that as a an important point and suggested approaches to it. Part of the problem there has been there are so many degrees of freedom that it's hard to sort out but efforts have been made. There have been analyses that have been source directed now on risks. So that has been done using sort of source apportionment approaches for the, the PM so that approach has been taken.  I mean, we've long argued that someday that there would be this sort of wonderful world in which we're developing probably Bayesian models that fold in underlying mechanistic considerations and epidemiology and to an extent I mean some of that's been done in the multi stage modeling approach example in a building biology, biologically based frameworks as Dan alluded to but you know I think your point is well taken, but we may need stronger mechanistic grounding. You know for example I mean, going back to Francesca's comment, maybe I'll well be that there are certain mechanisms that are important at very high levels of exposure that play in differently, the lower levels of exposure for some Asian so it's it's hard I mean I think rate on is just so easy because the biological hit to the cell is, we know what it is and we, we know that it's not, you know, does her exposure dependent. Francesca. So I want to quickly add with to what john said and which of course I agree completely is just in terms of estimating health in part of the component and so what I felt what I think happened there is, you know, there was a lot of interest in estimating healthy delta in different components and a lot of studies and to be honest with you I think we failed because we couldn't all the different and basically that we could never come up with and side of consistency. And I think one of the main reason was because we don't even have good exposure on the different components. But that is changing now and again I think it goes back to the chronological advances because I think we are making tremendous progress on estimating exposure to PM by exposure to the component through the use of satellite data through the use of the atmospheric chemistry model and through the use of machine learning. And so I think that hopefully in the next two to five years, we will see I think much more solid evidence of the impact of the components because we have better data and better better tool that what we had to just five or 10 years ago. Dan, we're going to have two quick questions and then we're going to be out of time. So, good, good comments. John, on the biology, I wanted to point to the key characteristics of human carcinogens which are summarized in that IR volume and some of our own papers which I can send to tyrannia. But those are 10 key characteristics based on 27 topological endpoints. So as we've actually tabulated for all the radiation carcinogens identified by our which key characteristics are expressed. And that gives you some information on biological mechanism, which can be tremendously important in deciding what you would expect the shape of the dose response curve to be not only at low doses but as you may have dose dependent transitions as you go up the dose response curve data things like saturation effect so the biology I think is very important. But the counterpoint to that is, well my second comment was to reinforce social portion of studies on air pollution, George Thurston has led a very nice detailed HCI analysis which Rashid can point you to which will tell you what the most important sources and types of mixtures are that produce the most potent PM 2.5 mixtures. And but my my hope is that with the data that we now have big data sets that we've never had before, better exposure metrics that we've never had before in some of these radiation dose registers, and better information on individual mixtures, we might be able to say epidemiologically and statistically at environmentally relevant exposure levels, make pretty pretty good statements about what the risks are so that's that's my hope would be the outcome of all your deliberations considering the different sources of information including the basic biology sorry for being a little long winded in my answer. Okay, we're running a bit, we're actually out of time for this session, but these, these discussions are really centrally important to what what it is that we're going to be writing about. So I'm going to let this session go on for another 10 minutes. I think there are downstream speakers that have a problem with just pushing the whole program back with you email run you and let her know about that and we'll dynamically adjust the downstream schedule but let's keep going with the q amp a for a few more minutes. Bernie. Her presentations, really exciting work. But the issue that we're dealing with for a long, low dose radiation is mostly not an acute effect, but mostly a chronic effect. And of course, Americans move a lot. And, you know, pharmacovigilance data is is great but it's mostly as old folks who are taking the taking the drugs and what if Jane moves northward because of global climate change or whatever I mean, there's obviously is that going to be a limitation for trying to do some of the things we want to do in using the kind of approaches that you suggested. Or how would you get around moving. So I mean I can just speak for a moment with respect to my presentation this statistical tool that was presenting wasn't the context of long term exposure and chronic effects, not in the context of acute effect so this was all in the context of, you know, chronic exposure. Clearly, you know, depending on which data you're looking for whether or not you can assess long term exposure to a contaminant will depend on whether or not the individual subject move. So this is really depending on the quality of the study that you're looking at in our context where we looked at the Medicare data we actually do know whether or not someone move residents and so we in all our study in the chronic or pollution study we exclude, you know, the Jane that decided to move from Connecticut to Florida we can identify them and we can take them out so it's definitely a challenge is something you have to look at you cannot assume that's that everyone is in the same place, but depending on the data sources, you might be able to assess whether or not they move and so you either assess the long term exposure based on where they moved or you exclude the movers from from your analysis. Thank you. Yeah, I mean, when you talked about a lot of all of you, I'm not sure who would be best to answer this. It seems like the major focus was still on cancer but their other health effects from radon and their other health effects from air pollution. How do you calculate that in risk models and how do you look at those differences. So I can.  So I think from my own perspective all of the consideration I made it's applicable for any type of outcome and in the context of our pollution research. We, you know, there have been studies from, you know, from all cause mortality to neurological disease to cardiovascular disease or respiratory disease so it's not, you know, limited to one outcome on another outcome and there is also additional research which have not been able to, you know, I've not had the time to talk about but I think it could be relevant is also to assess the effect of a continuous exposure on several multiple outcomes simultaneously and I think that's also something that if the committee want is interested, I can definitely provide some of some of some of the references but there was no any focus on a specific outcome from, you know, from my own consideration. I guess what I'm thinking about is that the dose response curves and those things are likely to be different for different outcomes and we're certainly interested in non cancer outcomes as well as others. So I'd at least like to see that paper that'd be great. Thank you. I'm going to ask a question. I'm going to pick on John and Dan for a minute here. So both of you in the radar on study, linearly extrapolated across a low dust point, which was slower than your linear extrapolation curve. I think you justified that on the ground on mechanistic grounds that read on a distance damage we know how that happens and so that's a reasonable assumption but but there is an emerging body of data that says that in not necessarily in that circumstance but in some circumstances, exposure to radiation actually stimulates the immune system to have a long term efficacy against radiation induced damage cells. And if that were the case, then in fact, there might be an argument for there being some non linear effect there and so I guess what I'm trying to get to is what when when one of the things we're thinking about is a lot of biology that's coming to the fore now that has to do with immune responses that has to do with the impact of stress in the brain etc etc on the immune system on damage repair process and so on that aren't necessarily linearly related to dose up. How do you think about considering that is we develop the next generation epidemiological models. Now I'm going to talk about things I know absolutely nothing about, but I will say that on beer six there were people who were and I, you know, I actually think, though that the sort of the high le t rate on example is not a good one really for your context of what you're talking about and I, you know, understand, you know, certainly the issues you raised so, you know, and that again speaks to some of the elegance of the work even available back in the 90s the single cell radiation experiments, you know that air call and others were doing I think we're pretty directly informative I mean there were bystander cell effects in these systems that they use but I mean they you know I think the particle example is a little bit different I think from the sort of more general radiation concerns you're having which are probably less about high le t general argument here. Yeah, so, you know, I mean I think and I don't think I believe the rest of this answer to Francesca and Dan but I think it would be willing to be potentially to say well look. Here's our prior idea of what the dose response curve might look like across this range of doses versus that range of doses based on the considerations that you have it and the kinds of Bayesian models that might be used to structure into it. What the prior is and now having said even more about things that don't nothing about leave it to Francesca and Dan to comment but I think that would be the general approaches to build in biologically based priors into the models. Well, okay. Excellent. Francesca Dan either of you want to elaborate further. I mean I think I think john john is is is right I just say that because he's here but anyway. No, I completely agree I think it's hard, but I do think that we have now the scientific understanding and the technology would be really exciting to start within a biologically basis shape of exposure response function and then see the degree to which the data that we have that even if there's a lot of data is often perfect to see the degree to which is consistent with our biological hypothesis. Okay, thank you. Dan quick answer and then Ben last question. I like the way you framed the question how do we dial in the biology. If you go back to my slide on evidence integration there was human animal and mechanistic read biologic evidence. I think it's a critically important part of the the evidence lines that you want to consider. How would I do it if you looked in dear seven there was a long discussion of the different biological mechanisms by which different types of radiation at different doses at different routes of exposure might act and that discussion the biology was a big deal in that group. I think I end up trying to see do I understand the biological mechanisms. Number one number two what would those mechanisms imply for a dose response function and number three if I had sufficient data to test whether or not I could validate that mechanism and under real conditions would be the icing on the cake so that's my short answer I could talk a lot about this but you asked for a short response. Great that's very helpful and look at your son thanks man. Yeah, and my question is a yes or no. Thank you for for for Francesca so I love that we're getting into the leads because that's where biostatisticians live most of their lives so you know you you're estimating the the dose response function non parametrically based on you know biological knowledge would you ever think about restricting you know putting any restrictions or constraints on that dose response function based on your biological knowledge. You're muted. Yes. But you will have to face the criticism of people that don't agree with you on the biological hypothesis and say that you're not going to listen completely to the data. Yeah, yeah. So yes, the caveat. Yeah, thanks. Thank you. All three of you this has been an incredibly helpful session. I think it really advanced our thinking a lot on this and I will have to go back and read the tons of literature that you've brought to our attention so again thanks for taking the time to And at this point we need to move on to the the next session, which is really going to continue the discussion of risk evaluation in this context with stakeholder participation and some of the lessons that have been learned in that space and Now we have four speakers. David Cossum from Vanderbilt. I think Catherine Higley from Oregon State. Steven Crown from Vanderbilt and Michael Greenberg from Rutgers and so we'll have presentations from each of these four individuals and then have our questions at the end and again I apologize for letting us run about 10 minutes long but hopefully that won't perturb your schedules too much today so with that let me turn the floor over to David and ask you for your presentation. Thank you for joining us. Thank you very much for inviting us. And if you bear with me one moment to get this going for us. And unfortunately, Kathy is not able to join us today. So I'll cover her slides for us but we've one set of slides and I hope that they are made available to the committee. But the study director asked me to present on Cres with the idea that Cres may be a potential model for this organization for the committee recommendations as to how to approach the low dose issue and build credibility there. And that's the way we organize this with a couple of case studies also. I also tried to answer the very specific questions that Reina outlined to me in her email and in our discussions. So as you mentioned co presenters here today. Again, thank you for inviting us to join you. To give you a little bit of background Cres mission is to support the safe effective and publicly credible risk informed management of existing and future nuclear waste, both from the government defense and civilian sources. And we do that through several modes. Strategic analysis, which is usually multidisciplinary, including experts both as part of the Cres team and beyond the Cres team review in specific areas. And as we're requested as we believe is appropriate and applied research and education applied research usually involves graduate students doctoral students and postdoctoral students. We have a webpage that you can go to for more information. And over the course of Cres history which I'll talk about a little bit. I'm involved with most if not all of the DOE complex from the defense sites, the small sites, the large sites, etc. And throughout all of this, what we've been trying to do is improve the confidence in the environmental management decisions being made by providing credibility, providing additional credibility capability through the academic resources and trying to provide certainty and understand and communicate the uncertainties where they exist in all these decisions. And within that Cres is organized, if you look at the bottom around different topical areas of waste processing and special nuclear materials, the remediation issues, the nuclear waste policy and strategy that they take in stakeholder engagement and communication. We've had this organization for quite some time. And as I mentioned we support each of those areas through strategic assessment applied research review and education, and we map that into the different needs that DOE has, as it works through the issues that it faces. Cres is a multi university consortium that has evolved over time. Originally the result of a competitive request for proposals, and we were the awardee back in 1995. It was originally founded under the leadership of one of your committee members, Dr. Bernie Goldstein with as the PI and Charles Powers or Chuck Powers as he's known as the executive director. I'm Bill Oman, John Moore, and Art Upton as being the original management board for Cres, many of these folks I'm sure you recognize, and it was in response to a National Academy's recommendation that we move forward. To talk about stakeholders, we're talking about not only the public and interested groups, but we're also talking about different organizations and entities that have different defined roles in the management decisions that are made that includes the federal regulators, the state regulators, the tribes, the defense nuclear facility safety board, we've got site specific advisory boards, we've got local state and federal elected officials, and we've got the public and advocacy groups, both at the national level and at the local level for the major sites. And that becomes very complicated. I'm not going to go through all this, but I've got two slides here about who is Cres, it's led by a management board which is highly multidisciplinary. We have eight universities that are part of the regular Cres core, and as necessary for specific expertise we go out to other universities, or other folks that are not aligned and bring them in for review or specific challenges that we may be asked to address. I'm sure you can read through that on your own, so I'm not going to spend the time now with that. But why Cres? It's an independent team, it's arms length from the Department of Energy, whereas a cooperative agreement, we're not a contract, we're not a grant. And it's multidisciplinary experts that are pulled from all over the country and international experts in nuclear and environmental law, social science and policy. And then the specific technical issues that they face, landfills, performance, leaching assessment, nuclear waste processing, safety, environmental health, physics, ecology, just to name a few. All of us have our primary roles as academics, and our employment and salary doesn't depend on this. Basically, we're all tenured professors. Most of us are at the most senior level at the universities. So it allows us to speak freely and speak our minds. We highly leverage what we get from EM both with the Nuclear Regulatory Commission activities, EPA, the Defense Board, National Academy committees that many of us serve on GAO, we have regular interactions with, etc. We also have the ability to tackle sensitive issues. And to do that kind of off the radar, or to bring things to the table from different perspectives at the national labs, or that DOE itself cannot do. Essentially, we have wide reach back, and we are also very flexible. We have an annual scope of work, but we also have ad hoc requests that come, and we reorganize our priorities when things come up that are high priorities for DOE or for some of the other stakeholders. Also, since we've been in place since 1995, we've been along and through, I've lost count of how many Assistant Secretaries for EM, and we have quite a long institutional memory of the issues that they face at individual sites and at headquarters. We do offer graduate training, we have a host of doctoral students and postdoctoral fellows throughout the universities involved, and also when requested we do professional training for DOE, or for professionals at other organizations that is in the interest of moving this issue forward of cleanup of the DOE complex. We're unique in the fact that we have multi-disciplinary teams, we can bring the types of expertise such as law and policy that the national labs do not do. We often work in concert with the national labs or collaborate with them, but we also provide independent activities from them. And depending on the nature of the problem, the engagement with stakeholders can be very broad and it's not a one-size-fits-all. I really need to emphasize that a lot, that understanding the context and who the stakeholder groups are that are critical for a particular problem changes by location, by the nature of the problem, and you'll see some examples of that. I'm going to send us some specific questions about operational aspects. CRESP has been awarded in five-year intervals that have been renewed. We do an annual scope and budget allocation, which the CRESP leadership, the Management Board develops in dialogue with the DOE and environmental management. Sometimes we have additional tasking from the Office of Nuclear Energy, sometimes from NNSA, sometimes from the Legacy Management Office. Ad hoc reviews and projects are often requested during the year. They may be small issues that can you take a look at some risk communication pieces that they're putting together. They may be larger issues, you'll hear about a few of them today. Sometimes they're just can you provide some advisory information to DOE or to some of the other folks such as to the Nuclear Regulatory Commission. We're asked to comment on specific proposals, sometimes informally, sometimes formally, same thing for the GAO and for other federal agencies. The results in the projects and the reviews are briefed to DOE and other interested stakeholders. Typically project reports and reviews are posted to the CRESP website from 30 to 90 days after their completion. At least to give DOE and other involved folks in the studies time to respond before it's publicly posted, but we're committed to everything that we do being public. Publications and presentations don't require any prior review by DOE. That's important because they can't tell us not to say something or they disagree. We obviously seek their input, we seek their responses and discuss things with them, but they don't have veto power. And that's been very important in the credibility of what we've done and interacting with state agencies or with EPA that often can be had strong disagreement with the agency. We provide them quarterly progress reports. There's no formal process for evaluating the effectiveness and implementation of CRESP recommendations. There have been some cases where they have implemented a formal process. For example, when we were dealing with issues for the pretreatment facility at Hanford Waste Treatment Plant, there were some high profile issues that were raised by the Defense Nuclear Facility Safety Board. We were asked to form a review committee of experts on some of their mixing problems, and we generated about a dozen letter reports that came out of that, that the Office of River Protection formally tracked and responded to. That was different than other cases that we've had. But again, it's not always a one size fits all. Okay, current projects relating to engagement with stakeholders. So we have projects that are focused on the engagement aspects themselves, as well as the engagement and communication with stakeholders being an integral part of many of our projects, if not most of them. And as I mentioned earlier, the relevant stakeholders vary by project. Often we've been requested to help DOE with stakeholder communication, and that may be by reviewing documents by providing training to some of their senior management, or it may also be participating in the risk communication at some of the troubled areas and we'll give you some examples of that. Right now, ones that we've been involved with in the past short time, meaning the past few months is the restart of unfiltered ventilation at the waste isolation pilot plant in New Mexico. And we'll have Steve Cron will talk about that in more detail, the review of the Portsmouth environmental reports I'll talk about that a little bit because Kathy couldn't join us. Risk communication of the Portsmouth detection of neptunium 247 out of public school. And again, Kathy was going to speak to that but I'll speak to that for her. And then risk communication workshop for the site specific advisory board chairs meeting. I'm going to bring Kathy Higley put on a very nice workshop for them for half a day, and then followed on with for future discussion of how to engender better communication engagement at the individual sites through the ssa base. Right now the communication centered projects that we have are measuring and communicating EM objectives and accomplishments, improving risk communications, a special issue of risk analysis, and the role of social media and public engagement. We've got some more details on these as supporting information if time permits at the end of this discussion. So here's just a few notable successes over the years of Cresk. Amchika Alaska was, this is a project that Chuck powers led with many of us involved, and also we brought on the state of the University of Alaska at Fairbanks as part of the team to carry this out. At Amchika there were three underground nuclear test shots that were carried out, including the largest one the US ever did the canakin test shop. And the question raised by the state of Alaska and by the native alley youths was, were residual radio nuclides from the test shot cavities migrating through the subsurface and up into the sea, through the sea floor, and being taken up by the various areas there, which subsequently became a large fraction of commercial fishing and subsistence food supplies, coming out of that region, the North Pacific for that Cresk organized a science advisory board that then came up with a science plan that science plan was done in communication with the state with the alley youths with many visits up there. Then it implemented that science plan and had representatives from the alley youth communities as part of actually carrying out fielding the ships being on it collecting samples and the like. The analysis of the samples and some other aspects was carried out at Vanderbilt, we did split samples with the national labs. And then when all was said and done, we went back Joanna burger from Rutgers was a masterful person with all this of meeting with the individual alley youth community she spent a couple of weeks up there, going from community to community, and discussing the outcomes as well as presentations to the state and to other stakeholders, like fish and wildlife. And the outcome of that was the development of a bio monitoring plan, which subsequently was implemented, and even Greenpeace came out in favor of this, the headline so it was really quite a success that we were able to achieve. The Hanford site wide risk review was a different type of project that was taking a holistic view of the Hanford site and all the remediation issues that were there. The risk communication was very much engaged with the site specific advisory board with the local town elected officials as well as with DOE and with the communities. We went in there first and told them what our tasking was what we're going to do and get some input on that we held a few public meetings and met with the advisory boards. And then after that, we came up with a methodology, and we put that out for public comment had additional meetings in the communities got feedback on the methodology, altered the methodology based on that feedback, and then completed the study. The study at that point then was briefed back to the communities, as well as other interested parties like GAO we briefed it to. And in turn, this wall had quite a bit of controversy from what I would say a political perspective with it. So, that's the question of what our experience and what our recommendations were had not been questioned to, to date. And that's over the past few years. And many of the things that we stated in there has been the basis of further actions. as well as the vaults, tanks and other cement and concrete materials that are used in the nuclear complex. And that was dealing with an issue that was between the department, some of the state agencies, the Nuclear Regulatory Commission, and some of the plans that the department had. So we put together a group of independent experts that worked on this and it included representation from the Nuclear Regulatory Commission, that they had some of their experts with representatives of the national labs. This was also briefed at public meetings, but also came out with a series of reports and recommendations that set the foundation for some of the studies that are going on now, including the one that the National National Academies is involved with for low activity waste alternatives at Hanford. The landfills partnership was a different type of studies led by Craig Benson. And in turn, this was to deal with issues of near surface disposal. And here are the state regulators, the state. And in addition to a swarm of state environmental agencies, as well as the Nuclear Regulatory Commission, have all been active stakeholders in this. First asking what are the issues that are most concerning to them, then executing a research program that address them what their involvement, and some of the outcomes of that are reports even that are published under the new reg framework by the Nuclear Regulatory So some key lessons learned engage with stakeholders early and often. You have to do individual small group and open public meetings they're all important one's not a substitute for the other. Different stakeholders have different information needs you cannot take a one size fits all and be approachable and familiar to the stakeholders, don't just show up and then go away and don't come back. You have to have regular ongoing engagement. You have to listen to the input and provide feedback on how you address the input and how it influence the outcome. Make the science and the facts and the uncertainties clear and communicated in a manner appropriate and understandable to the intended audience. One of the cases was where people wanted to communicate to a very low income community about some of the risks and relative risks from radiation exposure, and the first suggested analogy was flying on airplane across continental flight on airplane. None of these people had ever been on an airplane. So coming back to examples that were more relevant to the community was very important, and Kathy Higley was masterful at that. Use as that's examples and be clear about your role in the process. We're not decision makers, but we provide input to the process. So here's some recent case study examples. Kathy was going to talk to this when she's been the lead on it. An independent consultant working for the community identified that neptunium 237 was present in the ceiling tiles of a middle. Yes. You're running a bit long on this one. I'm wondering if you can expedite some of these. Yes, we are. We're going real quickly through it. Thank you. So they detected that that precipitated a series of sampling and engagement with the communities crest provide insight into radiological collection. They had a member participate in the science advisory board as well as ongoing meetings. And most recently we provide a member of the crest team two communities to meet in coffee shops and the like, and answer questions. Plutonium fishing plan I'm going to turn this over to Steve to talk about. Thanks, David. In 2018, there were two offsite contamination events at at Hanford that were linked to the DND work at the plutonium finish being plant based on that the deputy secretary of energy formed an expert panel. And DOE shut down the project and the deputy secretary set up an expert panel to advise DOE on actions needed to safely restart the work. It's important to understand that that all of the special nuclear material and high radiological material had been removed from this facility at the time of the effort so they were into deconstruction of the structure that you see on this slide. Next, next slide. Sorry, the first paragraphs will repeat. But if we move on to the second bullet crest provided two researchers, one experienced in radiation protection, Kathy Higley and myself as a nuclear facilities, nuclear safety expert to the panel. We assessed the Richland office, corrective action plan and root cause analysis. We reviewed the contamination patterns to determine what the root cause of the contamination issues were, and then it evaluated DOE restart process and their new work practices to reduce the possibility of that. And the possibility of that in the future. And Crest provided specific formal comments on both the root cause analysis and the restart plan to the deputy secretary. And still ongoing is a project that the waste isolation pilot plant, the transuranic waste repository in New Mexico. It's very evaluating restarting a ventilation system that is not filtered with high effects with heap of filters to allow them to increase airflow in the mind during operations that are that do not involve moving radiological waste. It also allows them to operate more equipment, most of which is operated by diesel engines. So there is a limitation on this operation that no waste can move during the time periods that this plan would be in operation. The anticipated advantage with advantages are as shown on the bottom of this slide. Next slide. So what have we are involvement that Kathy higgling myself has been to review the nuclear safety analysis associated with potential accidents that could lead to releases from the facility. And in independent analysis of both historical radiological monitoring data and any potential release releases associated with the tests that have been ongoing of this system. And we've provided formal feedback to do we and the Carl's bad field office we have also provided independent members to attend public meetings where the restart of this system has been briefed to the public. And we are ongoing with review of the next stage of testing. So let's turn this over to Mike. Now, if you can talk about the study Mike. So this was a very large study that took over two years to do. And it came from Congress, Congress wanted to find out how effectively the DOE was identifying its programs and executing its plans to address risk. And the DOE is remaining environmental cleanup liabilities and putting out in English what what that means is whether using the money they were getting in their budget as effectively from an economic point of view, and from a public health point of view as possible. And in order to do that, we knew that we were going to have to talk to a lot of people around the country, everything from individual stakeholders to a lot of people in DOE, but also a lot of people in other organizations, like state organizations, and EPA organizations, all of who did not necessarily get along famously with each other. I think that's probably an understatement. So we interviewed over 100 people in various states on the phone we went and visited them. This was the pre zoom era. We did many, many documents. The result was a report, and their report was fact checked by independent peer reviewers that Cresc retained. And we got back a lot of questions, and they will also peer reviewed by the agencies, whose points of view we were representing, and some of them didn't agree with what we said so we had to talk and negotiate with them. And then these were ultimately submitted to our Congress so have the next slide Dave. So we came up with 24 recommendations not on purpose to turn out to be 24. And we found that not surprisingly, human health and safety is plays a very important role in prioritization and budgeting, but it's not clear exactly how much. And that's because there's so many other factors involved, most notably consent degrees, which do we sign with these sites, mostly in 1990s to say that they were going to accomplish a particular set of tasks by a particular way at a certain site and they couldn't do it for larger reasons, political and other reasons.  It was very important all of this to not only have the legal implications taken care of what you have to have staker inputs from the citizens advisory boards, the local chambers of commerce to worker groups. The citizens advisory boards are really good.  The tribal nations have very strong feelings about these things, and they don't hesitate to voice them. And then so the DOE needed to take into account some of these recommendations they did. I think it's fair for me to say and Steve you were there so if you, if I'm terribly wrong you can correct me. Most of the recommendations we made to the agencies, meaning EPA DOE were accepted. Most of the ones we made to Congress or not. Congress was reluctant to make the kinds of political and what I call political and economic decisions they needed to make. So we wrote a very, very large report, which I have a feeling no one's going to want to read. But there is this short paper that we put in risk analysis or short, a short time ago that you can read and that will sort of give you the reader's digest version of that study. So Dave, do you want me to take the next two slides. If you want to do them quickly because I'm sure the committee's got questions and I know they're running behind schedule already. Okay, so the DOE now retired official came to us and said, you know, it would be very interesting if we could judge the extent to which the stakeholders, the community stakeholders actually understand what we are telling them about what we are accomplishing. I didn't know what we were getting ourselves into we found out shortly thereafter by looking at the media content of DOE state and EPA publications in on the media on the computer of what they were accomplishing. And the bottom line of the story and we wrote a paper on the stove, which you're certainly welcome to read, which essentially says they tend to contradict each other. That is what DOE at the site with the states in their version of a DEP or EPA. And sometimes what EPA is saying contradict each other. So we're talking about the way for the different the accomplishments are expressed in different ways. If I was a stakeholder living in one of those places, I wouldn't know what information to trust. So we have put this, read this paper, and I'm hoping eventually, and we would love to be able to work with the site specific advisory boards to have them take the lead on figuring out how to get messaging that is not totally consistent, but consistent enough to where the stakeholders can read and say okay now I understand what they're doing. So that's what that project was about and I'm happy obviously to go into more detail. And finally, based on again discussions with DOE and our own interests. We've been trying to help DOE with what I'll call actionable recommendations regarding risk communications. And so we have worked out an arrangement with risk analysis which many of you know for a long time I was editor in chief and Tony Cox was took over from me and Tony's agreed. And next year we're going to have a special issue of risk analysis, which is going to talk about not only the role of planning risk communications which I have to have. If you walk in, and you haven't got a plan you're going to be in big trouble, but also technically when you're speaking, what kinds of things should you be doing and not doing and David briefly overview some of them before. But we've interviewed more than a dozen experts who have a lot of experience and speaking to different audiences, and the vast majority of those are not risk communicators. They are people like the people I'm looking at on the screen, and it includes some of us, including me who talked about how to deal with media which is very interesting. I also have a set of academic papers written more by academic people who essentially talk about how this can be done better. So it's advice on planning communications, and advice on implementing it when you're the one that has to be the implementer. And I'm hoping it's going to be out in some summer 2022 I'm going to say it's about 90% done. And that's, that's the short version. Mike. So at this point if there are any questions we're happy to answer them. Thank you all this has been very helpful. One of the things that we've been discussing our models for how it is that we might manage the Lotus program this is an interesting model. So let me throw it open to the committee for questions. But while we're waiting for the committee to come up with questions let me ask one. You're sort of operating at arm's length from DOE but you're funded by DOE. What, I guess what's the magnitude of your funding and what does the funding portfolio portfolio look like over, let's say 20 years. Since you come in every five years is DOE managing you at a distance in terms of the content of things that you study. They've never said for us not to do a study that we felt strongly about. They often will come to them or they'll come to us with things that they think are higher priority or that we think are higher priority. And ultimately it's a discussion as to what that portfolio looks like each year. And we review it each year though there's ongoing discussion. Many of us are in contact with leadership or senior management folks at DOE or at the sites. At least several times a month if not more frequently, depending because of the range of it at any given time there are probably a dozen or so substantial projects going on. For 20 years, I think our low point was about $3 million per year. Our high point was probably about seven, seven and a half million dollars in a year. And that, for example, when you take a look at the Amchika project that was probably getting close to a $3 million project in and of itself. So it depends on the magnitude of what we're doing. And what's involved. Thank you, Rania. Thank you, everyone for your presentations. One question to follow up on Joe, which was not my original question. Am I correct that funding for Cres is earmarked from Congress. Yes and no. Over the 2526 years that we've been in existence. I think three years it was earmarked the rest of the time it has not been. So they've had various funding mechanisms that they've used sometimes it's come out of headquarters budgets. Sometimes they have taxed the sites, the sites operating budgets to provide our support. It's been all over the place. Thank you for that. So coming to my original question. How, how has your work been affected based on the different priorities that the many different assistant secretaries of VM have had over the years. I think that some tend to use you more than others tend to tend to come to you for with questions and other assistant secretaries. Can you talk a little bit about the relationship with the leadership. It as you're correct that some are more engaged than others. I think it's time, depending on who the assistant secretary is we've had very active engagement with the assistant secretary sometimes at higher levels, most frequently at somebody that reports directly into the assistant secretary. For example right now we're reporting into the person that's policy and communications, though we interact with all the principal deputy secretaries and the like, pretty routinely. They, a lot of what we do has longer term perspectives, you know we'll see things on the horizon and say, you know, this may be an issue that you're going to be facing. Five years from now 10 years from now we're longer. Can we start getting research going that will help you in that area. For example one was just brought to us to start on is the movement of the graphite reactor cores on the river at Hanford, and looking at what the risk profiles will be when they have to move them to the central plateau and how to get better at what the release will be of carbon 14 of chlorine 36, etc, that are coming out of that. And what we're, that's not an issue that they're going to have to face for another 30 years. Why are we being asked now because they look at their overall risk budget for cleanup of the site, and they see that that has the potential to be a significant contributor, and they don't have the tools to deal with that right now so they're asking us to participate in that. So it's very variable, but the fact that we have a portfolio of sometimes urgent things that they need like the whip example, and longer term things like we're talking about, or like the cementitious barriers partnership. That was, I don't know six seven years ago, but now it's coming to fruition with helping them at Savannah River, and at Hanford address issues that otherwise they wouldn't have been prepared for. Simon. Yeah, thanks Joe and thanks for the presentations. I wonder, given the crest as I understand is funded by DOE and presumably many of the stakeholders or some of the stakeholders you engage with but it's not best disposed towards DOE. How do you sort of manage those relationships and balance them to make it work. We take the same examples that you use the National Academy studies are funded by the DOE, the state regulators are funded by DOE. They're all arms length it's the same sort of thing that we use as examples that many of the stakeholder groups such as the site specific advisory boards are funded by DOE. The difference is we're not under contract, where they get to control what we say. Okay. So you don't really encounter any problems with the credibility with some of the stakeholder groups. Some of them that the credibility depends on whether or not they are welcoming to our message. I don't think it's under a lying issue of whether we're credible as individuals or an organization, but there certainly have been some strong disagreements with some of the messages that we've related. I can speak to some of that. So pass. Thank you very much. So just to ask that question a slightly different way. When you deal with your adversarial stakeholders. Do they think of you as Crest or think of you as DOE as Crest. Okay, so so you're you're confident that you're disagreeing communities think that you're sufficiently independent from DOE to be trustworthy. I think so. And there have been places where we've publicly disagreed with DOE and DOE accepts that and DOE has recognized that our independence is important for us to be helpful. That and we're going to have disagreements or we're going to have recommendations that don't allow a line with what they the directions they thought they were going. Fortunately, there have been a number of times when our recommendations have caused them to change direction. So that's a positive impact from my perspective. Thank you. Shane, did you have a question or my question was answered. Thank you. Okay. Okay. Thank you. Thank you. So David and others who reviews whether Crest is doing what is meant to do. And how is membership of Crest determined and what type of rotation method do you have. And membership of Crest depends on what our technical needs are. We have folks that have been from the original proposal when Bernie pulled that together. And given that that's coming on almost 30 years ago. We're going through an intergenerational transfer with some of that at this point. Basically, it's been through the management board and myself as PI to reach out to identify folks that are of like mind that being where academics that want to do good science and engineering and research, but also have an impact and be strongly collaborative multidisciplinary. That's not everyone. And it's certainly not a good place as a pre 10 year faculty member to be. And a very recent example is DOE reached out to us that we're looking for someone who really specializes social media, and you can tell by my age that I don't specialize in social media. So we identified a, a tenured professor at Rutgers who's very, very good social media who has worked with the Defense Department, and now he's got, he's going to be working with Crest but social media issues for DOE. And I think that's a very important thing to emphasize the, the, the membership of Crest has been very responsive to what the challenges of the environmental management program are at a given time and those change and that that causes us to change the research membership of the Crest Management Board and the Crest researchers. And the continuity and the independence of who's participating Crest and the flexibility there, I think has been essential to its success. And David who reviews the work of Crest be any entity. There, there have been reviews by DOE of what we do we do an annual portfolio review with them that's a little bit more formal will we present and they provide feedback and we've representation from headquarters and from the sites. There have been many times when we've been asked to present and provide results of our studies to the Defense Board to nuclear regulatory commission to the GA. National academies. Thank you. Bernie, this is all your fault. Yeah, absolutely. No, it's actually all tough powers for. But let me just emphasize a point that David said, but he said we're almost passing. I'm not sure what which is only got senior people, Bernie get a little closer to your microphone. He's only got senior people on that on Crest. When we form Crest, we were cognizant to the fact that we are likely to be biased by working for with the OEM these funding. And to me my greatest vulnerability was to have junior faculty, whose careers were at stake and now suddenly if I tell truth to power, we're going to lose the Crest money and their careers are going to be at risk. We avoided not all of us I mean the folks the University of Washington had some junior faculty been in New Jersey our group had only had faculty who had their own NIH grants or other grants, and who could survive losing the Crest money. So we can always tell truth to power. Interesting. Other comments from the committee. Well I'll just ask one, one further question then and that is. How do you feel that DOE as an institution is nurturing the Crest program. In other words, are they are they serious about maintaining expanding etc, the enterprise or is this not a labor of love for them. I think so. And I think you know it is always keep in mind DOE has a lot of turnover. Also in its political leadership and it's senior management as political leadership changes there always tends to be reorganization he's full history I'm sure you're all familiar with that. So it's always a reeducation process. But the fact that we've been supported this long, and we have very good and frequent engagement with a lot of folks there at the sites at headquarters, and they respectfully say yeah I think they're nurturing of it. The level of interaction support can vary from year to year, and from individual to individual, and DOE is going through a generational change right now at the end. And it's going to be another reeducation and challenging process I believe for the next couple of years. Thank you run you can I answer that question in a slightly different way. The DOE wasn't happy with crests overall accomplishments. I think it would have done what it has, what federal agencies often do is get angry Congress would have pulled the money and it would have ended up going to their state schools of public health and schools of arts and sciences of the particular elected officials who are on that committee, and that hasn't happened. So it must be satisfied with what we're giving them even if they don't always agree with it. And when they don't agree they tell you. They're not bashful about it. Thank you run you. What are your views about DOE em using the Office of Science to address some of its scientific challenges related to cleanup. My understanding is that maybe in the mid 90s, they had stronger relationships but it's not the case anymore. And that includes the Lotus program but other office of science capabilities as well. I think it's been variable and it's been leadership driven. I mean it's also somewhat personality driven. So it's leadership both at the Office of Science and at EM, and above them, and how that's organized so as I said it's been variable. There have been some very good things have come out of the Office of Science. So, supporting em. Thank you. What about today David sorry what about today. Well there are still things that Office of Science does that is very helpful and supporting em part of the problem is the and there have been reports and studies on this. That overall the science base and the technology or the research development and demonstration base for em is woefully underfunded. And when you look at the magnitude of their program it's about $7 billion a year, and you look to look at it relative to any other organization that's looking forward for at least another 3040 years of operations. They're woefully underfunded in that area. Thank you. One of the recommendations we made in the omnibus report that they needed to do more in that area. And the Secretary of Energy's committee that I served on a few years ago. And that report made the same point probably a little bit more artfully of in detail of what level should be in what areas but that includes relationship with Office of Science. Thank you, Simon. Yes, just a really clarification from my own. Assistance, because I don't understand the US system so well. So you want you funded by the DOE office for environmental management. Is that the name. So you're separate office sides. Yeah. Okay. The sites actually report to that office. But the sites also get direct congressional allocations in the annual federal budget and EM gets some and that it gets divided out. Okay, okay. Thank you. Thank you. Any other last questions from the committee. Seeing none. Well, thank you. Thank you all very much for great presentation. You've given us a lot to think about here and we appreciate you're taking the time to do it and very helpful. Thanks again for inviting us. Yeah. So with that, I think we're going to conclude the presentations and move into the public comment session. We have about 20 minutes set aside for this if you are not one of the committee members and material that you would like to bring to the attention of the group here. Now's the time again we would appreciate the you're being brief to the extent of limiting comments to about five minutes. Just to keep in mind that we're likely going to have another public session sometime in January, and there will be additional opportunity to a comment then if you don't have an opportunity to a comment now. The details of that will be announced on the academies website and lists are so you can pay attention to that if you don't get your voice heard today but for now, if there are members of the public who would like to make statements about what you've heard today or in previous meetings, please raise your hand. Yes, I am. Did you call on me. Yes, I did. Let me just get my statement. I want to keep it short. Second year. So, a couple of things. One is following up on what we feel is unanswered questions about the credibility of this committee still that really have not been answered. There are some organizations sent a letter in July with several specific questions and then follow up questions in a September letter, specifically asking who at the academy concluded that no need no changes needed to be made to the committee who was involved in the reviews of concerns. We asked specifically did staff or officials of the academy consult with or discuss with DOE and relevant congressional committees on the selection of the committee and establishing the low dose committee did the academies consult with solicit receive comments or otherwise give me the chance to influence or the congressional committee members, the chance to suggest the perspective members of the committee, and so forth. So, so you have those letters, and then we got a response from Dr. Ferguson, which basically did not answer those questions his, the extent of his response was that the academy invites, and I'm quoting anyone who's interested in providing nominations for committee service, including self nominations to do so. The decision on who serves on a national academies committee rests solely with the academies. And then after the statement of task is agreed on committees maintain an arms length relationship with the study sponsor in order to preserve their independence and sponsors may offer suggestions but do not select committee members. So that's how he answered the specific questions. Then when I raised this at our last public comment period. Dr. Costi said, and I believe said, she said that in his letter, he was clear that DOE and no offices of DOE had a role in the makeup of the committee, but that is not what his letter said. So that's different what we heard verbally, and what we got in writing were not the same. And so I don't feel like we've gotten a specific answer to that question. It does leave me with no alternative but to do a FOIA request to the DOE for all of the interactions with this committee. And of course, we know that takes a long time to get an answer, but that's what we're going to have to do. So I wanted to lay out this continuing concern, which was raised by over a dozen groups that closely track radiation issues and the national academies efforts. Also, I was disturbed so that that's that issue was disturbing to hear the chairman you saying asking questions about how to really basically supporting the idea of hormesis. It's not supposed to be the goal of the committee, and perhaps it's not a direct one, but it raises the question of if that isn't what what this is about and we've suggested that's our concern. So that that was distressing. Finally, Cresp does not appear to have any stakeholder participation in what it does, at least that wasn't described in the presentations. So it's hard to say if there were stakeholders that were happy or unhappy. Public participants, I mean, I take it back, not stakeholder, but community and public residents that class of participant that so that we're continuing to watch what's happening and again raise support for the presentations that took place the second last time around with the community folks and the arguments that Dr. Marco Johnny made about having people who are impacted community people considered respectfully on a power with the scientific experts and and to have more to share more power with them over the studies that are done. Thank you. Thank you. I'll leave it to a run you to a comment about the interactions with the Academy directly in the context of the letter but there is one important point that I want to make here and that is all of the materials that this committee receives. So if you're in public or in writing or in any form are made publicly available via the Academy's website so if you want to know what it is that the the committee is seen in the way of input from any any group. It is available. So right now you may want to Sure, maybe I can repeat again for the record the statement that I made that I think was two public meetings ago and in my view at least does not contradict what is in the letter from Charles Ferguson to the group is that the Department of Energy's Office of Science had no role in any decisions about who serves in the community. And no doubt the FOIA request is something that all of us have the right to do to get the information that we need from the government. And if there is a role for me somebody will tell me from the government and I will of course comply. And I I really know is that the DOE did not play a role but that is not what his letter says. I mean, when you're saying that his letter says that I read exactly what his letter said and it didn't say that it said here's our usual process. Here's what happens. I cannot answer who looked at this did DOE have a role and you've taken one of the agencies within DOE and said that this department had no role and you're saying that that's what his letter says, but his letter doesn't say that so we don't have it in writing. And it's just, you know, I waited until the transcript was played till the video came out so that I could compare what you said to the letter that we got from Ferguson. And that is what I read to you. It's very general, and it does not answer any of our five questions. Now you're answering it here. And so I guess we've got that on record, but it's not what we got as an official response from Dr. Ferguson. Okay, I'm going to let Ron you take this one offline with you and continue the dialogue with the academies I don't think that this committee can do. The reason I'm bringing it up here I know it sounds picky is that I think the panel needs to know that there's a concern about the legitimacy here and this is at the root of it so I don't mean to take unnecessary time from important work but if you want credibility then it would be good to resolve these issues officially. We are we are so surprised. Are there any other comments. Amy and Cronenberg. You may be muted. Yes, thank you. I'd like to go back to the presentation from the press folks this afternoon, and I have my question is directed to the concept of only having very senior people involved in the work of the group how we want to call consortium. I can understand some of the advantages. But I would have liked someone to discus