h care access. So we looked at this and what I'm showing you are three measures our own area deprivation index which is commonly used by CMS today, and then also social deprivation index which has also been commonly used across different literature. One of the things is that from the six categories of this very simplistic approximation of neighborhood we in fact can improve whether or not a patient will come back in 30 days, over 31,000 plus heart failure patients admitted to the Emory health care system by almost 10% improvement, which I think is really exciting. So this suggests that in fact, you know, ADI and SDI both may not capture neighborhood or community factors well, we really want to think about how we can get at better data. And somehow I jumped off sorry seems to be unstable. Somehow I don't like the internet here so I tried to tether to my hotspot. Sorry about this. I don't know if my phone will be any better. Yeah sure if you don't mind if I can use a quicker that also works as well. I'm pretty agnostic. Yeah, so, you know, thinking about this we actually dealt into the six categories a bit more. And you'll see it along the six dimensions and interestingly enough, even if we adjust for patient characteristics so this is looking at age gender comorbidities. What we found was that if you can advance one. We found that pharmacy and grocery actually have protective associations. So that means that if you live near more pharmacies and more grocery stores in the census track, you are less likely to be readmitted in 30 days. And whereas surprisingly, well, at least for me, restaurants have a negative hazardous association with 30 day readmission so somehow, if you live near more restaurants you actually were more likely to come back and surprising. What can we do with this data right what can we do this with this information. We've been thinking about how you can pilot pharmacy interventions. So the idea is that if you don't live near pharmacies in a census track that's not nearby. Maybe what we can think about doing is doing a meds to bed program where we send pharmaceuticals to them or we think about you know sending someone out there and making sure that they're taking their medications and have that. Now in thinking about this a little bit further. You know what we also want to consider is how can we actually do better right so this is with very course measurements. Surprisingly restaurants is not better, and we really couldn't understand why because our intuition was, if you lived in a food desert right that would mean that you aren't anywhere near restaurants. The opposite is true so we dug into some of the census tracks, and what we discovered is that fast food restaurants are the prevalence right. And so that motivated us toward this next idea of how can we actually get at more fine grained measures. So as an example we might think of health care facilities, we might talk about it differently. So if you don't mind advancing to the next slide. What we're thinking about is how you can actually do human guided category refinement and what I'm going to show you is a use case on one of the six categories and we're looking at in the context of health. And we were looking at Twitter in particular because Twitter tends to be on structured data. So I wanted to say well, how you preprocess the data, right, how you maybe think about topics that are emerging in the tweets are the things that we want to consider. And in thinking about that we really want the domain expert to come help us guide these clusters or these refinements because for every category we're going to think about. You really want it to, you know, sort of reflect some, some notion right, or at least maybe potentially improve it. But on the next one. What we discover what we've done in the interactive under the hood is that we've developed this unsupervised machine learning model that can take these tweets. Extract keywords get at the representations or the embeddings right so we're sort of agnostic to any big changes in the natural language processing world thinking about large language models they're sort of plug and play for what we've developed is this mechanism to figure out well how do I learn word representations where I push similar words together in the space and to distinguish this similar words and what this allows us to do is identify topics that might make more sense. So if you click on the next. What I'm now going to show you right is a comparison of sdi. If we looked at just the health score in terms of Twitter, you'll notice that it does a little bit better. Now if we just do you know this topic refinement without human guided annotations, you'll notice that we are in fact not any better than sdi. But if we go through this iterative process of defining and pushing keywords that make sense together, we in fact can do a little bit better. Right. And so this is, you know, an example of how I think human domain knowledge can help guide these refinements. But also in the process when we've been thinking about this right is that we've been treating each data stream as separate right and in fact every data stream is likely to be correlated with one another. So when we've been thinking about the pharmacy one what we did is we pulled the Georgia board of pharmacy list. And what we found is that there was very little congruence between our data and the pharmacy data. And so if you advance to the next slide. What we really want to think about is how we can match any two objects between different data sources. Now traditionally in the computer science domain, this is known as entity matching. So if you look at these two entities right one is from Zomato which is you know, years ago and one is from Yelp, would you say that these two are the same. And based on right yes right it's clear to us that these are the same. This is what's an integral portion of database right and it's very well studied in database. Now if you click one more. The biggest problems with current models machine learning models that are trained on this task is that in fact these models assume the two data sources share the exact same columns. Right so someone's gone ahead and align the databases, and then someone's gone ahead and clean the values as well so you can imagine this is actually a lot of work. And so if you click on the next one I'm going to show you what it looks like originally right in the pure original data, because real data is messy real data is ugly right things are not going to align and we cannot expect that we want to pay the price of manually annotating thousands and thousands of these to get which pharmacies are really true and which ones are not. So if you click to the next slide please. What we so what I'm showing you now is for common benchmark data sets. So version one is the original benchmark data set that has been cleaned has been standardized version for us to take the original models without any cleaning. And this is an existing state of the art model using large language models. So what you'll notice is in fact is there is a significant degradation and accuracy right in fact if you look at the last two lines it's really terrible. So if you click on the next one. What we've developed recently is this new deep learning model that really learns nonlinear relationships between column names and values. And the idea is that we can learn which columns we should really focus on which values are the ones that really matter. And if you click one more time. What I'm going to show you is the results from our model right and what you'll notice is that because we're thinking about this from the perspective of real world data right we're really modeling interactions across the columns and the attributes. You'll see that we actually have a relatively stable performance. There is obviously going to be a performance drop right as you could tell it was a much harder problem. But in fact it's not terrible right it's actually fairly usable. So last set of slides. So hopefully what I convinced you of today right is that really we can think about these new data streams and thinking about integrating them in right beyond traditional environment. We really want to think about how you incorporate human interactions to get at these better fine grain metrics. And really when we're thinking about these data integration models these entity matching that we want to do we really should train it on real world and messy data sets not on the clean ones that we assume will do well. And with that I'm going to if you click two more times. Thank my collaborators and my students who have made this all possible and obviously NSF NIH and J&J. Dr. Ho. Our next speaker is Dr. Thomas Harten. So Dr. Thomas Harten is the chair for evidence based toxicology in the Department of Environmental Health and Engineering and John Hopkins. Dr. Thomas Harten is the vice president of the University School of Public Health. He also have a joint appointment at the Whiting School of Engineering. And he also have affiliation with the Georgetown University. And he is also a consulting vice president of XSIM in your audience. So without further ado. Dr. Harten. The University of Constance where just give a lecture. Okay. My slides I've made available I will show the QR code at the on the last slide again. I think for this discussion important I'm also the field chief editor of frontiers and artificial intelligence so I share a very general interest in AI. Next slide please. At the moment the synergy of three elements the increase in data the increase in computer power and the increase in our AI algorithm. Our data are increasing by more than 60% every year. They turn the world so which means more than 90% of all data in the world have been produced in the last two to three years. Moore's law gave us over the last 60 years a doubling in computer capacity every second year. But AI for the last decade has been doubling in capacity every three months and altogether next. This led to about a billion fold increase in the computational power of the systems over the 60 years of my lifetime. Next. And this is leaving nothing unaffected. It's no longer the world as we know it. This is also environmental health and toxicology as I want to show you in the next slides, please. We have seen that on P day, the 14th of March, GPT for was released and it is now outperforming most human tasks. We have simulated by exam of a lawyer in SAT. These are astonishingly powerful models as everybody is witnessing at the moment. Next. What we see is that over time, something is happening that we have moved to deep learning with which the AI really became powerful. We are moving at the moment to deep reinforcement learning, which has made this tremendous possibilities happen. And now we are increasingly also further working with distributed agents. So not just one central computer doing the entire job. Next, a very strong feature of this is the so called foundation models. These models are now these big models like chat GPT. And these models are no longer trained for one specific task, but they allow us to fine tune them use them as a foundation and add our problems to the body of work which has been done in the past and they especially are being trained on what is called multimodal data, which is using pictures, they're using videos, they're using text speech and all of these together. And this is the basis of the current foundation models next. Yeah, you choose Lewis Carroll. Sometimes I've believed as many as six impossible things before breakfast. And in fact, several things I would have called impossible a few years ago have happened in recent time. For example, number one. Alpha zero showed us that without ever studying a human game with an eight hours. Alpha zero is playing so far better than any human and so different than any human that human players now study how chess could be played more effectively. And there's several others. Just to mention, while in 2020 there was no AI designed drug in clinical trials in 2022 there's already 18 AI first drugs in clinical trials at the moment and you can marvel at some of the others independently. Next. In toxicology and safety sciences environmental health. There is for my point of view a very strong need for such type of technologies to revamp. What astonishingly has not changed for long. Most of the methods we are using in regulatory toxicology have been introduced when I was not yet born on kindergarten. I had the privilege to lead for the Department of Defense, a future directions workshop. I found to court chairs with Anna was a scene from Columbia and by tissue from Texas AM and this 30 avant-garde stick toxicologist and 15 agency observers next. We published two weeks ago. This workshop is document, which is in essence a call for human expos on project. Next, it has three pillars. And I already described them in this paper earlier this year. Next please. These three pillars are to make toxicology back please more exposure driven technology enabled and here AI is one of the core technologies we refer to. And important for today's discussion evidence integrated. Again, AI is the methodology here. Next please. That's exactly what it says is AI different way of evidence integration and toxicology. Next. The starting point of machine learning and AI is big data. Actually, I like to define AI as making big sense of big data. And the important part is AI really shines not just if the volume is big, but it needs variety needs the multimodal type of data. It needs the velocity is which this information is accumulating. And next. We actually do have a lot of sources which feed in our field. We see that more and more legacy data of the of the past are created in databases. We have a lot of scientific literature accumulating. Pub made alone gets about 1 million entries per year. 80% of them have show some chemical effects, not as drugs and toxicants but often as inhibitors or effectors in our systems. The internet is full of data also in the safety sciences. 100,000 safety data sheets, sensor technologies, robotized testing, the omics technologies and high content imaging are new sources of data rich technologies. Next. Interestingly, at the moment, it is an enormous part of the work of any data scientists to get access to later data and clean them. About 45% of the time of a data scientist is used for this purpose next. We, in this case, I have to praise Tom Lüchterfeld, this talk track who is working 50% of his time in my team and has is entertaining two companies in the ice space. He has been programming so-called bio bricks, which is a one line command code to import an entire databases and he has done 50 of these, so that essentially all relevant databases can be uploaded within minutes to an hour in total to give the largest database on safety science data in the world. Next. What you can see here is one example, the first five databases he combined in chem harmony. For example, they are giving us 200 million cases where we have a chemical, a property and a result. And this is a treasure trove for training now models on these it is definitely a first time really a big database of chemical effects. And we are going to make these 50 bio bricks soon publicly available that really was a one line command without knowledge of the structure of databases or the programming code being used. Anybody can import the relevant information. Next. A big part of the progress at the moment is natural language processing. The big advance at the moment is really that also this becomes multimodal already now scientific literature is essentially read by a computer, almost as good as by a PhD student but not one paper per week but millions a day and never forgetting anything. And the big challenge is actually multimodality. The reading of tables and figures is the challenge we are seeing at the moment. Next. But if you see the number of scientific articles logarithmally increasing at the moment. And this is really of key importance. Interestingly, next. If you ask for what is the information coming from which mage maids chat GPT and others at the moment intelligent. 9% of all data, which went into training this are in the science and health area. And you can see here the top sites. Very clearly the open access journals of of plus and frontiers are representing a very important source. So this is really a reason why we can expect these models also to help us with some of the of the task attend. Next. Here you see in a logarithmic scale, the increase in these large language models and already the contribution of bidirectional encoders which Google made in around 2018. The bird system led to next, specifically pre trained systems of cambert and cybert next please. So these are systems where the context of scientific articles is considered that the system understand it is reading now a scientific article. And then, next, in February of this year, Microsoft released bio GPT. Bio GPT is now specifically trained for scientific literature. And next, you see the effectiveness of this bio GPT is performing better than a human annotator in annotating scientific articles to retrieve the respective information. And this shows you the more enormous progress in this field of integrating information. Next. Our own interest is very much into systematic reviews. And one of the products talks track has developed is called sis ref. It is a publicly for free available program software, which has given rise to more than 10,000 systematic review projects, which are registered here already. And is using system semi automated systematic reviews machine learning. So it trains on the inclusion criteria and then the actual choice of abstracts and after 100 to 200 abstracts by human assessor has learned as good as a second human to find which papers should be included and can then assess hundreds of these 1000s of these next please. So what this is next, what sis ref is doing is auto extracting annotating. We have we are boost during this at the moment to recognize natural entities such as genes enzymes, and others, and the causal relationships between them. And our goal is really to fine tune this to in order to import talks relevant information out of the literature. Next. In the context of the oops. This is really forward. Please go for a click. And again. mistake. Okay, so what we have been doing ourselves is now developing predictive algorithms already in 2018, which built a large model for 10 million structures for 900,000 we had some type of information 74 properties. And we showed that we were able for 190,000 chemicals where we had classifications done based on OECD animal tests that we were 87% correct in predicting these, while the reproducibility of the animal test was only 81%. In the meantime, next, we have shown that also with respect to human data. This model is superior towards the best animal test in predicting human skin sensitization for example. Next, in November of last year we showed we can run this on thousands of chemicals in this case we chose 4700 foot relevant substances and carried out in less than an hour. The equivalent of 38,000 animal studies which would cost more than $250 million. And we were again about 83% correct in a small validation set we analyze so at least as good as the animal tests. Next and at the Society of toxicology in March we showed that also the more complex toxicities such as cancer and reproductive toxicity can be predicted with reasonable predictive values accuracies of 75 and 82%. Next, in the context of the European on talks project, a $20 million project with 18 partners, us and talks track as the US partners. At the moment we are applying this methodology to liver kidney and the developing brain. I have the privilege to lead the AI part next, which is essentially trying to use his ref to extract data from the literature. Next, it is using databases through these bio break tools, which I described and third. Next, based on another product from one of Tom Lichtenfeld's companies chem chart, we are crawling the internet for soft data this together next is giving us the data. We then use in the second phase next, in order to train a variety of models. And the cost based structure activity relationship is the method I described for chemical structure based predictions, we have been developing. But we also feed automatically in physiological maps and AOP networks to understand the causal relationships. And this next is being used from understanding the perturbation of physiology and the chemical structure properties, a probability of hazard. We can predict, ultimately, how probably how problem is it that a certain chemical of interest has a certain property. Next, this is leading us to a key problem from my point of view in environmental health, which is that we like to work in a black and white environment of a substance being toxic non toxic carcinogenic non toxic carcinogenic. The probability is a lot of grades of shade shades of gray. There's a lot of uncertainties. Next, and we have made an AI based probabilistic risk assessment, the core of evidence integration in our project, and we're going to have next month the second workshop on probabilistic risk assessment. This is the paper from last year, where we are looking into perturbation of biology to train for probability of hazard. Next, I think this is essentially coming to an end here. However, we are at the moment witnessing that our traditional system of hypothesis driven research experimental tests of one aspect after the other is being enhanced by two things. Next, one is machine learning and AI through big data we are synthesizing evidence and combining the various pieces of information. We have evidence based methodologies. You can say this is the best what humans can do systematic reviews quality scoring and similar of the privilege to have the first chair for evidence based toxicology and we are hosting the evidence based toxicology collaboration, and we're trying at the moment to bring these two together so that we can use for reinforcement learning the evidence based methodologies and help to mine the big data, even more effectively. Next, with this, I would like to close with john men at Keynes an economist who very rightly said the difficulty lies not in the new ideas but in escaping from the old ones. I hope I've shown you some of the new ideas brewing at the moment with talks ecology written with an AI with how I like to call it at the moment, a science and transition. Thanks a lot. Thank you, Dr. This is a wonderful talk. We don't have our third speaker today so we are going to go directly into a discussion and our panel. That was short. I can ask a just a little question and I think it's been been coming up. Not just in the context of this talk but I do I really appreciate all the, the really great overview of all the increasing increasingly available resources with respect to AI and increasing ways that we can use it for, I don't know, lots of things. And so what the question it's raising for me is around public trust in these tools and just recognizing that because they're so new, because people might be aware of sources of biases that already exists in various systems and technologies. How are we thinking about or how might you be thinking about in your work, how we can provide sufficient information for the public to understand the way these tools work in a way that also reassures them that it can be trusted, I don't know. Yeah, you nailed this. This is really the problem. But what I'm not arguing for is to give these tools any autonomy and see them as the ultimate decision taker is at the moment, a way of integrating data, which a human brain cannot handle anymore. And giving on the silver platter some suggestions how to interpret these. But we need the human in the loop, not only by training these models but also by evaluating them. We have to ask, does it make sense. What the system is suggesting. But I think it is at the moment, a prime tool to enhance our decision taking to look what smells like problem, what can be put on a back burner, perhaps, and increasingly learn what the quality of these predictions is via the very beginning.  So I'll go with the computer science way of saying that the, I think trust is potentially overrated right so you know there's all this discussion about whether we care about privacy and security but if you look at what everybody posts right like in a public forum. I do wonder whether you know a lot of it is we're just hiding behind the need to convince the public that this makes sense. I think what we can think about, you know, some perspective is to think about well how can we utilize it to make to help better decision making. In the end it should still come back to the human right and that's that's the best we can do for now. In the first set of sessions there was a discussion right if we keep training these AI on on human decision making. Maybe that's going to exacerbate biases right but I think a lot of it is well what if we integrate more data would that help mitigate some of that and I think those are questions that we don't know the answer to. And part of it is convincing the public that maybe that is the perspective that we should be taking. I mean when humans can find the bias machine can also find the bias we only have to train the machine to learn that bias is a problem. And, and this is why I think that that the machines have to learn from our evidence based approaches which are the most systematic way of objectively and transparently analyzing information. And there's also a big move to explainable AI, so that the AI explains why it comes to a certain conclusion. This makes is a game changer, if you want to take decisions on the basis of predictions. I'm in my latch off for those who are online can't see in the room. I'm almost afraid to say this because I feel like it's anathema to Americans but you know I feel like the games that I play on my phone I'm giving away more information than I give away through my medical records and trying to get one doctor to share my medical records with the doctor is ridiculous so are there folks talking about this in the US I mean it just seems like we should be doing a better job to advance the science I think somebody had a slide on that in the in the in an earlier session. But maybe that's a recommendation that can come out of this workshop is rethinking what we as Americans share about our health data. It's a private it really needs to be. I feel like people are going to really push on me so go ahead David. Yeah so so I'll push back a little bit and then I'll also give you words for encouragement. You know, so I think you have to look at the reasons why we have the protections that we have in place, because there is a history of abusing the power that we had. So, you know, I think we need to be very careful in in stepping back from some things that we put in place to protect the most vulnerable. You know, that said, the all of us program has an extraordinarily aggressive data sharing policy, you know so all of the data that they are collecting on everybody that they are collecting is made available. You know, in a pretty transparent way, you know, we are going through an effort now and I don't know if Allison's going to talk about this to add location information to there and so there's going to have to be, you know, as we were having the conversation earlier today, there's going to have to be some level of protection, you know that's not going to be as transparent as as you know the so the HR data is going to be. I think there is reason for optimism that that we can work in other ways within the existing systems that we have so we don't have to walk back on the requirement for IRBs to look at things that we're doing. So I would also say you know there's been a lot of research done at the federated learning and privacy preserving world as well. I think what's holding it back a lot is that databases aren't standardized right, even if you look with an epic any two healthcare institutions have different ways of doing it. And that's really unfortunate because you can't even share within the epic ecosystem. And I think part of this is pushing, you know, those problems to the forefront. A lot of it we've been sort of hiding behind. Maybe we should develop fire standards or maybe we should think about Odyssey or omap or different standards. Probably needs to be some push towards just saying this is the one standard we're going to go with and this will enable more of the sharing across institutions. And in that perspective then the sharing will be more along what we've doing with our games or posting on social media right about ourselves. And the good thing is we're not alone. The same problems hold for every aspect of life and these technologies pose new challenges but they also give some answers. We heard some of them already. You can train things systems behind firewalls and never hand the data out. And that's the federated approach you have blockchain to exchange information in a secure way. You can produce synthetic data sets which still contain the information but nothing is identifiable anymore. There is really a number of tools coming up because we have to solve this. But as a society we should be interested in encouraging data sharing by telling the people what we do and what we will do and why we want the data. And then there's an astonishing willingness to share if they have trusted players. And this is something which is really important if you tell them what they get in return for making data available. People are astonishingly willing to share. I would like to ask Dr. Ho a question if you guys don't mind as important as the privacy conversation is. So going back to the social determinants of health point that you were making and I love all the sort of new and novel data streams that you are using and exploring. I guess my question is you know when we talk about things like the area deprivation index or the SDI. These are these constructs about latent processes that we can't really measure or ground truth in a specific way but we think. They're capturing a larger context around deprivation issues right and I completely agree with you that there's a lot of ways they're sort of misused or not used to their full potential. But based on your talk are you saying like you know these novel data streams are they doing a better job at capturing factors related to social determinants of health or are they just seeing something very different that's perhaps closer to the human experience in a way about you know things that we think of as exposure measurement error classically right like I think the novelty is they can really get to what people are experiencing on the ground. But also layered onto that do you sort of worry about like biases and who's contributing the data or who geotags their tweets versus not or like open street Matt. I know that's a huge topic but I guess I just want more of your thoughts on what are they really capturing. Thanks. So we've had this conversation a lot between my collaborators and myself. And part of it is, you know, gaining acceptance in a field. So I, you know, I feel I'm an outsider right and people really don't like me. Maybe because I, you know, computer scientists stomping over other places. But I think what we're capturing right so a lot of the ADI and SDI are very census based measures and they're captured every 10 years or something and you really don't see changes in them fast enough right and if you look at you know how populations have been shifting towards cities there's there's a lot of that at play. I do agree a lot of these streams will have biases right but the hope is that in the integration of more of them right. And looking at how you distill information and get it. So let's take the pharmacy one for instance right so what we found actually is when we asked the Georgia Board of Pharmacy for the list, they sent us everything, including the small mom and pop shops that typically patients won't go to right most of us will use CVS Walgreens and those things. And so it's unlikely that someone will go to those places because that's just not what they're used to right. And so from that perspective, you know in some ways you're capturing more of what is likely to be used right, and it is a different construct than what ADI and SDI were set to do right. And we've been having this discussion as to how you can showcase that it works, because everybody, you know. I mean, I think Thomas said it best right like it's really hard to get rid of the old ideas right now and replace it with new ones. Everybody thinks deprivation is probably the best indicator of neighborhood measures or at least you know it's very commonly used and very widely accepted, and to propel new measures is actually hard unless you sort of like validate against existing ones if that makes sense and so I think what we're seeing is there's mismatch of information across the two sources. I'm not really sure which one you truly trust right and really there needs to be more analysis to determine, you know, what makes sense so all the different categories that we presented. They'll all have different impacts right healthcare facilities might look quite different than what we might see in pharmacies and so I think we've started to tease and unravel it I don't know that I know the answer yet. Definitely, but I'm, you know, all this work that I presented is all under review at the moment and so it's all very new, even from my group. If may quickly comment on this. I think that these type of indexes are 20th century type of science. What humans can do to handle complex data but we are also losing a lot of information by squeezing everything into an index and the opportunity of machine learning is to leave them as they are and find the way through all of this. In the end, identify what are that what were the what were the parameters which really informed best for meeting our result or needs. I think the we should not sort too much humans can only handle seven variables. The systems can handle billions. There's hands up. Thank you very much Thomas I agree with you to a certain extent but I think that what Karen was saying, you know the trust so as a public health practitioner, I tend to be contrite with respect to the history and that trust thing. Just look at the National Public Health Study of simplest right so that's you know that trust thing is real, but Joyce, in terms of, I think the deprivation index right well, for example with COVID. The CDC is now considering and has put into motion, taking away SBI and replacing it with HOI. Okay, so you know next time I see y'all would like to answer to. Run that versus HOI right EJ and EBD environmental burden. Plus you was and let's see what happens. Okay, terms of the malls okay. Yeah, but yes, it's a difficult question. I'll make my graduate student do that. So we also have a question from Christian. Yep, so I am not there to be with you all today it's, and you don't really want to see me on camera so I apologize but going back to this question of these deprivation indices versus sort of the newer version, it opportunity for using natural processes and AI to really think about it. I think we also as scientists really need to really put everything in context and really think about the questions we're trying to ask with these data that we have and so, and the mechanisms by which these processes work right so we're trying with the long term impacts of deprivation over a life of course on an individual in accelerating disparities and disease, ADI might be appropriate, right, because we can't use, but if we're trying to predict hospitalizations and or better care and or exacerbations of disease and a more acute framework than these other models and AI really do make a big difference so I think it's like anything we've always done in epidemiology but we really have to think about context that we're and the questions that we're trying to answer, and and the mechanisms to really tease out the value validity and capability and value of these different metrics in these different analyses we're trying to accomplish and I think that's going to become really critical as we start to think about all of these different opportunities for data linkage that we have available. I think the basic questions always is can the answer be in the data, I will always find an answer, but the common sense should tell you, can it be inside. If you train a prediction of stock markets on weather data, you will get a result but not something I would invest on and this holds for everything you're looking. Yeah, and I guess it goes back to that classic, you know, back in the day, just to the number of flamingos predict births in Florida I mean I think that was a critical question 25 years ago and AI could be you know so how do we avoid those, you know, those false discoveries given that we know that I will always find something so what is it as humans we need to be training our students thinking about things to avoid as we start to make use of this technology in these innovative ways. Sorry, you go for it, go for it please. Alright, so I mean I think education is a crucial thing and I think one of the things that we have sort of masked under the hood is critical thinking. And I see this as a faculty member these days right especially thinking about the impact of COVID. You know, I even see it in my own kids right they don't have much attention to focus on some particular thing. So part of this is figuring out well what what can we engage them in. And how can we make them. You know, I think the biggest problem is, we teach them black and white, right but the bigger thing is nothing in life is black and white and so how can we think about ethics and legality and can we teach that at a younger age and I think a lot of this can help address stuff. So when should we think of false discovery as false discovery one might it be something correlated under the hood. And I think that might be the bigger direction to push as well. So amazing series of presentation really and one thing that I've been trying to do when everyone was present is to go online and see whether I could find the method you had for instance by your GPT. Really a lot of excitement I really thought perhaps one day I will have an helmet that's not only typing in the question where just someone can read my mind and just write for me. But really, I'm joking about this but the part of the question is what what what has been presented today is ready for prime time and available to everyone. It's against what is still being developed. And it's a general question about to the speakers but also I'm thinking, perhaps this could be one of the outputs of the workshop where we could create brackets of different products that scientists or practitioners can already use and they're available online and hopefully for free some of them perhaps others under the paywall versus a project that we might be on the lookout for and perhaps in one year 10 years that might be available to us. So would you see this what type of reaction you would have to this idea. I mean we are at the moment very impressed by the large language models and generative AI. There's only one of the many many flavors AI comes in. And these systems are because of the nature how they are trained and what how we are prompting them to work. They are very much hallucinating. But already now they're extremely good. Yeah. I was asked to write a comment on an article by a colleague and for the fun of it. I put it into GPT for said summarize it, praise it criticize it and put it as a supplement document to my own comments. It was pretty good. I was very impressed. I could have just submitted and everybody would have been happy. This systems get better and better. They were not made for scientific reference texts. They were made to make things up to fill gaps like we do when we are chat when we are chatting their chatbots. Yeah, they're trying to to just get through with whatever things are close to reality. But if as soon as we start really putting in more science as input data, more scientific processes as the way we reinforce, they will do scientific jobs. I just read an analysis by a venture capital company. They say that now already a draft of a scientific paper is reasonably good. But in five years they expected to be of the quality of the most skilled scientific writers and writing scientific papers. We might be the last generation who learned to write papers on our own. The next ones will just know how to correct them and polish them. Do you have a question. I saw the race. So I think, you know, it depends on what you're thinking of as, as available. Right. So I think Marzia sort of alluded to this that a lot of this stuff is not reproducible and I don't know that you want to, like even thinking about the chatbbt right, you prompt it with the same question, and it regenerates answers for you. And that really harms reproducibility, because you, you know, any two patients, do we really want different answers depending on slight language tweaks.  Is it to write papers in a deterministic or less, you know, generative, where there can be a lot of craziness? Do we need to put it behind a paywall? Who's going to use it? And I think if we think about along those dimensions, there will be a bunch of tools that are available. But I think in the end, reproducibility is the biggest question mark in my mind towards achieving. And I think there's been this push towards fair principles as well. Right. And if I'm going to use that as the ethos, then I'd say a lot of these tools are not there. But if for, you know, for discovery, for thinking about it, I think most, a lot of it will be available here. We have Genshin, Nix. Very nice discussion. I want to add on to the discussion about reproducibility. Because I'm from the geography background, there is a very good paper on PNAS by Professor Michael Goodchild about how to do reproducibility, how to achieve reproducibility across space, especially for this larger model, because our topic is about environmental health. So it's care about how the model prediction across the space. So usually all the times when the foundation models, the prediction is what performs very well, even if you're looking from a geographic space. So it can achieve very good in US, but not in Africa or other places. So I wonder, like, is there any suggestions on, but this model is very hard to modify or even to, so is there any suggestions or what will be the solution or suggestion for us. So I think, oh, sorry, go for Thomas. I mean, the first thing is, I think actually the opposite is the case. This is one of the most democratizing technologies we have ever seen. If you see that GPT-3 found 400 million users in two months, there's never been access to technology so fast anywhere. And the prices are extremely affordable for what it delivers actually. I don't want to advertise for them, but I'm just saying it is delivering already astonishing good products. And if you now unleash all of these clones, all of these open source activities, we will be seeing tremendous improvements of these models. But we have certainly also to manage expectations, what can we expect from such a system. We have to set in place things to control that you cannot assume another identity that you should be transparent where you use these tools because it is plagiarism if you just squeeze out something from the similars in the world who have done something like you. And so these are challenges for society, but the access is I think the least is the one I see most positive of all. So I think there's been a big movement towards making all of the data trains on open. So there's been a lot of discussion, especially in thinking about GPT models and chat GPT. A lot of it is reinforcement learning base. There's human in the loop. And those prompts are not available actually for anybody to replicate. And so I think until we get to the point where we can get all the data that went in and all the human annotations that came along with it, it will be very hard. But that's exactly where, you know, they're making the money right is they're not going to release this data. And so I think part of it is, you know, maybe thinking about open sourcing some of this so that the community creates it helps curate it. We think about sharing prompts and how we're using it and exact samples that we're using for it and clearly, you know, if we're thinking about health care, or health diagnosis there will be HIPAA issues but thinking about maybe synthetic data to help train it. So there are all options that are feasible for these large language models. Yeah. I just want to come back to Andrea's point. One of the things that I would love to see come out of this workshop are some tools like a really short just toolkit for people who are interested in integrating these areas so that's just going to inquire about ethical guard rails. Thomas, could you say something about that we'd be remiss if we didn't discuss that a bit with regard to the language models. Overall, actually. The first thing is the, I think that ethics should be the guidance for regulation. You cannot regulate on the basis of technological developments. If you see that this is doubling in capacity every three months here. No legislative process is fast enough so we need to understand what are the principles and they're not very different to the principles which are behind current regulations and laws here. You cannot cheat. You need to be clear who is talking and you cannot take what other people produced and sell it as your own. I think these are important things. Then there are specific things where we have to find ways of validating these tools. We need people we can trust who give it a check and tell us how well does it work. I mean I've been heading a validation body for cell culture models for the European Commission. It is a big task but I think the. It is exactly what we need if you want regulatory decisions to be taken on the basis of AI, then somebody has to write it and and really explored extensively to understand what is the risk in using such a tool. I have a different type of question for the panel, which is honestly inspired by the comment. One of the comments that Thomas made that one of the tools you presented can be better or similar now to a PhD student. And by the way, my PhD students are better than me so I'm really worried I'm going to be phased out soon. But the thing that I really wanted to ask is that there are two issues I would like to bring up. One is I read at some point a comment by Noam Chomsky about charge GPT that the difference between machine learning artificial intelligence and human is that artificial intelligence will give you answers that are probable based on the data, but only humans will give you answers that are improbable. I want to see whether that is true given given your knowledge I seem possible to me but I'm not an expert so whether I mean and I to say probable is very possible that we overestimate creativity we have in our jobs. I mean, it's very possible in most of us and certainly this true for me come up with probable hypothesis and solutions rather than improbable. The other question I guess is more related to another issue which is the issue of especially inspired by the personal experience that I have with charge GPT about hallucination that charge GPT seems to lie flat out sometimes I tried. I asked the writer biography of Andrea Baccarelli, I look much better than I really am. I have so many degrees that some prestigious schools never even thought about having and and I to say I mean, when I work with my team, I know the level of trust I can put on people. I started to learn to learn that and I know how to gauge I mean some someone can be great at epidemiology but I cannot trust them in toxicology or vice versa. Sometimes some one is can be 99.9% honest and transparent other people less so I mean I work with any type of people. So I'm wondering how it is a way to gauge the confidence we have in the systems I mean how to do that of course every system should be perfect but but as myself I also understand that I now started to learn what I can trust in charge GPT and what I don't I mean clearly I cannot get get charge GPT to write an article about myself. And not without checking it. I mean I did I did the same I tried because I said who is Thomas out and then I was also assigned an invention which I have never done. But I can check it yeah I can I can check the possibility most of what was written was pretty good short summary of which would have been for me difficult to do so condensed. And I, we also have to see, we have to train these models for their purposes. If you want to write a scientific text and say you restrict yourself to actual sources which you find on the internet. That's a completely different beast than a chatbot here, which is trying to give an answer. And it is trained on on on all of the BS in the internet. So we get to get this back. Yeah. And but also humans are chatting this way. I mean, I would be happy if they would give me the most probable answer they know. Very often they give me the most provocative answer the one which they just come up with and in this moment yeah. I think that we are really, we need to define what is not chat GPT doing for science know what will be science GPT doing for us, if we now give engineering criteria what we need to make it really useful for our purposes. That reminds me of Isaac Asimov foundation series. Did you guys remember that it basically everything is predicted based on statistics but they can't predict the one like rogue actor. He was biochemist if I recall correctly. I think so. Almost time. I want to thank our panelists again for the wonderful presentation and the discussion. I think one thing that we can all come back, go back home and think about is all with all these AI tools that we talk about today how would we as experts in certain area, work with all these tools, and how do we trust them how do we have as other help other people to trust them as well. It's going to be there so we need to be ready. So, thank you. Thank you Thomas. We are going to have a 10 minutes break, and we will come back at 320. Welcome back from the break. We will get started with this rest of the session today with a brief remarks by Rick Vojcek, and then followed by Eric Topolski note and finally a fire set chat with the two of them. So first, I'll introduce Rick Vojcek, who became the director of the National Institute of Environmental Health Sciences, one of the National Institutes of Health and of the National toxicology program on June 7, 2020. In this roles he oversees federal funding for biomedical research to discover how the environment influences human health and disease. He and his staff receive input from several advisory boards and councils to accomplish this significant task. Prior to becoming director and since 2011. Dr Vojcek served as deputy director of NIEHS. In this role he assisted the former director Linda Burbaum in the formulation and implementation of plans and policies necessary to carry out NIH as missions and the administrative management of the institute. So please welcome Dr Rick Vojcek. Terrific. Just doing a sound check. Can you hear me okay? Yes. Okay, good. Lucy, great to see you by the way. It's great to be here this afternoon. And so thanks for this opportunity to provide just a few kind of remarks. I suspect just looking at the program is sorry that I haven't been able to join for the rest of the day. But I just want to reinforce. I think some of the things you probably already heard. So I'll just start off by saying that I think we're working in some very exciting times and actually just to be honest, we're working now at a time and we're doing things that I've been waiting for my entire scientific career. So we can now imagine employing fundamentally a transformative approach to understanding health and human disease. And this involves integration of multiple different types of data. It could be involving genomics experiments where we get about go beyond one gene at a time to look at it whole genomes and whole transcriptomes. We can incorporate data from environmental sciences and we can also incorporate very importantly data from research that evaluates the social determinants of health. And so now we're taking this more holistic approach. So it's not just about transcriptome or not just about genome or not just about one environmental exposure. But we're not taking this holistic approach that recognizes the interconnectedness of all these different factors and how they influence human health and well being. So a key driver of this approach of course is what you've all been talking about. It's having available and our ability to generate omics data. And this is genomics data, epigenomics data, proteomics data, transcriptomics, and more recently, this whole notion of exposomics data, which as Chris Wilde defined back in 2005, it's the totality of exposures over the life course. And I'll say, you know, recent developments and AI have the potential to provide us with the tool is to integrate across these various omics data sets and other types of data that we're collecting. And we can do this now in a way that will help us to understand human biology and to understand the etiology of human disease. The expectation is that by leveraging AI's analytical capabilities, biomedical as well as, you know, environmental sciences can join forces. And they can begin to elucidate those relationships between environmental exposures and human health with a degree of resolution that I predict that we haven't seen up to this point. Hopefully you've been participating in the meeting today. So, I mean, the purpose of this workshop is really to gather experts from diverse disciplines and from various sectors across the biomedical enterprise for the purpose of delving into some of the latest research findings and to explore opportunities for integrating environmental and biomedical data. So we hope that by leveraging new and cutting edge developments in multimodal AI, we can begin to unravel the complexities of an integrative health model. And I'm a big believer of the types of collaborative product platforms that will foster discussions on innovative approaches methodologies, as well as technologies that can enhance our understanding of the interplay between environmental exposures and human health. So at the National Institute of Environmental Health Sciences and IHS. So we recognize the significance of AI and data science and driving innovation and advancing our mission. And IHS has already initiated several AI and data science initiatives and we are actively involved in developing and supporting policies for data integration data management, ensuring and ensuring that in IHS data and here to the fair principles and making them findable accessible interoperable and reusable and also the some of the new efforts around developing standardized vocabulary is an ontology is on how we handle environmental data. The convergence of AI and big data, including that from environmental health scientists, I think represents a powerful opportunity to drive scientific knowledge and improve the human health outcomes. So NIHS will continue to support and to develop resources that promote the responsible and equitable generation integration and utilization of environmental health data. We are committed to collaborating with other NIH institutes and centers to integrate our data resources across the broad the broader biomedical enterprise. I know you've been talking about this today. And we feel that by leveraging the synergy of AI, big data and environmental health research data, or you can expect that we can make significant strides and unraveling those complexities of human health, and to ultimately pave the way for innovative approaches to promote healthier and more sustainable lives for all. So that's, that's it for me. I will turn the virtual podium back over to, I don't know, is it Lucila. Yeah, thank you so much, Rick. And everyone will hear from Dr. Vojcik again at 345 Eastern, when we will start the fireside chat. And we have a short keynote address, and I will introduce someone who needs no introduction, Dr. Eric Topol was Professor of Molecular Medicine and Executive Vice President of Scripps Research is the founder and director of Scripps Research Translational Institute. He has published over 1300 peer reviewed articles with more than 300,000 citations. He's an elected member of the National Academy of Medicine and one of the top 10 most cited researchers in medicine. His scientific focus has been on the use of genomic and digital data along with AI to individualize medicine. He's also a practicing cardiologist in 2016. Eric Topol was awarded a significant grant from the NIH to lead a part of the precision medicine, now called the all of us program from NIH. And prior to coming to Scripps in 2007 he led the Cleveland Clinic to become the number one center for health, heart care, and I was the founder of a new medical school there. He was commissioned by the UK to 2018 2019 to lead planning for the National Health Services integration of AI and new technologies and has published three bestseller books on the future of medicine. So please welcome Dr. Eric Topol. Well, thank you very much. I'm glad to join and of course be mainly getting into the AI in the medical sphere. So let me share my slides so I can get going here. Looking forward to our discussion. And hopefully, you can see my slides. Is that right? Yes. Okay, great. So I'm going to be talking about multimodal AI, which is really a very recent and exciting opportunity. We're just starting to get into of course. And the first thing that we get to when we start to see AI impact is reducing the diagnostic errors. There are over 12 million serious errors a year in the US and there's a classic NAM report about each person, each American will experience at least one of these diagnostic errors in their lifetime. And when it when the diagnosis is thought of in the first five minutes the accuracy is very high but after that it precipitously drops. And we have a big problem with overconfidence of physicians who when a patient dies the autopsy is demonstrates that the wrong diagnosis is the case 40% of the time. The term precision medicine is a real problem because in effect if you keep making the same mistakes over and over again it's very precise but inaccurate and we need accuracy in medicine. And so what's happened here is that we move from the deep neural network like convolutional neural networks and recurrent neural networks architecture to a whole new architecture called transformers, which allows multi attention inputs. And this is where we get to how we get to multimodal AI. So just with deep learning we've had an enormous impact before unimodal focused on medical images and what the term I like to use of machine eyes obviously machines don't have eyes, but their ability to interpret medical scans is is a remarkable and far beyond human capability. So by training from millions of chest x-rays, the radiologist experience radiologists will miss in this chest x-ray, the presence of a nodule, which turned out to be cancer. And this is across all types of medical scans this is mammography the largest study that was conducted by NYU. It's much bigger than that. It's seeing things with machine eyes trained as inputs with large numbers of annotated scans with ground trees that you then can get to look at through the eye for example the retina ability to trap kidney disease blood pressure glucose diabetes control a window into the likelihood of developing Alzheimer's disease predicting heart attacks and stroke gallbladder and liver disease high lipids in the blood heart calcium score. It's remarkable. These are things that we can't see as humans. And then for the electrocardiogram as a cardiologist. I could never tell you the agent sex or the hemoglobin or the ejection fraction make difficult diagnoses be able to tell from the electrocardiogram the presence of valve disease and its severity, whether the person is likely to develop a arrhythmia such as atrial fibrillation or develop a stroke diabetes and prediabetes the feeling pressure left ventricle kidney disease hyperthyroidism. This is the ability of machine eyes which is right quite incredible and it's already being implemented in the particularly in Japan and Asian countries that are doing machine vision during colonoscopy and endoscopy picking up polyps that otherwise would be missed. And also, in real time, reporting the likelihood of whether they need a biopsy whether they are potentially cancer. So that was the unimodal world of AI until March, when we had the first GPT for multimodal large language model. And that has led to something that's not just remarkable with respect to the parameters, the interactions of the neurons as shown here on this graph, exceeding a trillion, but bringing together all the different modalities whether it's speech and voice and video and images and text structure and unstructured text. This is not something that happened overnight. This has been building up for decades and getting to this extraordinary levels of computing power pedophiles. And when we look at the parameters trained. We see GPT for as mentioned was a trillion and the tokens were trained. They've gone up considerably. And this is the meta llama model, which has got over well of 1.4 trillion. So basically, to simplify this, these are the building blocks, they should be connected but I haven't done that yet. But these Lego blocks. They needed transformer models, they have had requirements for massive amounts of GPUs. But with that, the ability to get into computing power of flops that are is just unforeseen. This required not to be able to annotate the input but to have self supervised learning, and that got us to this level of multimodal AI. Now, for the, the ability to understand each individual's uniqueness, all these layers of data can now be captured. And that includes the environment of course, the expo zone, but all these other biologic layers beyond just DNA and RNA and microbiome, the physiology through sensors, the anatomy through the scans and of course the electronic health record and the immunome. So we had a recent review about this topic of multimodal AI in medicine, and it leads to all sorts of remarkable possibilities. I'll touch on at least one of these but it's very broad applications, once you can bring together all these different field domains of data. This week, a couple days ago, there was this publication in nature biomedical engineering, which interestingly showed that with multimodal AI, this specifically Irene, it was far better than previous models for being able to make the diagnosis of pulmonary disease by integrating chest x-rays, all the lab tests, electronic health record unstructured text, also for predicting adverse outcomes of COVID-19. So this is one of the first validations of the remarkable increase in accuracy from multimodal AI bringing together these varied input. The hospital is a dangerous place as it turns out. This is a study from last year of 11 Massachusetts hospitals, some of the leading hospitals in this country. The adverse event rate of people in the hospital was almost 35 per 100 admissions. And you can see the breakdown about adverse drug events and surgical procedures and infections. That really sets up the hospital at home in the future, which of course would rely on multimodal AI. So we've done several reviews on this topic beginning a couple of years ago about the update on health and medicine, self-supervised learning, how big a step that has been because we didn't have data sets that were massive that could be annotated in this sphere. The one I just mentioned and the one that we just published in April, which was the Foundation models or generative AI, large language models that we're discussing here. In that nature review, we basically predicted what GPT-4 would enable. It wasn't yet put out, that was in March, in the middle of March this year. The idea of taking everything known in medicine through publications along with the different inputs of images, electronic health records, sensors, biologic data, all those layers of data that you could take medicine to a new plateau. Now importantly, this is summarizing the different phases of a large language model development. The biggest part is this pre-training and in GPT-4 that has used approximately 30,000 GPUs. So there's only a few places in the world would have access to that many GPUs, after which the fine-tuning that we're seeing now in medicine for various functions is much less computing requirement as are these reward modeling and reinforcement learning phases. Now, just to show you a fine-tuning, this was for medical imaging and it already very trained very quickly in 15 hours. And it could do things that's really remarkable and challenging, like looking at a chess x-ray and saying all the different devices that are present, which would be hard for even radiologists to be very accurate in this. And this has already superseded the performance of GPT-4. But there's lots of problems here. It's never so straightforward as you'd like. So we see we can have great summarization of lots of data on a particular patient. We can improve and even promote empathy. We can do all sorts of administrative tasks like getting rid of or reducing the need for data clerk function and keyboards. But there are lots of concerns about hallucinations and data security and bias that are very well founded, no less as I alluded to carbon emissions. And we need, of course, a lot more work to validate their presence and utility and safety in healthcare. So this is a good summary just a couple weeks ago in Nature Medicine on the chasm. I call it chasm of AI and healthcare. The fact that we have lots of myths and we have these realities that people need to be familiar with. So I'll just stop here, which is that there's this amazing two-edged sword with AI in medicine as it will be applied in many different directions. And the one, of course, that I'm most excited about that I'm not going to get into is the gift of time that will be hopefully afforded and propelled by AI and medicine to bring back humanity, which has suffered so greatly over several decades. So with that, let me just acknowledge a lot of my colleagues who I get the privilege of working with our funding support and open it up for our conversation and discussion. Thank you. Thank you so much. And I think we have about four minutes before we do start the fireside fire chat. So why don't I abuse my privilege as facilitator to ask a question to Dr. Topo and then we, I'm asking if people who are present in the room there can also help identify who has questions. My question is precisely regarding the large amount of resources needed to train the initial models. Do you think that will widen disparities in whole nations that will be able to do it and others that will not? Yeah, this is really troubling the CLO because only a few tech titans like Microsoft and Google and limited others can do this. So we have dominance, hyper dominance. So not even just countries, it's companies and globally. So some people think that there will be a much less requirement to do these do this free training in the years ahead that that will be compressed substantially, but we haven't seen that yet. So this is setting up a very awkward situation where the role of academics and so many others working in the space is on the fine tuning piggyback or not the actual development. That's okay because in effect, at the moment, these models like GPT for and barred and made palm these were not really medically trained at all. And so they do need fine tuning, but ultimately, we can't go on to be fully dependent on just a couple of few companies in the world that can string together 30,000 GPUs. It's preposterous. We're lucky to get a handful of them in our effort. So this a lot of people are not aware and not only that but the energy requirement, the cooling requirement. And I mean this is not helping our environmental crisis. So, you know, the people I had a recent discussion, a podcast that will be posted soon with Al Gore, who of course has been so deep in warning about the climate crisis and I asked him, what about this large language models, what's that going to do. And he thinks, you know, it can help come up with new ideas to solve the problem, but it's also going to engender more problems. Yeah, I wonder if Rick would comment on that environmental effect as well. I'm sure that there's much else I can say on this that I think Eric very accurately pointed out the environmental parameter here, and that we just have to be conscious of that. It's there. So, yeah. So I think we can officially start to the fireside chat, but I see a hand raised from Darryl. Thank you very much. You can unmute yourself and ask the question I think. Dr. Topol, that was just great. I, you know, I thought that we us that work on the public health expose on side the ecto expose on had a great catch all phrase being from the cradle to the grave, but you just topped us with that free tune to the womb. Wow. Thank you for that. But but no, but but what I'm more concerned about, and thinking about this, should we be concerned for an example with Irene, the input, are there equally weighted inputs of the five that you show there, or because this is iterative and machine learning, is that something to be concerned about. Right. Well, thanks, Darryl. It's funny just you pointed that out. One of my colleagues and mentors, Bill Kelly, he once said I didn't have the right title for that paper that review he said it should have been from lust to dust. So, yeah, a lot of different ways to express that. The inputs are the whole story. And obviously that's where we see the potential biases emerge. It's not so much the transformer architecture of the deep neural networks it's our, it's our culture. It's putting in the whole internet and Wikipedia and literature so inputs are big. And I think the biggest thing here is that also I study I didn't mention from last week from NYU, they took 330,000 patients from NYU house health system, and just basically use all unstructured text and structure text to come up with predictions of readmission of morbidity of insurance denials and prognosis of survival, extraordinary. So what we're learning is this basically machine ability to classify process, massive amounts of data. And what what we got to this point if the inputs are are good, and that I think we could say and obviously large enough. We're starting to see things that I don't think we expected to come up with. This is of course, the whole debate in the AI community, which is, is this some sparks of agi is the preprint of Microsoft put out is this and level of understanding that we did not expect to ever see if not this early in advance but of course there are many still I think this is a stochastic parrot. And it's just a matter of statistics and word prediction. It doesn't look that way to me, but it's somewhere in the middle here and it's, it's a source of big debate so I think the inputs is not is a big deal, as I mentioned, particularly the potential of biases but the bigger concern is what are we seeing here is this understanding is being called a world model understanding the world. I, I, you have to decide yourself about that one. Yeah, maybe let's say like maybe I can chime in here too. So, you know, I, you know when I when I look at some of the key challenges and opportunities, especially as it relates to say environmental data, you know, it's the, you know, one of the concerns I have is the just the broad heterogeneity and the diversity of different data. So it's very easy to say expose home is the totality of exposures over the life course, but what are we actually measuring. You get to the totality of exposures over the life course, you'll have less dust, whatever. And I think that that's, that's a real challenge. And the different data sets, you know, you compare say transcriptomics to you have the, the, the chip based transcriptomes then you have already seek and you're just the heterogeneity of the data.  The other question too is, is it, is it useful to do, you know, you know, transcriptomics on a whole kidney, or do we have to ultimately get to the point where we're looking at which genes are being expressed or proteins are being expressed in those transplanetial modifications in specific cell types within say the collecting tool, the kidney. So I think there's some challenges. So I just, I just worry a lot now about what we're going to get out is going to be a function of the quality of the data that we get in. The last thing I'm going to throw on the table here is the, just the overall. I'll talk about consistency and the sustainability of our data repository system. It's been very in my mind has been very kind of ad hoc and someone has some data they developed their own database and now we're, we're, we're hemorrhaging with databases. No good standardization consistency and how the databases are constructed. I mean, in the environmental health sciences we haven't agreed upon a standardized set of vocabularies and ontologies and how we collect the data.  You know, what do we do on a global scale to eliminate some of the redundancies that occur. Everyone establishes their own database because they know they can manage this. But I just worry that, you know, if there's no thoughtful integration across different databases, and that could be an exbosomics or it could be in genomics or other things. I'm spending a lot of money to, to support things and, and the other thing too is that I've been doing a lot of exploration on sustainability frameworks. I mean for many people sustainability is getting your grant redo it. Well that's not terribly sustainable. It's, it's, you know, what are the elements of a database that that make it worthy to continue to be funded. Or can we imagine that there is a life cycle of data repositories where, you know, they, they have usefulness for some period of time but then after some period of time, you know, there is a framework where we can measure whether they're losing their usefulness so we can, we can take the resources out and then read, redeploy them to, you know, to another data repository, but it's, so I just raise a number of different issues I'll put it in the context. Yeah, the one that you raise really resonates with me, and I should have emphasized it as well, Rick is the incompleteness of data. So, as you aptly pointed out the biologic layers, even though you might have a DNA sequence of a person. The other ones are cell specific, whether it's, you know, RNA, epigenomics, I mean, so you can capture all different types of the body or tissues. And also the same, you know, the, we're just scratching the surface in the area of the environment, we might get air quality or something like that but what about all the other things. So, it's, it's terribly incomplete and that's why it depends on the tasks that of interest if you're trying to that Darrell asked about Irene, the model that was used for better pulmonary diagnoses, you know, that you have to understand the limits of those inputs, but we, we right now, even theoretically would have all the layers, each of those layers have an incompleteness that's noteworthy. Right. I mean just the physical and chemical complexity of the exposures that we have are, it's complicated enough, but then, you know, as I'm sure Darrell would accurately point out, psychosocial stress is probably one of the, the biggest factors. So how do we measure psychosocial stress in a way that we plug this into these multimodal AI systems where you can be factoring in say epigenetic modifications of key genes that may be downstream from psychosocial stress or sleep behaviors, or your blue and green spaces and so it's very complicated, but I think these are things that we as the, especially the environmental health sciences community needs to pay attention to. So we can actually develop better, better data sets. It would be great to get to the point where we could do the types of things you were pointing out, Eric, with with the, you know, the imaging capabilities that we have in hospitals having, you know, AI capabilities to actually read the images and, and potentially have, you know, more accurate diagnosis. We eliminate some of those, some of those problems that we see, you know, in hospitals and with physicians. Yeah, one of the biggest challenges in the multimodal space is the high frequency or continuous sampling through sensors. So we can get to stress these days with risk sensors with heart rate, heart rate variability. We can get sleep data through sensors. I mean, we can get it. But when you have immense data sets of an individual, you know, with that type of continuous sampling, how to process that no one has yet done that by the way. Okay, taking very multiple sensors and then integrate it with all the other data. So this is at the current at the moment this is one of the analytical challenges. You can take electronic health record and image. Okay, when you start throwing in a bunch of sensors with weeks, months worth of data, try and understand their stress level and their sleep health and, you know, all sorts of other things. It hasn't been done yet. You know, Eric, I totally agree. And that's where I think quite frankly where you know the AI and ML can come into play, because it's it's these are, you know, any given data set is just very complex just take transcriptomics which is probably the simplest of, or genomics. Okay, I mean, we're also still not factoring into the equation the somatic mutations that may be arising during the life course that may be influencing in very serious ways, development of cancer or other other health outcomes. And that's where I think it's, it's, you know, we have these these capabilities to go in and let some of these powerful tools integrate across these these different data sets and hopefully give us, you know, greater insights into into human biology. I have a question on the chat, though I don't know if Rima Habri has a microphone. So I'll just repeat it here. So it's a question for Dr. Topol. You mentioned it's not more quantity of data but rather higher quality and more complete diverse label labeling of data that is needed. Investing in creating systems to generate better and higher quality labels that are unbiased to better serve discovery as much as we are investing in better predictive power of this AI models. What wouldn't we need to get there to incentivize this. Yeah, well, I think it's fair to say that the labeling is going out the window here in healthcare. There were only a limited number of well done annotated data sets.  And fortunately, around that time when that realization was being made there was also the whole cell supervised learning, letting the data, you know, basically, if you will label itself. We are now we're not unlikely going back it would be great to have cross validation between annotated carefully annotated by expert and cell supervisor unsupervised data sets but it doesn't look like it's likely to happen. We just don't get that many expert physicians to be willing to put in the time at scale to do this with hundreds of thousands of, you know, of images or, you know, whatever inputs, you're talking about. So, that's where things are headed now is is it's basically abandoning that idea that Rima is brought up. But on the other hand, we're learning that you can get a lot out of unlabeled data. Like I mentioned that NYU study. The study I mentioned with the Irene model this week for to pulmonary. These are completely self supervised model examples. So, we'll be seeing a lot more of that, but that goes back to what Rick is emphasized with the incompleteness and accuracy of the data, as well as the bias of the data that goes in, which you can, especially when you're not even trying to label anything and it just compounds the problem we could with large language models we could make these problems worse. Hey, Eric, I'd like to get your, your point of view on the, the issue that I raised around heterogeneity of data repositories and you know we have this this new efforts that are happening I think primarily in Europe around the global kind of data coalition where maybe we could be a little bit more proactive in planning. What are the data repositories that the world community needs. And if they don't exist, how do we create them and how do we fund them with sustainability frameworks under what are your thoughts on this whole concept of data repositories. Well, you know the one that's had the biggest impact in medicine actually has been the UK Biobank, which has been just every week there's new insights papers being published. So that I think has proven now that these are extraordinarily important with open access to the research community. And all of us of course aspires to when it gets, you know, fully loaded with the million participants and all the deep data for each. But we don't have as you nicely point out the kind of collaborations and standardization and, you know, we haven't seen many data repositories that have made contributions at the level of the UK Biobank. It would be great if we could do that now. What's interesting in Israel they form what's called the human phenotype project, and they've had 14,000 people who have actually about 5000 already been back for the second visit, where they do, you know, every layer of data that you could possibly collect, including cognitive testing and retinal photos and you know everything in medicine you can imagine.  And Japan countries in the Middle East to try to come up with as you're getting at a standard way of collecting data of organizing the data making it accessible all open access. They're getting a little traction on this, but they're one of the first groups I know of Wiseman Institute that's tried to do it. We haven't really had the kind of global collaboration that we need. And essentially we're relying on you know just very limited sources even though there's so much activity in this space. I think that that's where I mean that being an IC director I we've been talking a lot about this what's the role of the NIH to help kind of nucleate some of that centralized governance. And so we're trying to get some things happening along those lines it just we have to get away from this model where everyone does it on their own. And you know it's a lot of both that UK has done this with UK Biobank it's a lot of both that is real is doing this the Wiseman Institute. Yeah, but we got to get away from everyone realizing something needs to be done so I'm going to pick up the mantle and then I'm going to run with it. You know there needs to be more coalition are you familiar with the global bio data coalition. I've heard of it but I don't know any of the details. That's something maybe we can all pay a little bit more attention to I know one of the things to we're struggling with the NIH is that maybe we need a GBD model for the different databases that are funded across the 27 institutes and centers at the NIH. So that's a challenge as well. Yeah, that would be great to pull that all together. So I have limited time I just also wanted to throw in an era kind of touched on this the issue of, you know, wearable biosensors and various things how we collect data I think that's, that's, you know, if you look at I think one of the questions that were the advance and in the coming years, I think getting to the point where we have those individual wearable technologies and that can dump data in some structured way into these data repositories we could take advantage of this I think it's going to be really important. Yeah, and that's one of the things that came out of UK Biobank because they had 10s of thousands of people over 50,000 with Fitbit. And so that was, you know, exemplary of what you can learn from that sort of thing and yeah and the other thing that you're touching on which is a theme that can't be emphasized enough is diversity of inputs. And, you know, the UK Biobank is largely European ancestry, almost exclusively, and it's only by combining forces. As you've touched on that we can get to the diverse race ethnicity and every other aspect that we want to have as inputs for for these models. The one thing I appreciate almost anything you look at is going to be a function of genetic background. Exactly. Genetics slash epigenetic backgrounds and yet. Now you've got if you got a factor that is the equation. It's really laudable that that the leadership at the all of us program are really making a real effort to make sure that that's a million person cohort is represents the diversity, at least across the United States. Of course in the global community as a challenge of looking at the global diversity that exists in the human population. Yeah, with just about half of the people of the 600,000 are underrepresented minorities. This is a feat that's never been, you know, even approximated in the past so it's a really stand out aspect the other thing of course is returning the data to participants which is what all of us aspires to do. And that's something we've got to do better as well. You know, gather all these large language bottles and build all these publications but when when are we going to start helping patients right and so getting this to be things like virtual health coach in the future are exciting opportunities and you know I could foresee where, you know, where you were someone's at risk for asthma. And giving them all their inputs like we've seen with the Louisville study on hotspots for asthma and reducing the toll of asthma. It's just one example, whereby people having that feedback that they're getting continuously for for themselves so that's another thing that we haven't done yet it's nice to publish all these papers, but let's help patients. You got it. I totally endorse that and it's really getting putting putting our science to work for the benefit of public health. Exactly. Excellent. I see that Kristen Malaki has a question maybe she can unmute. Oh, sure. Sure. I mean I think this is all really helpful and I think that last answer, or that last example really starts to get at my question. You know the goal of these workshops is really to think about are we ready now to integrate into these large predictive health AI type, you know mechanisms and if so, where do we start how do we begin what you know what are the current barriers and, you know, are we at a point where chemical structures could be integrated in here. So what do we do with that information, and if not what are the barriers and how do we get health care really just start thinking about these environmental factors as critical to sort of patient vulnerability and susceptibility and or or response. Rick may be able to answer that better than me. It's, you know, the, I guess, we need an operational kind of definition of how we collect the environmental data, you know, do we have chemical structures but it may not be just one chemical. I mean, if we're really, you know, if we, you know, Eric, you may have heard this but the environmental health sciences community is absolutely embracing the notion that if we want to understand the effect of the environment on human health. We have to erase this totality of exposures, because it's not just air pollution. It's in what's not just all of the different components of air pollution air pollution is a catch all term for PM 2.5 for ozone for a whole variety of different chemicals. But it's going to be also the, the flame retardants you're probably breathing in coming from, I don't know you have a leather chair, maybe you don't have it in years. It's, it's this totality of exposure we've got to figure this out and how to put this together in some way where we can ultimately use this to to our better understanding. And, you know, the other approach to is a lot of the exposomics work and we're not talking as much about, well, can we measure some of the biomarkers and biological effects of exposures. So, you know, my mother was probably exposed to things that she wasn't even aware of, but my epigenome is probably reflecting that it's influencing how my genes are being expressed so we, you know, it's a it's a it's a complicated issue but it's one that we need to be focusing our, our planning activities around. I couldn't agree more. I mean, there's a lot of things out there that are very disturbing like seeing the rates of cancer increasing substantially and younger people, which brings to mind what is it about their environmental exposures that's doing that because, you know, seeing people in their 20s with colon cancer, more and more now, what is going on here. And there's nothing to indicate that there's genetics in play. So, I hope we can make some headway here. And this is when we barely have our arms around in the medical community, because the only things we can tap into are things that are, you know, nominal marginal of very there's no depth there, as you're getting it. I hope those will see a real improvement. The other thing too I just want to throw on the table here is that, you know, in the environmental community we focus so much on all the bad things in the environment. Well, the fact is that there are actually a lot of good things in the environment. I don't think we we still have any clue of why exercise is so beneficial. And there, you know, there's plenty of data now emerging that those omega three fatty acids. And from fish, even though they may have methylmercury, they're there are health benefits associated with those omega three fatty acids, and the polyphenols and blueberries and blackberry is this for real. So they could be positively impacting our health but you know figuring out how do we integrate that into an exposomics framework is also part of our challenge. Yeah, the nutritional side of this is is is ginormous. And just another part we in our studies we try to get two weeks of everything a person eats take pictures of it so that we can use AI to analyze that that's also just scratching the surface about nutrition. And I think the, you know, I really applaud the, the all of us program that you probably know that there's a small segment of the million person cohort that they're doing the high resolution mass spec and actually doing metabolic profiling specifically for this purpose of precision nutrition. You're not relying just on what people tell you that. Yeah, you're not be accurate. You're actually looking at the metabolic profile as a way of objectively assessing what actually made in the system and then also factoring in individual biological variability. So the people who could you can be eating western McDonald diets and stay rail thin for their entire life. So it's, it's good I mean I think we're finally beginning to really embrace that we can do something about this. Yeah, I do think as you're getting at the metabolomic layer of data will be more helpful than, you know, it's been difficult to get at that because of some of the expense to do it at scale to do it properly. I mean that may give us a lot of important insight. Well along those lines Eric, you know the, the other thing is, you know with our page kind of coming on the scene now. I know that at least, you know, Renee Wagersen is the director of our page, you know, it's very interested and and are there new technologies that we can actually imagine where we can start capturing some of the high resolution mass spectrometry, what are we doing on a scale that we need to do and with an affordability that needs to happen. I mean we've seen this in the genomics community. I mean some of the primitive technologies we started off with you know those two color micro arrays, you know spotting CD days on microscope slides. You know it's, it's, it was a start. But, you know, it's, you know where we are now we focused on those technology development what are the things we actually need to need to get done between do the same thing environmental health sciences. Yeah, I mean, a really recent and I think notable example is the wastewater surveillance. We didn't do that. And here now we can look at you know every pathogen, whether it's SARS-CoV-2 or polio or M pox or in the whole list. You know we've never done that we were a terror on the ranking list of the world we were, you know at the bottom of the rich countries for doing it and so we've learned how important this is. And that's just one example of how you know it took a pandemic to figure this out right. So I'm pretty familiar with a lot of those, the wastewater experiments and so I'm actually part of my evening that weekends I spent on the Radix project and Lucila and no knows that well managing all that data is important. But it's interesting some of those, some of those, those, you know, digital PCR strategies that are being used as part of wastewater. Actually, they came on a DARPA projects. So, so that's why I'm thinking it's, you know, we also talked about frameworks people approaches, you know, we need more collaborations. So people developing, you know, new technologies, you know know what to develop, you know, the engineers, so that they have a really a solid framework of what do the physicians and biomedical scientists what what do they actually need, rather than having you know just an engineer is building the things that they think we need, which may or may not overlap with where the actual needs arise. Sure.  I think we're out of time but I love to see how positive both of you are. So my question is, what is coming up and what impact all this will have in health. So my function is to thank you both for taking the time to speak to the, the panel here and to pass the baton to the organizing committee with closing remarks. So thank you very much. Thank you. I'm here. Bye bye now. Great. Thank you very much for leading that session and for Dr. Weitschek and Dr. Topol for for really, I think, ending us on a very positive note so thank you to everyone who's joined us all day and who's stayed with us for this really compelling discussion. I'm Carmen Marsit from Emory University, and I've had the great pleasure with a group of some hardworking colleagues and National Academy staff of developing the program for this workshop. I'm going to take a little time now to try and summarize some of the themes that we've heard across this tremendous lineup of speakers and panelists. So we started the day in a first session that was really designed as what we described as a level setting opportunity to bring everyone to the table to understand the foundations and challenges of bringing AI to environmental data to address health questions by talking about the current state of that environmental data, AI methods and their application to biomedical research. We started with Dr. Bryce who highlighted some of the biggest opportunities and challenges in is an in environmental data. In particular, he noted the explosion of data with increased temporal and spatial resolution and opportunities to think about how data that is being collected for one purpose could be repurposed to consider the health related consequences with examples including pathways, water, heat, satellite data and traffic. In addition, he also highlighted opportunities that are developing because of the growth and personal measurements and we had that highlighted a few weeks ago in another one of these workshops. And this is really allowing for a movement more towards the concept of what we talked about as precision environmental health. These are the biggest challenges that come with those opportunities, integrating this data, making use of the data to predict risk and prioritize efforts, and being sure to assess the effectiveness of this kind of data. He also highlighted the importance of training researchers and our public health workforce to use this kind of data moving forward. Dr. Grotto talked about opportunities to use data and AI to mitigate inequities in healthcare by developing new algorithms and tools in informatics and data science to bring together individual health data, like clinical data from from electronic health records with biological data like genetics or genomics or microbiome to build predictive models she highlighted really requires large data repositories to train the models, and this needs to be done really moving toward a more global effort and with privacy protections. She highlighted the NIH all of us program which is enrolling a cohort of diverse individuals who are actively giving permission to share their health data, and it is growing its body of environmental health data as well. She also highlighted we're considering genetic risks and add mixed populations, and the need to be able to begin to include other types of data such as social determinants and environmental data in building risk models. And to do this in diverse populations so that as she said, no one is left out. Dr. Gosemi reminded us that our goal should be about identifying actionable insights in human health with the ultimate goal of identifying models that can better perform than humans in making decisions about a person's health. At the same time she noted that AI learns from humans, and so there are biases that are being introduced, given known biases in existing biomedical and clinical research and implicit biases that are in the clinical landscape. To help address these shortcomings, Dr. Gosemi talked about the need to consider the effectiveness of these models across different groups of people to see if the models are being developed perform equitably across different types of individuals, including those at different intersections. She also noted the importance of building in fairness constraints to try and avoid inclusion of these biases and to learn from other regulatory agencies how best to regulate the use of AI tools. In the second session we dove into more concrete examples and use cases of how AI and machine learning have been used to bring together environmental health data with biomedical data to address important health questions. The session started with Dr. Patel who noted a number of large cohort resources that are starting to do these types of data integration. This includes examples like using metabolomic data of diet to understand internal exposomes and predict outcomes, polyexposure risk scores, which bring together multiple layers of exposure data, and considers those along with polygenic risk scores, and novel measures of aging based on MRI phenotyping. Given that there are all these new possibilities to integrate multimodal data, there are needs to think about the ways the models are developed to ensure that what we're doing is really robust. Dr. Zhang focused on how to bring various modalities of biological data from often large multiomic data sets together. One of the biggest challenges that we see when doing this is that there's often missing data, missing modalities, and that was traditionally addressed with the imputation. With AI, he introduced different approaches that could be used by looking at similarities between individuals and using different methods to improve the models and address the missing data problems. These methods may be more powerful as they don't need prior knowledge and they may produce less bias in their results. And then Dr. Hansen brought up the promise of being able to follow an individual throughout their life to be able to provide early identification of risks so that intervention and treatment can happen, and then to continue to monitor those individuals to understand how the environment can be impacting their treatment as well. Importantly, this needs to happen in a scalable framework so this can be done at a population level. She noted that to address this challenge requires interdisciplinary team science and used as an example of collaboration between her group at Oak Ridge National Laboratory and the SEER program at the National Cancer Institute, where they've developed machine learning models to auto code information on tumors from pathology reports with built-in accuracy checks and have developed methods to integrate information pulled from residential address histories to build external exposomes. The goal of this is to build foundational models from such data streams that could then provide flexibility to address various downstream questions. And those foundational models is something we've heard as a theme across a lot of what was talked about today. She also emphasized that work needs to be done in an open science framework to assure reproducibility, replicability, and usability for real-world data applications, and also to assure appropriate protections for individual privacy. In session three, we focused on new methods that can be used to aid in data integration. The session started with Dr. Ho, who motivated her work based on health equity. As example data streams, she suggested using common social media, app maps, mobility data, and community-based forums to aid in developing potentially different measures of social determinants of health on shorter time scales that could then be related to health measures. This could allow for more fine-grained examinations of information about neighborhoods and their characteristics, particularly if you bring in human domain knowledge to guide refinements. The challenge there is that these types of data streams are messy, but that new models need to be trained using this type of real-world messy data. Dr. Hartung pointed out the convergence that is bringing us really here today, the huge increase in data, increase in computing power, and the development of new AI methods. He particularly highlighted the growing importance of foundation models, which underlies GPT, that can be applied to a variety of tasks. He called for rethinking and advancing regulatory toxicology and the need to have a human expose on project that incorporates new technology to drive this new toxicology with an implicit incorporation of AI to help in extracting data from the literature and synthesizing that information to inform policy. He highlighted a number of existing tools to demonstrate the growing potential for this approach, including the utility of AI tools to replace expensive and time-consuming animal testing protocols for various endpoints. He challenged us to all think about how we can take advantage of this new science and be ready to leave behind old ways of thinking. These presentations led to a discussion on public trust and privacy. The speaker suggested that trust may be supported by continued reliance and understanding of evidence-based solutions, and the use of approaches like explanatory AI, which describes how these tools are getting to their answer. There may also be a need to rethink what privacy is and how willing people are to share their information, particularly if they understand what the potential importance could be. There was also some discussion of thinking about the questions being asked, the ability of the data to have an answer, the ethical guardrails, and the continued incorporation of human thought when applying these tools and utilizing their outputs. Finally, in our last session, we welcomed Dr. Rick Wojcik, Director of the National Institute of Environmental Health Sciences, and our keynote speaker, Dr. Eric Tobol. Dr. Wojcik set the stage about the incredible wealth of data that is now available to help to understand the environment's impacts on human health and the real potential we have to bring this data together using new AI-based approaches. He highlighted the importance of collaboration and how he is strongly supportive of identifying synergies between NIEHS and its researchers with researchers across the NIH. Dr. Tobol talked about potential for AI to help improve diagnostic errors. He provided a number of examples where machine eyes, based on unimodal AI, have already been successful at identifying information about patients and their diagnoses beyond what a physician would be able to see. Now we're moving into multimodal approaches, which can take the expose on with multiple biological layers to improve medicine, and he showed examples about integrating various layers can improve the performance of diagnostic testing, and potentially more importantly prevent errors. He also cautioned on challenges, including bias, which we heard a number of times today, carbon emissions based on the computing needs, and the need for validation of utility and safety of these models. He ended that he is most excited that there's an opportunity to bring humanity back to medicine by using some of these approaches. I wanted to thank all of you for joining us today, and then we hope that you will join us tomorrow starting at 10 a.m. We will talk about governance and infrastructure for AI, technologies and tools to advance environmental health and biomedical research, and provide an opportunity for everyone to get involved in the conversation. See you then.