Good morning, everyone. Good morning here from Washington, D.C. We have a full room here of scientists and speakers ready to go for this fantastic workshop. My name is Andrea Bacarelli. I'm a professor of environmental health sciences at Columbia University, at the Mailman School of Public Health. I had the great pleasure to serve as one of the two co-chairs with Kristen Malecki of the Standing Committee on the Use of Emerging Science for Environmental Health Decisions. So I would like to introduce first what the committee does and then give you an overview of our two days together for this workshop. And of course, I mean the National Academy of Science, the National Academy Standing Committee on the Use of Emerging Science for Environmental Health Decisions, short ESEHDS, as I used to say from now on, examinees and discusses issues on the use of new science, tools, and research for environmental health decisions. We convene workshops that provide a public venue for communications, for discussion, for brainstorming new ideas, and for government, industry, scientists, environmental groups, and the academic community. We want to propose topics that are around scientific discoveries and advances in methods and approaches that can be used in the identification, quantification, and control of environmental impacts on human health. The committee is organized under the auspices of the Board of Life Sciences and the Board on Environmental Studies and Toxicology of the National Academies of Sciences, Engineering, and Medicine, and sponsored by the National Institutes of Environmental Health Sciences. And of course, I mean, this is all about the people. We have an amazing committee here that you see on screen, that is co-chair by, as I mentioned, by Christian Marecki and I. And I would love for all of you to be in the room when we meet about, and hear all the amazing ideas and the proposals that you can listen to being there. It's really a very proactive and very diverse and very enthusiastic group, and of course very qualified. We also work with a multi-agency group of federal government liaisons, who, along with the standing committee members, help shape and guide our activities. And here on this slide, you see some of the topics we have covered in the past few years. Since 2019, we have hosted numerous workshops on a wide range of topics. And depending on the maturity of the field, the topics can be broadly categorized as emerging research strategies and analytic methodologies, emerging areas of convergence, and emerging advances in science and technology. And the good news is that many of us missed these workshops in the past, but we have proceedings online, notes, some videos as well. So please look us up on Google and you will find that the committee, the proceedings of the workshop over there is a very rich repository. As I said, this is really about the people, sorry, this stopped working. Okay, got back online. This is really about the people and that includes of course you. So if you have ideas about workshops that we want to consider, please email us. There is this email on the screen and send us any idea about workshops we should do in the future. We are organizing new ones and we look for new ideas. And just a few housekeeping rules. I encourage all of us to be active participants. We are in this new brave world where some of us are in the room and most of you are online. So we want everyone to be active, everyone to participate and contribute. And please use the chat box or the comment box online to submit your comments and questions. And of course, those of us here in person, we have a microphone, remember to bring them up and to take them down and to switch them off when you are done. And we lost the video here. Fortunately, I'm almost done. I don't need the slides really. But what I would like to say is that also thoughts and ideas shared during this workshop, considering this workshop is public, are really attributed to individuals and not to the National Academies of Science, Engineering and Medicine. And lastly, recordings of the workshop will be available. So if you missed the workshop today or you missed any part of the workshop, please come back to the website. You will be able to stream them at your own convenience. And at this point, I would like to start to introduce our two days together. The title of the workshop is Advances in Multimodal Artificial Intelligence to Enhance Environmental and Biomedical Data Integration. And of course, I mean, we are all here in person or online because we are excited, so I really don't need to give you much about why this is particularly important. But all of us know and everyone knows me, know that I'm over-enthusiastic about this, so someone will have to slow me down during these two days. But really everything from online searches to chat GPT seems to be powered by AI today. And clearly it is ubiquitous, it has become very much part of our lives. And this workshop will consider what artificial intelligence and related techniques mean for environmental health, for biomedicine and for health. And of course, we would like to thank the planning committee for organizing this workshop and give a big round of applause for the hard work and make this workshop that we are all participating in possible. And the planning committee includes Carmen Marzit as a chair from Emory, who is here in the room. Yao Ye-Chung from the University of Minnesota. Thank you. Christopher Duncan from the National Institutes of Environmental Health Sciences. Anindita Dutta from the University of Chicago. Megan Lat Show from Johns Hopkins. And Gwen Ellen Ottinger from Drexel University. Although I had the smallest slide ever seen by humanity, I've been able to read them, but I don't need an eye test not yet. So, and I'll give you just a quick snapshot of the next two days. As I mentioned, we will hear from experts, speakers and panelists who are trailblazers in the field who had done new advances and have substantial experience in this area. We want to hear what they're learning, what they're doing and where these still need to go. We will wrap up the first day with a keynote address by Eric Topol and a chat with both Eric Topol and Rick Vojciak. Then please join us again tomorrow, 10 a.m. as same as today for an equally exciting lineup of speakers. The agenda is online and you can access online and download it for your own convenience. I'm sure it's going to be innovative, it's going to be exciting, it's going to be helpful. Our workshops, and this is no exception, intend to be stimulating, intend to be novel. We want to be a venue also in which new ideas can be generated and where new collaborations can be made and new areas can be brought forward. With that, I would like to turn it over for session one to Megan Latcio. Thank you, Megan. All right, welcome everybody. We're excited to kick this off with the first session which is meant to sort of be a level setting session. And I guess I should start off by introducing myself. As Andrea said, I'm Megan Latcio. I work at Johns Hopkins University in the Department of Environmental Health and Engineering. And as you'll probably see throughout one of my longest stints in my career was here in DC when I worked for the Association of Public Health Laboratories and also the Association of State and Territorial Health Officials. So I have a very practice oriented approach to environmental health. And also I served as the chair of the APHA environment section. And so that's a general summary of who I am. So to this session, what we're going to be focusing on is what are the foundations and challenges to using AI to integrate environmental health and biomedical data. And so we're going to lay out the current state of research when it comes to environmental health, when it comes to biomedical science, and when it comes to AI. So we'll all sort of have a sense of where things stand in these three fields. And most importantly, we want you to be thinking throughout this session and throughout this entire workshop about the opportunities for how AI can advance these fields and bring together these two areas. So our first speaker is Dr. Patrick Bricey, and he is a professor of environmental health sciences and medicine at the Johns Hopkins Bloomberg School of Public Health. Dr. Bricey recently concluded an eight year tenure as director of CDC's National Center for Environmental Health, an agency for toxic substances and disease registry. And he earned his MPH and his PhD from Johns Hopkins. And I'll turn it over to you, Dr. Bricey. I have a single slide. Not sure we're going to get it up or not. I don't really need it. I took a minimalist approach to this talk. And since I'm really just trying to level set, I want to start off by humbly stating that I certainly don't think I have my finger on the pulse of all environmental health research might be relevant to this. But I'm going to talk about it from one perspective, something I've been thinking about for a while. And so we've heard many statements, things like, you know, place matters. And so we know that there's many challenges to understanding what we need to know to identify quantifying control factors as we just heard from from Andrea during his introduction, but I would add one thing to that humbly. In addition to identifying quantifying control input anticipating. And I think that anticipation is probably an important component, especially when it comes to AI. So can I have the next slide. So I think the biggest challenges we have and biggest opportunities we have rolls around access to new data that allow us to look at environmental factors. I'm going to speak on the environmental side. Other panel members were talking about the biomedical side. Understand factors laid to the temporal spatial resolution of hazards and risk factors in our environment. I think there's an unprecedented opportunity here. And along with that comes big challenges. So I put a number of things on this slide I could have put dozens of them on the slide I didn't want to make this a big slide heavy talk but, but we certainly know that in terms of internet of things and smart cities. There are things that are happening right now in cities that they're collecting data for all sorts of purposes but usually for how do they manage the city functions how they manage the city assets how they manage resources and services, and to officially improve the operation of cities. And the environment is usually not part of those discussions at all but absolutely should be and so we need to make sure we're integrated in all that going forward. So some examples of things that are happening now that we can easily see how we can build on those. So for example, in Baltimore City. I read recently they, they're putting noise monitors, a certain place in the city to monitor for gunshots and to target police operations for gun violence going forward so, you know, obviously noise monitors to be used for a lot more than gun violence going forward and, and I was thinking as one or if anybody in the city of Baltimore is talking about, you know, where those things go what else that data can be used for going forward. And I'm sure there are other cities doing the exact same thing. Many cities collect electronic water use data to manage water sensing water use, especially, you know, in the city of Baltimore having a drought right now. There's a water source that has to be tapped when the reservoirs get too low. And so they need to manage that when you figure out where the water is going who's using water in addition to just how to build people for their water. They need to manage that but it's very easy to think we can easily put water quality parameters in that data collection as well. We know for example that excellent water crisis that people started developing home led water sampling detectors in real time. It's an opportunity to monitor stuff that we probably need to start thinking about it for if we're building these systems to manage city functions. Let's see how we can use it for environmental health data. Temperature is a big deal. So, we all know that, you know, heat stress and health effects of extreme heat events is a growing concern in environmental health, and many cities are measuring temperature around the city, but they're doing it to for environmental assessment. They're not thinking about how that can be used for health environmental health assessment. So I'm talking now about how do we, how do we integrate data streams just from the physical side of things. I haven't even touched on the challenges for the biomedical side of things going forward. Traffic monitoring is another one. Many cities are monitoring traffic. We all know that traffic is being very carefully monitored so we can access it on our phones every day. Traffic is being used for air quality management in the future as we go forward. So but how do we think about how do we use that data more efficiently. So in addition, there's explosion in opportunities for satellite remote sensing going forward you see a picture on this graph. So this is a this is a graph of wildfires in Canada affecting the east coast. And you might think this was last week. This was in 2002. So this is from a paper that we published at Johns Hopkins looking at the health impacts of wildfires on the east coast in 2002 so this has been around for a long time, and actually 2012 sorry. And so, you know, but that's just an example of one satellite data but satellite data is being used right now in addition to tracking weather wildfires. This is being used to assess water quality. You can you can look at the color of the water to assess algal blooms and growth of algal blooms going forward. There's all sorts of land use vector quality vector ecology issues that can be assessed with satellite data that might help us understand changes in infectious disease parameters. There's there's measurements of soil moisture and drought that can be used. All that have health consequences as well going forward. Obviously, there's big uses for climate change extreme weather events being used for that right now. And, and, you know, as we watch the radio this morning listen the news this morning, the extreme weather that's about to hit the southeast today is a good example of that. And so we're already using data like this for environmental health for emergency management and emergency response conditions. Now I'll split down your sensor technology and something clear. I think near and dear to David's heart is, you know, there's been an explosion in capability of measuring things using small portable sensors. It has huge implications for for data management issues data quality issues going forward. And then has big opportunities for citizen science issues we need to think about how to, how do we use citizen science better, and how do we take measurements that people take on themselves better and so we're now getting the realm of, of what's been called precision environmental health. You know how do we take measurements that people can take on themselves and how do we use that an individual basis, as well as a population basis basis to assess health impacts going forward so a good example that is, is heat stress. So for example, my colleagues at CDC and I published a paper a few years ago that said in in 2000. 20 I think it was, there was a heat wave in the Northwest, and we were able to show that over a four or five day period there was a 70 fold increase in ED visits from that nothing startling about that going forward. Those ED visits don't have to happen why do they happen because these are vulnerable people we can use. We can use data sets to identify vulnerabilities. These are people who don't have access to transportation we can use data to assess that. We know that when we looked at that why cooling centers aren't being used in another community in the Southwest. We saw that people who need the cooling centers don't know where the cooling centers are, and they don't have access transportation to get there these are all things once we understand we can fix using data and how do we make it so that people don't have to go to emergency departments during a stream heat events going forward. So there are many challenges to using these data. This includes how do we integrate it in a meaningful way which is part of what we're going to talk about today. And how do we anticipate I think one problem how we use these data to anticipate when problems are going to occur not only just to kind of quantify them and address them and how do we use them to evaluate the effectiveness of whatever control procedures we put in place. Perfect example that so we spent a lot of time doing things that we think are improving people's health and reducing people's exposures, but we don't always evaluate how effective they are and so there's optimistic effectiveness research here these days that we have to look at as well. So how do we identify the risks are greater greatest how do we prioritize their efforts. This leads us to questions that inevitably about environmental justice issues, because you know, oftentimes, we have we address for the exposure to greater. That's where we're looking at disadvantaged communities because that's where the exposures and health risks are greatest as well so so these are there's a challenges here associated with using these data to address environmental health that inevitably leads us to addressing environmental justice issues as well. So I'll just end right there and I'll say these are exciting times. There are many challenges we need to start training people to utilize data like this. Our training needs to include more data science assessment and then environmental scientists we trade. We need to think about how do we use these data how we take advantage of them how do we address the quality issues associated with the data we collect, and how do we make decisions about it. But I'm very excited about these opportunities hold. Thank you. Great. Thank you so much, Dr. Bracey. Next we're going to hear from Lucila Ono Machado, who's the deputy dean for biomedical informatics and the chair of biomedical informatics and data science at the Yale School of Medicine. Her research focuses on privacy for healthcare and biomedical sciences. Before that, she was associate dean for informatics and technology and the founding chair of the UCSD Health Department of biomedical informatics. Also, she's the PI for the California Precision Medicine Consortium for the NIH, all of us research program, and she received her medical degree from University of Sao Paulo and her doctoral degree, medical information sciences and computer science from Stanford so I'll turn it over to you, Dr. Ono Machado. Hi, thank you so much for the organizers and for the facilitator here. I would like to share my slides now I think I need to be granted permission by the host. Sorry, I still cannot. Okay. You have permission. Somehow it says host disabled participants screen sharing. Oh, now I got it. Thank you. And please let me know if I have the correct mode or I need to swap displays. We see the notes. So I think you might want to swap displays. Yeah. Thank you so much. I'll briefly comment on precision medicine and the role of AI environmental health and biomedicine. Again, this will be a very overview of some things that we are doing and colleagues in various projects. So to enhance health and healthcare using data, we definitely need new algorithms tools and systems that use informatics and data science AI statistical learning and so on to enable the personalized health that personalized medicine via harmonizing and data from several modalities, including electronic health records, EHRs, human genomes, microbiomes evidently environmental data that we're talking about today, surveys and so on. And then we build predictive models that can be used at the individual level. Most importantly, we also not only characterize but use that information to help mitigate inequities in health in healthcare. So this circle here is just to illustrate some of the sub specialties in data science that are involved in terms of improving human health. We obviously won't talk about them all, but you can see that several types of data and specialties revolve around those data types. And in a particular interest today is sensor data environmental data and integration with electronic health records and omics data and other modalities. So I'll talk primarily about models, but keep in mind that, you know, the data court studies are super important. I'll speak a little bit about that. And then the implementation aspect is something that we totally need to work more on the science of implementation to get our discoveries and find is disseminated in practice. So precision care in health. I spoke about we have a genomic data for variant based therapy and cancer and HIV treatments and so on. I'm phenotypic data, which it's my area I'm very familiar with, but also environmental and social determinants. So where quality exposures diet access to care and so many other items play a significant, if not larger role in someone's health, but then the genetic makeup or the particular diseases that the person has at the moment. So predictive models in medicine, we require a lot of data in order to build reliable models. So AI requires a whole lot of large and representative data sets. So we need to build this access to large data repositories to improve research, but we need to do in a way that protects privacy of the individuals and privacy of the institutions involved. We need to aggregate data from different countries. If we are to make this covers at a faster pace than we do today. So there's the dilemma that gets to all of us. electronic health records, for example, is it right to share them, since people have not been explicitly asked. Now, is it right not to share because the numbers and acceleration will depend on that sharing. And can we share without moving data around can share with explicit permission of the individuals, and that's one aspect of it so I will briefly say that the all of us that we participate in isn't NIH large initiative is exactly doing that is a building a large and diverse data set by asking people to actively contributed their data from electronic health records as well as physical measurements and other items that roll into the protocol for the this development of this cohort. And right now, there are many institutions already making use of the data available through this program to researchers at large currently in the US but soon in other countries as well. There is a treasure trove of data there, and it's important that it includes several items including some environmental items but perhaps not as many as we would want. Another aspect of it is the diversity of the people involved because traditionally many segments of the population were left out of research studies, and one segment in particular that we're studying is at mixed populations and this is part of another NHGRI initiative, because we know at mixed populations improve the power of variant discovery portability and genome wide association studies as well as in predictive models and other areas. One of those predictive models that is coming along is polygenic risk scores. So to use the genetic information as well as phenotypic information and hopefully social determinants as well as environmental data to be able to assess risk for particular individuals and then do something, preventive or therapeutic about the diseases for those individuals. The problem is, as this polygenic risk scores are produced, methods need to be improved. For example, the scores may be have a need for continuous update. Will all of this generate or introduce more disparities and more discrimination than already some healthcare algorithms introduced. Can we also protect privacy, particular with some of this forgotten populations were also the ones the least likely to trust biomedical research and what do we do with individuals from mixed ancestry. So it's a center of mixed science and technology for genomics for everyone. And I'll tell you how that relates to environmental data, which is the topic of today. So think of a trait of disease, for example, as a function of genetic determinants of health, as well as environmental and social determinants. So if we are to develop an individualized risk, we need to know all of those items and we're working hard on getting some sequences from from individuals, for example, in order to assess variant effects and also ancestry hereditary. In fact, but we also need to collect exposures diet education and also to determine it's in order to determine the risk for a particular trade. So we need to do this in, in a way that the algorithms to analyze data and stay in their end. In short, our vision is that no one will be left behind and we will increasingly replace concepts that are being used today for race and ethnicity with a combination of genetic environmental and social determinants of health, because each individual is different, different not only in the genetics, but in the way the course of life proceeded. So we will develop new methods and tools that allow model findings to be applicable to all in not be as they are today, mostly referring to the majority of population that does participate in clinical trials and observational studies. So with that, I want to thank everyone. And thank NIH for sponsoring several projects related to this. And again, thank the organizers for the opportunity to talk to you. Thank you so much, Dr. Ono Machado. Very interesting. I've been taking notes all morning, lots of good ideas for how we can advance this field so getting really excited for the conversation. Next, we're going to hear from Dr. Marzia Ghassemi. She's an assistant professor at MIT in electrical engineering and computer science and at the Institute for Medical Engineering and Science. Professor Ghassemi focuses on creating and applying machine learning to understand and improve health in ways that are robust and private and fair. And previously she was a visiting researcher with alphabets, verily, and an assistant professor at the University of Toronto. Before she got her PhD in computer science at MIT, she received a master of science degree in biomedical engineering from Oxford University as a Marshall scholar and a BS in computer science and electrical engineering as a Goldwater scholar at New Mexico University. So I'll turn it over to you, Dr. Ghassemi. Thank you. So today I'm going to talk to you a little bit about designing machine learning processes for equitable health systems. And the thing I want you to keep in mind is we all have a focus and a goal, I think, of creating actionable insights in human health. But to get there we really need to understand how we build models to perform well or be healthy. How we decide which data is most appropriate to be used in model training, what kind of health care we want to represent, and what kind of behaviors we want to encourage in end deployments when we couple machine learning models with human usage. And so let's say that we have a patient, Simone, who's having trouble breathing. And so she goes into the emergency department. It's very late. And, you know, she's told, hey, we took a chest x-ray, you might have pneumonia, but we really need somebody to come and look at it. It's swamped and so it's going to be three hours. You can wait here for three hours for a doctor to look at your chest x-ray. We have a machine learning model that can look at it right now. And this machine learning model performs at or above humans at this task of saying you're healthy, you're in triage, you can go home, you don't need to be seen, or you should wait around in the hospital. And so, which would you choose? You can have this AI screen you know, or you can wait for three hours for a human. This isn't a hypothetical situation. This is actually the situation that we're in with several clinical AI that perform at or above humans in a range of tasks across the human lifespan. And this is a paper from our, to be keynote speaker soon, Eric Topol, that I really enjoyed from a few years ago, really demonstrating that across these different settings, we have AI that is no longer a calculator or a simple toy. It's now performing in ways that are really impressive. But the issue is, you know, even once a model is regulated and the FDA does regulate a software as a medical device, we have some questions maybe about exactly the performance that we might get in different settings. And the reason we have this is AI learns from human. Both in terms of the data that we generate and feed into the model and in terms of the design decisions that we make when we do optimization of these models towards some training objective. And so we have some issues with the current medical data that exists. So if we're looking at just the data that exists, for example, in randomized control trials or top tier journals, medical journals, that data is often very sparse because RCTs are very hard to run. It's often very narrowly scoped. So most of the RCT populations are not diverse. And so the findings may apply in varying levels to a more diverse population. And some of the new reversals happen more commonly than you might imagine where we run in RCT, we get a result, it's standard of care, and then there's a reversal years later saying actually that's not the way that we should do things. If you look at patient records in large databases, so let's say 250 million patients, you might imagine, well, can we just mimic what happens with patient care, can I just look at your nearest neighbor in sort of treatment space, but even for extremely common conditions like hypertension, depression, and diabetes, it's been found that large proportions, so almost a quarter actually of hypertension patients, follow completely unique treatment pathways. So that means they would have zero nearest neighbors if you were saying, you know, there's no RCT for you. Let's just look up what worked for a patient like you. But unfortunately, you know, the solution here is to have better improved data, right, and larger that 250 million patient population is not large enough. Health data really lags as compared to other machine learning subfields so when we compare machine learning for health to natural language processing or computer vision or just general machine learning. Health papers tend to not release code as often not release data as often and not leverage multiple data sets as often, which leads to a substandard of reproducibility that is not common in the machine learning space. So, let's walk through what I just talked about. Let's say that you've engaged me as the machine learning researcher to go through and train a model. Here's a triage example. So if I take the three largest chest x-ray data sets that exist, that's over 700,000 chest x-ray images for the United States, and I train a dense net, a kind of convolutional neural network to predict no finding. It means the patient is healthy, you can send them home. And I get the best possible performance state of the art just like the paper. Maybe one thing I would want to compare is the false positive rate in different subpopulations, and I could call that an under diagnosis rate, because if you had a higher rate of false positives, I say you're healthy when you actually have pneumonia or another condition. In one subpopulation like female patients, this would lead to this deployed model having a higher rate of no treatment for patients who actually need it in that subpopulation. So we do this and we find that the state of the art model has the largest under diagnosis rate in female patients, young patients, black patients and patients on Medicaid insurance. And intersectional identities have it worse than aggregated groups.  And you might think this is a very simple point that we should just audit models, maybe we should ask the FDA to create very specific categories of insurance type, sex at birth, self reported ethnicity, and all of these software as medical devices that get approved they also have to hit certain bars within those categories. That would be a great start. But this can get really complicated. So everybody loves language models, right? And note completion is actually one of the tasks that GPT is being used for right now. In fact, Epic has an add on with open AI, or if you start out the patient's note, it'll auto complete it for you. So we took Cybert, which is a transformer model, so a contextual language model. And we took a real sentence from a real patient's note in the Boston area. And we filled in the first word, the patient's race, so blank patient became violent, sent to and we asked the model to fill in the blank. If we say that the patient was Caucasian or white, the model fills in the rest of their note with they were sent to the hospital. If we say the patient was African African American or black, the model fills in the rest of the note was sent to prison. This is not a simple audit category that you could just check for every model. These kinds of associations that we learn are going to be very deeply proxied within the data that we use, especially as we get into large language models and other high capacity models that use lots and lots of human generated data with human biases and biases are a really strong part of the clinical landscape and they're not something you can simply escape. How strong are they you say? Pop quiz number one. So this is a real medical note as well, but we've redacted the patient's self-reported race from this note. Can you tell from this nursing progress note what the patient's self-reported race is? In the clinicians that we surveyed, they could not. So this is not something humans are good at at all, they just guess. But machine learning models can and some of the clues they're using are maybe fair game like in the Northeast, which is where we did the study. There are more African American patients that have dialysis due to hypertension. However, we tested this in two data sets in Boston and New York. So that's over 4 million notes that we evaluated it on and we found that a lot of these statistical power that these language models are gaining from running over all these notes are things that maybe we wouldn't want a model to generally use, but it does use because that's actually what's in the data that we are generating. Like if you talk about a patient's skin at all, that patient is probably white because there are very, very few references to skin related diagnostics of disease and darker skinned patients. Or if you use the word difficult to describe a patient or their family, that patient is probably African American. And maybe you can see how that would happen. Tiny ways that I describe a patient or their family over time and the notes would add up and a large language model can, you know, read between the lines and see what kind of patient you're talking about. What about chest X-rays? Surely there is nothing proxied in here. Can you tell that this patient's self reported race, not genetic ancestry, just self reported rates, which I will say has a very heterogeneous genetic ancestry associated with it for most categories we are talking about. So is this patient black? Radiologists cannot tell. We surveyed them. We tested them and they can't. But when we look at several different data sets from both major chest X-ray releases and also from a data set that was more race balanced from Emory University, we find that here neural models can tell the self reported race of a patient with extremely high performance. And then we look for the proxies. We ask the radiologist, tell us every cheat code you think that a machine learning model could be using here, just like in the note example where it was looking at, do you talk about the patient's skin?  It's not body mass index, breast density, bone density or disease distribution. In fact, this is information in the frequency range, which we suspect, although we cannot prove because we do not have photos of the patient's skin is related to the melanation level of your skin. Darker skin has a more fraction of X-rays and of most radiation. And these tiny, tiny differences in the frequency spectrum are not things that are ever perceptible to human doctors. It's trivially obvious to a convolutional neural network. So much so that even when you band pass filter these images to where they do not really look like chest X-rays anymore, you can still tell a patient self reported race. And so what are some ways that we can try to improve these models if these are issues, these deeply proxy dissociations. One of the things we can do is explicitly include fairness constraints and design things for what they're intended for. For example, if you need a parsimonious model like a decision support checklist, build a complex model and explain it down, just build the thing that you actually want. Now, when humans come together to create decision support checklist or risk scores, often it's very difficult. They have to come to consensus. And then years later, we'll figure out that actually those risk scores we created, they're really biased and they over or under estimate risk, for example, in this paper for African American patients across many different clinical sub specialties. If we learn an optimally predictive chest checklist from ICU data as a mixed integer programming problem that directly minimizes error with fairness constraints. We can do things like predict mortality after continuous renewal replacement therapy in the ICU, but ensure fairness across intersectional groups. And so if you put no fairness constraints, you might get a checklist like this predict mortality after this treatment, if three or more items are checked. This is not referencing your gender or your self reported ethnicity. It's also not using any obvious proxy so an obvious proxy for sex at birth would be height or weight. Those aren't in here there's nothing obvious for self reported race, but this checklist with no fairness constraints is no better than a human made checklist. It is a max false positive rate gap between black women and white men of over 50%. When we include constraints from the get go, we get a model that performs much more fairly and could potentially be used in a deployment. The last thing I'll talk about in technical problems is subpopulation shift, which is a huge area in machine learning. Generally, there's many categories of it. You could have serious correlations attribute imbalances class imbalances, or just attribute generalization. And we have many data sets that fall into each of these categories. Some of them are medical such as mimic notes or chest x3 data sets. Some of them are not medical at all. They're just large machine learning data sets that people use as standard benchmarks when they make new algorithms. Well, we benchmarked all of these state of the art algorithms that we could find that perform really well in published papers on these tasks. And we found that these existing algorithms really improves various correlation and class imbalance, but they really don't improve other shifts because often shifts get lumped together when people evaluate their models. This is a problem because we have to improve both representations and classifiers to get to these better attribute generalization results that we don't currently have with state of the art models. A final note if you use some of these subpopulation shift papers is they often optimize for worse group accuracy, but that is often inversely correlated with worst group precision, which is maybe the thing that matters more in many healthcare applications. So I'm going to skip the content for the last part because I don't want to cheat the panel. The last thing that I'll say is we've done some evaluations where we've deployed models, and we've asked specific loaded questions in high stakes decision making settings. We've given bias GPT advice to people in different ways, and we found that the the use of specific AI doesn't matter sometimes as much. If you have a biased AI or not biased AI as the exact way that you give the advice to people either with descriptive or prescriptive advice, and here we found that clinicians listen to bias prescriptive advice, but not to biased descriptive advice. And if we want to get to safe integration, we should take lessons from places where there are self safe technology deployments like aviation. There are many federal agencies that could have a role in regulation and, you know, advice for technology and recommendations for developers for how to safely integrate AI into healthcare spaces. But there are some things that are going to be unique to help that we need to figure out as a community in panels like this. So inequity and underlying data processes that will be learned and automated, like gender concordance increases a patient's probability of heart attack survival. And that effect is driven by increased mortality when male physician street female patients. That would be like if male pilots crashed more when they were flying with female passengers, or patient physician race match improves medication adherence, and many other things. That would be like if black patients only got in flight safety announcements when a black pilot was flying, or a majority of Muslim women experience poor quality care maternal services with indicated stereotypical or discriminatory behavior. That would be like if most of the passengers were always randomly checked by the TSA, which is true. That one's in there for me. That's an actual true on just FYI. There's no simple fixes here. This is going to be an ongoing process. We should consider sources of bias in our data, evaluate models comprehensively and recognize not all gaps can be fixed. This is work from a fantastic team. Thank you so much. Thank you so much, Dr. Gosemi. All right. So now it's time for discussion. This is my favorite part. So we please put questions in the chat if you're online, here in the room, if you have any immediate reactions, things that you're excited about. Feel free to jump right in. Well, I'll ask a question. So this was probably to Dr. Ono Machado. I was really excited to see you thinking about the incorporation of other types of data into those PRS models that you talked about. And I'd love to hear if you have good examples of that working. I mean, so where do you see that happening? Where is that actually being used or is it being used and can it, where do you think it might have the best implications for use? Yeah, I think currently it's more use of social determinants than environmental data to be completely clear. But the concept is the same, right? If you want to individualize, we should, you know, get the more specifics we can about the person, which of course in order to build models would require even more and more data. However, every variable we add implies a lot of observations to be added as well. But I think in healthcare algorithms as well, I think AI was very instrumental in bringing the issue to attention, but it has been there all along with simple statistical models as well. There is currently a lot of interest in understanding more the biases created by algorithms and how to regulate, how to mitigate those biases or eliminate. So it's an area of active research right now. I would say algorithms that have been historically used in healthcare are being rethought as to whether they are fair and whether they are actually improving care rather than, you know, creating more disparities. Great question. Oh, yeah. Fantastic presentation. I have a question for the last speaker, but sorry, I don't know how to pronounce. Sorry. Fantastic presentation. I have a question for the last speaker. Sorry, I don't know how to pronounce correctly for the name. It's very interesting for you to show the example of how the language model generates different things for different rates. And I read some paper also on the same direction. But the issue is devising for machine learning models become a little bit difficult when, say, when a large-language model comes because, and you also notice that when this kind of large-language model can be used on a wide range of tasks, this kind of bias will be propagated to a lot of different downstream tasks. And the devising become more difficult. Do you have any, like, suggestions on that direction? Sure. So I think what you're asking is when you train, you can decouple in machine learning processes, training a representation, right, that maybe could apply to many different downstream tasks, which large-language models are an example of, say, a classifier that you apply to the representation. And I think what you're asking is, when we train representations right now, like large-language models or other, you know, large high-capacity models that contain biases, it may be very difficult to remove them because they are going to be applied to many, many downstream tasks. And so it's difficult to do, like, a one spot fix. Is that accurate? Yes, yes. Okay, great. So there are a few papers. We've looked into this a little bit, but there's also some topology papers, actually, at machine learning conferences, that are trying to focus on if I know that there is some statistical bias in my data that is not desirable, or a part of this larger ball that I'm learning, right, if my representation space is a big sphere.  I haven't seen tons and tons of examples, and so I don't have this smooth space that I can generalize on. Maybe there are ways for us to use augmentation, right, so take an example in a space where there are very few examples, and do small perturbations of it so that we fill in sort of that part of the space and we get less anomalies. And there's also other work on instead of trying to, you know, generate data that sort of addresses this either by actually sampling it from real people or augmenting the data you have. Just knowing when your model is out of distribution, somebody hands you a trained model, how do you know that it is in a, you know, the popular term for this is hallucination, right, how do you know that it's in an extrapolation or it's not interpolating between answers that it knows, it's kind of out here beyond anything it's seen it's in this extrapolation regime. And so there, there are some works that are also focusing on that so you could think about it from either side. Thanks, and we have a question from online from Claudio Sorrentino, who asks, how would you suggest applying the development of AI models to evaluate cumulative impacts of pollution to communities. I might throw that one to you, Dr. Bryce. So, you need to do two things. First of all, you need to be able to understand what those cumulative impacts are so you need to have data to understand kind of what the breadth of the environmental risk factors are going to be and how do you quantify those. And then you need access to the health data to look at kind of the relationships there. And those are those are the challenges think we're talking about today how do you integrate across those two dimensions so I think it's hard to do right now but I think it's where we're trying to get to. Going forward by having access to environmental data in a comprehensive way, and as well as the health data that we've just heard talking about right now so how do you, how do you look at somebody's how do you predict somebody's risk factor for heat stress. For example and then how do you use that in a in a clinical setting to minimize their risk by making sure that if you know that somebody doesn't have air conditioning, and you know somebody lives in inner city where it's hotter. And you know somebody is, you know, got the physical out of condition that puts them at risk, and then you know something about what the temperature they're being exposed to over the period of time. How do you collect all those data how to integrate all those data and then how do you do something to minimize any heat stress or hospitalization or ED visit the might be a consequence of that. And as I go back to a component earlier then once you figure something out how do you go back and double check and make sure that you evaluate whether it's working or not so that's that's just one example but I think I think you're asking a good question and that's part of the challenge before us today. I was thinking about that also you know one of the things that I feel like can be the focus is like this individualized approach but I think you know the word communities and that question was really interesting. I always come back to the idea of, you know, Flint Michigan like if we had AI looking for trends in blood lead levels or in lead levels in the water, could we have prevented something like that and what can we be doing to prevent something like that in communities or populations I don't know if you want to comment. Just a quick example, I think that's a great idea so we collect a lot of environmental surveillance data, and I'll put blood lead in there is as a environmental status data even though it's a human based measurement. And then we don't do a good job of mining it and we don't do a job, a good job of doing exactly what you said how do we, how do we use that to anticipate when a risk is about to happen, rather than waiting. And for a year or so after the switch to show that the blood levels.  You know we need to be able to use the data to anticipate that problem as I said before and that's a great example. And then also Claudio thank you for your question so the next question you posed is can AI help identify critical parameters to evaluate and predict cumulative impacts and you know this term cumulative impacts might not be well known by everybody so I'll take a step back and just sort of explain my perspective on what cumulative impacts are. So it's the idea that communities that may be unfairly impacted by environmental issues are unfairly impacted by a lot of environmental issues so if you're exposed to high levels of lead and you're drinking water, the likelihood that you're also exposed to high levels of crime in your neighborhood or that you're also exposed to higher levels of unemployment or, and so all of these different factors become cumulative and they may add up in ways that is not additive but actually like exponential they the impacts that they can have or more than you would expect if you added those risks together. What I think Claudio is trying to get at is, we haven't been good in risk assessment and environmental health risk assessment we tend to look at one chemical or one exposure at a time and estimate risk. So can we use AI to look at all these risks because it's sort of like that expose on type of approach. I'll just throw that out to anybody on the committee, the ones to answer and Claudio I hope I didn't take your question in the wrong direction. Yes, Darrell. Yes, Megan I wanted to second that motion in terms of we do have frameworks that we're building with, of course, the support of the NIHHS but more often than not, these chemical and nonchemical stressors, as we were talking, Lucinda I think talked about that as well, one exacerbates the other. And so the polygenic approach in terms of risk trajectories that we should be looking at and turning that into, you know, risk model but the cumulative we've been talking about that since Charles Lee in the early 2000s and I'm happy that we are finally getting there. Yes, definitely moving in the right direction but I see a lot of opportunity for AI to help with it. I don't know if that's happening yet. Well no I and I worry from the standpoint of underrepresented communities with respect to, you know, the medical records and the inherent bias I mean, we're covering it we're getting there, but not we aren't moving fast enough but the speakers did a wonderful job I was really hopeful now that we can close the gap. That's great hope is a good thing. There's a great Langston Hughes quote I think there's a Maya answer about hope is a caged bird. Is that right. My poor English teacher. So I think the question though hits at the heart of what we're trying to talk about is data integration, how do you, how do you combine all these data to identify all these cumulative stressors. And they're sitting in different places and they're, they're accessible in different ways. They have different data structures they have different time frames they have different geographic frames. So this is a perfect place to think about how do we integrate that. Yeah, and I'd like to add to that when we're thinking about these cumulative exposures. The action items and thinking through very carefully our methods and our techniques, which levers do you pull right if you're measuring everything from employment opportunities commuting outcomes on road emissions water quality. Pollution crime stress and indoor air pollution outdoor air pollution where your schools are what are your schools exposed to when children are spending eight hours a day school right on top of a 10 lane highway. So, AI allows us or just you know big data allows us this data integration, but as policymakers as clinicians as local community organizers advocates and so forth. What levers have the greatest pull, and how do we integrate these and accurately tease out where can we make some biggest impacts where our biggest hotspots. And that comes back to the methods right and this and finding these various correlation so I just kind of wanted to add that little fine point to what's our end goal. Once we put these things together. I agree yeah so what how do we actually impact health how do we make a difference with limited resources where are we going to drive those resources yeah absolutely. Great point. Thank you. I think I might say on the health side is. Sometimes people try to propose technology as a solution where it's not a good solution. So, even if we look at the levels and Flint Michigan right like that was a result of state emergency managers changing the city's water supply right and maybe we could have predicted that this is something that would have happened and I don't know if an AI telling them. Hey do you realize that these old lead line pipes are dangerous would have changed their their action right like this is a cost cutting measure. So I think one of the things we need to understand is AI is this really great really powerful tool for data analysis for giving us better information. We're not always great at using that information when we get it. And so there's a big push in the machine learning community right now. Some of you may have heard for your sort of two camps you know this sort of alignment camp that saying we need to align AI with with human objectives and then there's. This other camp called the safety community that says well, you know we shouldn't just try to align it we should just use it as a tool and use it safely. I think these are two different mindsets but they're both hinting at the same thing that unless we know what we want from AI as a society, people will use it for whatever they want and that might individually create really poor effects. I would agree and in the medical area, there is also, you know, you need to know why you're creating a model and whether you're listening to it or not, and if not, again why. So it's not simply saying well we could have detected this the fact that we didn't right and the fact is also that we didn't in many cases, no not this specific one but in many cases because there are so many false positives of others that we would not have created each one of them so I think the whole area of post marketing surveillance of devices medications, or even surveillance in general safety surveillance is, you know, has been evolving. Even before AI became more popular but it's always caught in this dilemma right how, how do, do we have ability to respond to signals and do we have enough specificity in this alerts and alarms so that we can act upon them. So it's very tricky area. So we have some more questions from online Olivia Harris asks, have there been AI analysis of situations where most medical errors occur, or would fatigued health care provider factors overwhelm analysis. So I can say that there've been several studies of medical errors and many areas prior to AI. Again, AI analysis would be possible, as long as there enough data labeled data, I would say about this, because just counting on outcomes. would not necessarily indicate the errors, right. Sometimes you do the right decision but the outcome is not favorable. So I would say there've been analysis and in many analysis also indicated fatigue being a factor or a poor user interfaces and things like that. And I think was so is exactly right. We know this is a problem without AI. Right. And so one of the things that we have to think about is when when we train models we're training them on the data that exists. And I love her point, it would need to be labeled. I can imagine asking a hospital to go back through their records and label every single decision made by a clinical staff member that could amount to malpractice. I don't think that would happen. And so one of the things we have to you know consider is machine learning runs either on millions billions of examples with no labels and it learns. This is the way you act you haven't told me how to act but I see how you do it and I'll just do it that way and accept that it's going to sometimes act like a fatigue doctor a doctor that used a poor user interface a doctor that didn't have their icon, whatever those situations are and make a mistake or we have to decide that we want to go back and somehow create an accounting of these are things that should not happen. If they ever happen in the data. Don't use that data that should not be used but that's not in the data that gets sold to private companies for the models that they create that gets even without large language models or any high capacity models. Even if you're just training with just a regression or XG boost models, you're training with error in there and you're training with maybe error that affects different people differently.  So I wanted to dig a little bit deeper into the Laura's question, because I was actually formulating a very similar question. You know, when we are trying to integrate you know use AI to integrate all of these extremely highly correlated factors. Dr.  High proximity to traffic because of different policies, you know, all of these are going to be showing up as drivers and how do we identify which one is the one that we want to target for intervention. So, I think there's maybe two parts to this one is what data do we even have for a long time we didn't have any social determinants data, and so you could not use it. And now we do in many data sets thankfully thanks to efforts like with those with the all this data set that I'm a huge fan of. There are social determinants questions in that data. So, I think we're starting to look at whether we can integrate that data into an analysis. We have a paper that's coming out in a month at the ACE conference which is one of the AI ethics and society conferences that the community runs demonstrating that in hospital tasks specifically so acute care tasks, social determinants data that we found at a state level in Massachusetts did not improve prediction in general population. It did improve specificity of machine learning models in specific tasks for minority groups right and so for example for looking at data and trying to predict outcomes for diabetics when they're hospitalized there having social determinants data did help a lot with specificity, but just as a general thing you throw into a model. It's not a, it's not a solution a simple solution because you have to imagine things like self reported race, and social determinants are deeply proxied in data, and they are deep proxies for other things in data. So I think in that CRT example where we made that checklist. I can't see what the proxies are with what could this be. It was still an incredibly biased checklist right and until we included explicit fairness constraints it was not addressed. And so this is an ongoing area of research that we need to work on. We tried as researchers to decouple information about self reported race or sex at birth or social determinants from clinical data, and then take this decoupled you know sort of washed data and then use it for clinical tasks. And so models don't perform as well in all cases right and so for some data. There's information that's being deeply proxied in different parts and so we have a couple papers we're working on now on the completely machine learning side just trying to understand. How do you remove enough information about somebody's for example self reported race or sex at birth if that's information you don't want out or about their social determinants if you don't want stratified care you don't want to learn these sort of weird social things that we have or poor systemic injustices that how do you remove that while retaining enough information that you can actually perform the clinical task that you care about. And it's a very hard question but it's one that more diverse open health care data, like the all of us data set and others would be extremely helpful with. I'll give you an example to in the COVID crisis in which we analyze data from several health systems 14 health systems, large ones, including the VA, and we were showing in hospitalized patients that the mortality of patients identified as Hispanic was lower than the mortality of Caucasian patients, for example, which is against the, you know, whatever everything else we were hearing. So I want to emphasize the importance of having experts who know the data and who can explain certain things because once we did the multivariate analysis it was clear that age was the factor that was really leading to higher mortality and it happened that the white population was older than the Hispanic, or the ethnicity Hispanic in that case which again age being the major driver if you have a correlation of Hispanics that were hospitalized being younger, you would see a lower mortality of that group anyways. Right, so the interpretation of all this models and what, you know, AI or some modern predictive models telling you it's something we need to be extremely careful about because we can make again this whole movement of liberating it's interesting is important but the, you know, misinformation is everywhere and miss an interpretation of models would be even worse because now they are data driven and someone is using the data and just interpreting in the wrong manner. Lucila, with that data that you just referred to. Is that at the county level, or census tract block group. Those were hospitalized patients with electronic aggregate. Okay, I was just trying to figure out your source. Yeah, from those who, you know, access to care was no longer an issue because they were already hospitalized. Right. They were received here uniformly that that's what it seemed to indicate to us that there were no adjusted for other factors there were no mortality differences. So I think we have time maybe for one last quick question. So, and I'm going to kind of change it a little bit dip on John Amalek asked if you think historical database on stressors and biomarkers should be developed using routine health check up data and environment data and that was one of the questions I've had as we've been talking you know everybody's like all of us is great. But there was also a comment that all of us doesn't really have a lot of environmental data in it. So, yeah. So maybe that's sort of thinking about next steps right that's the whole point of this workshop where to help how can we advance it. Yeah, I think that's exactly right. And so that's the challenge before us. I mean how do we, how do we get better environmental data and then how do we get it into a system that can interact with health records so it's very easy for childhood asthma for example, to talk to clinicians say the treating asthma is the you know the kids who have uncontrolled asthma because they don't have access to a physician and they don't take their, they don't have access to medications. And then of course you talk to our health scientists you say be able because they got cockroaches and in their house and they got bad air quality in their house and there's all these environmental determinants of asthma. They're exposed to violence in the community there's huge stressors and there's all these environmental things that cause asthma. And we don't really start this out very well because we don't those two data sets don't coexist in a way that allow us to look at kind of what the, what the whole constellation of drivers are. And it's obviously some combination and speaking back to David's point before we can use that to determine what's the best bang for the buck in terms of, you know what do we want to do with that you know there's a study. There's studies that show that, you know, putting an air cleaner in someone's home is is as efficient as putting them on controller medications for as many, you know, an air conditioner cost a fraction of what controller medications do. But the health system won't pay for an air cleaner in someone's home so so but how do you build the system to allow us to make answer ask questions like that and answer questions like that is think is why we're at the table. That was a great I feel like that was the perfect summary we have one minute left so I might just draw us to a close, unless either the other two panelists wanted to throw something else in there quickly. I just want to agree with the Allison Motsinger comments that the all of us program is working hard to incorporate environmental data. Yes, and David said the same thing in the room. Yes. Thank you. Well perfect so thank you so much to our panelists and to the folks online and in the room for a great conversation I think this is really setting us up for some diving deeper into a lot of these issues and coming up with some plans for what we can actually do to better integrate all of these different fields so I think we have a five minute break. Enjoy. I'd like to welcome everybody back to session to title of the session is leveraging AI ML for environmental health and biomedical data integration. Introduce myself first for the speakers. My name is David right. I'm chief of the predictive toxicology branch at NIH s and what we do is very relevant to this workshop. Our branches expertise in data science toxicogenomics spatial temporal exposures computational methods and new approach methodologies to advance predictive toxicology applications. Prior to joining in IHS last fall I was a professor of bioinformatics at North Carolina State University, where my lab was focused on data integration for environmental health sciences and toxicology. So I'm delighted to be part of this workshop and really look forward to doing the engaging session. Our objective with session two is to provide research studies and use cases of first how integrating environmental biomedical and health data is important understanding human health and disease. And secondly, how ML can be used in the pipeline for these data. So for this session we're going to have three presentations, followed by about 30 minutes of q a panel discussion similar to the last session. So, you know, hopefully try to keep each speakers to, you know, 10 to 12 minutes for the presentations, but we do have that buffer time built in at the end for discussion so hope we have time for a good robust discussion. So I'll start out by introducing our first speaker. Very lucky to have Dr shrug Patel, who's an associate professor in the Department of biomedical informatics at Harvard Medical School. Another research interests include developing multi-scale computational and data science methods to dissect the role of environmental exposures and genetic factors in complex traits and disease. With an emphasis on a trajectory from obesity to diabetes and its complications. Dr tell is recognized as a leader in exosomic science, developing methods to map systems of dietary environmental exposure factors with disease. So like to turn it over to tell thank you. Thank you, David. And thank you all for having me it's an incredible honor to be here to talk about multimodal AI. I think a extremely timely, timely topic given the emergence of some of the the approaches that we've seen earlier today in the data. You'll find that I'm reiterating a lot of the things themes that we've talked about already so hopefully towards the end I'll point out some things that we can that we can add to the field here and compliment to those awesome stocks we heard before. So as you know informatics for us in informatics it's been a deluge of data coming from biobank scale as resources things like UK Biobank now all of us and existing resources such as and hands the CMS Medicare data. There are a lot of opportunities that we should discuss on how to make them useful, in particular with multimodal data coming from expose own genome and phenome and we've seen great examples in the literature from genomics and phenomics and so can we leverage these data sets, along with multimodal AI techniques to make them useful. We've talked already about a lot of approaches in clinical AI some fantastic talks earlier. So the key questions that a clinical AI people are asking just to reiterate or to try to integrate these data that may come from diverse places such as one's healthcare institution, like a acute care setting, and try to mash up things that are high frequency time series or low frequency time series a when you think of time to death. Things like tabular data spectral data 1D spectra 2D spectra free text. And so you can see the diversity of data that are being tackled in the current approaches for looking at multimodal data. And the current approaches might use some new cool ideas of in neural networks to create a new data process or new data structure called an embedding essentially it's a, it's a new way to represent data that you can then input into your favorite algorithms of choice to make some sort of learning algorithm work or make some sort of decision. So questions that emerge from these are, first of all, does when you have have new models of modalities of data, do they actually help you with with prediction. And so there's multiple evidence that coming from our folks in a collaboration in AI, suggesting that here's a fantastic paper from so angst and colleagues, which I think we'll hear more about this when Eric Topol gives his keynote about how predictive capability here you're seeing AUC curves, in fact increase for several outcomes that are captured in this mimic data set used for for machine learning research to look at high frequency data coming from different sort of modalities. And you can see here from the AUC curves that when you increase the number of modalities for these different outcomes, you fact get a bump up on average and this is also showing, but in fact it is diverse, it does depend on the type of outcome you may be looking at so can we tease apart the outcome from the modality in in dissecting when our capability increases. For us environmental health studies, we're dealing with this, this complex expose own type of data set so they come might come from targeted mass spec which may be tabular or we may be looking at the spectra, like biomonitoring of lead we're looking at geospatial by digital markers so area level, we're seeing 2d spectra coming from images which are, which many of which I'll have talks about later tomorrow I think so and aggregating this to zip code or even a personal level. We have self reported questionnaire information very important that are also tabular, giving us behavioral information, new modes of getting behavioral information from sensors and of course on targeted mass spec so diverse types of data diverse types of examples here that we need to integrate across to achieve this idea of the expose home of analyzing things in the multitude to, to in fact, increase predictive power. So he's a great example of looking at the sort of the blood expose almost small molecules that might be indicative of environmental exposure. And complementing this are geospatial markers, other analytes that may soon be commoditizing in, in, in, in addition to those, those markers. And of course there's geospatial information emerging from satellites, again, diverse multimodal data sets, how do we stitch these things together. So first is, I'll show a couple of examples here of how people are doing this. Here's one from nutritional, our friends in nutritional epidemiology and AI. So here, Ravi Shah, bank, more than colleagues, we're able to look at what we call the internal expose home or the metabolomic signatures that are related to dietary measures, and then connect those signatures to to cardiovascular related traits like incident CBD and these. And also to use these metabolomics measures or the internal expose home or how the exposures respond at can detect some biological response to get a pattern of dietary exposures using this idea of canonical correlation analysis, and then connecting those patterns to incident CBD and chd. So I saw this I was like wow you could actually get something that's trained on dietary behavior, those biological measures of the internal expose home or metabolome. And they'll, once they're trained they can actually do better in predicting the outcome there's just something is biological signal that's also emerging from essentially predicting these behaviors. This is essentially what this figure is trying to show here from Ravi Sean colleagues is that when they're predicting dietary behavior, based on these metabolomic measurements and then going forward to predict disease, you're actually doing better than the the the self reported recall measures themselves. And then they use this idea of dietary patterns using canonical correlation analysis a very old school approach but with given these new sets of data have found new found uses. And then they've used a this way of finding a pattern and then also looking at how these are related to the health, healthy eating index and found that in certain circumstances they might also be more predictive of diabetes and CBD. We and of course committee members like Allison have developed the poly exposure score so this supervised approach of developing a score that summarizes exposures across diversity of domains. Here we use one to look at diabetes undiagnosed diabetes in the UK Biobank and our colleagues use the peg score to also in concept kind of validate what we had. And what we find is and I think this is key for us looking at multimodal AI techniques is how to compare it to the state of the art. And so one thing we did was looked at polygenic risk scores and we found the poly exposure score did better with our with our measures. But of course you also want to compare to how people may be using this to screen populations in the clinic like using blood pressure like using family history, like using glucose. And we're not as good as as as as the poly the clinical risk scores, but you could ask the question of how well you can reclassify individuals who already have some sort of risk latent risk for diabetes. Maybe their blood pressure is very high their glucose are in the pre diabetes range you can ask the question. And that data the clinical risk that you might have how much does it the exposures adding them actually add to reclassification, we have some evidence that it's doing much better than, than genetics, it might be even helpful for looking at individuals who have not been able to be screened yet for diabetes using the gold standard measures such as undiagnosed diabetes. Building these approaches of looking at risk factors that are already strong for particular outcomes like COPD, and looking at how the other exposures might push people over the edge for, for example, if you are already smoking already at high risk. But then if you start incrementally adding this information, how much better can you do in prediction for things like a lung function. And we'd say that in the UK Biobank we're comparable to using a polygenic risk score across smoking status for COPD. I think the use of the deep learning technologies is also going to be very useful in integration of exposome information. So totally another approach of looking at new phenotypes for for aging. So here what we did was ask the question can we predict abdominal age or age for biological age given magnetic resonance images from individuals in the UK Biobank so here's examples of images and we just apply some deep learning convolutional neural necks against these these data to ask the question how old these people are, and then figure out the residual them so given the prediction and the their actual chronological age. Is there any biological signal in that in that residual. And in fact we did find some biological significance we did ran a GWAS against those residuals. But we also ran an x was against the that residual to find that things that are that are lifestyle factors such as packers of smoking alcohol, all connected with this accelerated ages as measured by the deep learning algorithm, algorithm, exercise behaviors, accelerometer based physical activity all connected also with with decelerated age of abdominal agents unpacked ways of looking at the data in different ways. For example, do these this exposome that intersects with pancreatic or abdominal aging actually come first before diabetes and or or their independent sort of any lot independent etiologically. These same tools can be used to look at accelerometer data, as we've done here I showed this as an example, we looked at biological aging with respect to how how people exercise. I think one word of caution is that it when looking at these very complex approaches multimodal AI. It's unclear how we even do the studies to begin with the studies to begin with our have a lot of things under the hood that we haven't really figured out. Here's example from just doing one at a time simple epidemiology where we where we where we look at ingredients from the Boston cooking school cookbook and we see that there's a lot of, if you will vibration around around these these studies that that lead to different relative risk estimates so simple regressions. What happens when we start adding parameters to these models and understand trying to understand the risk. So, here we would say that the john shown fellow johnny and needs and colleagues that are weak statistical evidence against some of these non replicated inconsistent effects and so the ideas once we do we exacerbate the problem by looking at multimodalities and and multiple parameters. Today's in meta science to sort of elucidate the role of study design. One is to throw the kitchen seek of compute against all these different factors and seeing which one stick which ones don't, the most robust should be very robust to the data assumptions that might make. So we asked the question for for all cars mortality for example, through all these study design parameters in it could be different variables that you're adjusting for your model. Do any of these affect the estimates that you might get or the hazard ratios you might get and in fact for many environmental factors if you're looking at all cars mortality. In fact, do some of them are very consistent like like coating, but some of them are wavering around that estimate and so this is very, this is generalizable across data sets here's an example using UK Biobank. And so the idea is you could use this type of approach to screen for those that are non robust to your assumptions that you make in the modeling and for example here's one that has a risk profile for some of your models and a and a protective profile for other models and you'd may want to say that this is something that's not robust the assumptions you might make and filter that out from your, from your future studies for example. So the technical opportunities and challenge for environmental health research are, are, how do we integrate across the these diverse scales such as geography tissue time model systems. We integrate findings from new approaches and compare to the state of the art, and what data sets and approaches are need to establish benchmarks which are very important to help us interpret findings for the field. I thank all of my collaborators and look forward for the discussion. Thanks very much for having me. Excellent yeah thank you so much and, especially for you know the theme of the session is having examples and use cases right so I think we'll have a lot to see our discussion later. I appreciate that very very much. Thank you Dr tell and our next speaker is Dr. I don't john, who's a full professor of computer science and the School of Engineering and applied sciences at the University of Virginia. She also holds joint appointments with the Department of biomedical engineering and the School of data science at the University of Virginia, and her research interests include machine learning data science bioinformatics and health informatics. I look forward to talk. Thank you Dr john. Yeah, thank you for the introduction. So that I share my slides, or. Yes, I think what we're seeing up on the screen is as your, your main slides that looks correct. Yep. Okay, but should I share mine that you know so I can control it, or this is a from the audience right. Lily, are we running the slides from the room or should Dr john shares. You can share your own if you'd like. Okay, someone start did start sharing them for you but it looks like you're pulling them up now. Yeah. Okay. I think you just want to go to display mode on the slides. I don't understand this. So you. So, if you go to the bottom right hand corner there's the little like thing kind of looks like a screen icon if you click on that. That's a way to get it to presenter mode. Yeah, I. Okay, I so I cannot get to that. We can, we can share them if you'd rather that's fine too. And then you can just say next slide. Oh, well, let me see. Okay, but do this. Just still get here. I think it works now. That looks correct me. Okay, sorry about that. Okay, so. Thank you for invitation, you know, to give this talk, and I've been listening and so I think it's great workshop to organize this topic. I'm going to focus on more on the more fundamental part of how, you know, people have mentioned how machine learning will help with multi model data. And so I will talk about the specific approaches that we have developed on the multi model learning. So, basically, as you know, all of us have seen in the previous talks, people have been integrating all different kind of modality data. And these are from different sources, different sensors, especially currently there are all different sensors measuring all different kind of data, and it's actually becoming a very critical issue how to integrate them. So, in the past, you know, I, you know, from the previous talks showed the integration of the multi model data will help. We also have experiments to show on the multi omic data when we integrate different modalities together, and you will see the prediction of the samples increase significantly. But the issue is how to integrate this different modalities. But the challenge here, if you look at it, so the most important challenge, I would say is the missing part, missing data, because in one modality, you don't see that much, but when you combine different modalities, for example, you look at it here. And you will see most of, you know, mostly very common, you will have missing data from different modalities when you combine them. So, then how are you going to deal with this if you want to fuse the data and for prediction or other, other any research you want to do the follow up. In the past, people have used the imputation. They have used most of these imputation, but now with the advanced machine learning models, we really should look at how, you know, beyond the imputation and how we are going to fuse different modalities and to do a more effective follow up research for this. So that's what I am going to focus on. And so this is part of an SF project is funded for multi-modal learning for multiple domain data with incomplete modalities. Now, so what our approach is, is to, first of all, with the input data when you have different modalities with incompleteness, we want to be able to construct this data and to represent it in a way we can avoid the missing part. And at the same time, we can handle each data represented in a unique way so we can use the machine learning to process it. So people have used deep learning or other kinds of learning. You know, if you have data missing, you cannot fit into the deep learning model. So you have to be able to represent in a way you can fit into the model. So graph model is a very popular strategy to do that. So that's where we propose to model the multi-modality data into a graph model. Then we can use graph neural network to map the data into an embedding representation for the follow up research. So to be more specific, how we will handle such, you know, different modalities with a lot of incompleteness into a graph model. That's the central part I want to talk about. So for example, here we have three modalities. And so with three modalities, you know, all different combinations, you will have, you know, seven different kind of pattern, right? So you have seven different patterns. So with more modalities, so normally you may have hundreds of modalities. Then the missing part becomes more complex. You will have more combinations of all the patterns. That's what I hope to show with this slide. So when we deal with the missing data, we can convert, treat this as a different pattern with each different kind of machine. So with that, you can work, so you have different patterns. So then each sample, for example, this patient one has only one modality and two other modalities are missing. So this is one kind of node, we call it a hyper node, but it's one kind of pattern. And then the patient two have a different missing and will have different patterns. So this is another hyper node. So each node is represented as a different hyper node with different modality it contains. So I hope that's understandable. So with that, you can represent each data with a unique hyper node without this, you know, you don't have to impute the missing data. So then we got the nodes, and then we have to establish the links between the nodes. So the hyper eight edges we established is calculating the similarity between the different hyper nodes with the each modalities similarity. So, for example, this node has two links to this one to this one because the MRI image has a MRI image in this node and MRI image in this node. So you will calculate the different similarity. And if it's above threshold, it's very similar and you will establish a link. So that is called happening. So you will link different nodes with happening. So for example, this one has the second modality and third modality, it will link to this one and to this one. So I hope it's understandable that way we can convert the multimodal data with incompleteness into a heterogeneous hyper node graph this way. So we don't have to deal with the imputation of missing data. So after we represent this in a graph, then it comes to the issue, how you're going to handle this in machine learning algorithms. So it becomes the graph model and the input into so we machine learning you normally map the original data into embedding space, which is just a vector space, and then you will do follow up or follow up research, for example, prediction or other decision making. So how do we do this? So basically, this is a purely machine learning and nothing to do with domain anymore. So basically you have different modality that you can convert into vector space. With each modality, you may have images, you can use a CNN convolutionary neural network, you may use other kind of neural network, if it's texture data. So different modality can map into a unique space. And then with that, this is just modality. But each hyperlink, remember what I said here, you have an interaction between the different modalities inside of hyperlink. With that, we can use the embeddings to catch that using the neural network to generate embeddings for the interactions between the features. So in the end, with each hyperlink, you will be able to convert into internal space with the node representation, and also the interactions between the modalities inside the node. So for example, this node has three different modalities. So not only it has embedding for the node, it also has interactions between the nodes which are caught by the embeddings inside. So this is what AI machine learning is very advanced now. We can do all this automatically in the system. So after that, you have the internal nodes, and you have the hypergraph. So then it becomes a very technical neural network issue of how to convert this graph into embedding space. So, which I will not talk very detail, but on the high level, we can convert this heterogeneous graph, which is a hyperlink, the graph into a homogeneous graph with each node is embedding and connections are also interpreted in the embedding space as whether they are near or far. With the embedding space. Okay, so that's how you can basically represent your multi-model data into embedding space, catching all the features between modalities in machine learning space and to do follow-up research. So in that way, you don't have to do imputation, and so how this works? Does this work? So the experiments here shows, although this data set is more on the object recognition, not really on different modality in the biomedical data, because in that paper published, we need to compare with the state of art and baseline, and so they all use this data set, so we have to use that. So basically, we have all this data set, which has a huge number of classes in each data set to recognize the object faces, and they have all have different modalities. So in this case, we use the two modalities. So the object has a different way to catch their image. So combining different modalities together, and we showed here with complete data, only the complete data, and what are the performances, and then randomly dropping 30%, 45%, 60%, and 75% how each different model works. So you can see our model can work very well, even dropping the 75% of the data in the experiments. And you can see with the data missing gradually increasing to higher original, and then you will see the drop of accuracy. So that's understandable how different approaches work. And you can see all the other approaches actually impute zeros into the missing data. So, and that shows how this approach work. That's basically I hope to introduce how using advanced machine learning, we do can, you know, fuse different modalities but without using imputation data, and we can actually make a new representation for the complex data set to represent the complex data. So I will not go to more details, but you can see this paper we presented in the top data mining conference KDD 2020. And we also have a project on multimodal machine learning for database in completed modalities. So this has been worked, you know, applied to different domains and to show how effective the approach can be. So I will end here. And so I think, you know, in summary, the multimodal fusion can be achieved in graph structure. And as I shown, so it is effective. And by modeling data in graph, we can deal with heterogeneous modalities with incompleteness. And this shows the effectiveness. And we also have worked out the in distributed cases, you have different mission data and this approach can also apply. So I will stop there. Wonderful. Yeah. Thank you so much for the presentation. I think I look forward to the discussion about, you know, not just fusion of data but also fusion of machine learning and AI elements right. So there's a lot going on there that I'm looking forward to hearing more about so thank you again I want to introduce before the discussion. Our third speaker, Dr. Heidi Hansen. Dr. Hansen is a group lead of the bio statistics and multi scale systems modeling group in the computational computing and computational sciences director at Oak Ridge. Her research is focused on disentangling the interactions of genetic and environmental influences on disease risk throughout the life course. And she aims to link her findings to clinical measures that can be used to improve precision strategies for screening and treatment. So that like to welcome Dr. Hansen. Thank you. All right, thank you so much. And I'm honored to be here to present to you today. So I'm going to start off my slideshow with this question, the power of large multimodal biomedical data. And I have a question there, question mark there for a reason and the reason is is a lot of the algorithms we're talking about are super data hungry. And they don't always transport well into other scenarios. And so this is the vision that we'd really like to have where we can follow an individual over their life course, collect as much data as possible so that we can identify who's at risk for disease early on and treat them. And then not only stop following them at that point in time but really follow them through the course of treatment. But to do this well it takes a lot of data and to do this well across many different diverse communities. It takes thinking about things for more than just the perspective of a single place. So, this is just a look into some of the projects that we have going on at Oak Ridge National Laboratory one of them is a partnership with Cincinnati Children's Health. And we are working with them to build a model for anxiety trajectories for children. And that model should the goal that models to have something that's rolled out into the clinic can identify children at an early stage of anxiety diagnosis. So that they can jump the doctors can jump in and help treat them. We also have similar projects ongoing with the Veterans Administration. And then one with the National Cancer Institute that I'm going to talk about in just a little bit, but you can see, there's lots of data available when we start to combine it with the environmental data sets. We get even more data which is exciting, a little bit overwhelming and thinking about how we deal with that. It is still a question that we are dealing with. But this study is designed just for Cincinnati Children's Health and we're using data just for Cincinnati Children's Health. I'm going to walk you today through something more of the infrastructure that I'm very excited about not that I'm not excited about that that study. But how can we start to think forward so that we're making sure that as we're designing these things they have the ability to scale up. So can we start really thinking in a way that allows us to design early on so that we can talk studies can talk to each other and really start to build a full population health models, rather than just models that are specific to different clinics and I know that's a hard question and we're not going to be able to solve it in a day a week, a couple of years, but I really have been thinking about how we can do this in a more broad population level way. So what's required to do this well I'm going to talk not a lot about this. We have a lot of compute at Oak Ridge, but probably our biggest advantage is we have a spot where there's strong interdisciplinary team science. So when we talk about multimodal data we're talking about omics we're talking about imaging we're talking about the environmental exposure part electronic health data. The person who knows all of those things really well is brilliant. And most of this maybe don't quite hit that level. We're still brilliant but not not quite that level so in order to do this very well. We need folks that are experts, not only on the compute side in the domain, but they have domain expertise and expertise in their own specific problem set to be able to come together. And talk about what it means to really do multimodal data analytics and what kind of things should they we be thinking about so that the algorithms that we come up with aren't biased and don't have issues when we actually roll them out into the clinic. So I'm going to walk you through a use case this is a collaboration that we have with the National Cancer Institute, it's called the mosaic project, modeling outcomes using surveillance data and scalable artificial intelligence for cancer. The reason I picked this project to highlight is because it actually is one of our biggest projects that is truly a population level. So the data that we have to work on this project is coming from the surveillance epidemiology and and results registries.  They're also involved in the national childhood cancer registry so through that partnership we get information on imaging information on omics and electronic health records. So we're collecting a lot of data on cancer, and by we, I mean, the partnership MCI is actually doing the data collection. The day that we're training on includes data from Louisiana, Kentucky, Utah, Seattle, New Jersey and Mexico and California. We train our algorithms using that data, and then we test it using the full ser data. And basically what we're doing right now and I am going to walk you through some of the architecture types of things because these problems are so big is we are extracting information from unstructured data using large language models. We take pathology reports which is what you see on the right hand side, and we basically extract the information that seer needs to be able to do their cancer reporting so we categorize site sub site histology, laterality and behavior. The reason that I'm pointing this out is right now this is rolled out in production at the national level so these algorithms are used on a daily basis and seer and seer DMS. We have built in basically uncertainty quantification. So we're talking about health space which means we're high risk and our algorithms are always going to give us an answer. Whether or not we can trust that answer is a question that we should be asking if we're talking about rolling things on to the clinic if we're talking about a causation we can deal with that in a different way for thinking epidemiological studies, but really truly the science that we're rolling out needs to incorporate uncertainty quantification so what we do is we built that in. We don't predict on reports that we don't trust, we set the bar very high at a 98% accuracy across all data elements. And then we predict on those ones that we trust and so right now across the US, we're able to take a report and predict 17. Those are interesting percentages that I need to fix but we basically provide report somewhere around 23 to 27% of path reports across the US so we can take that information rapidly phenotype those cases, and then we use that information for downstream tasks. And that's the part that's really important here is that we can extract information from one modality. And then we can actually use that information for downstream tasks if we need the interpretability there, as was talked about in some of the other talks we can also use the embeddings for that. But this gives us that intermediate step if we want interpretability. I'm not going to get super into the details of some of the things that we're doing but outside of the large language model that we have in production, we are working on foundational models and the reason this is exciting is we can take all of the data that we have pre train and then we can predict on many different types of tasks. So basically we come up with one foundational model that specific to pathology reports and we can start to predict on many different tasks, which is a domain shift from what was done before where I train one model for every single task that I'm doing. It saves on compute, it makes us more flexible, and it also allows us to do what we're trying to do right now so we're partially in the building part of this infrastructure, this is the vision for it so essentially what we're trying to do now is build a library of foundational models. So what I mean by that is we have all these different modalities, can we start to build these foundational models that can be used on imaging data, pathology data, clinical text data, survey data, social network data to have those models be trained and then start to predict different tasks and combine them in ways that are specific to the study at hand or the prediction model at hand. So right now we're in the stage of building some of these, these models. And so we hope that this will allow us to scale to different types of questions across the US and make something more reproducible and replicable. Let's talk a little bit more about what we're doing on the information extraction side from electronic health data. Now I'm going to talk about the fact that Sear is also linked to residential history data which is super exciting to me. So what they have is linkages for from Lexus Nexus across 11 of their Sear registries right now. There should be 15 Sear registries by the end of the year. This means we have for those same pathology reports that we have records for we have 3.2 million records with residential history, and that residential history goes back to it goes back to 80, but it's high quality from 1995 to 2020 and 83% of those records of geocoded at the point location. So we're working with NCI to start linking to these three data sets first indoor radon exposure RSE I micro data and air pollution data, so that other folks can start to do research using this information. So this is the way that we're doing this and this is, this is going to be I'm going to walk you through how we're doing this, but it's applicable across many different data sets that we're trying to collect in our environmental health data repository. Essentially we are using over age hexes, and the reason that we chose uber age hexes as we can take that point information or we can take polygon information, we can overlay that information with a fishnet of hexes, and then we can extract the information that we need. So one example of this is with RSE I micro data so we took RSE and micro data, overlaid uber age hexes on top of that, then we were able to summarize what the value was within that uber age hex. The beautiful thing about uber age hexes is you can scale them based on population size. So what you see on the right side is the bigger hexes have smaller populations, and we need to scale out to so that we don't have to deal with privacy issues. So we have those hexes large in spaces where there's very little population so that we can scale to the level that we need to to protect privacy in urban areas that are like here, they're really really small. And that allows us to get specific and the types of exposure areas that we are looking at. So that's an example with the RSE I data but what we're actually doing is doing is we're dropping those hexes over every single data set that we have. So we start to build an external expo zone measure that we can stack on top of each other and measure as many different things as we can, and start to describe all of the MOLU of what you're exposed to an environment. The other thing that we do is we drop a population mask over that so we use the land scan data, which identifies where populations reside. That way we can throw out information where nobody's living, and get rid of it, and only store information that we actually need based on where populations are living. The beautiful thing that allows us to do is it allows us to assess missingness so this is what you're looking at on the right hand side is from individual home rate on test kits. And there is a selection bias into what we actually see here so in order for us to get a test kit someone has to put it in their home test, which means certain areas or certain populations may not have testing in their home. We assume that they don't have high rate on exposure but it may purely be for the fact that we don't get those measures. The population mask allows us to identify areas in green here where populations live but no testing is happening and those may be targeted areas or things that we need to think about in our modeling. And again, similar to what we're doing with the electronic health data is we're building basically libraries of models that we can use for downstream tasks so libraries of foundational models for environmental exposures. And this includes behavioral social data and chemical exposure data that then we can pick and pull to create our multimodal models, and also combine together for looking at disease risk. Yeah, this is a question I get asked a lot is like that's great and all you're doing some of this for mosaic but how does it apply to the rest of us. And this is a part that I'm really really passionate about is making sure that what we're designing is reproducible, reputable and usable for real world applications. So at the project level what we are doing is making sure that all of our code from start to end so data processing pipelines. Actual code for the models is publicly facing and available, so that folks can run things in the exact same way that we're running things. And that's one way to get to that scalable population level models that I talked about. And another way that we're kind of thinking about doing this is federated learning. And so we have with the same project that we're doing, we're setting up federated learning runs, everything on the electronic health is cross silo, lots of data, one center. And we're basically playing around with the privacy accuracy trade offs that are required in a federated learning space, so that we can enable folks to participate in some of these runs that historically have not been able to because they don't have the compute or they aren't able to because of data use agreements. I'm not going to dig into all the things that are there. There's a lot that actually goes into that, but it's an exciting direction that actually will, in my opinion, allows to scale up to the population level. And I will stop there. Yeah, thank you.  So I do want to invite the three presenters to turn on the cameras for some panel discussion here. And they're my no doctor tells in the room in the actual room there. So it might not be doable. But I feel like the presentations that we just saw we actually did a fantastic job of meeting that first session is that objective of actually presenting people with use cases and studies where they can see this right so I think we've covered that really well. I think people follow up with the speakers if you have questions on those. I did want to ask kind of the second objective is some thoughts about how AI and ML can be used and in a pipeline and maybe even thinking about what a pipeline is right and I think maybe I think the next slide or the title can comment first. Because I think clearly, you know, the Miss a project is, I would call that a pipeline right it's, it's things are being watched. You know, there's automated learning, you know how do we turn some of these other things into what we might consider more of a pipeline. So that's a very good question in the application you can see from the presentation, he had just give, you know, there are a lot of different modalities and sensors and you you have to integrate them. So I think the integration is the most important part and I do believe the advanced machine learning approach can help the best that we can, comparing with other approaches to integrate in the pipeline. By doing that integration, a lot of the issues need to be resolved. But one of the like I said, you know, how do we integrate data when they have missing this and also heterogeneity, right. Other part I did not mention the heterogeneity all different data have different, you know, different models. So how do you integrate them. I think this really need the machine learning approaches to do the job. So if I may add to that David I think there's a couple of things that need to be put into place or are able to do right now, in terms of building pipelines the first our access to clinical data that are linked to potentially some of these other banks that we're already collecting so you can imagine a scenario in which you run a process some multimodal AI to develop your algorithm or to do some sort of discovery, and which is linked to electronic health records where you can then prospectively evaluate some of these methods on real patient data or even call back individuals for to develop some sort of AI based intervention, all in sort of electronic health records scenario. The second thing is I think we need and I think our speakers touch on this are libraries of models if you will so we have libraries of data I think one thing is to make them also accessible. As we've heard in a in our fantastic last talk here by Dr Hansen, make them available so that others can quickly evaluate them and for that that would require data standards to be to be agreed on so that we can implement them on, you know, for example, all of us to UK Biobank to electronic health record setting etc in real time evaluate all these model performances across different core data. Yeah, I think, you know, one before we, you know, there's a good chat question that I want to get to.  Dr tell would be the idea of, you know, for building a pipeline. Some of these studies are also life course, right. So we saw Dr giant presentation as well right now had, you know, we're predicting things for different stages and modalities of people's So in a pipeline sense, what does that mean right because we you'll have a the model for today. But then that same corpus of data, right that's the same cohort of people I'm sorry. We need to ask questions about them tomorrow you know so what is in terms of a pipeline what does that mean are we constantly updating outputs or are we offering interpretations or how does that work for actually communicating these things. That's actually very good question. That's, you know, if people have heard shift, right the domain shift, you know, over the time, the domain changes all the time how are you going to build your model will will be able to generalize to all to the domain shifts. And I think that is a major major topic in machine learning community to deal with that. And I think it's still an open problem. How do you deal with the domain shifts. Yeah, it's I would agree with that that's a great answer and it's still very much an open problem. Some of the things that we've started to do is build in temple check so basically we're, we're assessing temporal drift as we, we go forward in time, and that's part of our data profiling so not only do we have data profiling, but we have model profiling where we're assessing different types of biases or temple shift in the data. And then that might inform what how we make decisions about how frequently we update it, but it's really trying to think about things is rather than a single endpoint, but the full pipeline and an iterative pipeline where you're constantly reassessing and updating things. I think that'll be especially relevant for. I'm sorry, Dr. Oh, sorry, sorry, David, I just wanted to add to that. You know, many of these modalities may be redundant. And so one sort of experiment one needs to do is understand the correlations between them I think Dr. Valshaw mentioned that the hyper correlation between exposures so it could be that some of these modalities are simply repeated information and that you don't need to measure that and you can implement as far sort of a set of measures to get the capture the same amount of variance explained.  Right so you know clearly all of our you know, the environment is not static right we know that much talking about pollution but you know all of it gets affected so you know you know on that. Like as a segue here the check question you know do we have suggestions for dealing with the need for privacy and spatially heterogeneous environmental contamination data that can be associated with localized exposure patterns right we saw. I think that Dr. Hansen presentation somebody had about spatial joining and how to scale up and down there but you know do folks have thoughts on that. That's a very interesting question and it's a very good one and one that folks don't always catch on to is the fact that when you start to combine all these modalities you really run into privacy issues so I may identify my health data. But then I join it to some sort of information about environmental exposure at a small spatial extent, maybe one KM one KM. And those patterns become very unique to that space in some cases right and so I went through all this work to de identify I joined into an environmental exposure and hey, I have an identifiable information in a way I didn't really think about. So I would say there are, I think that there are ways we can start to think about making sure that these are privacy protected. So I think this question is is a very good first step so a lot of the things in the privacy, privacy space haven't been really well quantified. So, at what level do I need to think about this at what happens when I get to certain patterns I think we need more research into that sort of question asking or critical thought. I also think that we need to start to think about other methods and that's kind of why I'm interested in the federated learning spaces. At some point when we're talking multimodal data, it's going to, it's always easy to triangulate a person so we need to think about other ways to deal with this so can it be privacy preserving federated learning. There's always going to have to be IRB, DOA sorts of things but can we start to get innovative in the ways that we're processing this data would be my answer or question. Maybe I can add the other aspect of the question on the contaminated data. So, you know, machine learning goes very advanced in this serial learning if you have heard of serial learning. I think that's a similar approach, you know, contaminated kind of fake data or similar to those adversarial and also generative AI has gone, you know, advanced to data augmentation to try to reduce such problems. So I think that we do have a future on that. And I think as we go back to the, you know, from the first session to some of the questions about, you know, the dangers of identifying folks and having, you know, vices built into our, you know, our models I think that the map you should, you know, Dr. was 48% of the country, but they were certainly geographically contiguous, right.  And I think a lot of cohorts end up looking like that right and so you're going to, if you're not building out at the beginning it's going to get big and if we don't think about the deidentifying things, you're going to end up with some of those non desirable outcomes about where the learners bias themselves. So I'm sorry, Dr. to tell you something, I'm sorry, Mike. Oh, sorry. Thank you very much. Yes. David, yes, I just wanted to comment on Heidi's. So everyone did a great job. Thank you very much. I learned a lot. And Heidi, yes, that was a great trick or, or the methodology with regard to using land scan, right, to derive without crosswalks, you're now at a what one, basically a one almost two kilometer level in terms of, yeah, your aggregation there. So without a gridded data set, generally you would link, you know, modus we do that, you know, we use modus 2.3.5 and then link it to our public health expose on to be down near about that and thank you for bringing those issues up. The IRB really doesn't like to see that. But, but I want to compliment you on that. And the reason why this is very, very important, because the imputation in Columbus.  Generally, these are purple air monitors. Generally, these monitors aren't placed in communities, vulnerable communities. Columbus didn't so we got a grant to put them in those census tracks where they weren't, and found out that we were almost an order of magnitude off in the imputation. That's what I'm saying with respect to real versus imputation so I want to caution less to be careful in that regard but that was without AI or some sort of supervised clustering methodology. Yeah, but other than that. I was just asked Dr. Are you are you saying that those were placed with the benefit of some learning models where you guided your placement of those or they're replaced based on just where you had missing data. That's where we have missing data which is always in vulnerable, low census to low socioeconomic status census tracks. Right. Thanks. Yeah, well maybe you know I'll ask as a follow up to maybe Dr. Particularly I think Dr. Jung might be able to comment on this of, as you're doing these multimodal data and you're finding holes in the data, the missingness of the data. The methods didn't really care about whether you had prior knowledge of of the, you know, anything about that particular data set like did I misinterpret that or is it beneficial if you're having ex bosomics data on one particular kind of contaminate that you sort of know a spatial pattern. The graph embedding method is that actually adding information or is it, can it just be, what's the right word agnostic to, to any sort of prior idea about the nature of that missingness. Yeah, well that's a very good question because in the past, people have used the prior knowledge to impute the data right and, and sometimes they just simply use the data to impute. In this approach I presented we try not to use those but we utilize the interactions between the modalities. That's the key part because the, why do you want to use a different modality, because they have interactions. And they have a common features that you want to extract. So I think machine learning can extract those information automatically from the data set. Yeah. David to this I think for certain critical questions I think we need to do be better at measuring. So I think several topics that emerged from the last session were about non additive relationships between cumulative exposures, or versus multiplicative effects and I think that while I'm very optimistic about imputation I think eventually we need to measure everything such that they're not missing so you can really ask those questions about non additive versus, you know additive relationships between exposures and health health status. I wonder if you have thoughts about your. I think you presented it as I think the word used was like a old method or traditional method you're talking about a canonical correlation. You know, this is an AI ML workshop and we're all pushing, you know, the methods throttle as fast as we can. You know, but I wonder about like, you know, you have some thoughts about the benefits of like for interpretation, maybe some methods that were around for 100 years and just slow. I want to speak for bank murphy and David, and, and Ravi who are the authors of that paper but I would say that that that was enabled because they had complete cases on both the metabolomics and the dietary behavior. And you're right they're able to get something that was easy to interpret and look at out help outcomes with that so I think that matters and it adds to the impact of the things that we can we can do. Well, I would add that when you do integration or you look at a correlation, especially using machine learning models that you always face the interpretation problem. That's, that's also one of the problems that machine learning communities trying to solve. I think we, we are optimistic to have a good solutions to do that. And maybe I'll just ask everyone if you know if you have a, you know, closing thought as we're about time, you know, for the panelists, especially on that key piece of like, you know, going into the next sessions. Are there things we can do now to help interpretation, you know, are there like quick things that are reachable right now, you know that we can start doing. Yeah. And so I think, you know, look at the, you know, like it was mentioned, comparing simple method to more advanced machine learning method now there's a trend going whether more advanced will help and with interpretability. So I think this is a very important topic to have the balance between two. Yeah, I would agree. I like that answer. It takes a little bit more time and we all are methods geeks a little bit so we're all like, yeah, but I think it's very important way to go forward. I also think thinking about building uncertainty quantification in what we're building is extremely important. Again, kind of a hard space. There's a there's a lot there a lot of open questions but rather than just accepting what we're modeling can we start to build in uncertainty quantification and then incorporate that into some of our pipelines. I think those are I also like those answers I think enhancing our existing cohort data or existing surveillance data like the National Health and Nutrition Examination Survey, so they do not have missing information so that we can build baselines for example, when we do implement these data or measurements multimodal measurements and things like that, like all of us in biobank samples. So I'd say baseline or baseline metrics to assess how we're doing. Thank you all and I'll let the, you know, the thank you again to the speakers for everybody I'll let you all have the last word you've earned it. And we'll now I think move into a lunch break, and we're set to reconvene at two, and I think we'll want to start exactly at two because right after that, you know we have our key notes and a visit from Rick way chick so you know want to make sure we start session at the same time so unless the organizers have any more comments I think we're going to take a pause now and reconvene it to. Yeah, thank you all again to you know all three. Thank you. Hello everyone. We'll come back. We're going to start our session three. So the goal for the session three is to provide an overview for the emerging AI and ML methods approaches for data integration and applications. I have three wonderful speakers today. And my name is y'all each time I am an associate professor University of Minnesota I'm in the computer science and engineering department. But my background is a mix of computer science and spatial sciences I'm also a GIS professional. My area of expertise is on spatial AI so we work on applied machine learning methods, data mining methods systems to look at different problems that involve spatial data. Without further ado, today our first speaker is Dr joey's whole. So Dr joey's whole is an associate professor in the computer science department at Emory University. Her research focus on the development of novel data mining machine learning algorithms with problems in health care. She also had co founded successful healthcare analytic company. So today we're going to hear from her about her work. The floor is yours. Thank you so much. So today I'm going to talk about the integration of novel data streams to capture neighborhood level measures. And to start this off I really want to motivate it from the perspective of health equity and sort of take a case study, which is to think about heart failure. Now, it's well known that social determinants of health and neighborhood really have this effect, this unknown effect, right and it's unclear exactly why it is. So if you look at existing literature what you'll see for heart failure and for peripheral artery disease is that there's conflicting evidence of whether or not neighborhood deprivation index actually plays a role in this. And given this you might say well maybe there is no neighborhood effect. One of the main premises is basically deprivation indices really fail to fully capture the neighborhood or community factors. And so the real question that we want to think about is, how can you get at these factors without asking the patient without burdening healthcare providers. So what we thought is that you in fact can tap into novel data streams that are already publicly accessible, you know people create. And so I'm going to cover four different ones that we're quite interested in thinking about. So the first one you can think of as market segmentation data. This, this would be things like Instagram, Facebook, Twitter. Another one is thinking about geo data services so this would be Google places for square yell and open street map. Another one might be mobility data that's covered you know so for Atlanta we have Marta. We've got Department of Transportation and Apple Maps and Google Maps as proxies for mobility. And then there's also community based forms so this would be things like patient, like me patient info next door and even thinking about Reddit. Now these are all already publicly available to some extent they all have API is that you can sort of get at the data and users already posted so it's not something that we necessarily need to collect. So the question is, does it do better. And so we went about to answer this by thinking about prototyping new neighborhood measures. So in particular we focused on three different sources of data. This is going to be Twitter. So you can see that in the middle an example of a tweet. We also thought about four square which has, you know, thinking about groceries in New York. And we've also been thinking about open street maps which is, you know, curated by the public essentially. Now for four square and open street maps what we're looking at is the points of interest within the census track, and we're focusing on census track because you have to validate against something. And the natural ones that are typically used when thinking about health equity is census track level. Now for our Twitter is we've also been thinking about well how many of them are geotag with specific keywords. And we're going to focus only on six categories of interest. And these are the ones that we sort of curated initially because there's been some literature to support that there is some effects thinking about these. So these would be the number of parks that you live nearby. Right so recreation thinking about pharmacies so how readily available is medication grocery so do you live in a food desert and as a proxy that would also be restaurants. We also looked at sports right so sports team sporting events things like that. And then we are also thinking about health as well so looking at healt