Okay, thanks. So our two moderators for our first panel. Excuse me, Rex Chisholm and Renee Ryder. You've already heard, let's see, start getting messages from the system. Rex is a professor of medical genetics in the Feinberg School of Medicine and was the founding director of the Center for Genetic Medicine at Northwestern. He's a member of the Genomic Medicine Working Group and very deeply engaged in the Emerge Network research and he's in the perfect position to facilitate this discussion because his research has really focused on how basically to have genomics informed personalized medicine in the care in care delivery settings. And then Renee Ryder is a genetic counselor and the program and the newly hired program director for the National Human Genome Research Institute. She's for the implementing genomics and practice pragmatic trials network or ignite. And as you've already heard Renee has been one of the key people in helping make this meeting possible so a personal thank you to Renee as well. Rex and Renee. Hi, thank you so much. I wanted to welcome you today to session one which is laying the groundwork. This session will have two speakers which will be followed by a discussion panel. Throughout the session, please feel free to use the chat feature to engage with with each other. However, if you have a question that you'd like the speakers to respond to please put that in the Q&A. After each of the talks today we will pause briefly for clarifying questions but we asked that you hold the big questions and discussion for the panel section of this session. And now it's my pleasure to introduce our keynote speaker Peter Hewlett from North Shore University Health Center, who will speak on the state of genomic learning healthcare systems. Terrific. Thank you. It's a pleasure to be here and help set the stage for this 14th gathering of genomics medicine. Let's get the slides up. I'm very excited to help set the stage here. As mentioned by Renee, I am outside Chicago at North Shore University has North Shore University Health System, a learning healthcare system and implementation is dear to my heart. I look forward to hearing all the great things that come out of this conference to help us move the world of genomics in the healthcare systems forward. Again, laying out the objectives, exploring the real world examples of how genomic learning healthcare systems can apply cycles of genomic medicine and implementation to bring just evaluation and adjust and implement practices across delivery systems. And then examining the barriers and identifying potential solutions really want to look at the opportunities here and not look at barriers as something that we should run away from and really develop solutions and that comes from collaborations and listening to others and how they tackle problems. There really isn't one solution. And as we work towards as a goal together we can really bring this field forward. So the challenge is why do we really need a genomics learning health system personalized profiles and degrees was really must capture all facets of health genomic information included. The reality is there's an exponential growth of health data and we are under a deluge of data and how do we really bring it together to provide these inputs and discover insights that we may have blinders to and really validate them and then activate around them so that we can provide the best care possible for our patients and learn as we implement. So where are we, how are we learning well this is certainly a learning health system concept really has started taking off if you use publications as a proxy. Really this is somewhat of a lagging type of marker but it does show there is gaining interest and this is critical because we need this inertia to really move the field forward and start to take advantage of all this data we're collecting. We are in a data generation mode of medicine and really we have to start separating the noise from what really matters in terms of the care of our patients and providing the best next step and simplify it so that physicians and other caregivers can know what is that next best thing that I should do for my patient. What defines a learning health system well there are different ways to describe it but it really a health system which internal data and I would argue also taking external data and experiences are systematically integrated within external evidence and that knowledge is put into practice. As a result patients get a higher quality, safer and more efficient care and help delivery organizations become better places to work as well. We are streamlining all facets in terms of improvement of healthcare delivery.  It needs to be an iterative process to continually improve, and it requires partnerships not just clinical partnerships but across the domains and moving away from a culture that's just business as usual. It takes all stakeholders engaged in coming with that common goal is how do we learn from our experiences and apply them directly back into patient care. So elements of a learning system is having that leaders who are committed to that culture. It can't though just be top down and has to be something that has some grassroots to it is as well. And really we have to be able to systematically gather and apply evidence and ideally real time to guide care. This requires new and novel it methods to incorporate that evidence. We need to include patients as part of the system is there are vital members because ultimately this is why we're doing it to help improve patient care so they need to have buy in because we also need and recognize that there's data sharing elements and patients need to have a voice in that. So capturing and analyzing this data in different episodes of care is necessary and then that continuous feedback you know how do you find the process how do you train the algorithm how do you make sure that this is really working for every patient within a system regardless of what their background is. So learning health system doesn't fit into one size off. There are different components to it in the traditional model of clinical care you try to make a diagnosis and then the care team the physician makes a treatment decision. But now we are starting to move into more of a comprehensive data model and then trying to apply real time techniques to start to gather this and a large process of this has been the adoption of the electronic health record creating data elements that are portable different different standardizations like a lot models etc and really start to hone in on how do we aggregate this data and start to build that evidence. A lot of times we look at it retrospectively and this needs to be filter back into the treatment decision over time. We all know dissemination of guidelines and uptake of guidelines can take years if not decades to really reach that. So how do we really start to move into this real time aspect of a learning health system and really start to recalibrate our treatment algorithms as well as our treatment care plans based on real world evidence that is well vetted and that could be deployed at the point of care and then ultimately really have that standardization across not just an individual organization but across multiple organizations as we start to share this experience. The comprehensive data model or learning health system is also viewed as a first step, mainly because it is relying and is driven by the data that's in the electronic health record. When you think about it so North Shore is on epic. There are over 162 million patients with 5.7 billion encounters and patients all across the United States. When you think about the US health system in terms of what is that potential what is that data how can we best utilize this and this is just one electronic health record vendor. These tend to resemble longer to cohort studies so again we have the opportunity to see how things develop over time and start to learn what inputs could inform our learning health system and how to we build on that. This was critical certainly when we were going through the COVID pandemic and are we still trying to figure out what is our way out of that and trying to understand well what is the impact and looking at this data to see how can we better position ourselves to improve care. When there frankly were a lot of unknowns on how best to manage the pandemic. This is critically important getting as real time evidence as possible out there so that our clinical teams could make their appropriate decisions and then be able to take a look back and say, you know, let's take another look at are we making an impact which is critical because as we implement if we're not having the desired effect, we need to see that signal, or if there's another way to enhance it, we need to be able to iterate and continue on that process and the pandemic really highlighted that need. The challenge is how do you speed this up and do it in a safe and effective way and quote unquote real time, whether it's instantaneous or on a daily updated basis or monthly, but increase that speed of iteration. This really builds on early clinical certain decision support system, so that medication errors drug drug interactions, etc, which some can argue can be white noise but there are value in these alerts when they are at the right time in the end with the right information so that patient can have the right decision made for their care. Moving this data to the clinical care beyond just observable rate research is key we need to actually apply it not just observe and report although that's a critical first step, and use it to impute for these algorithms so that we can better be better effective with our care oncology to serve as an early opportunity for this in terms of patient journeys looking at well my patient doesn't exactly fit a clinical trial, but I can see what's happened to other patients who were treated with similar circumstances in terms of their comorbidities or other aspects of their health care and determine what happened, if I'm trying to prioritize what treatments might be more effective or better in line with the patient's goals of care. The key to this is traveling is translating these novel insights from clinical care of prior patients directly back in the patient management. And in some instances generating new hypothesis of what is the best care approach in the full learning health system is sort of the holy grail is also building in some randomization that it actually becomes in essence a prospective interventional trial that you really test in a rigorous way, did that outcome really improve with our proposed intervention. Again, the COVID pandemic really highlighted how this was important to really implement overnight. It forced some institutions to really say we need to adapt this now and was a real catalyst. And certainly, this led to some very novel insights that helped us try to get ahead of the curve in terms of creating a pandemic. In some ways this has an opportunity to help us as a learning health systems in general adopt this methodology, but then now we can start to think about it in the care involving genomics as well. So where are we with this in terms of what's been done what's been studied what works. Well a recent review attempted to look at this and see, well, how did learning health systems effectively work what was the empirical evidence that this really did improve care. And there are a number of different aspects of what these studies focused on but most were focused on a specific disease or clinical context which made sense we have to take stepping stones towards implementing as opposed to a broad just across pan health approach to learning health system. The graph here really shows that you know each circle here represents a note or keyword in terms of what was the focus, not surprising learning health system, given the scoping of trying to figure out where are we involved with learning health systems and electronic health records. So let's come up to the surface here. The HR is generally the primary where we're going to get this data in terms of how do we implement a learning health system type of care model. But as you can see there are there have reported outcomes of what they were trying to assess whether it's evidence based policies whether it's public opinion and trust, testing out machine learning techniques, all these are important. But I think what was notable is that there really were a limited number that incorporated the implementation of some sort of implementation science framework. So that seems to be an area of opportunity where, how do we make sure that we're appropriately measuring the effectiveness of these learning health system environments. The challenge there has been traditionally these mixed methods approaches have been very labor and intensive in terms and resource intensive in terms of deploying. But we are starting to see novel ways of incorporating them, which hopefully will make it easier for us to learn from other examples in the reported literature of what worked why didn't work and also the lessons learned from stakeholders and input, so we can actually avoid making mistakes that other people unfortunately had to tackle and overcome in the first go around. So the takeaways in the learning health system scoping in general is that the implementation is increasing and there is an opportunity here to use that implementation framework to make it more robust so that we can really translate these innovations at certain institutions into other organizations and really understand more the how and the why the success or the lack of success occurred that's equally important we need to, you know, fail fast if we're going to fail and implement on that. COVID-19 just underscores the need for these types of environments and the use cases will only grow over time. So then we talk about a genomic learning healthcare system. So is it just simply just adding genomics and everything will be fine. Well, as this group is well aware of that's unfortunately not the case there are other aspects that make some additional challenges of genomics by argue we need to continue to walk away from that genetic exceptionalism approach. The good news here is we do see more and more academic and community centers incorporating genomics information as discreet data that's sort of a no brainer foundation that we need to occur. We are starting to see this evolution of more data donor culture where both patients and health systems are getting more comfortable with using data in more novel ways. And now we start to look at well how can we start to make the scalable decision support tool the knowledge contents that it's effective all across the organizations really start to understand how do we make this interoperability problem. Go away or at least be reduced because unfortunately a large part of this is being reinvented at each organization. Then finally seeing those implementation wins carry out will help guide this process forward and really help other systems who may be on the fence of going into a genomics learning health system kind of model tip over to the edge. So just describing how complicated this can be this is example from our own institution where we had a number of use cases where we knew we would need to have genomic data integrated in the EHR to start to work on building that learning health system genomics guided goal. So we have use cases that we were taking raw next gen sequencing data off our own sequencers we had our pgx data that could could come from in house or externally. And then we had our population primary care at our DNA 10 K where this data needed to come across and be integrated into our platforms as well as our clinical decision support and the relationship of this, for example when a patient would have a result come back from a genomics which was our partner for the DNA 10 K project report came in had to go to the HR went to our variant repository, our different bioinformatics clinical decision support areas to really bring it back out so that we can use it in a street way, and it is informed in the HR in terms of care pathways and other downstream patient care activities. And this is really critical. If you look at pharma genomics as a proxy in our system of incorporation of genomics information is really taking off dramatically with some of our efforts and not surprisingly it's coming from different data sources as as well. So, on the positive side, you know, we have crossed now over 25,000 patients who have discrete data in our system. However, when you think of the scope of the number of patient lives nor short takes care of we still have a ways to go in terms of having genomics guiding patient care for everyone. But this is really in credit critical as although probably 99% of our pharmacogenomics testing occurs in the primary care setting where the encounters are being had across our system where patients had this information in the system, at least 50% if not more is occurring outside of primary care and all across our organization. So this interoperability within our health system is critical if we're really going to have genomics guided care and in the setting of a learning health system. And when you even look further where this clinical decision support is fired it's even more of a mosaic care of the different areas within our organization where there's this information is firing. So critical need as we look at data data is being generated all across our organization and we need to harness that and start to help feed that into the algorithms that move our care forward in a logical and patient perspective or patient friendly manner. So how do we gauge the incorporation of genomics into healthcare systems there's not just a single measurement, you know, standard tool that says okay you have 90% incorporation. So using personalized medicine and programs that have decided they wanted to adopt a personalized medicine program. In some instance, a proxy for genomics incorporation because those that are typically wanting to put genomic information discreetly into their health system, have some sort of personalized medicine model or precision medicine model across. But in the personal medicine coalition, they put out this framework, looking at where these adoption approaches are occurring anywhere from the range of healthy patient screening to rare on diagnosis diseases and then the continuum in between and looking at well where are they in this journey of a genomics adoption and personalized medicine. And we see of the organizations that have identified that there is an interest or appear to have an interest and they are implementing. Certainly this is not all encompassing but it gives us a spectrum. And I think the key takeaway here is there is interest and it seems to be evolving and for but there's no one who is probably or there's very few institutions who are quote unquote doing it across the spectrum at the same death of integration. And so there are going to be different learnings depending on where an organization is and where they choose to embark on their initial efforts of getting genomics into the HR and building that clinical decision support that learning health system around their disease interests. So is the glass full or half empty when we think about the progress of a genomics health learning system. Well, on the half full side. We do see implementation continues to gain traction, and we are starting to see new tools tailored to this development to help us understand better the why and how things worked or did not work. And importantly enough focus on incorporating genomics into each year, personalized medicine continues to gain traction and interest both from clinicians and administration but also patients importantly as well. COVID-19 pandemic did highlight some sex successes but it also highlights some gaps in terms of where we are in this process. We still have to understand you know how can we better adopt implementation science framework so we can better relay why things worked or didn't work and help others who are just starting on this journey learn from prior experience and also inform us what are next steps for those organizations and for further along this process. The reporting of outcomes and implementation methods still remains a barrier to identify this important insight genomics information does add a layer of complexity to privacy and data concerns if we talk about the scale of scalability and interoperability across organizations. And then finally, there's a variety of ways that genomics is being incorporated and approaches organizations are taking in terms of personalized or precision medicine, which ultimately is it's somewhat of an early proxy for genomics guide learning healthcare systems. But I take the approach that you know technically the glasses always full it's just what are your ingredients what are your components of our program and to me, a genomics learning health system is just that we have to take those initial steps, each of us are using different ingredients but the themes are the same and we're moving the field forward, and I really look forward to this conference to better understand what others are doing, and also to share in the dialogue of how we can make this work and so hopefully we continue this progress that we've made in the learning health system. So I appreciate again the organizers for inviting me to help set the stage for this 14th conference and I look forward to the next two days of dialogue and presentations. Thank you very much. Thanks very much, Peter that was a great overview. We have time for a couple of questions there was one that was put in the Q amp a earlier that asked the question, how do you address the quality of the MR data longitudinal courts often have strict data collection standards. The standard data collection is entered by multiple individuals in a health care system, and more variability and not the same level of standardization. Do you want to have a few comments on that. You're still on mute, I think. Sorry about that. Yes. So, quality of the MR data is, I think part of this depends on well what problem are you trying to solve. So some of the initial things in the learning health system. What we did as we implemented some of our primary care programs initiatives, we started to look at what were the some of the operational barriers and delivery of care bear so there's a different level of quality in terms of, we could see different trends that why one site may be performing better than others. Was it simply, you know, a question of patients didn't know what the next step was I mean sometimes going to be as simple effort as that. When we started getting people to the next step we noticed there was a lack of follow through in some of our breasts, high risk breast cancer referrals, and we were trying to figure that out but when we went and looked at our data, we saw something as simple as that we had an over aggressive schedule that if they only called once and they didn't get a response from the patient they removed them from the queue which wasn't part of our process of how many times in the ways we reach out to patients. So when you start to get more downstream in terms of the rigor that's needed in terms of implementing care to someone have that breast MRI or what was found on it was earlier detection of a cancer made possible screening for unrecognized disease then it starts to you start to have to have more parameters on the data that you're extracting and not just necessarily relying on discrete data but also taking unstructured data out of progress notes and that's where I think the field is really starting to advance as well. And I think certainly things like machine learning are going to help a lot with improving, improving that we've got another question from Alexander reikevic who says, our physicians buying into the clinical utility of your efforts. Again, you're on mute. Yeah, sorry. Again, this is something that is an evolution. We started probably two top heavy in our experience. And then we started to kind of, as we learned and some of the work that Amy Lemke has done for organization we look at some of the barriers to implementation, using some mixed method results. And that we really need to engage a wider net of stakeholders for this process improvement. And we create something called the genomic ambassadors where we have initially it was about 10 primary care positions who were early adopters. So we had to identify the early adopters, and we had them commit to an hour per month combination of education on genomics in general what the goals where we're trying to go and, and then we started to teach them how to talk to their peers in terms of conveying this information to get by it and we've we've seen this culture changed is everyone 100% on board know but we've certainly come a long way over the past really five to six years we've been working on this. Okay, thanks very much. I think we can go ahead and move on to the next presentation so Renee I'll let you introduce the next speaker. You're on mute Renee. Great. Next up we have Terry Manolio from NHG who will give an overview of the genomic learning health care system barriers. Great, thank you Renee and Brex. So, I did want to mention I forgot to say it when I was describing the overview of our meetings very often before these meetings will do a survey of the people who are presenting or attending in that. Typically we do these through Duke which ran this one and we appreciate that just to find out a little bit more about the state of you know what people are working on or challenges they're encountering in this space. And so we did that in this case we wanted to gather information from existing genomic learning health care systems that we were aware of and I have to say that's a small universe and I'm sure we missed others that would have fit perfectly in here and so our apologies for that we didn't get complete response you know a terrific a perfect response for you but we did pretty well. And we wanted to gather information on how their system is organized how genomic information is integrated into it. And again what barriers they're running into and what solutions very importantly they found especially to genomic data integration. And then we use those responses to help plan for these meetings for this meeting to share experiences and identify opportunities, the 10 groups that responded are listed below there. We kind of, you know, masked the, the institution information because it isn't that it's very important but it's it's not critical to the goals of our discussion here so so that's masked but you'll see there, the various results so are masked about enterprise wide EHR as we had the impression that if a system was doing this that they must have an enterprise wide EHR it turns out that not necessarily. Some were in development some were in selected systems somewhere incomplete. And in, in kind of summary of that there were only four they had system wide EHR is what they described as as that for in selected places and to head incomplete development of those. You can see the EHR is used. I think reflective of the US ecosphere at the, at the present time, seven groups used epic one user and to use multiple systems. And one takeaway from this is that if we want to do useful and effective implementation of genomic learning healthcare systems using EHR we need to engage with these EHR producers. So, and then a variety of, you know, whether they included structured genomic data seven said yes three seven. And asked about what evaluation metrics and what framework frameworks they were using you can see that many of them but not all assessed health outcomes, almost all assessed process outcomes. Several measures of satisfaction including two groups that looked at it for health systems and one that looked at it for researchers. Many looked at health system costs. One that didn't have process metrics yet and then the frameworks that were used were some of the, those that we've heard of and we'll hear more of today see for and reaim others not necessarily using a framework or using other frameworks. We also asked about gaps in expertise in genomic medicine. I think it's fair to say that probably all of us recognize in all of our systems we have gaps everywhere so we said you know give us your top two so you can see here genomic educators and informaticians were were really These numbers are very very small but so those those kind of one by one person and genetic counselors and medical geneticists were not far behind again not surprising. And then people with expertise in genomic medicine interestingly pharmacogenetics experts I think is that's an area that that we may have addressed, or at least it's been addressed better than some of the other shortages, and we have, you know, can offer some insights to schools of pharmacy and and other groups that have really taken this on the clinical pharmacogenomics implementation consortium or CPIC, the pharmacogenomics research network and others that have really taken on trying to get pharmacogenomics implement clinical care. It's not to say that that expertise isn't needed in many places but it's not one of the sort of top five, at least as judged by these 10 groups. But well okay so you have these gaps what are your approaches for filling them and the emphasis is mine here the bold blue are just things that sort of jumped out to me. And one had had to do with a genomic medicine track in an internal medicine residency and so you'll hear about one such track. Tomorrow morning. I'm going to talk about the ambassador form which seemed like a really interesting idea. I don't know if we'll be hearing more about that today but that would be our today or tomorrow of 12 primary care physicians sounded sort of like a focus group almost and established that one group established its own genetic counselor training program as one solution. Other groups are growing their own in terms of adult medical geneticists and genomic medicine experts but they're, you know they're having trouble getting getting appropriate candidates so something else we can talk about. And another group established a dissemination implementation science in omics unit to study model supporting translation to clinic. So, some interesting approaches to filling gaps and expertise. So we're going to talk about barriers and obstacles I only show this slide because again we asked people to give us their sort of top two obstacles so that we can then go and say okay what solutions have you implemented to address those education of patients and clinicians and systems, as well as bioinformatics infrastructure were the top two again by only one sort of one vote, but that you know goes along with what we saw previously in terms of the expertise gaps. And also that close behind was acceptance by patients clinicians and systems and the shortage of genetic counselors. So okay so how are you addressing those barriers and again, a large number of potential solutions. One that that I believe we'll be hearing a little bit later about where automated care navigation pathways so can you can you automate at least some steps in the process to simplify it and reduce the sort of human person burden. Education and counseling. It wasn't a solution as much as sort of a, you know, cry for help is difficult for any system to create original educational content. Although there are groups now that are doing that the is cc is is one source of it and there are other places that we can talk about another group was working also working on educational programs, we need to bring those groups together and share this information. We also need a sustainable model for that because if you can't charge for it. It's not clear, you know how else it's going to get paid for and if you charge for it then you're less willing to share. In terms of acceptance and bioinformatics developing a shared evidence knowledge base and literature archive is is a solution that one group either has implemented or is suggesting that we try to implement. Diverse multi institutional cohorts were suggested is a great way of generating evidence we would agree and as well. Recognize genomic data as a longitudinal health resource that can follow the patient from care source source to care source and should do that and currently does not. And then potentially novel education models a few of them are listed there and they hear about them. And that's why it is so I will stop sharing and turn it back to you Rex and Renee. Thanks so much Terry. Next we're going to move on to the discussion panel section of the section. We're going to ask each of our panelists to just make a few short opening remarks and then we're going to open it up for discussion. So our first panelist that we're going to have is Gail Hanan from the Desert Research Institute. I'd like to start with the obvious and that not. Not all projects are alike. And it's very important that when you design and plan your project that you do it in the context of the community and the sponsoring organizations that you have. And it requires constant reevaluation. Our model here at the Health and Nevada Project in northern Nevada was patient empowerment at the start of the project. Basically we provided the finding positive findings to the participants. We are along with consultation and recommended acts detailed recommended actions. And we trusted the participants to access their primary their physicians and act upon the findings what we found, which is not a big surprise probably that only about 70% of the participants actually acted upon the upon their findings. And there is an attrition in execution in every step along the way. What we found that contributes to a lot of attrition in execution is that healthcare personnel are very uncomfortable dealing with genomic findings. And it's the whole spectrum of genomics findings we provided we did the testing we provided the results, but they are quite uncomfortable with ordering genetic testing, interpreting the results conveying the results to patients and acting upon them and it really affect the outcome of the project. The way we designed our project. When we delivered result there was a notification process and consultation by genetic counselors and genetic counselors are not that easy to come by. Initially at the beginning of our project we used our own genetic counselors, and our notification and consultation success rate was about 90% of all the positive participants with positive findings. Eventually we switched to a third party vendor. And despite using the exact same supposedly the same exact protocol the notification and consultation rate became only slightly above 60%. And then we dropped in execution, which we understand by the way that it is kind of the cost industry standard in terms of success rate. We are currently at the result of this considering other ways of supplementing or improving that notification success rate, because it really does affect the bottom line of execution. Additionally, we found that eventually when you look at the electronic health record you find very low rate of documentation of the findings, especially in the problem list. We found that only about 10% of the medical records of our positive finding participants actually reflected very specific genetic diagnosis for those individuals. We also found that there is a higher rate of diagnosis that appear in other make possibly in other sections of the electronic health record but that was not higher than about 25%. Many of the diagnosis were very non specific diagnosis, which basically affect the utilization of the finding because if I have we report on age book Lynch and familiar hypercholesterolemia, but if you assign a diagnosis that says, a tendency for cancer, it is not very helpful and for providers down the road to actually act upon it. In terms of actually finding improvement in care. And actually when we looked into the medical record only anecdotal evidence that actually even the patient that had specific documentation in the medical record actually had any evidence of effective follow up in terms of the risk. And as a result of all of this we have a lot of missed opportunities. One of the missed opportunities is that many of our participants, because of the voluntary nature of our project actually were detected at a relatively later age, and as a result already had a presentation of the outcomes of their condition. Obviously, if you're designing a project like this and you want to maximize the yield of the yield of your effort. Selecting the patients that are going to be recruited may be very relevant to maximize that to maximize that. And we also found that as a result of the fact that physicians mostly feel uncomfortable with executing and acting upon genomic results that this is required as been mentioned already here multiple times. It's a decision support system but it's not enough just to present the finding within the electronic health record. We took care initially of educating and all the provider the providers within the organization and primary care about the project and about the action that they should take. But we found out that this requires constant renewal and hand holding and follow up about the actions that were taken. And this is very briefly about the findings from the Health Innovator project. Thank you. Thank you very much guy. Next we're going to introduce Bruce Korf from the University of Alabama at Birmingham. Thank you. So I'm going to very quickly tell you about a implementation project that we've been doing here in Birmingham, called the Alabama Genomic Health Initiative, which is a state supported project that is a collaboration of UAB and Hudson Alpha in Huntsville it's in its fifth year now. And initially we were doing a population based enrollment and returning results of secondary findings from the ACMG list with genetic counseling provided but we use the, the COVID hiatus and enrollment as an opportunity to pivot our approach because we didn't feel we were reaching the, the physician community and healthcare provider community as much as the population so we initiated a collaboration with family medicine at UAB which had expressed an interest in genomics. So, deployed a group of, of study staff to work with the staff and family medicine the covenant with them was that we would not complicate or, or add time to the workflow of their kind of activities. We also convened a community advisory board to get input from the patient community which continues to meet. We generated a workflow that would minimize disruption to the family medicine staff and implemented this. I guess it was June of 2021 so a little bit more than a year ago. We've turned secondary findings now pathogenic and likely path results to participants and to their provider who orders this test and the individuals who wind up with actionable finding which is about 1% so far are then referred to an appropriate clinic at UAB for continued care. In addition, we now return pharmacogenetic data, which is generated at Hudson Alpha. It's reviewed by a team of pharmacogenetic pharmacists led by Nita Limdie, who also have access to the medication list for each participant. And ultimately then they issue a report to the participant that is a sort of general report really designed so it does not encourage them to make individual changes of medications, and then a more specific report to their physician, who then is guided in terms of what may be relevant changes based on the pharmacogenetic profile and a landing place for the pharmacogenetic data and the genomic data has been developed in the electronic health record which is a CERNR based system but it's not structured honestly it's more of a PDF based system for right now. We have an implementation science project underway right now to look at outcomes. We also, about 90% of participants agree to share their data for research purposes it goes into the I2B2 database and has been used by a number of investigators at UAB and we're using it to take a look at results we don't return for various reasons to get a sense of what the correlation is between phenotype and at least phenotype as recorded in the EHR is in our first year 557 enrolled of which 69% were people historically underrepresented in biomedical research. About a third had a result that could affect their current medications and 1% as I mentioned before had an actionable result. The challenges we've run into some of the clinics we have one in Birmingham one in Hoover which is near Birmingham one in Selma are especially the Hoover one still mostly telemedicine, which is a bit of a challenge. The enrollments are done by our staff getting a list of patients coming into clinic the next day. They're contacted by phone they can be consented that way although many of them are consented when they actually show up in the, in the clinical office. We really aren't yet to the point of full EHR integration. We've had very good physician engagement but not 100%. And we've had instances where a change in medication was suggested, and it would turn out to be unclear exactly how it had originally prescribed it and trying to track that down and figure out how to make the changes has also been a challenge. So, I will stop there and thanks very much for the opportunity. Thanks Bruce. Next we have Casey Overby Taylor from the Johns Hopkins School of Medicine. Hi, thank you. So, first of all, a learning healthcare system can be considered broadly as an environment where biomedical innovations including genomics can enter the clinic before they're optimized and then refinements can be made through those continuous feedback loops. And so for my, my current focus, I've been fortunate to be able to interact with this with several genetic professionals at my institution. And have they've helped me to realize that that feedback loop has been in place for a long time for patients that are getting whole genome and whole exome sequencing. For example, the tests may be performed for some, for a patient that's presently undiagnosed or those so-called diagnostic odysses. And in some cases they may remain undiagnosed at the time of initial testing but then a later date, we may have enough evidence to better interpret the patient results and then they can be assigned a diagnosis. And so that process really is that genomic learning healthcare system feedback loop that is very familiar to genetic counselors. And so as part of my Genomic Innovator Award that funds my research largely, I've been working with collaborators to really better understand this process of monitoring and following up with patients that have genomic results and also identifying opportunities for technology to help to facilitate that process. And the initial focus in designing and building technology has been around supporting genetic counselors and notifying them at an optimal time and with appropriate information to support following up on genetic test results over time. And so what I've learned so far and some points I want to bring up to consider in this panel first, and this is a point that's been brought up a couple of times already, but EHR documentation is really critical and it influences the extent to which technology can be helpful at several points in this following up process. So with the current data availability for our project, we're able to support one feedback loop which is really to review test results for patients at a set amount of time after a set amount of time passes. Though ideally we'd want to be able to implement more targeted rules for when new evidence is relevant for a set of patients. So that requires having a structured test results to be able to do that. And I know some of the some of the sites are able to do this but there's kind of different levels depending on documentation that that's available at structured or unstructured. The second point is we want to design for a team of care providers and, you know, as I mentioned we focus on tasks of genetic counselors but they're one part of the patient's clinical team, and there may be ancillary systems that are part of this genomic learning care system. The entire ecosystem and I see that the EHR is the common thread between the care team. So while we're working on a solution for genetic counselors initially there's some areas where we're sinking broader input with respect to where actual information gets documented in the EHR and who is notified and went outside of the genetic counselors themselves. The third point is considering release software is just a start. And this is related to the objective of the meeting on having ways to be for solutions to be developed and shared. So for our solution we haven't used this and like an agile method of software development, but we have made some practical choices for what we can release as part of a minimal viable product that offers a software is most essential features and and doing this allows us to get feedback before expanding the scope further. And the reason why I'm including this point is because doing this has some initial drawbacks and, for example, you may not be able to fully leverage data standards right away and but I'd argue that resources to leverage the standards become more more possible after demonstrating value of stakeholders. So in some we focus on genetic counseling and these points are, however, possibly potentially relevant for genomic learning health care system solutions more broadly and with that I'll stop. Thanks so much Casey. And last we've got last but not least, Adam Buchanan from Geisinger. Thanks Renee. I'm great to be here thanks for the opportunity. So I heard a lot of similarities described already with the work that we're doing at Geisinger I'll focus on our work within our large buyer bank that's reporting secondary finance to individuals using a list that's really similar to the ACMG secondary finance list. And I think some of the highlights that have already been specified today have been consistent with our experience as well the importance of using implementation as my colleague's a lot of Ron and Laney Jones have pointed out and written about has been really key for understanding not just whether the particular health outcomes that we hope will occur when we introduce some genetic information in the care, whether those occur but actually how those occur and what conditions are most likely to incur some to her. We've also noticed that the more that we think about reporting those genetic information, those data points in a way that fits under the flow of care, the more likely we are to be able to see some of those health outcomes happen. So we've seen some similar experiences to what Guy described in terms of suboptimal uptake of risk management after reporting an actionable results. We dug into that a little bit more and in many ways it's very similar to what's been known for lots of screening behaviors and medicine for a long time it's multifactorial, and often takes multiple different types of interventions to your particular patients to go all the way to realizing that intervention. So, what that submit to us is that the more that we tap into some of those existing systems within the healthcare system that think about leveraging some risk information and using it to close loops in care, the more likely we are to actually fit into the regular practice of medicine that grows on a guy's finger. So, one example of that is that we have a care coordination group that works with a lot of patients with complex or chronic illnesses and that group is really a support network for those patients that helps them stay out of measuring and see measurement, for example, and other processes to manage those complex illnesses. They support the frontline clinicians in doing that work so the frontline clinicians are not quite as overburdened by having to do that longitudinal management. And that's a mechanism that can be tapped into by our genomics team as well in thinking about the risk that we're identifying through both our Biobank and through other programs as something that needs to be managed long term. So, once they've made already, that genetic information is necessary to understand something about somebody's medical risk but it's certainly not sufficient for changing health behavior for completely understanding that risk overall in the context of the rest of that patient's particular health status. And so I think the more that we both cap into some of those existing non-genetics infrastructure elements, the better off we are. And the more that we think about using that genetic information as one of many pieces of information about that particular patient, as has been mentioned already, the more sophisticated we get at being able to figure out not just what that patient's risk is but what are some of the behavioral impacts on how that patient either does or does not act on that genetic risk. And so it's partly that infrastructure with that infrastructure also including implementation outcomes, but it's also continuing to think about using genetic information in the context of the rest of medical care in a way that can leverage some of the distance systems. So, I'll stop there and looks like I got a chance to have a discussion. Great. Thanks so much. So, the first question that I wanted to just bring up real fast is I know Angela had asked a question about third party vendor genetic services that is going to be. Actually, I was going to say it looked like that was going to be addressed tomorrow in Cynthia's talk but now I'm seeing that it's probably not so why don't we start there. Why do you think your rates went down when a third party, when you used a third party gender vendor for genetic services. I can pick it up. I don't think we saw any risk of doing it are finding that the notification rate declined and it might not be a real decline. It's simply that may have been that we were simply overzealous doing it initially, although that's what we would like everybody to do. From what we heard from patients, they were extremely satisfied with the level of service and consultation and action plan that were formulated for them. And we saw no problems with doing it with a third party vendor compared to let's say using your own consultants within the organization. However, the rate of successful notification was eventually deemed a little bit problematic for on our part. So just to China we've used, you know, mix of our own team as well as third party genetic services for some of our population results and I think one of the challenges is when you look at the third parties, it's it could be harder to make the next step for the patient to make it easy. Yes, there can be a recommendation even when we said, you know, match sort of the genetic result to war specific providers are it was still that you need to call XYZ clinic versus in our system. You have the potential that that scheduling ticket could already be created and it's and almost more of a warm transfer to get to the next step and I think some of that interoperability is kind of key as we think about how do we move patients to the appropriate next step of their care. I think some of this comes down, sometimes to just basic fairly minor operational issues. So that's one that that necessity of making a referral and that one handoff. But it's also possible that that contact with a third party company isn't recognized by a patient, whereas a contact from your organization would be and, you know, we've seen that being itself a reason enough for those contacts might be successful. So I can hop in here and build on that a little bit. So a little bit earlier in the chat, Sandy Ernst and that may may note of the fact. I think it was the example that Peter gave where there was a scheduling a scheduler issue that affected the quality of I wonder if the rest of you would be interested in commenting on sort of how do we dig into the sort of operational details of these small little things that actually have a huge impact on the success of the outcomes of genomic medicine and the learning healthcare system so we want to take that one on. So this is where that implementation lens can be really useful and collecting data that are implemented now comes with implementation strategies that might be used really rounds out some of the detailed understanding of projects and so sometimes that can be done by doing some interviews with individuals engaged in that process the schedulers or others who are along the front lines there. Sometimes it is just watching what happens. So kind of an observation and someone to play that role and understand the workflow in that setting. But it seems like it's often those sorts of things where the progress breaks down and so understanding that implementations is critical. And Casey this seems right up your area. Yeah, I was just going to really add to what Adam was saying because I completely agree like, if there were a way to monitor for unintended consequences using standardized metrics that's the implementation science frameworks where there are measures that are both like qualitative and quantitative. But there are also some like EHR use metrics that might be leveraged to be able to monitor regular regularly, as well as things like unintended consequences because I have an experience where we have a, for example, a telehealth equity workgroup where we meet regularly and discuss findings from a dashboard that shows what's happening with telehealth adoption and I feel like something similar could be relevant for this use case as well. Bruce, you probably have some input on this as well. Yeah, so I would emphasize the importance of really deliberate engagement efforts and that applies both to the patient community and to the provider community. And we know genomics, we think pretty well. But, you know, marching in and saying here's how it's going to be done would have been a prescription for failure what was necessary was to listen and to understand what the concerns were and to make sure that anything we did was sensitive both to the concerns of the patient community and also the provider community and that needs to be continually monitored and tweaked as as time passes. And Peter. Yeah, so I was expanding on sort of how we found out that operational aspect of things, you know, we did have, especially for our primary care initiatives where we wanted to make sure things weren't falling to the cracks we developed tableau dashboards of data and what we would expect the next step would be, but I think one of the challenges is someone has to be monitoring that data or looking at it currently and that obviously requires some bandwidth so I think, you know, that is a future iteration that how do we start to bubble up those problems and benchmarks so that we become aware of them and it's not someone actually physically having to look at and dive into the data, because it only get more complex and what pathways you're managing or watching. Mark Williams has his hand up. Thank you. Yeah, I wanted to just add a couple of things to this discussion. I would certainly endorse Bruce's point about deliberative engagement but I think there's some tools that can be employed. It can be very effective in addressing this specific issue one is, you know, workflow analysis that can be done upfront as a talk through to kind of understand from the end user whether he's patients or providers or both. You know what are the aspects that they're seeing in workflow we've used our clinician advisory group at Geisinger any number of times to say, we need to return this information to you. We should use this function of Epic and they say well we don't look there. We actually use this which is not exactly what Epic would say is the way you should do it. But by understanding how they're doing it, we can make sure that we don't necessarily throw something into an area where they're not going to pay attention. And I think a new tool that's really becoming a much more effective in implementation science is a product is something called process mapping, where we can really look very deep at a very detailed way about the process in a given for a given workflow and how that process actually varies perhaps depending on clinician location clinician specialty, etc. And if you use that to inform how you build your dashboards, you can then identify process failures much more quickly and remedy them so those are tools that I think we need to more formally incorporate as we do this in the future. Mark, we've also got a couple more questions. Angela Bradbury asks, how will healthcare systems address the costs of reassessing results and the feedback loop on a regular basis and maybe I'll just broaden that in general. I think Peter, you initially made the point that it's really important to have buying from the leadership but how are the sites that you're involved with handling this question of both the upfront costs and then the costs that occur over time. Yeah, so I think you know that's an important thing so we had to rely on initial institution, no support foundational philanthropy sport to get this program up and running. And we saw some things in terms of having patient outreach focus groups, etc. What was moving and we saw some interesting things with patients in our community that they said they were more likely to choose a physician who started to incorporate genomics or genetic testing now you define what that was in the case of the care to the degree of you know something like 30% and you know our marketing team said that's a huge move we don't see that now whether that's actualized or not. That you know that's a little bit harder to measure but there are some ways where we've looked at new patients acquired through some of our efforts to promote quote unquote a more encompassing look at once help that includes genomics and we've had success. We'll start to look at well doing the right thing by the care of the patient you know there's downstream effects of getting patients the right carry the BRCA carries and the necessary screening and preventive aspects that are evidence based and guideline based, those all can contribute so it's something that becomes you start to build that win win approach from from the administrative and the clinical care and patient side. Adam or Bruce do you want to add anything to that. One of the things that we have the benefit of it guys singer is having a health plan that we can ask questions like this to so some of the questions that guy raised for example about, you know when do you start screening. We've had a similar experience of identifying risk to late in life. And so that begs the question of when you screen for certain conditions and ages. We can ask our health plan is there a mechanism that they would use to cover a particular practice for a period of time while we gather necessary evidence, similar to what Medicare calls a cover coverage with evidence determination. Sometimes the answer is yes sometimes it's no, but it at least allows us a mechanism to say these are technologies that we think will lead to improvements in outcomes that are important to the health plan. And is that something that they would support the trial for some period of time, but like Peter I think we're still in the institutional or other support as we're trying to gather those data. So Guy, do you want to do you have anything to add to that about the how the costs get covered and what your experience is over time as the, are there increased costs that arise from doing this kind of testing. We've had a lot of buying from the hospital board in general, in terms of sponsoring a project and we've also with the hospital been very creative in partnering with industry partners in terms of reducing the cost of the testing. So, I wouldn't say that we experience increased cost, and definitely, we, I think we are seen as improving the quality of care within the hospital. So I don't, I don't think that we really are facing that problem so much. In terms of following up and that's going back to previously, part of our consenting process with our participants is that we also have a way of reaching out to them continuously that they consented to, to get their feedback and buying into the process, and figuring out how to improve it continuously in order to partly reduce continuously reduce the cost as well. I can add we've had very high level institutional support for this but it's also a state supported project that we're doing and I think you know the key question for us is going to be sustainability, because that kind of support is obviously not something you can assume is going to be there forever and part of the implementation science analysis that we're doing is actually looking at what are the outcomes in terms of both quality of care and in terms of costs so I suppose it's possible to imagine coming to the conclusion at the end of the day it was too expensive and, you know, not having sustained state support it wouldn't survive but the other side of this is that we have an opportunity to look at how it improves quality of care in terms of pharmacogenetics and also in terms of recognizing individuals at risk that they may not have known that they had. And I guess, you know, we'll see what those results are but that's the whole premise of this is to show that it really improves quality, and at least as cost neutral and not improving that. Maybe we can pivot a little bit. There are several questions that have come up in the chat and on the Q&A related to health disparities and whether we're accelerating the problem of health disparities through these relatively expensive approaches. Anybody want to comment on how we make sure that we develop an equitable plan. Bruce you said you actually in a fairly large population of underserved in your study. But I think this question of health disparities and then more broadly, the social determinants of health as a as a counterpart to the genomics. So people want to comment on that. It turns out that our, I would say most robust enrolling site is in Selma, which serves a very generally underserved population, largely black and African American. And the uptake has been extremely enthusiastic and I attribute that mainly to the really close working relationship that the staff of the clinic have with their patient population and as I mentioned we do have a community advisory board. We really haven't found it to be so difficult to engage people if we understand what their concerns are and try to address them. But I will say, you know, one sort of key point here is that, for example, we provide the results in a, you know, essentially PDF printout which is a pretty low tech solution and we're very tempted to build apps and you know other kinds of web based things that people can use to get access to their information. The problem is that internet access in that part of the state is spotty at best and sometimes completely non existent people very often don't have smartphones never mind any other mechanism of internet access. I want to increase disparities, put everything on the web and tell people that's the modern way to get information and in our community that would not work for a very significant percentage but if we do it in a way that works for them, then we have found people really quite interested. To add that what we've observed we have disparities as well and it's very, it's clearly obvious. But what we've observed that if you have a community champion. It really, really does help in the increasing recruitment and underserved communities as for communication method, we also observed that the success notification rate was clearly higher for people of European descent compared to others and that's probably because of exactly that ability to owning, owning phones and communication method that are more available to the, or less available to the underserved. Peter, I think you were up next. Yes, so we've started to take a look at this with some primary or some personalized mess and driven initiatives. We acquired a safety net hospital in Illinois Swedish hospital and we have some clinics that are fully in the preferred language and we've gone in and done some mixed method, mixed method studies, looked at the, not only from the physician perspective and the providers there. There are thoughts on this, how they see the barriers and then also community engagement and we actually did some novel outreach as a result of it that historically North Shore hasn't done. So our program has evolved, we started to recognize to that, you know, social deterrence health disparities is, is well, it can be exacerbated by genomics. It's not just genomics problem as well and so we have started to take the approach of where are our, where are some things that are lying more broadly in our organization that we're trying to solve for example, the mammography screening rates within our organization that's something that we are tackling we have pockets where the compliance is not there. Well part of the health screening or breast screening can involve genomics and so how do we increase access to mammography. And at the same time we've leveraged some of our learnings in the primary care and now deploy a more targeted population approach at our mammography offerings where when you go in for a screening mammography, you get a structured questionnaire, and it tees up just like at the primary care visit whether you may be indicated to have genetic testing or not and our team helps supervise it so trying to find those common channels in the organization because our team is relatively small. We're not going to it's going to be much harder for us to have an impact across our broader organization if we don't try to find these other organizational priorities for social deterrence and solving the disparity issues. Casey. I'm trying to lower my hand. Thanks so, so I just wanted to make a note about genomics in general because there's the idea is that a lot of genomic discovery has happened in European populations and so the findings may be more. Those groups may may benefit more than some other groups and so just emphasizing be able to monitor like who's benefiting. And if anybody is not benefiting as much so that can help to guide a discovery research monitoring race and I know we're in genomics we say like maybe use not to use as much but for monitoring for disparities it's important to look at race and. And also, it's the genomic test results as well as other clinical data that goes into rules. And so there may be several people talked about clinical documentation in the EHR, if there are patients in some groups that don't visit the doctor as often because they're not often like that then their clinical data is less complete. And so when you take rules that have both clinical and genomic data together, that can impact also how well, like a ruler or an algorithm performs and so these are these are all kind of considerations that can lead to disparities, ultimately and so, so back to the just monitoring for those kinds of issues. Adam. I echo Peter's point about making things easy within the flow of care. You know, when we think about how previous referral guidelines for genetic counseling or testing have set really specific criteria for who should have that. And that has increased the number of steps that patients need to go through to get that information so when we democratize that and take more of a universal screening approach we can decrease some of those steps and that can be helpful. And so if you have fewer patients fall out along the way. So Casey's point to evidence base is really critical. One example from Clinton from an action ability perspective for secondary findings is that when we took a look at G6 PD, we realized that the evidence base there was not nearly as robust as we thought it would be based on how we were all trained and what we thought we knew in terms of the types of guidelines and other evidence that's considered to be really robust evidence and so when we are kind of doing downstream activities that rely on that imperfect evidence or incomplete evidence then it really does one of those because that's the way that this varies and going back and being really clear about how data were generated and who's missing from that generation, I think it's going to be really critical. I think another have about eight minutes left for wrap up and wrap up. Yeah, another thing that has come up in the chats and discussion is the question of transferability between health systems people don't necessarily spend their whole life in a single health system. They come and go and how important is it. I mean it's obviously going to be very important to address that problem of interoperability were I think a long way from that at this point. Any thoughts about how we do a better job of that. Really that's an issue that goes way beyond genomics. Genomics is the least of our problems with interoperability. I'm just getting imaging data or simple medical records can sometimes be an impossibly difficult challenge. Casey, were you going to comment on that. I was just, I was going to say that in the thread there's some, I added comment about putting genomics in the hands of the patient where that going from one one site to another that might help with consistency and data and I, there are others probably speak to that more than than I can. The other issue related to portability is in like software solutions and that is, that is where standards become important. But there, you know, there are some additional requirements locally to be able to implement standards sometimes, sometimes for software solution so, so that that's just something to keep in mind for for software projects. And Peter. Yeah, it's a real issue and I think it really was highlighted so we have merged with a couple other organizations in the Chicago land there are also epic shops but they're on different instances of ethics so even when you have the same EMR a lot of what we built. And that's kind of implement right away even if we wanted to. It's a little bit why we developed what's called our flight software our own repository that helps run clinical decision support so that we can get data in and out from a variety of different sources but that we have a very talented or have a very talented team to do that that's not scalable and I think finally we're starting to get you know more traction and attention from from the major vendors but a lot of this is going to have to be done in discussion with them if we're really going to have interoperability I think. And Mark I see you've got your hand up. Yeah, thank you. This is a really critical issue I'm hoping this did come up some at GM 13 so I'm hoping that Ken Wiley will cover a little bit of this in his talk, but I do want to take one point of discussion away from what Bruce said, where he said that this was the genomics is the least of the problem I would argue that it may be a slightly more important problem than things like labs and imaging and that has to do with the persistence of the information over time. You know, a lot of the information that we generate through transactional electronic health records, like laboratory data, imaging data, etc, is relevant for a certain period of time or episode of care, but it may not have relevance persistently, but there's genomic information, at least in theory has the ability to be relevant throughout an entire lifespan. And so I do think there are some aspects of having this in the hands of the patient in some ways with travels with the patient across the lifespan that's going to be critically important and should be an area of emphasis for research. I definitely take your point I think my point really was that, by far the most common reasons you want to see medical records are not so much genomics, as just, you know, simple who put this patient on this medicine. And even that we right now struggle to get just, you know, at the moment, in spite of intentions otherwise maybe medical records remain an incredibly siloed kind of enterprise. So we hand up from Padmaha. Sorry if I pushed your name, but go ahead and unmute yourself and ask your question. Are you able to. I allowed her to talk, but I'm not so sure that they're there. Okay.  Well, I think several times in this discussion the issue of sort of monitoring and assessing how things are happening has has come up and Peter you made a comment about currently having to do that kind of monitoring manually and obviously wanting to make it automated.  So that's something that we're trying to look at and see what can elevate it. It's more honestly a manual process still where at least we're getting more of the reporting automated so that you can see trend lines quickly and you can identify where things may be going differently. This is been important as we see looked at turnaround times for labs that are going out and coming back. Some initial baby steps and implementation but I think there's certainly more work to be done there. Adam. We focused early on trying to automate phenotyping, both through emerge and through some later activities but now we're more focused on trying to automate the performance of the recommended management that's based on that genetic information and that's just as time intensive and manual to begin with as as the other basis but once you get there, the goal is to have the ability to dashboard that information so that you can see that your cohort of patients with a particular genetic risk are either on track or not on track and drill down on those are not on track for that management so that you can then intervene accordingly. That's doable with some validation processes for looking at CPT codes, for example, but it just it takes time to build that out and we've taken a few diseases that were focused on first and then we'll expand that later. So I think we're coming up to the end of our time. First of all, I want to thank everybody for a really terrific discussion. I think they're for me there were a few important themes that emerged here. One was the importance of making sure that we're monitoring the uptake of recommendations and making sure that we're doing whatever we can to improve the quality of that. Another big theme that emerged was the importance of making sure that we do no harm that we're both increasing the diversity of the evidence base that we have. I know there's a lot of interest in NHCR and expanding to sort of non European populations to better track the variance of unknown significance and the variance of non significance or potential significance. I think making sure that we're paying attention to merging genomic data with not only other clinical data but with social determinants of health and I think that's going to be an important area of opportunity for us going forward. And then we've also heard a lot about the economics of this process and I think sustainability boost made the important point that sustainability of this is going to be really important going forward and we're going to need to figure that out. And we're going to need, you know, I know the genomic medicine working group has tried to engage payers in this process and I think there's a lot more work to be done there in terms of persuading payers to help cover some of these costs. So I think, and then finally, building on the work of Casey and others about the use of electronic health records and how do we better improve electronic health records, both to integrate genomic data into them, and then to actually broaden the utility of that data once it's in the electronic health records. So I think lots of things for us to work on and I know we're going to hear more about many of these topics in the course of the next two days so again I appreciate everything from the panelists in terms of setting us up for a really great discussion during the rest of the meeting. So thanks to the panelists and I think we're now going to schedule to take a 10 minute break so hopefully we'll see everybody back here in 10 minutes.


Our next step in SPSS and introduction is to look at basic graphics, because those are always a good first step in analysis. And the easiest way to do that in SPSS is with something called graph board templates. Really, you can just think of these as graphs made easy. The idea here is that if you set the levels of measurement in SPSS, then SPSS can suggest graphs that would be appropriate for those variables. Now in terms of level of measurement, remember SPSS uses three. Number one is nominal for different categories. Number two is ordinal for ranks. And number three is scales. That's for interval or ratio level measurements. And then when you're in the graph board templates, you have two basic choices. You have basic graphs. And those are where you choose the variables first that you want to graph. And then SPSS will show you suggested graphs, you can see what you want to do with them. There's also an option for detailed. And this is where you choose the graph style first. And then you choose the variables that go into it. Now these aren't exclusive, you can bounce back and forth between the two tabs. And it'll be easiest to see how it works if we just go to SPSS. If you're logged into datalab.cc, then you should be able to download the exercise files from the same page of this video is on, open up this file SPSS01 underscore three underscore one underscore graph board dot SPS to syntax file. And let's see what it looks like. Syntax file that you've opened looks kind of complicated. But this is really because I want to have a written record of the same things that we're going to do with the drag and drop menus in the graph board. We do need to open a data set. And as I mentioned before, depending on whether you're on a Macintosh or on a Windows computer, the path to the data sets is a little bit different. And also depending on the version you're using, I'm using 22. And so if you're using something else, change that number right there, most of it should be the same. And you can run this command and open up the data set and activate it. Now I've already done that. I'll show you. There's my data set right there is the demo dot save. And we can come down here to variable view and see the levels of measurement that SPSS is assigned to these. Most of them are scale, we have a few that are ordinal, we have only one variable in this data set that's truly coded as nominal. And that's gender, which is actually a string variable in this case. But I'll go back to the syntax. Now, I have some rather complicated syntax here. But what you'll see is that when we use the menus, it's actually pretty simple. The first thing we're going to do is make a chart of age. But I'm going to come up here to graphs to the graph board template chooser. And when I come to that, you see I'm in this tab of basic graphs. And this is where I choose a variable. I'm going to choose age right here. And it recommends three different kinds of charts, a dot plot, a histogram and a histogram with a normal distribution. We'll take the very first one that's available dot plot and hit OK. It puts in the output window, which I have to maximize. And there it is, it's a dot plot looks a lot like a histogram of age and years. So it goes down to 18 years, it looks like it goes up to about 77 78. And it's an easy way to get a feel of the distribution that we're dealing with. Again, the command in text and syntax is complicated. But the graphical interface makes this very easy to do. I'll go back to the syntax for a moment. If you were to paste the syntax for that command, this is what you would see right here. And there's a way of saving it. And you can modify it manually if you want. Now we'll do a histogram of age with a superimposed normal distribution. Again, I'll come up to graphs to graph board template chooser. And this time, all I have to do is come over to the right. I click histogram with normal distribution and hit OK, expand the output window. And it's really simple. Now both of those charts that I showed you were with age, which is a ratio level or scaled variable in SPSS terminology. We can also do this with categorical variables. I'll use gender and make a bar chart. Come back up to graphs, hit graph board template chooser. And when I come down to gender, you'll see that the recommended charts change. Because this time it knows it's a categorical variable. Now, if I had GPS data, I could put that in here. I can do a bunch of different things. I'm just gonna do a bar chart because that's the easiest to deal with. I'll hit OK, make the output window bigger. There's my bar chart. And you see that in this particular data set, we have an almost exactly equal number of men and women or data on them. Now, those were the basic charts where you choose the variable first and SPSS recommends particular graphs. You can also do detail charts. These are ones where you choose the style of chart first and then you fill in the variable. I'm going to do this again for a dot plot of income and then show you that it's really easy to modify it. Come up to graphs to graph board template chooser. This time I'll go to the detail tab, click on that. And I'm going to make a dot plot. So I'm going to scroll through this. You see we have a lot of choices. I'm going to choose dot plot. And then it's going to ask what I want to make a dot plot of. I'm going to click on this and I'm going to scroll to income. See the one that I want is right here, household income in thousands. I can click OK, then expand the output window. And here's my chart. It's a really basic chart and you see that most of the people are at the low end, especially because this is hundreds of thousands of dollars. So that's going to be a million dollars right there. But I want to show you an interesting thing about this. If we double click on the chart, that opens up the edit window and the graph board editor has some special options. For one thing, I can change the number of decimal places here. I just click on the decimals, come to format and change the minimum level or rather the minimum number of decimals to zero. That's better. But a more interesting one is if I click on the dots themselves, they're done as points and the modifier is to pile them. There are a few other modifiers that can be useful. One is to dodge them. And what that does is it puts them in the middle, expanding out either way. It might be a little harder to make comparisons from one level to another, but it's an interesting kind of chart. I can click on it again. And we can do what's called jitter with a normal distribution. And that takes points with the same value and it kind of randomly spreads them out up and down. And again, you can see that we've got a whole lot there at the bottom. One other choice is jitter uniform, which makes them stay within certain boundaries. But it's hard to tell really how much things are spread out there at the bottom. So I actually prefer pile or I think dodge is interesting in this case. And so that's one way of using graph board to both set it up and then to manually modify it by double clicking on the chart. Can close this because I'm done with that. And you see I have the modified version right there. Now, we can get a lot more complicated. So for instance, I can make a scatter plot of age and income with colors for point density. There's a lot of options and you can explore them. This time, I'm going to do a little bit differently. I'm just going to select this command. And again, the way I got these was by setting them up in the menus and then simply hitting paste. And it put the syntax into the syntax file so I could save it and run it later. And so I'm going to show you how that works. I've got the command here that I created using the graph board template chooser. And I'll simply come up and select run selection. And I maximize that window. And there you can see I actually have what's called a hex scatter plot. And it's showing a few different things. And it's a really neat way. And so you have a lot of options on the way you display things in the graph board template chooser. And while the code is complicated, the interaction with the menus is really simple. You can be creative and you can get different views on your data and try to get more insight as you're doing your analysis.