So, we're getting our act together as people I see are getting a little bit restless. We will do our best to be parsimonious and efficient with the time, but we want to be sure that we have a full discussion. One of the things that we do as a consequence of many of these kinds of meetings is to write a summary manuscript afterward and set it to, we've been fairly lucky, journals have been relatively interested in the kinds of work that we're doing. So what we generally do is have all of the people who were presenters or moderators be co-authors of those. That requires that you follow ICJME, ICIJCME rules of authorship and you do have to respond.  But if we don't hear from you, then we can't include you in the authorship. So please do respond. It doesn't require that you write lengthy and brilliant notes and that, but you do have to kind of respond back and let us know that you approve the final version and that sort of thing. So, are you linked? You are linked. Outstanding. All right. So you're showing on that machine. Is that a PC? Yes. Did you rather that I do the describing? Okay.  I'm ambiguous. You're ambiguous. Yeah. That's right. Yeah. That's right. All right. So what we tried to do was to come up with some overarching themes and then we do have several themes from sort of each of the sessions, but the overarching ones probably are the ones that we'll capture most of what we were aiming for. So what we've tried to kind of highlight, and I'll ask Carol to comment on these as after I go through them, is one of the major themes that we heard today in terms of dealing with variants. We kind of divided this up into sort of variants, phenotypes, and then people. And in the variants was really trying to prioritize where we should functionalize. So the biologic axes to organize these activities, you might remind people, Carol, of what that was in Nancy's, since it went by kind of fast in terms of the way Nancy had described it, or Nancy could remind us. So it was like just the last two slides. I pointed out that non-Mendelian genes, but also Mendelian genes, tend to collect on certain axes. So that is to say that altered expression of these genes are associated with set of phenotypes on one end and a different set of phenotypes on the other end. But it's several genes that have that same pattern of association. And some of those genes we already know work in the axis of, say, innate immunity wound healing or TGF beta signaling or apoptosis and growth. And so you can recognize some of the axes that are piling up a lot of phenome and a lot of genetically determined gene expression that help to drive those axes and people tilt in one direction or the other depending on genetic and environmental factors. And that, but the idea that if you pile on those phenotypes counts absolutely right. It means that the kinds of assays that you could look at for function are going to be useful for a number for sort of everything on the axis, at least at one end of the axis, maybe both ends. And so really delineating that can help with the functionality studies that we try to think about in high throughput now. But also with how we ultimately think about treatment, you know, thinking about going after this one gene at a time is exciting because there's, you know, you can knock down genes now. There's a lot of drugs that try to target genes. But in many ways, targeting the higher biological pathways might be even better, right? I mean. Great. So those were two ways, there are probably many others, in terms of prioritizing genes. So those that, you know, look like they're piling up phenotypes just from data mining, as well as those that clinically, you know, seem to have some importance, either the ACMG-56 or something else, other ways. So we have Wendy and then Callum. So the risk of talking up too much and losing my introvert status, one way would be to look at the, well, one way that I mentioned yesterday to Doug is to look at GTR for tested genes. Those tell us about genes that people care enough about the test. But even more so, perhaps, looking at ClinVar, you know, the list of variants I took a peek at that represent at least two labs have submitted it and there's a conflict. And three, let's see, three quarters of those are maybe, you know, one degree. You know, you probably care about those. But the other ones are 25% of them are, you know, two, three or four degrees of change. So, you know, that's really important, especially with the notion that our, you know, our whole endeavor of interpreting genes can be called into question by a number of groups, insurers, payers who wonder, well, why can't you labs agree?  Your methods must be fallacious. So that's one idea. That's great. Good. Callum? And I was just going to emphasize that a lot of the things that are on the biological axes that Nancy outlined are phenotypes that have very clear representation in fly and worm, in model organisms that could then be organized around those types of traits and inform and be informed by the clinical phenotypes. So I wonder, you know, we have several suggestions for how to do this. But then there's kind of a question of, well, who could do this? And how would we get that information to the people who could best use it? So any thoughts on that? Nobody wants to raise their hand. Oh, I'd be happy to. But I mean, it gets done. I mean, a lot of the stuff that Daniel's been talking about is publicly available through EXAC and through publications that sort of give the whole set of statistics. And I think, so it's just not been applied in some of the ways that might be useful in these more clinical contexts. And so the connection to what's actually publicly available data and information isn't always being made. And, you know, so that's, it's as simple as that I think sometimes. Well, perhaps, and one could say to Doug, go look in those databases, Doug, and, you know, and figure them out.  I mean, you know, I think back a little bit onto the clinically actionable gene problem where there was not only, you know, there were not only lists of things that might be relevant, but lots of disagreement as to how to weight them.  Melissa? Yeah, so I agree with Nancy. I mean, I think that, you know, like there's a number of groups that have done a lot of data integration and really have a whole corpus level perspective on what to focus on, but it's the lack of connection with the clinical activities that's really missing. So if we can come up with a couple different groups that are doing these kinds of things and actually all work together with some clinicians who are really invested in exploring those, because we're not going to go look at every single little database. We want to actually use the data in aggregate to inform these decisions. And there are different approaches that different groups have taken to integrate data, but it's the data integrators working with the clinicians that I think is the answer for that, for your pilot. Well, and perhaps not only the data integrators, but the data producers to some degree, I mean, the basic scientists who produce them, those who pull them together, and then the clinicians who are saying, you know, we really don't care about this particular gene for blonde hair or whatever. I think there's a couple of opportunities to build upon some of the groundwork that's been led. So both Cesar and the UDN might provide great opportunities to go back in and do that more focused discussion to try and pull out the use cases from the clinical folks that are already know that those folks are invested in this area, maybe not focused tightly on this question because they've got a bunch of other things that they have to do as part of those, you know, those grants, but it might be a good place to start rather than go in after something completely different. Great. Yes, Gail. So I mentioned this yesterday, but I think among the clinical folks is the clinical molecular geneticists and molecular pathologists, because they're the ones who take the data and are interpreting it at the laboratory level. And while the clinician hopefully is going to look at that, scrutinize it, maybe have their own opinion, I think those are an important group of people and should be included. And actually, I've heard now twice in the last couple of months, there's a new job out there, which is the variant caller. They're in high demand, computer and biology and probably make more than most clinical geneticists right now. But I think this whole area of bridging the informatics and the biology at the laboratory level where people you definitely want in there. I completely agree, but I also think that finding edge cases can often help you define what you need to build. And some of the best edge cases for this might come from people from different specialties, MDs from different specialties. The NICU one may be a great one, and then we could brainstorm about what those other folks might look like in terms of their background and then have them work together as part of that group. Yes, Mark. So related to that, that could be the role, a possible role for the ISCC, since we've had some challenges in terms of figuring out how best to use them that maybe for this type of specialty specific case, we could take advantage of the resource that you're funding at least to some degree to task them with a tangible work product. Sure. And for those who don't remember, the ISCC is a group of professional societies largely focused on education of physicians, but also they are linked into scientific experts in various professional specialties, and so that might be a reasonable way to go. So other thoughts on this? So Erin first and then Liz. This came up yesterday, but I would just add to the list that the ClinGen gene curation working group effort, so they're identifying curating genes that have, you know, strong intermediate, moderate, limited evidence for clinical validity, so those on the lower end, where if we had some more functional data, might push them over the edge, would be good places to start. I think that's something that they could do. I mean, we already make the resource needs to be improved, but all the assessments that they've done so far are available online through ClinGen, publicly available, and so you can see the functional evidence. It needs to be packaged better, but we could ask them to do it, but also put it out there for other groups to take it and go from there. Well, and one, just a second, Liz, one might question whether ClinGen's view of the world, while important, is the only one that you'd want to include. I mean, it seems like that was certainly one that you'd want to include in a discussion, but may not be the only one. However, I think it would be great to give this job to Erin because she gets everything done. And so that's a good suggestion. Mike, did you have your hand up? No, somebody over there did, no? I know you did, but okay. I keep thinking I'm seeing you, Mike. I'm sorry. Liz, please. Just to add that we could maybe stack the odds in our favor of being successful by using some of the data that, for instance, Nancy presented around what we might find and we could choose our specialists based on the functions of the genes that look like they might be attainable. So not to put Nancy or anyone on the spot, but it would be really useful to be able to use these kinds of data and recognizing that you guys are publishing and that's appropriate and that sort of thing. I mean, is that something that we could work with you to sort of mine your resource? No, I think, you know, with all the caveats that the kinds of phenotypes we're best powered for are phenotypes that come to tertiary care medical centers like Vanderbilt and require blood work because those are the ones that get into BioView and so forth. That said, it's just a list of phenomes, right? I mean, it's a list of phenome packets, as it were, that coincide with a set of top genes, those that accumulate disproportionate phenome and those that are loss of function depleted, right? So, yeah. So just to follow up on that, so what would be cool and what we were just talking about at the break was, you know, if I actually mapped your phenotypes, then we could pull from our system all the most similar sets of phenotypes from the model organisms and find all those model organism biologists that might be most related to those sets. So that might end the pipeline there at that biologist who ever said that earlier. Good, so it sounds like this could be one deliverable, for example, you know, we've heard several ways now that we could consider prioritizing these. It sounds like we need to get people in the room who sort of know how these kinds of things fall out and then do some debating with clinicians and basic scientists and data integrators to figure out, okay, how do you actually mesh all of these together? And it may be something that, you know, once done may need to be revisited at some point as knowledge accumulates at all. But it sounds like people are reasonably enthusiastic about that being a useful thing. And in the meanwhile, in the, you know, six months to a year that might take us to do this, Doug needs something to do. So, I think. I have lots to do. Yeah, I'm sure you do. But I think we, you know, we heard yesterday the ACMG 56 would certainly be a list to start with. No, and I'll, I mean, I've already talked a little bit with Carol. I mean, I think that's the plan is to go back and look at the, annotate that list with what we think is doable.  So, I mean, that's something we intend to do in the next couple of weeks anyway. Okay, Carol, anything else on the, on that aspect of variant functionalization? We have another slide, but I don't seem to be able to advance this. I don't. Yeah, sorry. So, the evidence to move variants from unknown to known function classes. So, you know, the question was, I think it had come up, can we define amongst us, you know, some criteria or some kinds of evidence that actually help you to move something from a VUS either to benign or to pathogenic? Yes, go ahead. I have a question. So, in Quinvar, does it get updated or reviewed? You know, if stuff is old and something, you know, data from 2010, 2011 said it's pathogenic and now there's more data, how do you get stuff out or what criteria are there to, you know, condense it down to this is what it really means and who does that? I can address that. Yeah. Yeah, I work on ClinVar. So, the major way to get data updated is for the submitter themselves to update it. We treat them like GenBank records where the submitter owns the data. Another way is for an expert panel to rule, but again, that takes a lot of time and effort to get those panels started and then to compensate the people for their time to do those expert opinions. We have had a discussion, particularly with the ClinGen group, about what to do about literature reports. Say old literature reports where it was a single family and the group said, yep, that looks pathogenic and now you get a lot of data from clinical testers and it really looks uncertain or even benign. I think this will come up for other types of assertions in ClinVar as well, not just those from literature. I think if we have labs who don't update regularly or research groups who submit, we're also gonna get out of date submissions. So, we are thinking about how to handle those, maybe move them into a legacy category and then they wouldn't contribute to an aggregate value, something like that, give a score to conflicts. We're open to feedback about what we are, I don't really think removing it from the database is right because it is an archive and you do wanna know and see what was the history here, but I do agree that we do need a way that when we have better, newer information, how can we take out the noise that was just incorrect information from the past. Great. So in terms of the evidence needed to move variants from one class to another, I don't know that anybody has any set ways of doing that or even ideas of how to do that. I think that's one of the big questions of dealing with VUSs that our meeting was meant to identify. On the other hand, we probably should as a field agree what it is scientifically to do that so that people who look in on what we're doing have some view that yes, we do know what we're doing, we have standards and criteria that we apply in a systematic way. I think would make it much easier for the world to interpret the findings that we have. Yes, Howard? Yeah, I guess I wouldn't overemphasize that part because at least with currently, there are so many different uses for the exact same variant, ranging from no action through to eligibility for a clinical trial. And we have trials in oncology right now where any non-synonymous or other likely functional variant in the FGFR1 gene makes you eligible for the trial even if it's a VUS. And so people don't look at page seven at the VUSs and they're missing out on a trial opportunity for the patient. So something that would move it totally, a VUS would not be in the clinical utility section based on our logical thinking, but yet there would be a clinical option for that VUS patient. So I guess if we overbake this cake, we will wish we had pie. No, I don't know, I mean, I couldn't know where to go with that, but we need to do more work on that, yeah. Yes, Mark? Yeah, and that's exactly what we've done in our patient-facing genomic test report. The paired patient provider facing genomic test reports is that we have linked in their information about specifically for variants of uncertain significance or for a clearly defined clinical disorder for which we haven't founded a genomic variant, information for the patient and the provider about how they could be involved in clinical trials or the research connecting to specific investigators. And so I think there really is an opportunity to crowdsource that type of, so if I had a list of people that said for this particular gene, send me your poor, your tired, your whatever, wealthy. It doesn't work in central Pennsylvania, I'm sorry to tell you Howard, but we could easily then embed that in a report and then measure to see how that is being used. So I think, again, that's a type of solution that could be broadly implemented across systems where we could test how would that democratize the process. So I guess I'm not as clear as I should be when we were talking about the evidence to move something from a VUS to something else wasn't saying being a VUS is bad or that we shouldn't report it or whatever, it was just should we agree as a field on what evidence is needed to do that? Is it a mouse model? Is it a, you know, an IPS cell that completely recapitulates the phenotype?  Or is that a fool's errand because it's gonna differ so much from each different setting? I wouldn't say it's a fool's errand, but I'm not sure it's our errand. It seems to me that groups like ACMG and AMP and I guess Debra's not here any longer. You know, they have groups that are, you know, tasked with doing that. And then there are gene specific groups that are sort of tasked with doing it. So I'm not sure we would need to accept that as our responsibility. I think we could accept the responsibility to say, well, there's a lot of folks out there and probably most of the data associated with those folks is not being captured in any sort of a systematic way so that the groups that are trying to do their work are doing it with inadequate data sources. And so how could we facilitate the data associated with patients who receive a VUS? That's how I'm thinking of it. I think that's true, but I think if we're the representatives of the genomics element, the technology people, then it would be good for us to at least engage them in a dialogue to make sure that the ACMG have whatever they might need from us to make the decision better. Yeah, I mean, I think my point's pretty well aligned with Liz's is that although I agree ACMG will be the ones who end up defining the criteria because that's what we'll guide what's done in clinical practice. We do have responsibility to make sure the resources we build are built in such a way that they can use that. And it's definitely one of those things that, I mean, EXAC is, we've done our best to make the data available, but there's many ways I think in which it could be improved to make it more accessible to clinicians. And I think for the community as a whole, making sure that we have that conversation very carefully with ACMG and with clinical groups, not just with the sort of the scientific side of that, about what do they actually need to make those clinical decisions and how could we make those resources better that would be extremely useful. We can't defer all responsibility to them. And just before I call on Dan and then Colm, one thing to think about is whether ACMG and AMP CAP are including basic groups or data generator, data integrator folks in their working groups. And my impression is probably not to the degree that they might. And is that something that we should bring the message, those of you high up in the ACMG could bring that, not looking at anyone in particular, but those of you who might bring that message back, that gee, as these groups are being put together, I mean, I think they've been very good at engaging the pathology community in that. They may not have been as aware of the need to bring in basic science. Did you want to speak specifically to that point? I think they are aware of the information, but the difficulty they've had is that engagement piece because it's not the home for most of the folks that are in this. And so there may be a matchmaker, if you will, function for this group to try and facilitate that. So we would ask those of you who have those kinds of ties maybe to make those suggestions. That's, yeah. I think they're certainly aware of it. In principle, I'm not exactly sure they know the best entry into that system. And so I think meeting halfway on that, they would, my prediction is they'd be very amenable. Great. Gail, just on this point and then we'll get back to the calendar. So one way would be for some of the more basic computer scientists, people who are trying to implement this and use it clinically to join our organization, come to our meeting, join our work groups. We can certainly encourage the laboratory groups to, you know, include you. But I think if you're there, it would be well received and if you speak up, you're much more likely to get involved, you know, and be asked to be on a work group. Be careful what you wish were there, put you to work. But no, I think it would be great. I think, again, the most important thing is everybody talking to each other. So you need to hear what issues they have and they need to hear what you can do. So I would email me and I will give names to people. And so just to dispel any misperceptions that I certainly have encountered in talking with people about the college, the college is open to people who are not geneticists, not physicians, not PhDs. And it's even really cheap. But anyone who has an interest in genetics and genomics can join, we have different categories, but affiliate scientists, things like that. So we would welcome you guys. I mean, if you can come and you do software to, you know, do a demonstration at one of our meetings, I know A-S-H-G is a lot easier, but it either won.  But to have more involvement from this group and hear what the clinicians, the laboratorians need would be really great and important. Good. Dan, have you forgotten what you were going to say by this? Almost. It's filtered through Mark. And I'll sort of, I'll remember it in a second. So three comments. One was actually a question for Mark, and that was for most VUSs, do you report them or do you not report them? I understand that the VUSs, where there might be a clinical trial, that might be really interesting. But in general, our inclination is to sort of avoid those. Well, the clinical laboratories generally will report a VUS that meets some threshold that they think should be known about. Now, in our internal project, in our exomes, we are not reporting VUSs. That was the question.  It doesn't mean we're not attending to them, but we're not reporting them at the present. Attending to them is, it sounds like a huge time sink. So that was one comment. The second comment is that I think this sort of business, Terry's question of, do we need to set up criteria for what it is that moves a VUS from a VUS to something else, falls into this black and white versus shades of gray question. I think there will be variants where you could probably, if you try hard enough, demonstrate some function that is different from reference. And in some person, that might be enough to be the last gene in an oligogenic or polygenic trait that pushes them over to having the trait. So I think it'll be very hard to ever say it is or is not a VUS. What we can say is, across the spectrum of activity, this looks like it is really bad, or sort of bad, or a little bit bad, or not bad. And we can't find any badness at all. And then the third comment was, in response to Gail, it seems to me that, as work groups get formed and go forward, and I'm not volunteering for anything, I know. So you need to have at the table, not just the pathologists. I mean, it bothers me that the pathologists are driving this, because in many ways, their expertise is in sort of generating the high quality reproducible assays, and delivering them into the flow of clinical care. But when it comes to the genetic expertise, the basic science expertise, the informatics expertise needed to aggregate all that, and then the content expertise. So the cancer guys, or the cardiomyopathy guys, or the arrhythmia guys, all those people need to be at the table, perhaps chaired or organized by the pathologists. But I think that it's a daunting task, and that's why I'm not volunteering. But I think that there are many constituencies, and I don't want us to forget the content experts, because at the end of the day, that's going to be a big part of this. Can I just make one indirect response? I've been talking about AMP and the Association of Molecular Pathology. And yes, while the pathologists are important, when we're talking about the molecular folks, those are boarded through pathology, but do an extra year or an arm molecular program, which is two years. It's changing. So we're talking about molecular people for cancer. And if it's somatic, more often AMP will take a lead, and they will involve the tissue pathologists and our organization. But we're clearly talking about geneticists and molecular geneticists taking the lead on these. Great. Callum. I was simply going to say that if you think about the spectra of criteria that we've used over the last 25 years for inferring causality with variants, it's something that actually encompasses the entire community that we want to engage in this space from the clinicians who might be managing a family and look at segregation all the way through to the fundamental cell biologists. And so I think finding an infrastructure that allows you to engage those communities leaves open the spectrum that Dan pointed out of a gray zone where there may be functional inference for one particular prior context, but not for all. And then even just thinking about what Howard said and imagining that actually in many ways some of this will be empirically defined as the genomes move into clinical practice in terms of exposures that we've never measured before or conditioning variables that we haven't even thought of. And so in many ways, the infrastructure for doing this is at the core of what we're trying to achieve. And so thinking about how it's set up ab initio so that it does all of those things in a graded and simple incremental way is actually not an easy task. But I think the first thing that as I was thinking about it that it needs to happen because it's something that doesn't even happen in our specialist centers is an infrastructure for efficient bi-directional exchange of phenotype around genotype. I think that actually is a rate limiting step. And if you had that then the people either end of that infrastructure can vary depending on the resolution and the granularity of the question, but fundamentally that is a necessity that we won't be able to move beyond. Great, thanks. And we will get to sort of our third topic is people, which is the interactions amongst us. So I have a feeling then our third bullet in terms of criteria for evidence to move variants to clinical utility, you'll tell us that that's even more the domain of these other groups, ACMG and AMP. Am I wrong on that? Yes, come on. So I may have, as I want to do, overemphasized one part of what I'm saying over the other. I think everybody needs to be involved in that. I think having, for example, if you think about just the fact that there are criteria we could lay out, I mean there was a paper in I think 1989 in Nature Genetics looking at the criteria for inferring causality with a Mendelian disorder. That's something that still holds true today in clinical practice. That's what we all use. That's what I use with every family that I ever see to the point that I won't even open the genotype until I have enough individuals to actually generate a large score that's sufficient for that family that I can make clinical decisions based on it. But in addition to that, I think having the average clinician understand the spectrum of evidence that underlies a genotype-phenotype correlation is important. So I think it is important for us to set out what those criteria are and then have everybody collaborate around essentially generating the data to fulfill those criteria for every variant. Does that make it, maybe I'm restating what I'm saying too generally, but that's what I'm trying to. No, I think I follow you, but it sounds like you're more in the camp of yes, let's try to define what those criteria are, where we heard earlier for the evidence for moving variants from one functional class to another was something that we as a community shouldn't do. We should see that to our professional societies that are sort of invested in doing that kind of work. So I would just say, I think that if we do that, we'll wait forever because not because of anything other than the number of N of one classifications we're gonna, we need a system that classifies. And my point would be simply that that system is the healthcare delivery system and not a series of professional bodies, that's all. Just along those same lines. I think if we look at the guidelines that came out from the ACMG around how to determine whether something's pathogenic, so that many groups are interested in that question. The ACMG put it together, and now for instance, informaticians at the various sites are building the tools that will allow us to test those guidelines, the hypothesis, and come back with recommendations on how it changes. That can happen really slowly sometimes, which I think is Callum's point. If we as a community can work out how to make that happen faster, that turnaround from one group to next, which may mean all the groups coming together, then that's what we should do. So there's lots of questions around what those guidelines actually are, but a big part of the work is how to get people to come together to do that process faster. Just another quick comment in there. So when you look at the CSER data, for example, where there's quite a bit of inconsistency between annotators, it's all about the basic research functional validation data where we see the most of the discrepancies in that functional data. So if we can include the basic researchers in that process, like you just mentioned, that would be really key to building better classification systems for the evidence models in that particular part of the ACMG guidelines, which are really problematic. Yes, Wendy. So since we're talking about the ACMG interpreting sequence variance guidelines, I've read them a number of times, and then I looked back at them recently with another lens, which was to say, how much does phenotype count? So if I know lots of HPO terms, and I'm pretty sure that my patient has this condition, but it's been described as a VUS, and I enter all those terms into ClinVar, will that help? Now, there's really not much of any role that I could find for clinical phenotypes. And in retrospect, maybe that makes sense because it's laboratorians that are making these determinations with the data they have. And maybe that comes back to the issue of this collaboration that we're talking about, where people have these different accesses to different types of information that need to be integrated. So maybe, Bruce is gonna have to come in here at some point too, because I don't run a lab. I found, I've tried to get through that whole document several times. I've gotten a little further each time. It's very complicated. But the one point I will make, and I think I made it yesterday, is the bar was set incredibly high because we're gonna test healthy individuals. And if you start monkeying around, and it's meant for people that may not have any phenotype whatsoever. And obviously, the phenotype and other stuff all helps, but I think, well, I'll just stop there. That it was set so that it would work in normals and to call something pathogenic, you had to have stuff way over to one side. That doesn't mean that the clinician or someone else couldn't say, you're calling it a VUS, but based on everything I know, it's pathogenic. So, Bruce, I don't know if you wanna add anything there. You're closer to a clinical web that... Yeah, no, I don't think I have anything to add to what you just said, though. All right, so I know Melissa and Liz want to comment on, sorry, Wendy and Liz want to comment on that, but I'd really like to move to our second time because we have three and we're already 20 minutes to three. So you'll have your chance when the manuscript comes out and we can debate it in email. So Carol, anything else on that one? That's enough. No, moving on to the second. Moving on to phenotype. So in terms of phenotype, I think we all agreed that there is a need for deep phenotyping and those with unusual genotyping, sort of the genotype first approach, how best to do that, I think you heard Daniel say and others would be awfully nice to come to agree to some standards that we could come up with on that. Is that something that we should consider as a group might be something we'd like to commission a small group of diligent volunteers in their copious spare time to do? Or could we task some group that an HGRI already has ongoing with doing this? Yes, Mike? I think one way to move the ball on that would be to potentially work with some journals that might be interested in, for instance, taking on the task of inviting a group to submit a paper on the minimal phenotype for X. You know, right now the ACMG list has mostly cancer predisposition and cardiovascular disease. So there could be an oncology journal or a cardiovascular journal that would say, you know, we'd invite that and then that would go into, you know, peer review, get submitted and people could then react to what a group of authors have done. But I mean, that way you wouldn't have to do all the organization from this end and it would get out in the public, there'd be some debate and you'd end up at a better place than we are right now. Great. One challenge with that is that that would be, it sounds like starting more with the phenotype than with the genotype. So if you don't really know what your gene, you know, the genes that Daniel was talking about are loss of function variants in genes that have no known function. So you'd need everything and nothing. Right. You could start that with the common conditions but these genes of unknown function, I mean, that's going to be a series of 1001 offsets. Great. Column and then Melissa. So, I mean, I would sort of turn Mike's comment on its head in a little way. I think that would, that is necessary. I think it would be very useful for the siloed conditions because I think for example, you know, most folks who see vascular disease don't think about TGF beta disorders also involving lung until the people go to the operating room and then suddenly they realize, oh, I wish I had done their pulmonary function test before I sent them to the OR. But the converse is true for the gene first where I think you need, and this is where I think Nancy's point about universal phenotypes are low. The other thing that's quite interesting is just even hearing Howard talk about tumor, somatic tumor is maybe an assay that is being continuously studied in a very clear therapeutic context where you also have universal perturbation. So that's one, that might be one assay that is universally available for all pathways. What happens to this pathway in a tumor setting when you hit it with a particular agent that targets this tires and kinase inhibitor. But I think in addition to that, we need assays that are, I would hesitate to venture, that we need a minimal set of some assays that cover the fundamental biological axes or at least a subset of those in a way that could advance us a little bit further that's not driven by the particular clinical silo that you happen to end up in. And before I call on Bruce, Rita, could I just ask you, if you could ask the hotel, they're doing a great job out here with trimming whatever it is they're trimming. But if they could just give us another hour, that would be great. Thank you very much. Yes, Bruce? I certainly see the rationale for having some standardized deep phenotype. I guess the part that makes me a little bit uneasy is, what do we mean by deep? And in particular, I'm thinking that as a clinician, you often don't know what you're not looking at until after you have a specific question addressed to you. So the phenotype that you provide for any particular individual may still ignore some of the critical things until somebody has actually identified a gene and then they go back and they ask you, did you look for this? And you might never in a million years have thought to look for that. And so I worry a little bit that we'll send people off on an exceedingly time-consuming and labor-intensive process and still not get what we really want. And I think the power may really be not so much in deep phenotyping as in this notion of iterative phenotyping, keeping some kind of two-way conduit to the clinician so that as questions come up, they can be answered. I think in the long run, it reminds me of a term that was used by one of my mentors, something called unproductive diligence. You spend tons of time doing something because someone tells you it's worth doing and it's actually not. And what's much more useful is targeting questions, but you don't know what those questions are until you know the target. I'm not sure if this is completely obvious, but I think it's important for everybody to understand how phenotype information is being analyzed in systems such as HPO and Monarch and others. And it's not just a matter of capturing phenotypes from your patients, but the resources in the database basically act as a sort of a prior or a pattern against which to search. And for instance, as a result of the issues with OMIM, we're now completely struggling to keep our database up to date. And on the other hand, well, for instance, 100,000 genomes is depending on this to some extent for their diagnostics. And I think as we move out into sort of common disease, precision medicine, there really needs to be a way of defining what models are actually useful for the community, how to find a sustainable model because it's not really that trivial. So just to follow up on a couple of those comments because I think one of the goals that we had for the phenotype exchange standard was that you could really put anything in there and decide depending on what community you're in, what kind of focus area you have, how deep is deep for you, but that you would then also use that contextualization of all the phenopacket data out there, all the phenotype data that's been made available to help define what that next phenotype is to look at based on the priors that are in the system. And so, but I think the one thing that we hadn't really thought so much about yet was how to do that iteration part because I think that's also really critical. So I'd like to think more about that. Secondly, we have been working with journals and in fact the format was designed first with that in mind to be especially exchangeable in the context of the journals and the clinical labs where the sort of just basic level information that we want about phenotypes coming out of these in these two particular use cases where it was so critical computationally and we just aren't seeing it. And so now the idea is that the labs and the journals would be providing it and we have some journals that are already piloting some early versions of that. So there's a lot of interest from the journals already. Great, moving on to the second one, just one brief last comment on this, Colin. So I was just gonna say, I agree with everything that's been said. It's a tough thing to actually execute but if you take away the one thing that we've learned from genomics, it's that lack of bias and comprehensiveness are actually incredibly valuable. And so anything that we do has to reduce that bias and everything that we talk about in iterative or clinician-based phenotyping is really just reinforcing the biases of the past. Well said. In terms of EMR phenotypes or clinical phenotypes, sort of a desiderata, I think, deseratum is for clinical phenotypes to be more data-driven models of clinical features rather than designed for billing. I think any of us who work within a medical record system know that that may be difficult to achieve. I think one question for us though is is there some way to potentially extract through NLP or other approaches the kinds of phenotyping information that would be useful in trying to functionalize without necessarily relying on billing codes or other things? And that's something that I know Emerge is struggling with maybe not in quite formatted quite that way but at least something that we're working with. What do people think of this? Is this just sort of so naive that we shouldn't even put it down or is it something that we should try to strive for and think of ways to approach? I see Mark isn't there and he usually jumps up at comments like. A colleague of ours from Monarch, Tudor Goza at the Garza Institute has developed a text mining system that can take for instance a PubMed article or a discharge note and extremely reliably extract HPO terms and basically what this does, it pops up the window such that you can say yes or no so the entire process of actually capturing the phenotype is gamified a little bit and maybe it might take you a minute if you're sort of familiar with the disease and so it's actually not that difficult to do. So this might be something that could be implemented say in interested sites that wanted to refine their phenotypes. I mean it would have to be an extra step I would think. Or in collaboration with societies, I can see if that was set up, there might be certain journals or certain lead areas that were pitched to any interested members of the American Society of, or the European Society of. And that was not a, that was supposed to be blank. And nothing else, don't read into it. So you end up getting gaming where you get expert curation gaming as opposed to, let's get all the medical students involved and they're like, where's the brain again? It's opportunity to really reach out and that might get us more involved with some of these societies that are hard to get into. The cardiology folks have everything solved but they might be wanting to get involved with this. So there'll be some opportunities there. And we heard a lot about in the past couple of days about patient-derived phenotyping. We're even thinking that that may be something we need to have another entirely separate meeting on as to how to engage patients in doing this. Seems like their head's nodding around the room for that. I understand, Peter, I think I heard you speak a few weeks ago about some patient-derived phenotyping information that you're building into HPO. Did I hear that correctly? No. Could you? I just assumed I had such a loud voice that everybody would understand. Melissa and a few other people at Monarch led a project to translate the HPO into leis and so that this is available, I think now already on our website and so we're talking with a couple of patient organizations. I've been talking with a few in the UK and I think Melissa has here in the States and we're hoping to pilot this by the end of the year and at least one site. Anything more we need to say on that one then? Seems like people agree it's a good idea. And then, as we've heard many, many, many times, the need for common vocabularies, mapping across vocabularies and across databases and resources. Something that I think we as a community can endorse and be positive about but would have to happen between each individual vocabulary and resource and sounds like it's ongoing now. Are there ways that you can think of that we can help to stimulate that? I mean obviously funding always helps but is there an organization of ontology groups that we could approach and say, gosh we really hope more of this goes on? The National Library of Madison, I know, is leading efforts to do this kind of thing. The UMLS actually just imported the entire HPO and so that's been an important step and we would also welcome more contacts with the NLM in the future. John Madison, who's at Kaiser Permanente has also been speaking with people at SNOMED as a part of sort of a GA4GH issue and I'm not entirely sure if we can do this without being basically swallowed and digested completely but I think it would be great to have HPO being mapped into SNOMED such that you could use this in an EHR setting, have the sustainability and et cetera that's necessary so it's not an academic project but still be able to extract easily and then use this for biological translational research. Sorry, just a couple of follow on comments. So I think one thing we were talking a little bit is that it would be great to have more interaction built into the people who are actually using the vocabulary so the clinic, the Mendelian Centers, the UDN, lots of other resources are using these vocabularies but we often don't get a lot of feedback and so having greater interaction would actually improve those resources for everyone. The other thing that's maybe something that NIH in particular can help us and others with is the licensing issues because these are all open resources and there's often many dependencies and things like that so if we want to try to keep the data available and the ontology integration available that there's actually a lot of complexities that are really hard to overcome on an individual basis but that the resources collectively could take care of. So I'll point to my colleague in the policy area, Laura, just in terms of licensing and that, I don't think we have anything other than kind of a moral Suasion argument that we can do and sometimes the hortutory language as we heard about some years ago but there's probably not much other than that that we can do. Correct, it's the short answer to that question so. On the other hand, there is something that the journals can do so Karen do you want to maybe comment about that? Well what I was gonna say is I think that a common vocabulary I think doesn't just include discoverability but has been said before there needs to be I think for it to be useful some kind of quantitative standard so whether it's you know or just if so for example if somebody reports in a model organism or I would say even in a human something in a basic science journal a variant associated with a phenotype or a set of phenotypes you know what is the basic information that needs to accompany it for it to go into a database and actually be used rather than you know I appreciate now it's being lost in PubMed in a suboptimal way but I think creating lists in database without something where you can then use it quantitatively or isn't gonna be that much more helpful so what's the standard by which you say was there a severe phenotype or not is it a percent different from wild type and how do you say you know not only what phenotype but what background of you know strain if it's a model organism or you know probably demographic ethnographic information is important to for humans so I thought it was not just a common vocabulary for discoverability but a standard set of information that then becomes mineable. So something that might be considered and this was considered certainly by a group of journals early on in the GWAS era is what you know what would be the standards for data deposition and if there are each of these databases if there are particular things they have to have presumably that's somewhere front and center on your website or elsewhere but maybe not I don't know and I wonder if that might be something that would be worth the journal sort of you know helping us to promote to encourage you know cause you already are very very helpful in saying we're not gonna publish unless you've deposited your data but then there has to be sort of some you know agreement as to what that minimal data set should be so that makes sense. Exactly same kind of model. Okay I do want to get on to the last topic so we'll let Mike have the last word on that. Yeah I'm wondering if genomic medicine 10 or 11 or 12 should be an invitation to editors to figure out standards that they could promote I mean editors across biomedicine have pushed lots of changes by having standards in all different ways so might be something to consider. Then last, Carol anything else on? Nope. So the last topic, the last overarching topic that Terry and I used to sort of integrate all the discussions over the past day and a half was this topic of bridging the gap and there's two slides on this topic so some of the themes we heard were related to data sharing and resource integration and I think one of the things I heard over and over was somebody would say well we need a standard for this or do we need a resource for that and somebody say well we're building that right. So a lot of things actually exist and one of the challenges really is awareness of what's out there, how to use it, how to access it and this relates to variance but many other things as well. So there's building those data sharing mechanisms and resource integration but increasing the awareness about what's out there and so ideas on how to achieve that goal would be welcome. So are there any thoughts on that particular aspect? Oh yeah, Rick. So typically one of the problems with this is it's not a knock on the people that developed them but they developed them for their own use or because they thought it would be a good idea but didn't really put much thought into the interfaces, the quality of the interfaces, the usability of the site. So I think thinking about some way of helping make them usable is really, really important because I think often what happens is you go to the site and it's not really clear how to get the data out and it's not really clear how to do the searches so I think that should really be high on the list to think about how do we assure usability and accessibility. Okay, anything else? So other topics on the bridging the gap, going back to the very first talk, Howard's talk about understanding the perspectives of the two communities of basic scientists and clinicians and when Howard opened up the SOP it was what evidence about a variant would be appropriate for a clinician to really consider it to put it in a medical record to take it serious. And this led to a lot of discussion about the differences in mindsets and cultures in the two communities and that is one of the big gaps that I think exists for us moving forward and that led to this discussion about the requirement that a clinical study is the only acceptable evidence that I'll take versus what Les had raised about, well, we've got to embrace the ambiguity, we need to have this conditional probabilities and the predominance of evidence for what a variant does and what its clinical relevance might be and that seems kind of a tough nut to crack so thoughts on what the role of this community could be in helping facilitate these cultural gaps that we have. Howard and then Camel. I think the some level of clarity of what the evidence is would be useful. I think that guidelines have their use but when you're, it's really the acuity of the problem that helps drive this thing so if I'm asked how do you treat colon cancer I can tell you the NCCN guidelines and we helped write them. If your mother has now is now his third line therapy where we're post guidelines, we will have a very different discussion offline and so the level of evidence needed for frontline, generalized, not a real patient in front of you is very different from the acute action and I think even in Howard's example, if there was a specific need that one of the nephrologists had to make a change it would have been a very different perception in use but it was more just in a general term so I guess I don't know how to put that into some words on there but we need to present enough evidence that when one has acute need, one can synthesize a way forward but if we go to the point of making too many guidelines people will forget that there is that, that flexibility that happens in patients who are beyond our general approach. Calum, then Howard, then Liz. I was gonna say something almost pathologically identical to what Howard said but for a somatic variant so we're seeing exactly this with the VUSs and now that I think about it in Lamin A-C there are clinical trials for people who are in maximal therapy, have a genotype that may not be completely interpretable that's positive for a Lamin A-C variant that looks like it's either pathogenic or likely pathogenic, who are in clinical trials that target those types of pathways and so I think there are incremental steps by which we can get genomics into the clinic. I do think that just to add one other thing which I always seem to be doing this afternoon I see is one of the ways of actually bringing everybody around the table is to put everybody in the same place it's almost like Esperanto for translation where you basically say here's a new phenotype and that may be all I'm really saying when I suggest that we have some of Nancy's axes be the traits that we think about it. If everybody who looks after patients began to think about how different conditions are related by their unifying biology and everybody in the model organism community began to think about those uniform biologies in terms of the diseases which they impact that itself might be a small and incremental step. Great, Howard? So yesterday's conversation and Les's and Bruce's conversations about use cases for I'll use that word they didn't say that of how do you deploy this clinically? So in today's meetings while I've been listening and paying attention I've also been looking at some of the things that are out there and are not out there. So for example, ACMG has got these guidelines of if you have TSH and you have a gene mutation that's you have different levels or you have different family histories there's actually algorithms that then say this is how you should use it and I think one of the challenges we have as a community is we haven't put some use cases together or a checklist whatever we want to call that that then gives some directionality around this. It's so open-ended. Well you could use genomic sequencing for X or you could use it for Y and I just wonder if it's not maybe a time to put some stakes in the ground and put some things out there about this would be directionality and how you could use this tool with the ambiguities built into this because I think right now that's a challenge is how do we get that? Now I don't know how you do that. That could be you get into the level of guidance on this but maybe that's a paper that comes out from this group is a position paper on things to think about around that how you could deploy that but I think otherwise it's just I can't figure out how to solve the problem because it's so open-ended. You wanna do this for cardiology you can do it for cardiology. You wanna do it for rare disease you can do it for rare disease. So, I don't know, Bruce has his hand up so he's probably got a good idea. Do you mind, Liz? So Bruce and then Liz? Okay, so Liz and then Bruce. I find myself in an unusual spot where I'm in complete agreement with Howard. It doesn't happen very often. It'd be something stronger. So just to say that guidelines tend to try and bring all possible occurrences into one set of guidelines and what then happens is the guidelines get implemented and they get broken and then you work out what really works and finding some use cases to test some of these guidelines is probably what's necessary. And what we could do is again bringing people together focused around those use cases. Bruce. I think part of the point that I was trying to make yesterday and I think Liz was too is that physicians are not unaccustomed to working in a world of ambiguity. The problem seems to be that at least sometimes and for some clinicians they don't pick up on the ambiguity that is reflected in a genetic test report and it seems a little bit incongruous because I always tell physicians a variant of unknown significance means it's a variant of unknown significance. How much clearer do you need to be? And it doesn't seem to sink in because somehow I think they just don't, they almost view that as fine print and don't read that far into the report. So I think the issue here isn't trying to reduce the ambiguity. I mean in the long run be great if we can do that but in the short run I think what we need probably is to be clearer in terms of the way reports are written and probably even more standardized as the consumer of various reports like no two is the same and you never know quite where to go for a while until you get oriented to the style of that report. And maybe we need to copy a little bit more from the radiology. Clinical correlation is recommended or some words to those effect. Variant of unknown significance somehow is translated by the receiving clinician. If they ever even read that as technical jargon, even though it seems like common English that they just figure someone else will worry about and so I think a lot of this can be alleviated by being a bit blunter and clearer in how the reports are worded. Howard and then we'll move on. I just came up with a title with the paper Embrace the Ambiguity. Yeah, yes. No, at some level this genetic determinism that the specificity, the functionality and all this that we have to jump over this extra bar but I can't think of a paper bruise that talks about genomics and using this clinically that has articulated what you and Lass have both said with great clarity over the last day and a half. And I think that needs to be captured somehow so that you're, so while you say physicians are comfortable with that. I think when they're facing a patient at that moment in time, obviously. But when they start thinking about genomics somehow it's whether it's the exceptionalism or what it gets lost. And I think what's obvious to you and Lass and to Gail is not obvious to the people that aren't practicing medicine with us. I'm glad to know I presented ambiguity clearly. I wonder following on Mike's suggestion for a subsequent genomic medicine meeting do you think that the various big labs would be willing to get in a room with us and talk about what standards there might be? I mean obviously they're in competition with each other but it's also to their benefit to have these things interpreted correctly. So is that something you think that is viable or is that again a silly idea? I was just gonna say that the BRCA Exchange Housework Group actually convened six different labs like last fall to do exactly that. Just to have that first discussion about those things and everybody was really positive about it and agreed about sharing data. I think that the challenge that they had though was we hadn't yet defined some of the data structures for the FINA packets yet was what clinical data they were willing to exchange at the patient level is just really challenging. Yeah, well and not even that. I mean I think the issue that was raised here was just they report them in different ways and you have to learn how to read each report differently and so could we agree on something like that? Dan? So I think the labs, I mean as you said are very willing to get in a room and discuss these things and I know Heidi Rehm for instance has convened a number of meetings and worked with them within the ACMG context as well as part of the coming up with those variant interpretation guidelines and they're certainly, they've also been actively participating in depositing variants in ClinVar with varying success. It seems to me engaging them is so important because so much of the clinical tests, so much of the active interface between what we've been talking about in the last two days and patient care will come through those clinical labs that that engagement's absolutely critical and it's also, there hasn't been as much discussion of this over the last few days but there were some mentioned just then about the, you know, people's ways of dealing with VUSs. My biggest concern is not the way that clinicians deal with VUSs. It's the fact that at the moment I think that the clinical labs are still systematically over reporting pathogenic variants and I've seen so many reports where it's, you know, there's a variant that's been reported back as pathogenic or likely pathogenic that is just clearly too frequent in EXAC or other populations to be pathogenic. There's a lot of work that needs to be done to push against that. And part of the reason I think this is occurring is there's a strong commercial incentive for clinical labs to have a high diagnosis rate. They're judged based on their diagnosis rate and so it will always be attempting to relax those criteria a little bit more clearly. So again, we can't set those criteria here but I think we can help to push things in the right direction and make sure that the clinical labs are very aware that people are watching and that people are paying attention to the reports that are going out there and that there are standards that could be assessed and there's evidence that should be used and you shouldn't be ignoring, for instance, large variant databases when interpreting variants that will result in incorrect assertions. Yeah, that was kind of reflected in Bob's presentation too, that exact point. So Howard and then Mike and Bruce. Sure. Which is aside from the commercial incentive to have high diagnosis rates, I wonder too if there's a concern about liability, if they fail to mention something that someday is found to be pathogenic. So there may be a bit of both that drives this. Howard and Mike. So I think getting the labs together is a good thing. I think there's some good strategy around that. We've been doing this with Baylor, which has been very informative. But I'll tell you that the problem set is not so much around that and I think they're, and Daniel, I agree with what you're saying about this, but there's errors being made in medicine all the time. And I'm not saying that we shouldn't try to minimize it, but the average physician makes 62 misdiagnoses a year, according to the IOM. So the challenge is that how do we balance that piece off? And so what I'm getting as, I think the labs can all get together. I think we can have some discussions. I think we can look at reports. I think we can figure out how to exchange data. But that doesn't get to the point of somebody using it in a clinical setting. The use in the clinical setting, to me, is the hardest part, is about how do we get this in a utility? I think that what we're talking about in terms of getting the labs together is the easy part. Because I think we can look at data. We can make exchanges. I'm confident that your group and our group and the others that are doing that, we can come some standards around that. But it doesn't get rid of this ambiguity of how do you deploy it at the patient? And I don't know how to convey that. But to me, that's what I'm trying to figure out is how do we capture that? And less about just decreasing error. Not to say we shouldn't decrease error. But I don't see that as the biggest problem, in my opinion. So yeah, I mean, we're probably in agreement. I think everyone understands mistakes are made. And also, very interpretation is just hard. Even if you have lots of information at your fingertips, there are many, many cases that sit on the edge. It's seen once or twice in EXAC, and you just have no idea what to make of that variant. And I have no problem with clinical labs making a call one way or the other when the evidence is actually ambiguous. I guess what I do have a problem with is clinical labs that continue to ignore evidence bases that exist out there. And I have continued to do so just by a year and a half of that evidence base existing out there, continuing to give back diagnoses that are flagrantly wrong when all you have to do is look at public data to say that. Yep, Cal. I was just gonna say, I think some of this is, oh, sorry, Mike, you were ahead of me, weren't you? Mike, on you go. Yeah, okay. I'll take it. So I'm really worried about your manuscript, Howard, of Embrace the Ambiguity, because I think that's the top of the list serve for insurance payers, and it puts us even further back in relation to some of the conversations we had. I think, and it would be going outside of genomics a bit, but I think that one of the future meetings should have people in the room that can describe the parallels between the ambiguities elsewhere in medicine and in genomics, because, you know, again, it comes back to we are like other areas of medicine, we are just more introspective, I think. So that was actually. That's the embracing the ambiguity. So that was actually going to be my point is, I think the problem is not that we're ambiguous about the VUSs, it's that we're not ambiguous about the pathogenic variants. That the fundamental problem is, for years, we've made it seem like you don't need to know any genetics, you don't need to look at anybody else, you don't need to understand the phenotype or the genotype correlation. All you need to do is look at the task result. And so everybody thinks they've got it, every physician thinks they've got it covered. It's either pathogenic or it's of uncertain significance. And in fact, you know, my biggest problem is seeing people who have what look like they should be pathogenic variants and they have different diseases, or they have diseases that segregate from the other side of the family. I mean, that's what we're actually seeing. And I think that's a good thing. I mean, I think it tells us that it's more complicated than it initially appeared. But I think the way that the field is responding is by trying to either back out of this or goes full throttle forward with the wrong metrics. And I think what we need to do is to engage, as Mike said, the medical community. And there are hundreds of tests where the, maybe not hundreds, there are several tests where the market is $200 million or more in the U.S. in areas for where there's zero data that there's any benefit. And those tests got into the market by simply showing one narrow area of benefit and then suddenly it has expanded. So you can see why the insurance companies are wary of this, but by the same token, these tests got into the market because they showed some fundamental delta in outcome. And that's why I think to some extent we're all seeing the same thing from slightly different angles. We need to be more involved in clinical care. We need to communicate more precisely the ambiguities or the quantitative precisions around our statements. But I don't think we're gonna get there by either making pathogenics more definitive or VUS is more ambiguous. I think we need to make the process very clearly something that has to involve the whole medical community. I think that was a great summary. I hope we caught that on the tape. So in terms of other bridging activities on the slide, I think we've already addressed this, one of increased awareness of resources and standards that do exist. The idea of needing, expanding like the matchmaker concept to other domains was raised in the last panel. And one of the thoughts we had was, I don't know how many of you are familiar with Innocentive, the Innocentive challenges. So it's a website where people can post, I need somebody to write me a code that will go from genotype to phenotype, right? And you put out a challenge and then there are solvers, there are people who wanna sort of engage in that. And it's a little more detailed than a matchmaker which is based on, I'm interested in this gene or this variant, imagine that you are a clinician and you've encountered this variant and you want somebody to build you a mouse model or you wanna know if there's anybody who wants to work collaboratively. It's kind of a neat site, if you haven't seen it. That's another sort of model we could think about for bridging these gaps. Last, oh yeah, Howard. Before you leave the page, I wanted to mention to the NCRI staff in particular that this would be a good opportunity to reach out to NIGMS and to NSF and other worlds that contain the other end of that bridge. It was, I think it was with amusement that the information came out about a certain amorous website containing lots of men and fake profiles for women. And when that website went live and people started losing their jobs over it, that information came out. And I just don't want us to build a site that is full of genomics people and has no basic scientists on the other end to match. And I'm afraid that we are heading towards that sort of thing and need to be very thoughtful about it.  So, just take a left at actually in Madison. Well listen, this is much less of an exciting comment. So I just wanted to say that the NSF Phenotype Research Coordination Network, which was a five-year NSF-funded program, just ended. And the program, the PI on that project is actually now moved to NSF as a program officer, our division director. So it is extremely keen on developing these phenotype exchange standards for across all of biology. So I think that's a great timing and opportunity in particular. Last slide. Well actually we have like 20 more slides, but I don't think we're gonna go through them all. Are we gonna make these slides available to everybody afterwards? Because what you guys are giving us feedback now is terrific, but I'm sure that upon reflection there'll be additional nuances or subtleties or not so subtle comments that you'll have on these. And they'll be available and it's really helpful because it's helping us sort of frame how we're going to write up this gathering as a publication. So the last slide we had for the sort of overarching topics or overarching themes to emerge from the past day and a half or so of discussion. Sort of has to do with enhanced interactions, fostering those interactions. Some of this we've already covered. I'll start with the last one. One of the things that I think is gonna be really helpful for bridging the GAMP is sort of having, kind of what Gail was talking about earlier is having people come to like ACMG and AMP and all those things and having people go to different meetings. Actually in the past we've tried this so we've had a hard time getting workshops on some of these basic resources. For example at ASHG they just don't get through that review committee. Somehow it's being felt they're not relevant. So having somebody in those organizations to help open the doors is really useful. So Gail has done that for ACMG, for getting workshops on model organizing databases there. So I think this is where we have to sort of help each other and those of us who help organize and run conferences and basic sciences need to invite clinical people to attend those as speakers and participants and I think that could help foster interactions more. So this whole last slide here is about enhanced interactions with some bullet points that we heard from the discussions. Are there any other insights or input on these general topics based on what you've heard or discussed over the past day and a half? Yeah, I don't think I do have any particular additional insight into that. I mean I just, I'm aware of efforts. I think that many people in this room also know at Baylor there's a large fly screening for disease. I know there was a paper in Selma, a lot of zebrafish looking for disease. I think at Duke there's also a center that Niko Katzan is running. So I am aware of centers and maybe they're actually all part of the undiagnosed disease network that was raised here. I actually don't know, but I do think that it would be fun to identify meetings that are held on model organism disease screening and invite clinicians or vice versa. There's a clinical genetics meeting to which there could be invitations for model organism disease phenotyping and screening investigators. But I guess I don't know enough specifics to make specific suggestions. Appreciate that. And maybe I could just stimulate my colleagues at NHGRI particularly in our division of basic sciences, particularly those who are working with say the ENCODE consortium. Is there an opportunity or some value in having clinicians come to your meetings and having your folks come to clinical meetings? Have you given, I'm sure you have given some thought to that and how we might facilitate that? Yeah, Mike. Oh, and then Elise. So from division of genome sciences at NHGRI speaking about ENCODE, we're having our second annual users meeting we had one last year. It's open to anybody. We invite everybody. So clinicians are welcome just like anybody else. We do outreach events at things like ASHG and at Biology of Genomes. Last year we had one outreach event. I think we did this with Common Fund Roadmap Epigenomics and this year we have two scheduled at ASHG and Vancouver and those are how ASHG works. You have to get into the meeting first. That's very difficult for NIH personnel. And then once you're at the meeting you have to apply to go to the workshop. Those sell out very rapidly but there will be two of those if people are interested. Elise. Two comments. One is ENCODE primarily focuses on non-coding regions and this much of the past two days have focused on coding variants. So I mean, well, we're definitely interested in that, it's not the major focus. I also want to bring attention. There are some conferences that do try to bridge genomics, genetics and human disease. There is meeting, I don't have anything to do with this but the Allied Genetics 2016 Conference in Orlando in July has sessions on, with a number of different model organisms and some of those sessions are specifically have some topics related to disease. So that may be of interest to people as well. Dan. Oh, Gail. The other thing would be with travel more expensive and it's hard to go places is to do webinars and that's worked really well. Where we can get CME grade, I'm not sure how easy, I'll bet we could do it, Jane is good at stuff but to get CME, we got it for the mouse one is to develop a webinar, we do the case conferences which are great, it'd be nice of the places that did that brought in model organisms and I think they do where they can but then if there's something on either using databases or other model organisms that you can relate to variants or human disease, doing a webinar is I think a really good way to go because you can watch it later if you can't do it at that time. Mike and then Caleb. A lot of the epigenomics projects including ENCODE roadmap epigenomics do post materials from outreach events as video or slides or whatever we have available. It's beyond my expertise to know anything about what it would take to convert that to something that you could get, you call it CME credit and or who would do that but I mean we're, we try and offer these materials and we get that there's only so many people that can travel and get into a particular meeting whereas by offering it as web-based, it's more broadly accessible. Caleb. I was just gonna suggest sort of non-traditional venues as well, there are industrial conferences where huge investments are being made in healthcare delivery and care redesign that might be impacted by what we're talking about here. Similarly, biomedical engineering conferences where folks are very much more thinking about how to move stuff into the clinic at mechanical or biophysical parameters into the clinic at scale. Those types of things I think would be conferences that would benefit from hearing what our needs are and what things we can bring to the table as a community. Terry. And I might just note that probably your average clinical geneticist would not see a data users conference as being something that's really sort of relevant to what they do. On the other hand, they may have some useful insights to provide in terms of the kind of clinical problems they're struggling with when they come across VUS and who knows what it means. Now granted, those are in coding regions because they're in genes, but there are also others that are coming up that are not in genes. And maybe not necessarily a users conference, but even at an encode steering committee where you're looking for priorities or whatever to hear from those groups and also have your investigators come to some of the meetings that we have might be useful, I don't know. So there's. Yes. Emily. Hi, I'm Emily Edelman. The National Society of Genetic Counselors also has created and is actively, is committed to continuing to create some of these types of resources for practicing clinicians. So very much on the clinical geneticist level, not, I don't think we're talking about quite as translational as how the discussion is going today, but there's an online course that will be available for purchase for any clinician that has continuing education credit for genetic counselors that'll be coming out very soon, specifically around these issues around variant interpretation. What are the different databases that are available? How do you use them? And the society's gonna continue to work on resources like that. So I think while that right now is very much geared towards clinical issues, directly for how do I solve this particular problem for a patient in front of me? I think there may be opportunities within NSGC to think more about, genetic counselors certainly are interested in thinking more about how do we feed this back into the lab and feed this back into research. Yes, Liz. We could work together to make a course on all the things we've talked about in the last two days and put it up on Coursera for free. Let people use it. Great idea. Mike? Yeah, we looked into, I think it was Coursera for our materials and the impression we got perhaps erroneously is that there's a pretty high bar to getting something into Coursera. You have to have something ready as an actual college level course with test materials and so forth. And then it gets to the issue of who's funded to develop this and who has the time to do it. But it would be a fantastic idea to offer something of that depth. Yes, Eric. I don't know if this is the right time to jump in or not but I'll just do it since there was a lull. So I think some of these ideas are terrific. They seem all potentially having a very valuable incremental role to sort of help address some of the issues we've talked about the last two days. Just everybody just try to be a little more audacious for a couple of minutes and think about if there was gonna be some sort of a larger research initiative in particular or some way to stimulate this on a larger scale. Obviously this would require resources. I can't make any promises but it would be awfully nice to have some audacious ideas incubating being thought about every once and also that if there is an opportunity we would be poised to have something in front of us that has been discussed a little bit. If people have any immediate ideas it'd be great to sort of thrash them around a little but also I wanna stimulate people to think about this as you're traveling back or over the next few weeks because I think the problem from what I can glean from here in the discussion or the problems from the last couple of days is not gonna go away easily and it's not gonna take this series of small incremental fixes to totally solve or probably partially solve it. I just tend to think there might be some unique opportunities to put consortium together or some way to structure grants to stimulate the kind of discussions we were stimulating here. So I don't know if anybody has any immediate ideas. I'd love to hear but I also want to get people to not go back from this meeting thinking wow audaciousness is dead because there's no room for it budgetarily. I'd rather hear the ideas than not hear them. Well I'll just jump in. So I don't think this is actually overly audacious but I think that these are really big problems and the idea of getting clinicians to cooperate and put invaluable information that we can all use. So the real idea is to go beyond kind of who's in the room and even who we typically know to go to industries that are really interested in this. You talk about precision medicine, I was out at Accenture and they just all wanted to know how to get involved. And I think they do have tools that can scale as we like to say. And I think that it also, I know that there's a payers conference coming up and I got myself invited to it. But I think it probably isn't enough just to have payers and genomicists in the same room. So kind of to think about what does this ecosystem take? Like who are the broad set of players because I actually don't think we can solve it on our own. And just to clarify, so you mean to get them in the room for the discussion or to get them as part of research networks or to get them to help fund some of this? I don't think it necessarily is gonna come all under the rubric of research but certainly NIH and NHGRI could have a really important convening opportunity. But I can even imagine a consulting firm like McKinsey being tremendously interested in just getting into this conversation. So something big like that. Public-pivot partnership kind of endeavor is what I'm sort of hearing. Yeah, yeah. So I think that the challenge here is that we don't know where the next big innovation is gonna come from, right? So but if we put, if we solve our data sharing issues that we've talked about several times during this meeting and all of us have been at meetings talking about this, sharing the genotype and phenotype data in a way that is open enough, protecting patient privacy but open enough that anybody with an idea can jump in and start to develop it so that we don't have to think, well, maybe it's this person that we need to bring in or that person. If we make the data available in a way that allows anybody to come in with a good idea, to me that's where the innovations are gonna come from. And if we keep these as siloed resources or closed consortium, I think it's gonna hamper our ability to make big advances. So Monty talked about UDN and the success that that small group had when they were able to share data within that consortium but that's not open to everybody. So I think the big audacious goal is to make, really go about making the infrastructure to make the data as available as possible. I'd love to say, we talked quite a lot about what Doug is doing and I think support for really doing that properly, taking the 100 most clinically impact for genes to define by some external group, defining that very clearly, this is the gene list that we need to go after that will have the biggest impact and then funding support for systematically developing high throughput functional assays for each of those genes, assessing the impact of every possible variant within those genes and making that data available in a resource that is completely available to anyone who wants to look it up would be transformative for a bunch of clinical applications. And the key thing here is doing that systematically, it's very clear that a lot of this work is being done by Doug and others and will happen in many different ways but the idea of building that in such a way that the people think about exactly what they need to do to validate those assays, engagement with the clinical community to make sure that there's clear contact with the way that the data is produced and the types of information that's actually needed for clinical interpretation and then making sure that there's that unified resource where the data is displayed in a way that everyone can access would be phenomenal, we would love it. That captures what I was thinking and I think the key pieces are the sort of beforehand crafting what we all believe are gonna need to be the sort of bars for validation and quality so that the data is actually useful to the clinical community because I can continue to produce one-off data sets and others in other labs will do that too but unless there's sort of centralization of goals and standards and then dissemination it may not have as much impact as it could. I am just taking away from some of the conversations I don't know, this isn't an innovative idea but in terms of the challenge it may be considered to have some audacity but I understand that it's really the numbers that are limiting and the populations that we actually have sequence for and so I mean if I taking, I mean what Cricket said for the first day, more of a free fraud to really find a way like can we get many more people in many more population sequence and of course that also goes with having a good database like with EXAC and so forth but I mean that to me is probably less incremental just to get the numbers that we need for the statistical power. Yes, I mean I think there is, in terms of just surely building up of numbers in particular populations that will happen to some extent naturally through obviously the common disease consortium so long as there is clear work that's done in advance of that to make sure consent is consistent with data sharing and so on which I hope will be done but I think you're absolutely right that there are some populations right now where there's just not a clear path at all towards getting large scale collection of those samples in the system and in particular I'm thinking of Middle Eastern samples which are just dramatically underrepresented in our reference populations compared to how often we see them in the clinic. There are many other samples that I think we could really do a much better job of systematically including in large scale sequencing consortia and it may just be a matter of helping to guide for some of the common disease consortia guide them towards favoring particular resources that include populations that are currently underrepresented in our databases that would help a lot. Peter and then Cecilia. One of the biggest areas of clinical need is what about the other 75%? I mean if you look at clinical exomes and genomes that the yield is about 25% in big studies and I think a lot of what we've discussed today is an incremental improvement on the paradigm of Mendelian coding disease. Completely necessary and wonderful but I think we should also think about what other models might there be. One lesson that you can learn from Mendelian disease is that probably genetic burden, so merging into oligogenic inheritance might be important. I think that's one very good hypothesis for the other part of the other 75%. Another thing is that nobody's really looking that hard at non-coding variation and we've started to look at that but by detailed biocuration we found 450, I think validated things in UTRs, promoters, et cetera. It's really little, it's very difficult to validate these mutations and I think it's an open question. Are they more common than we think or are they as rare as we think and are very, very preliminary results and a small number of genomes with an enormous confidence interval is that we could probably get you another 5% with stuff that we know now but I think those are two areas that would be more than an incremental improvement. Cecilia. Yeah, I would second that. I think non-coding definitely will have important roles to play but will be challenging but with whole genome sequencing data accumulating I think that will come but I also would like to support finding innovative ways to pursue complex genetics because I think there's a lot of the unsolved ones perhaps are due to more complex model of disease and I think we, and I know it's very challenging but I think we need to find ways to really deal with that question and how to model it, how to interrogate it. So to me that would be transformative because I think there's a lot of human disease that's really more in that category than simple Mendelian genetics. I'm preaching to the choir, I know everybody knows that. And the other thing I wanted to just comment is in terms of modeling the, I don't know, the minimal set of genes that are of high value and interest. Cell-based assays are really important and they are definitely very informative but I think depending on the gene and the phenotype there is a place for animal modeling and so I would also suggest that a subset of those genes should be systematically looked at in animal models and with CRISPR technology it's very possible that you could actually interrogate every base if you wish to not that it's necessarily gonna be informative but there is a way to that. And so I'd like to propose that animal modeling should be part of the toolkit again depending on what the gene and the phenotypes are that one is after. I think everyone's running out of steam but. I always have something to say. Okay, there's one and then there. I was curious if there's interest in this community for more efforts at humanizing laboratory animals like the mouse. I mean there's the certain organ systems, the immune system but I mean that I think I'm wondering if people think that that's another worthwhile endeavor in this community to do better or potentially more human relevant modeling of variants? Callum and then Melissa. I was just gonna say I think with all due respect I think the best place to model humans is in humans. I think that's the fundamental, that's essentially what we're hearing is there isn't enough phenotypic spectrum in an average clinical evaluation to get you past a single gene panel, far less a genome. So if we're not actually doing stuff in people we may as well not get started. That doesn't mean, I mean I work in model organisms. I think it's very important that a lot of the complexity can only be worked out in model organisms. They're just not enough, they're not of people on the planet to do some of these things. I really think we have to be quite careful about what we try and bite off. Actually to Cecilia's point there is one of the most interesting things that it may even be an audacious project would be to actually understand the genetic architecture of more than two diseases because at the moment we don't really know the genetic architecture of most diseases and those that are where it has been worked out like for example, Hirschsprung's disease, Arvinda Chakravarty's work where a decade before he started people thought it was an environmentally driven condition and now it's essentially very clear that it's a result of the interaction of only three genes. So can you imagine how few genes might be involved if you can already detect a heritability signal? So I think there's a lot of space that we haven't explored and part of the reason is the assays are not available or the conditioning variables are completely unmeasured and at the moment we measure four conditioning variables, tobacco, alcohol, weight, and a couple of other sort of rare toxicology issues. Melissa, then Rex? So two points. One is in response to that last comment. I think we've done some work in collaboration with the Comparative Toxicogenomics Database and with NIHS just to define some of the data structure needs around the exposome and there's some work going on in that area. That's really fundamental to a lot of, you know, at the interaction level. So that's one thing that I think is really deeply lacking. The second thing is just, you know, can we better choose the models in which to, you know, do these types of experiments?  And, you know, those collaborations that we want to try to match make between, you know, clinicians and basic model organisms, we need to pick the right model organisms and we have lots of data to help us determine which model organisms but we aren't really effectively using it to make those decisions. Rex? I just wanted to amplify Colm's statement in the preparation for the panel that we had from one to two. One of the topics that came up that we didn't really get to in our actual panel discussion was the role of modifier genes because we're not going to figure any of this out until we start to really get our hands around modifier genes. So I think in terms of thinking about audacious goals, maybe something to help us think about how to better do a, do a better job of thinking about modifier genes at the meeting that actually Melissa and Colm and I were at last fall where they talked about the phenotypes and Peter and some others in the room. You know, one of the bioinformatics folks there actually made the point that there might not even be enough compute power in the world to start to do all the kinds of pairwise combinatorial computation that would need to be done in order to really handle thinking about contributions of three or four variants in even the same pathway, much less three or four variants in different pathways, all of which I think are likely to be important contributors to human disease. So, you know, that's a pretty audacious goal to start to think about how do we tackle that problem. Just to segue from Rex's comment, I mean one of the things that I think we mentioned a couple of times during the discussion is the use of drugs as both a conditioning variable. But also there, you know, drug trials are amongst the most rigorously phenotyped cohorts that actually exist because they have been very intensively monitored by external CROs. And so that does three things. That A gets you the exposure, B gets you low hanging fruit in terms of time series, and C, it gets you into the clinic. So finding a way to have genomics be part of the next step in introducing new therapies or new therapeutic trials I think is perhaps something that is more politically audacious than it is financially audacious. But it may be a good first step in some areas. Cecilia. And one other thought is that, again, I don't know if this is audacious, but the idea is to, if we are to get basic scientists and clinicians working together, I think ultimately you go where the money is. And so if their application process NIH where that is a requirement that, you know, if you want to be able to apply, you have to put together a program where the basic science is very clearly integrated with the clinical enterprise. I think that that would really go long ways to really furthering this, this, this, the same. So we're looking into that. Anybody on the extramural world familiar with this at other institutes or people you know at your institution who are involved in grants like that because that was one of the things we were talking about internally, we just wanted, but there's 26 other institutes that may have tried this. Mike. I mean some, some institutes like, I know NIAID at least used to have a program years ago like this where there are program projects that would require clinical components and basic science components together or translational components and basic science components. And I don't remember what the specific opportunities they have are today, but they have these kinds of opportunities. And Deborah Leonard, who I'll channel because she had to leave, did lean over and mention to me that NCI also requires this and some other comprehensive cancer research centers. So there are some models for it. I was reluctant to suggest it because you worry a bit about you know the government telling you what kinds of you know groups to put together. On the other hand, if we can incentivize it in a flexible enough way that you can then model or fashion that to meet the ends of the science that you want to pursue, then that's probably a very useful thing. So thoughts on that, you know, as you're heading back or whatever would be useful. Actually the Department of Defense is now funding a lot of biomedical research. And this is what they do particularly well. This is something that they encourage and in fact, you know, there's a lot of the programs really, they're packaged to really encourage the integration of basic science with clinical translational efforts. And so I think that this might be a model to look at as well. And in a way it's different from the NIH PO1s where you are required to have a clinical component because in a PO1, each project is separate. So you have a basic science project and you have a clinical project and so forth. In the realm of the DOD, they have PO1s like that but they're very, very few. Mostly they're just equivalent to RO1s. And so you have to put together an RO1-like application where you integrate basic science with clinical research. Sort of it's a nice way where you really are forced to dovetail and really make the clinical research and the basic research come together. And so I think that that model seems to have worked well for the DOD and I don't know whether there are constraints to whether NIH may or may not be able to implement that. Constraints usually haven't held us back at our institute but we'll work on that. Terry? Yeah, so at this point people are getting kind of tired. I appreciate everyone staying to the bitter end. This has been a very intense two days and it's really been extraordinarily useful to us. We hope it's been useful to you. One of the things that was mentioned on here was the opportunity for informal conversations in various places and hopefully some of those have happened. It's a little known fact that we actually incentivize National Airport to have flight delays so that you guys all talk to each other and as our budget gets better the flight delays get worse. So at any rate, many thanks and I'll just turn to Carol for the last word. Yeah, no just adding my thanks. You never know when the organizing committee, so Howard and Terry and I and the Genome Medicine Working Group would get together and talk about this agenda and we were throwing out names and a lot of times there were names like well we don't know this person really so you're pulling together in a room, a group of people, many of whom you already know each other but for most of you you might be sitting next to somebody you've never met before and the dynamics of that are always an unknown but you all have really just, the conversations have been really great, the contributions and we really, really appreciate your willingness to come into this group and open your minds and state your opinions in such a constructive and positive way. So thank you all and safe travels. Oh, well yeah and again, Rita, Teji, Ellie, the video crew, everybody for putting this all together it would not have been possible without you guys for sure. So thank you.