Paščicola, bilo bude na slabrih, tako pozdravno so baro je Har Sandra Benedetti in today I'm going to talk about the neural search improvements coming with Apache Solar 9.1 and specifically talking about approximate nearest neighbors and pre-filtering. Welcome, and thank you for attending. A quick introduction about myself, and the company I work for. sem italijan, jaz sem tukaj na Traskanj Citi, v Italiji, pre-Roman, včasem resercije in dovolopčenju software inginir. Imam tudi direktor na moj kompani, da je to moja objeva. Imam tudi master degree in computer science in sem program komitim membera in number of international conferences in information retrieval, specifically the European Conference on Information Retrieval, the Special Interest Group, Information Retrieval and the Science. I'm an Apache Lucine and Solar Committer and PMC member. And I work also with elastic search quite a lot. So my passion is around information retrieval in general, integrating search engines with artificial intelligence and machine learning. And in my spare time I play beach volleyball and I go for snowboarding. So my company sees, we are at quarter in London. I live in London, actually, I've been there for the last 10 years. And we work with open source software, so we are open source enthusiasts. We just don't work with open source software, we are actually contributing back a lot. So we support the community, we contribute back code, we help people using mostly Apache Lucine and Solar, but also elastic search and open search recently. We also love research, so we are active researchers. We try to keep up with the latest papers and implement them where possible in Apache Lucine and Solar. Specifically, the areas of interest for my company are neural search, so the integration of deep learning technologies with search. The reason is called neural searches for neural networks. And natural language processing integration with search, so all the branch of information technology that deal with natural language understanding, learning to rank, which is the integration of machine learning to improve the ranking of search results in a search engine. And we've been also working a lot in search quality evaluation, so evaluate the quality of your search engine in a scientific way, so not just by trying queries and results. So this is an overview of what we are going to talk today. We are going to start with lexical search problems. So for lexical search I intend the traditional search when you just match terms from the query to the documents. Then we are going to talk a little bit about BERT and in general how large language models impacted the search and how this brought to neural vector-based search. Then we are going to talk about the Apache Solar implementation and the 9.1 release specifically, and some future works. So let's start with lexical search problems. So an example of lexical search, so traditional search problems, is the vocabulary mismatch problem. So this happened when the query terms don't match the terms in the document. So you have a difference between the vocabulary used at query time and the vocabulary used at index time. An example can be a query asking for how many people live in Rome. You could have two documents. One stating Rome's population is 4.3 million. And as you can see lexically, the document has just Rome in common with the query. Also another document could be hundreds of people queuing for live music in Rome. So this is a false positive. It has nothing to do with the query, how many people live in Rome, but it shares three query terms with the query. So people, live and Rome. So we can see that the first document is actually much more relevant to the query than the second. So this is an example of vocabulary mismatch. Another example can be for the query, how big is a tiger? So you may have a term like big, biggest, that match big. This is doable with traditional search engine using stemming. But you may completely miss another document, another relevant document that says panteratigris can reach 390 centimeters, noz to tail. So this is a false negative, because panteratigris is the Latin scientific name of tiger. And the lexical search engine may not know that panteratigris and tiger are the same animal. So this is another example of vocabulary mismatch. In this case, you get a false positive. In the previous case, you got a false positive, and now you get a false negative. In general, lexical search engines have problems with semantic similarity. So we are just matching terms. So sometimes you may have queries that are completely different semantically, such as how are you, how old are you, that share mostly the same query terms completely different meaning. Or you may have, on the opposite side of the spectrum, semantically similar queries that don't share any query term. So how old are you, what is your age? You may also have problems with disambiguation with traditional search, so what's the price of an apple share, or what's the price of an apple, clearly have two completely different contexts. One is more likely to refer to apple as a company, and another one is more likely to refer to the fruit. Another area where traditional search... This has nothing actually to do strictly with traditional search in comparison to neural, but with like the day passing, we realize that sometimes information is not just in one document, so you may have actually the information, some part of the information in the document, and some other part of the information in other documents. So neural search can help you in aggregating documents into a response. Effectively, when you have an information need, you just want a response, you don't really care about documents. Most of the time, it depends of the domain, of course. Sometimes you care about specific documents, but sometimes you just care about an information. And over the years, lexical solutions tried to solve these problems. For example, with manually curated synonyms, or hypernames and hypernames, so like specializations or generalizations of terms, stemming and lematization that are algorithmic attempts to solve the morphology changes in terms, like a tense of a verb, for example, or gender slash number variations of terms. And also knowledge-based disambiguation, so linking your documents and query to knowledge bases, such as Wikipedia, DPpedia, to disambiguate potentially, like the query terms or the documents. But in the end, it's always term matching. So what happened with large language models then? So first of all, a very short introduction about the way, different ways to represent queries and documents. So in traditional search engine, you represent queries and documents using a sparse vector representation. So you are encoding your query and your documents into a vector, a numerical vector. In the sparse representation, the cardinality of the vector is the number of terms in your dictionary. So for a document, you may have a vector that is pretty much all zeros, except the terms in the dictionary that appear in the document. And the same for the query. So the reason it's called sparse is because only few terms are in a document in comparison to the entire dictionary. So your vector is going to be zero, zero, zero, and then some ones. Potentially, you may encode in the vector the term frequency in the document. So the number of occurrences of the term in the document. So if a pool appears ten times in your document, the vector value at the position of the apple term will have ten as a value. And all the other terms that don't appear in the document will be zeros. On the other hand, the dense representation will have a fixed number of dimensions, so your vector will be shorter, and all the values are pretty much all the values, and maybe there are going to be some zeros, but pretty much all the values are going to be different from zero, and will be normally like float values. And this is the current approach used by neural search. And large language models are the way to produce those vectors. So what is a large language model? So first of all, it's a neural network pre-trained on a very large corpus, or very large corpora, so a group of corpus. This may be like Wikipedia, maybe the entire web, a portion of the web. Anyway, I mean a lot of data. So normally large language models are pre-trained on a very big corpora, and then you fine-tune them to satisfy a specific task. So with the pre-training, what you get is a, let's say, pattern recognition of a language to a certain extent in understanding to the level of how machines currently understand a language of a specific way of interacting between the terms in the language and how the grammar, for example, works in a language. So you capture meaning to a certain extent, but not related to a specific task. So large language models can satisfy many different tasks. They can generate text that may be used for essay generation, for text summarization, for translation, for many things. And one of the tasks, and specifically the task we are going to talk today, is for dense retrieval. So for searching and retrieving documents from the corpus. So what we are going to do is learn a representation for our queries and for our documents. And this can be done offline. So currently, even if it's quite expensive to encode documents as vectors, you can do that offline at indexing time. So it doesn't affect that much your query time performance. So that's decently usable nowadays. So very shortly, how you pre-train a large language model. You mostly, you may approach, for example, this problem as a masked language model approach. So you're hiding part of the text in input and the model will learn how to predict the missing word. And this happens in an unsupervised way on a large number of sentences. And an example of pre-train large language model is BERT and all the family of BERT models. There are like many others, like Robert, Eddie Stillbert, and many, many variations. But one of the first transformers, they are called transformers, sometimes like B encoders. It was just probably one of the first. So it's the most popular, but you can find many, many of them. So as I said, pre-training, this is normally something you don't do.  And potentially you can just use a model that has been already also fine-tuned, but fine-tuning is something that potentially you could do in your company, in your problem. And fine-tuning for dense retrieval work in a way that you want to maximize the distance in terms of score between a positive score, so a document that is relevant to the query, and the negative score, which is a document that's not relevant for a query. So you encode the vectors, you calculate the cross entropy between the scores, and then you try to extend as much as possible the difference. So you refine in various iterations the weight in the neural network to make sure that you have a biggest distance between a positive and a negative document. But anyway, this is just to give you an idea on how you fine-tune. So fine-tuning happens with a set of samples that you must design and implement for your specific problem. And there are many models already available, especially in English, so you may not even need to fine-tune for dense retrieval, but if you have a very niche domain, a very specific problem, you may want to do that. And if you're curious about the state of the art, there are various literables around for various languages, and you can also directly download models on your own. So the current integration with Apache Solar happens effectively after you encode the query and the documents into a vector. So you encode the documents outside Apache Solar and then you index the vectors already in Solar. And at the moment, because there's still, of course, development in progress, but at the moment, there is no model management for inference in Apache Solar. So you need also query time to encode your query outside Solar, get the vector, and then ask Solar for the vectors. And no update request processor at the moment, so what we are working on is a way to hide completely the neural side to the user, so you are going to just input text to Solar and maybe specify, your admin will specify at configuration the model to use and Solar will do all the rest of the work. So what is actually neural search then? You get input documents, text documents, you get the input query text, you use the large language models to encode the vectors, you index the vectors in specific data structures that are easy to be searched after that, and then you look for nearest neighbors of your query vector. So effectively, the similarity between the query and the documents is translated to a distance in the vector space. You may have studied a school different kind of distances between vectors, so there are various in information retrieval, the most used is the cosine distance, but also Euclidean distance is supported by Solar, and you are effectively just calculated the distance between points in a multidimensional vector space. And the closer a vector is to the query, the highest the semantic similarity. The initial approaches were using just exact nearest neighbor, so you were getting the query vector and a document vector calculating the distance and getting a score, and you were doing that for your query and all the documents in the corpus. This is actually quite expensive, and nowadays, researchers move to approximate nearest neighbor techniques, so you don't want to calculate the distance for the query and all the documents in your corpus, but you use data structures you built at index time to find the closest neighbors to your query. And these strategies are based on trees. There are various family of solutions. The most important ones historically are tree-based techniques, ashes-based techniques, and graph-based techniques. In Solar we use graph-based techniques. So how we model vectors in specific data structures to have a quick retrieval at query time. We use hierarchical navigable small world graph. It's normally abbreviated as an acronym, as HNSW, and there are also a couple of nice papers about it, so if you are more curious about how it works, you can take a look to the papers. I will give you just a quick understanding of it.  So first of all it models the vector as vertices in the graph, and the links, the arcs between the nodes of the graph, is a proximity relation. So two points are going to be linked only if they are close each other. And it's called hierarchical because this data structure works on many different layers. So you have a first layer, a second layer, a third layer, and you go down the number of layers. And the reason it's hierarchical is because it follows an approach similar to skip lists. So skip list is an algorithm to get a nice insertion time and retrieval time in lists of sorted integers. And the way it works is that you have a certain probability for a node to appear on a high layer, and then you have a probability equal to one for a node to appear at the zero layer. So you will have all the vectors at the zero layer, the one below, and in a more difficult probability of having the same node in an upper layer. So this means that at the top layer you will have a number of nodes that is much lower than the number of nodes in the zero layer. And the reason instruction in this way is to have a quick and fast retrieval because you start from the top, you quickly explore the graph, and then you go down to refine the search of nearest neighbors. So effectively the longer edges on the top are for fast retrieval and the short edges on the final layer is for getting like accuracy in the approximation for the proximity of your nodes. In solar we use internally apache-lucine. Apache-lucine and apache-solar used to be the same project. So many of you are familiar with apache-lucine. So raise your hands if you ever heard of apache-lucine. OK, and maybe one. So apache-lucine is a library, it's a Java library for search engines. It's an open source library by the apache software foundation and it's the core internal of apache-solar. And how many of you have ever heard of apache-solar? Never use it. So apache-solar is a search engine built on top of apache-lucine by the apache-softer foundation. It's open source and written in Java. So lucine gives you the internal library's implementation so you will get like for example the vector-based search is implemented internally in lucine and then solar is the server that exposes this functionality. So you will be able to just get solar, spin it up and then you can interact with solar to HTTP requests, for example. So I don't know if any of you have heard of elastic search, for example. Elastic search is a competitor of apache-solar, is another open source project. Elastic search uses lucine the same as apache-solar. They are very similar as solutions. The only difference is that elastic search is from the elastic company. So it's open source, but nowadays it's a little bit more controversial. Apache-solar is fully open source. And the neural search capabilities in lucine were implemented starting in November 2020 with dedicated data structures and the latest version in lucine is apache-lucine 9.4 that also offers you the possibility of using potentially smaller vector from a memory outprint, so instead of 32-bit vectors, I mean, each value of the vector 32-bit is now the possibility of having 8-bit vectors. So if you don't have a language model that requires very specific values for a vector, you can basically don't waste memory for that. So you can specify, first of all, as I said, the vector encoding. So if you're going to use a single byte for value or four bytes for value, and the distance. So you specify the distance you want to use, so Euclidean, dot product, or cosine distance, you do that at index in time because you want to build the related data structures. So Solar implements neural search starting from Solar 9.0 and with the latest release, 9.1, we added some improvements. So, of course, as being open source, you're absolutely free to take a look to the code, to contribute back, and we have a JIRA project from the Apache Software Foundation, open to everyone, for tracking the issues to work on and the issues that we, committers, are working on. So the way you can use Apache Solar 9.1 to implement neural search, first of all, starts from defining a field type for your dense vector field. So the way Apache Solar works is using a JSON structure for your documents, where you have, like, a field name and a value, or multiple values for that field, and your unit of information is going to be a document, which, effectively, is a map key value, or values, or some multi-value that supports it. And Solar uses the schema to understand the type of data that you're going to index in that field. So it can be a textual field, it can be a number, it can be a geographical point, and specifically to what we are talking today can be a multidimensional vector. And in your schema, you specify the dense vector field and the dimension, so the cardinality of your vector, and the similarity function, like the cosine similarity, for example. You have also, like, more advanced parameter, like the way the data structures in index in time are built. This, for fully understanding them, I would recommend to, like, read the original paper or at least a couple of blogs about the way the hierarchical navigable of more work-graph work. But anyway, they are exposed in Solar, so you can control them to affect the accuracy and the index in time, and also memory, actually, used for building the graph on disk. So index in time, what do you do? You just push your JSON document with an array of integers or float numbers, depending on the kind of precision you need for your algorithm to Apache Solar. So you have, like, your JSON with the ID of the document and your vector field or vector fields. Potentially you have multiple fields in the index. You can do the same in Solar. You can interact with the search engine using a JSON payload. Nowadays it's the most used, but you can also use XML, if you want, or potentially the SolarJ APIs. Then, at query time, what you do, is you just run a REST-H TDP call specifying the KNN query parser. KNN stands for K nearest neighbor. So you specify the field where you have your vectors and the top K you want to retrieve. So this is going to return you the top K closer neighbors to your query. So the documents that are more likely related to your query. And then you pass the query vector. So just, it's pretty much a standard way to interact, very simple. You don't need to add many information in there. And for your own curiosity, if you want to take a look to the code, anyway, I'm going to share the slides after the talk, so you can also refer to the slides. But if you're curious about where this is implemented in Solar, being open source, you can take a look to the dense vector field implementation in the KNN query parser implementation. So you have the reference to the code. You actually find in the slides also the reference to the losing code. So if you're interested, that part, of course, is a little bit more complicated because you have all the implementation of the algorithm, like car. But if you are willing to, for example, understand it better or contribute, you can take a look. Another thing that is quite important is the ability of combining dense vector search, so dense retrieval with parser retrieval. So you may want, for example, to look for your top K nearest neighbors or documents containing a certain amount of query terms, for example, or potentially you want to pre-filter your corpus first with lexical search and then run top K on top of the reduced results. And you will get in response documents with the score, which is a combination of the lexical score and the neural score, so the distance between the vectors plus this score calculated by Apache Solar, which is the standard BN25, pretty much. It's a bit more complex than that, but in information retrieval BN25 is pretty much a standard approach based on term frequencies of query terms and document frequency. So how many times certain terms appear in the corpus of documents. Now, what was happening before in Solar 9.0 with filter queries was that you were getting a bit set of the results, so all the results matching the filter query. So filter queries in Apache Solar is something you use, for example, after the selection of a facet or an aggregation. So you are searching for hotels, for example, and then you filter by four-stars hotels. So you are reducing the result set using some sort of additional condition. And a filter query will return you a set of documents. Your query will return you another set of documents. Potentially you have other filter queries that will return you different set of documents, and what was happening in Solar 9.0 was just doing an intersection of these different sets. So you were getting the documents matching the query, for example, hotels with star rating 4.0 with the hotels in Tokyo, available these days. And you get the intersection. And this intersection potentially could be just zero items intersection. So with KNN search, so with KNR as neighbor search, doesn't work that well because you are looking for the top K closer vectors. And if you first run the top K and then post-filter with, for example, only a specific condition, you may get back zero results. Because maybe none of the top 10 documents from a vector perspective contains hotels, which are four-stars. On the other end, what is actually much more likely is that you want to effectively first pre-filter, so you filter your hotels for star hotels, and then you get among those hotels the top K. And this is called pre-filtering, and this is available in 9.1. In another functionality that we are currently working on is re-ranking, so you first, for example, retrieve documents using lexical search, so query terms just matched lexicaly, as traditional search does, and then you re-rank, so you can recalculate the score of the top K documents in that, using the vector distance functionality you want. We did some initial benchmark. It's usable for small corpus. It's actually usable in general. Of course, you can tune it a bit, so if you have billions of documents, you may need to do a little bit of fine tuning of your parameters to build the graphs in a way that is friendly with your memory in this space, but it's usable at scale. It's usable in production, so it's production ready. And the latest changes we've done with 9.1 also include the fact that you don't need to configure it. I mean, the configuration is a little bit easier now, even when you specify the more advanced parameters, and you can just effectively specify the algorithm you are using, which is by default, and the only current algorithm is hierarchical navigal of more world graph, and the specific parameter. So this is actually been released a couple of weeks ago, the 21st of November, and it's available for download and use. And pre-filtering is available as well in 9.1, so you are able to first pre-filter your documents using lexical search, and then just get the top K. So some of the things we are working on, and we're going to release with future version of Apache Solar, first of all, the encoding of vector values, so currently only 32 bits vector values are supported, and a maximum cardinality of 1,024 for a vector, so the longest vector you can have in Solar is currently 1,024 elements, each element 32 bits. And as I said, maybe you don't need that precision, so you may want to use less memory and less disk space, so we are going to support 8 bits elements for vectors. And another thing that is coming potentially in 9.2 is a way to hide the neural side of things. So your administrators will be able to push large language models to Solar, and then automatically, when you push text to Solar, text is going to be converted to the vector for the documents, so at indexing time, through an update request processor, for example, or at query time by a query parser. So the final user, or anyway, also the applications that use Apache Solar won't need to do the encoding, can just push the text to Solar, and Solar internally will convert it to vectors. I link in here also many additional resources, so in the blog of my company, we are anyway trying to update it as much as possible with the latest researches we do, and the latest contributions we do, and we also tend to write blog posts to help people using this thing. Of course, you have also the official Apache Solar documentation that I personally curate among other committers, but there are also various blog posts, so if you are curious to know more about this, if you want to contribute, for example, if you want to help in the developments of additional developments of neural search in Apache Solar, you are very welcome to do so. So this, of course, has not been a work just of myself, so a huge thanks goes to the Lucine community, which I am part of, but there are other people working on this topic, a colleague of mine, Elia Varchani, that helped me on these developments, Christine, Cassandra, and Michael for helping me in the review, and then final merge of the code. So thank you very much, and we have time now for some questions from the audience. So, arigato. So now it's open to questions, so if you have any questions, please feel free to raise your hand. Okay, thank you very much for the great presentation, one question from my side. So I think this was about the different languages, so is there any difference perspectives relative to this lively, like for Japanese language, or Japanese iconography, like kanji, characters? Can we apply this logic also? Yes, so there are, as I put it in here, there are, for example, various models already available on Aging phase, which is like a sort of repository for large language models. So the way you train a language model effectively starts from text organization that may depend on the language, specifically for Chinese, Japanese, and Korean. There are very specific rules, because we move from alphabets to logographic-based language. So you have some differences, but then once you do that organization and the text analysis, then the pre-training is pretty much similar. So what happens is that you may find many more models for English language, of course, but there are large language models already trained on Japanese corpora and fine-tuned on specific tasks. So there's support for that. So let's say that the specific to the language part is what happens at the beginning of the pipeline. So then you have a similar behavior between languages once you effectively split in tokens and trained. Thank you very much. You're welcome. Would you have any other questions? Thank you. Arigato.


We're going to move into the discussion phase. We have about 49 minutes until lunch. And first, let me also just make a comment and maybe start off with a question, but I'm not going to let Carol not have a chance as well. So first, as a somewhat outsider to this area, I am really impressed by the extensive number of tools that have been developed. We heard from Simone this morning, but also in this session about an extensive number of tools both for researchers and potentially for clinicians. So it just seems to me that this is sort of begging to create a universal pharmacogenetics repository that can be accessed by the research as well as the clinical community because I'm not sure that I, maybe I missed it, that these things may reside in different areas and not all the users may actually understand the value of the tools that you guys have developed. So that's just more of a comment. My question is, I heard, we heard from Mary that even now a number of the CPIC guidelines are undergoing updates, not surprisingly, as more data is generated, more those guidelines will change. But the guidelines are also being used, in many instances, to derive the clinical decision support rules that are used at a variety of healthcare systems. So how do we avoid the situation in which if a patient goes to two different hospitals with the same data, that they're actually going to get the same recommendation? So how do we think about the utilization of this, of the CPIC, foundational CPIC guidelines to ensure that the CDS is actually uniform across the institutions that are developing CDS rules? Well, the first step is to have CPIC be a reliable source of updated information, right? I mean, I don't think it would be appropriate for us to say because the healthcare system moves slowly and in mysterious ways that are non-controllable that we should not try to update CPIC guidelines when the information needs to be updated. Do we have control over how these myriad EHR systems deploy their CDS? No, we do not. I mean, we're trying to have only one version of the CDS for every CPIC guideline out there on the website. And unfortunately, we don't have a seamless way of populating the myriad systems that healthcare systems use to deploy their CDS. At St. Jude, I've been the biggest loser in terms of having to update our CDS more constantly than anyone because we started so long ago, Vanderbilt's probably about the same. I mean, we've had to completely redo much of the details of our CDS based on CPIC guideline updates over the last seven years and it's a problem. I think that's why what Sandy's doing is so critical. If we can possibly get to standardized terminology for the test results, that would be again, huge, right? Because it's the test name that's probably the most important thing to drive. It may be the phenotype designation as well. So the test names and the phenotype designations, if those could be standardized, then when the CDS updates, it's not a crazy idea to update the CDS going along with those test names. But if you don't even have a way of looking for what test names and pharmacogenetic phenotypes are affected by those updates, then we're all gonna be in really bad shape. So if I can just say we shouldn't hold ourselves to a standard that is different from medical care, which is that there is no guideline that is uniformly implemented in any healthcare system in this country. So to say that that's our goal, I think flies in the face of the unfortunate state of our healthcare delivery system. That being said, in addition to what Mary has just beautifully articulated, eMERGE has created something called CDSKB, the CDS Knowledge Base, where groups that have in fact developed clinical decision support artifacts can upload those so that groups don't have to start from scratch in terms of trying to understand the logic. Now at the present time, I don't believe that any of the artifacts that we have uploaded there are directly computable, meaning you could download the code and implement it, but at least has the descriptive terms and in many cases a logical flow diagram about how to do it. And the CPIC site and the CPIC Informatics Committee is also creating those types of flow diagrams, as we saw earlier, that will help people to get started so that hopefully there would be some consistency related to the implementation across healthcare systems. Bob, and then I wanna make Carol. Sure, so the goal is that the patient gets the same treatment no matter which health system they're in, not that everybody uses the same underlying infrastructure. So in some ways, the best approach would be to have a test panel or a sample panel, a standard panel, which once it goes through each individual system, the same recommendation in effect comes out the other end. So that the details of the implementation are only important to the extent that they generate the same data. So one could think about leveraging something like the NIST genome and a model for standardizing this kind of approach as well. I mean, I guess I don't quite see how that translates into updating for individual patients. And again, we have this experience. I have 4,000 kids that already have genotype results in their EHRs. And then I find out that for the 15% to have Diplotype X for this gene and the 30% that have Diplotype Y for another gene, my interpretation has slightly changed. I have to go in and update every single one of those results for those 4,000 patients. So sending through a test thing is not really gonna help me. I have to have a way of doing it in our EHR that's gonna do it for all those individuals. So what you're looking for is that when that allele interpretation is needed, the interpretation system is updated. Not only updated, but is able to be updated for historic past legacy patients because the other challenge with genetics is it's life long. Yeah, I agree with that completely. Yeah, that should be part of the standard. Yeah, so Marilyn, this is really for you. So the system you described, which is great, and thank you for putting it in GitHub so that other people can access it, starts with the VCF file. But there's a lot of steps that happen before that VCF file, right? There's alignments to a specific reference. There are all kinds of different variant calling parameters that are probably different that each group uses. So has there been any testing to see how robust different parameters in getting to the VCF, in terms of the interpretation that you get from the code that you've written? I mean, to me, that kind of falls in line with some of these other issues about how one updates and things like that. So variation in the variant calling that gets you to the VCF is gonna be just as important as what you're doing. Yeah, so you're absolutely right that that step, or not step, the series of steps that get you to the VCF. So the quality of the sequencing, what QC parameters you use, what reference, whether you do joint calling or multi-sample calling or single-sample calling, all of those things. However, so we have not done extensive testing on our end because for us, it's whatever genotype is in the VCF and whether it matches one of the genotypes in those haplotype tables, that's important. So how you've got to that genotype, while very important, the genotype, so these are hard-call genotypes, not kind of dosage probabilities from an imputation. So if it's, whatever the genotype is, Farmcat assumes that it's accurate. And so we've kind of felt that that comparison of all those other steps is out of scope of Farmcat because the haplotype table says these are the alleles that are important and these are the genotypes that those alleles and we're just interpreting those. When we write the paper, which is something Terry and I are working on, kind of have this section about all of those other elements and how critical they are. And Farmcat is starting with an assumption of clean genotype data, which it's a big assumption.  But we haven't come up with a way to check those because they are what the user says they are. Well, and it's not even nonsense calls, right? It's the state of the art for the algorithms that call, it all goes back to updating and interpretation. You're being able to update things as the algorithms for predicting variants improves and then the annotation for those variants changes and keeping this, this is not a one-time thing, I think to Mary's point, you need to be able, and to Jeff's point about interpretation, you need to be able to update as technology changes as the algorithms for interpretation change and as the annotations change. I think that's, for implementation, that's a huge challenge, I think. Yeah, no, that's right. So every time the CPIC tables would change, you would have to run it again. If you, and we would update, like whatever the new CPIC table is, that will be the table that's in there that generates the report. If a new reference genome comes out and you redo your calls and some of the genotypes have changed, you would need to run through it again. So I think, and Teri, correct me if I'm wrong, but I think in the report, it will have things about the dataset used, the VCF, and whatever genome build you were on, whatever details you provide, it would be there so that you would know that it will change if you have new genotype calls. But I'm, go ahead, Heidi. I just, just to drop for once again, how hard would it be to just simply take, because there's increasing use of similar standards around variant calling in terms of QD and mapping quality, for example. And if you just took those two parameters into account, could you, in the same way you're essentially dealing with the fact that there may be missing genotypes and flagging that, could you also incorporate flagging of low mapping quality and low QD scores? So I'm guessing what it would be is a pre-analysis that we could run on the VCF prior to running Formcat, where you basically just get a report of, these are the variants that have low QD or low quality scores, and that maybe those then get kind of wrapped into the report at the end. So if one of the critical alleles had a really low quality, at least it's a caveat or a warning that it's low quality. Yeah, I don't think that, I'll repeat what Terry said, if anybody's listening online. So she said, we don't think that we currently take into account the quality score, but it's a good idea and something that we could look into. So Josh and then Sandy and then Lynn. I just wanted to make sort of a Bill on Mark's point about CDSKB, it is ignite and emerge, one small little asterisk there. But the other thing is some of the reasons that some of those elements are not computable is that we can't actually share computable elements that are derived from instance from Epic. And so we are limited to sharing in a lot of cases things like screenshots or texts that come from the BPAs or things like that from those of us who may use certain vendors systems. I will suggest though, having the logic and the wording that you actually have around a decision support module, even if you have to re-implement it is a help, I think for other people implementing. Sandy, Lynn and then Julie. In my view, relative to the question of how can we get clinical decisions support consistent for pharmacogenomics, I really think that there are three components to this, three phases. The first is we need to be able to be transferring the genetic results in a way that's computable. And I think that Marilyn's work could really be helpful here relative to the advantage we have in this area for pharmacogenomics is that the results we found in digitized results are easier to describe and accessible to describing through LOINC specifically. So like the idea of taking FarmCat and potentially using that to inform a standardized set of LOINCodes, I think could be really powerful. It won't work, I don't think, if we were talking about like HCM, but I do think for pharmacogenomics it can. So I think that's the first step is getting the data moving so that it can be computed. The second step is we need the knowledge represented in a way that's accessible. I think that could potentially be accomplished by putting interfaces on top of ClinVar or ClinGen that are intended to be used for clinical purposes. Maybe not real-time clinical purposes, but intended to be used for clinical purposes. And then the third phase is representing the knowledge itself, which I think can be represented, I totally agree with the issue that Josh brought up, but having that represented in the CDS repository I think can get to that. Thanks, Andy. Lynn, before you ask your question, I'm gonna try to put you on the spot because you represent an important constituency here, which are the community-based health systems trying to implement. And so one of the things I'd like you to at least speak to is what is missing from the tools that you've seen that will enable you to be successful? Well, I was gonna comment on what was there that is very useful. So let me do that first. Firstly, as a community health system, especially in the non-cancer arena, we couldn't do our CDS support without CPIC, without digitize, and we're very interested, I'm very interested in what Marilyn and so many others have shown today. I am the person that's supposed to be integrating all this information along with my clinical pharmacist, thank goodness I have one, but it's a lot to do. It's wonderful, and again, we couldn't do this without your support. But as we go into some of our pilot studies that we're doing in the community with primary care docs, we are seeing obviously not just drug-gene interactions, but drug-drug-gene interactions. And a lot of our clinical summaries that we give to the physicians include not only, you know, you might want to consider an alternate drug or an alternate dose, but especially for the intermediate metabolizers, you may want to also monitor other CYP2D6 inhibitors or CYP2C19 inhibitors. And, you know, we're still on our drug-gene CDS and hoping to get those alerts running, and then looking at across the board, a CYP2D6 variation, and what does that mean for a variety of polypharmacy, which is what our population is. So I guess my question is, is that a next step that, you know, these nuanced interpretations where you do have drug-drug-gene interactions as well as drug-gene interactions? Still, I would love to see that as part of our clinical decision support because it's just about 50% of at least what we're seeing right now. Do you want to respond to that? Yeah, so I think, you know, absolutely, we want to be able to support that. I do think that the largest from an IT, from the IT component of this, I think the largest challenge is getting that base data accessible. You know, each individual test that you want to consider, getting that test represented in a uniform enough way such that you're going to be comfortable with the clinical decision support acting on that. And then once we have that, then I think that it's a little easier to build the overlying CDS rules architecture. I guess, you know, I totally hear what you're saying. For the clinician, the clinician needs to hear the recommendations that take into account everything with the patient, not just the pharmacogen status. Drug-drug interactions, renal function, liver function, age, history of surgery. You know, there's a long list of things for allergies. We have a diverse set of CDS built into our EHR, and unfortunately, there's separate underlying databases for every one of those pieces of information, and it's very rare that we integrate multiple different types, like now we have one that does age and genetics for voriconazole, but it doesn't include liver function, it doesn't include re-function, and I'm at a place where, you know, I think we're practicing at a pretty high level and we treat only children with cancer. I totally hear what you're saying, but I just think that that's a big ask, and I don't realistically see it coming from the genetics community. I wish that we had better integrated CDS for all of healthcare, but I don't know of any place that's doing it really great right now. Mary, I completely agree. I think where we're coming from is a, you know, how do you, we have one clinical pharmacist for an entire health system that is expert in pharmacogenetics, and to scale up, first we need the clinical decision support for the drug gene interactions. I'm completely in agreement with that, and that keeps me up at night because that's not happening, but then just within the space of pharmacogenetics, there's also this other aspect of this, you know, drug gene interaction that may also be just as detrimental or problematic, and I just don't want us to lose sight of that. I think that the key here is that since this session is on resources, I think this is a key resource gap that right now we don't have that integrative piece, and I think my belief, and I think it's shared by some, if not all around the table, is that ultimately this has to fall into the realm of the clinical pharmacologist. That pharmacogenomics has sort of been an unusual case in the sense that it's sort of emerged outside of that realm, but everything else that we're talking about is in the realm of the clinical pharmacologists at the present time. So I think that there's a gap in terms of training and synthesis that really has to take place if we're going to successfully implement over time, and so I think part of the things that I would want to see coming out of this meeting would be how do we engage in the training programs for pharmacists and pharmacologists to ultimately transition this information, which is now standalone, into something that is going to be part of it, and I know that Julie's program and everything. That only has Dean Johnson. Yes, who's next up anyway, but that to me is a key. Yeah, so as Dean of Ecology Pharmacy and immediate past president of the American Society of Clinical Pharmacology and Therapeutics, I think many of us in the room would disagree that this hasn't been driven by clinical pharmacologists. I mean, if you look at the Ignite affiliate members, I think we're at 15 or 20, they're almost all focused in pharmacogenomics and they're most all clinical pharmacists. So, I mean, I think this field is being driven by clinical pharmacists and clinical pharmacologists. So, and I think there are many, I mean, they are in the colleges of pharmacy, there are acquired accreditation standards around pharmacogenomics. That's better in some places than others just because the manpower isn't quite there, but I mean, I think pharmacy schools are way ahead of other healthcare disciplines in educating that space. But I wanted to comment on sort of the CDS support and issues generally. So, reflecting on several things that we heard. One is, the only way that those of us around the table have ultimately been able to make this work is by using grant funding to buy into our clinical effort for implementation. That's not scalable. That is not gonna work if this is gonna work. I mean, systems across the country are not gonna be able to get NIH funding to help build in their CDS. And then there are the challenges of epic upgrades and how do you upgrade. And so, I mean, I've really begun to wonder about whether we are approaching this completely wrong. And I use the drug interaction as an example and it's not that that's a perfect system, but people don't go in and build drug interaction data into their new epic build. They buy a plug-in from one of three large vendors and they use that. And so, I think you have to wonder if something like that, and this is much more complex than drug interactions. But the reality is that we really need a tool that probably is a plug-in that's constantly updated that includes drug interactions, drug-drug, gene interactions, drug-gene interactions and maybe renal function and liver function. But as we have thought about scaling in our own system, as you hear about all of the challenges, I mean, I have just begun to believe that this whole idea of trying to build into your epic system or whatever system you're on is just never gonna work. And so, this means really probably a commercial product but how we get there, I've begun to believe it's the only way we can get there on any scale and move kind of beyond this room and a few other major academic medical centers. Yeah, I wanted to pick up on that idea too. I think that we have been going about it wrong in the sense of waiting for the large randomized trial to justify putting one gene in. But also from the standpoint of changing the EMRs that we all have, I had the same experience as Mark, your 1,000th on the list of upgrades that they wanna make. And so, we just got a grant yesterday actually at our hospital where we actually took a different tact and said, this is a quality and safety issue and it can't be ignored. And they finally recognized that in that way and so it moves then to the top of the list. And I think we can make that case for a lot of the drugs we're talking about, certainly the patient story we heard this morning. And maybe that's a different way to convince people. Dick. So certainly at our place, it is a quality and safety issue. We've taken exactly the same tact. In terms of what we've learned and we've learned a good deal and what people like me thought we knew has not always been correct, in terms of both the underlying science and the implementation, it's been driven by, I agree with you Julie, by clinical pharmacologists and clinical pharmacists. I think that's true. You look at the PGRN network and who the PIs are. It hasn't come out of basic pharmacology and I'm based in a basic pharmacology department because they haven't been that interested in human variation. And so that's okay. This is where it's come from. But we're coming back to talk about training. Who did we really find at our place where we have 19 drug gene pairs that fire for every one of our 1.4 million patients. So they're through our entire system. We put great deal of effort into educating the physicians and Eric served me say this when I was at NHGRI. And so that's right there. It's called Ask Mayo Expert. As soon as an alert comes up, the doctor can look at that information. The fact of the matter is almost none of them did. What they immediately did was pick up the phone and call the pharmacist. And what did the pharmacist do? They were terrified because they had never heard of exons, introns, splice junctions, none of that. So we immediately realized we had to educate the pharmacist. And we have hundreds of pharmacists on our campus. I mean, it was scary when I heard the number, Julie. I think that's great. So they have an obligatory education program before we implement any drug gene pair of rules because they're on the front line. What's happened is the young pharmacists are embracing this as a career path. And I know that that's happened at your place too. So now we find we've just hired two additional pharmacogenomics trained pharmacists from PharmD programs. And they take care of the difficult consults because we don't have enough doctors to do that anyway and certainly not for training. But this brings us back to training. In RT32 for clinical pharmacology, there's an overdose of pharmacogenomics. And there is at Steve Leder's place in his program. I just visited there. But we need to be thinking about who actually is going in both in academic medical centers and elsewhere for the difficult cases, be available to consult one of these patients. And my conclusion, based on our own experience, is primarily it will be the PharmDs who have been trained with a PGY2 year in pharmacogenomics. I see the dean nodding her head. It makes me feel a lot better when that happens. So I think those are among the things. I mean, we're now doing this in many of the centers around this table. We've learned a lot of lessons. And I think what we need to be thinking about is, all right, how do we train the person power for the future, who are really going to be on the front lines as this happens? And I don't think I've said anything that the dean would argue with too vehemently. Is that correct? So I'll just follow up with a couple of one comment related to what Julie was saying about scalability. And I think it's, in terms of funding this off grants, it's not so much an issue of, I mean, it's partly an issue of scalability, but it's also sustainability. I don't know of any NIH funding mechanism that goes ad infinitum or ad nauseam, whatever the term might be. In terms of Dick's point with respect to training, we've taken a little bit of a different tact at our place. We've had a pediatric clinical pharmacology training program for about 20, well, 18 years, let's say. And we always had trouble recruiting into the program when the product was a pediatric clinical pharmacologist. They have no billable services. So it's really difficult to make a living. When we started marketing it as a pediatric subspecialist with a unique toolkit, then things changed. And so now, all of a sudden, we have 10 pediatric subspecialists who added an extra year to their clinical subspecialty training. And they now think about problems related to variability and drug response in their patients in a very different way. And so I think it's all about the way we present it. In our program, we actually have probably three to four physicians for every clinical pharmacist or PharmD that we've taken into the program. I think ultimately, that's the group that will be making the decisions, or at least will be responsible for the decision. So they need to understand what it is that they are deciding about. But it's still education. And the final point I wanted to make is that sometimes we have to change what it is that we say we're doing to meet whatever is hot. And so we're starting to find that adding the word analytics to something gets people's attention. So don't call it therapeutic drug monitoring anymore. Call it something else and add analytics. And all of a sudden, yeah, call it big data, big data. Or something analytics changes the perception by the people that you need to actually help you do it. Thanks, Sandy. I just wanted to comment on the idea of externalizing this support into a commercial product that's separate from the EHR. I totally agree that that's where this needs to go. Hopefully, we in this room can sort of orient towards open source based commercial models. But I think it does for broad adoption need that. But just a couple caveats that I think that we should be aware of. So there's a lot of things that make pharmacogenomics CDS simpler than other areas. But one of the things that makes it more complex is that it invariably has to be event-based. So you have to be interacting with the EHR's underlying event model, understanding when things are being prescribed and intervene as opposed to pulling things out into a totally separate app that can be brought up based on a user action. So as a result, I don't think externalizing this is going to totally solve the problem of needing internal clinical IT resources in order to get these projects over the goal line. And I do think once we're into that realm or remaining in that realm, I think that the key is really demonstrating clinical and ideally economic value in order to mobilize those resources to make that happen. Which I do think that the kind of grant funding model can help generate. So that's why I use the drug interactions as a model. Cause it has to interface with the prescribing that's happening in the record. So it does have to link into that to work. So it clearly has to be that kind of model. Yeah, but I mean there are problems sometimes with those three drug interaction databases when we get updates from Cerner and everything falls apart. But I agree that sounds like the way to go. When you said Sandy that you thought it really is, it takes more effort from the in-house IT people than the EHR vendors are ever gonna be able to provide. Can you just expand on that a little bit? I mean I would say I know Terry and I have been contacted by representatives from the major EHR companies and they've asked can we steal all the CPIC content and build something off of it? And we say yeah. But I don't know exactly what they're doing but it seems like to be scalable. It would be ideal sometime in the future if EHR vendors take responsibility for this. Doesn't it? Oh sorry. So I 100% agree that that's where we wanna go. I think that there is a standard that's emerging called CDS hooks where if that was truly to take hold that would give a standardized event model into this area. I think that it's more just a question of in the short term what's the most pragmatic way to make this happen and just within digitized and just found it's migrated. Epic and Surner have been great in this particular area where they've really supported the effort but it still kind of comes down to the internal efforts but I do agree it can mature into so drug-drug interactions. There's a mature infrastructure surrounding that and that's where we'd wanna get to. The three vendors that do drug-drug interactions got me on another gap here. One of the challenges is and any buddy that deals with these knows that you can't prescribe a medication without being flagged for some type of a drug-drug interaction unless there's been extensive work which I'm sure has happened at your institutions to try and get rid of the ones that are useless. Like Methodrix 86MP. Yeah exactly. So every time you and when you talk to the vendors about that you say well why can't you create a smarter thing and they say well we can but we don't want the liability. We want you to have the liability so if you wanna turn something off you can turn it off but what it really reflects is a larger problem which is we always default to alerts and reminders when we interact with our clinicians and we all know the problems that are related to relying on alerts and reminders which is our clinicians tend to click through them because they just wanna get their work done and so I think one of the aspects that we need to incorporate as we think about implementation is is there a different way that we can re-engineer this workflow so that we can get the information in their hands and operate under best practices but not necessarily have the clinicians aware that we're doing it. So an example would be and this is a fictitional workflow is that if you had a variant that would predispose to an adverse event. So let's say you have star five in SLC01B1 and I wanna prescribe a statin because you have elevated cholesterol. If I could go into a pick list to say I wanna pick a statin for this patient and Simvastatin didn't show up on the pick list because the system knows that this person has star five and the highest risk for an adverse event relates to Simvastatin. Well we've just operated a decision support that the clinician doesn't know has happened but we've reduced the risk for an adverse event. Now if the clinician says I wanna order Simvastatin and they start to type it in then you alert them and you say hey there's a reason we didn't show you Simvastatin on the pick list and this is it. But these types of workflows where you can really get high reliability and guide the process when you have evidence that's at a sufficient level that you can sort of automate it is something we need to be thinking about more and that's not part of the current research agenda related to implementation in the space. So a couple things as we're thinking about who we're training I think two groups who wanna make sure we don't leave out one is the nurses, they're the ones who are directly interacting with the patients probably spend more time than a lot of the providers and there is definite interest in that group as one of my close collaborators is the Associate Dean of Nursing at IU and they're doing this there's a lot of people that are very interested in it but there's within NINR at NIH there hasn't been much funding or it seemed like they've had other priorities so I think that's one group also that we can get a lot of information to the patients and the other is we need to make sure that we don't forget the patients themselves as people that if the educational materials are made available in the appropriate avenues so they can actually get it as we heard this morning they post a worker within a year becoming a pharmacogenetics really expert in one year. I mean when people have their genome they get very or their variants they actually as patients get very interested in that and they have a lot more time to actually go and look some of these things up of which when they go then to the providers they can actually use some of that so I think we need to make sure we don't leave out those two groups. Thanks so we have about five more minutes before we break for a photograph and then launch Lynn you've come up with the gaps that you want to see filled in your place. No I just checked the website and it's really really nice compared to eight months ago which was you were building. No I wanted to follow up on Mark's comment and it's something that we get a lot of pushback from our informatics committee our physicians informatics committee of oh not another rule and certainly what we try to do is minimize the physician seeing what's going on in terms of the clinical decision support so two cases in point carbamazepine and abacavir. Now carbamazepine our folks really weren't aware of the risks in carbamazepine. Abacavir every ID doc says look we order this it's in the EMR. I'm like I can't find it where is it you know some place it's under the immunology tab. So as we develop the CDS support it is looking for results first and then you know firing only when you don't have a result or if the result is positive if the result is negative nothing is going to fire and so that aspect of being you know having a lot going on behind the scenes I completely agree with you if we could do more of that we would love to we have a Surner system and it's been flexible enough but not completely flexible. So Terry. Just to sort of ask in terms of resources that we'd like I think Marilyn or maybe Julie mentioned there are three vendors that you get your drug-drug interaction tables are any of them interested in developing drug gene interaction tables? So one at least one or two of the companies were at our precision medicine conference so my sense is they're starting to pay attention and figure out I mean I completely agree with Mark the drug-drug interaction stuff is a mess you know and there's so many things that are not really clinically important so I'm not sure that that group is the one we want doing this because I think they've created a bit of a nightmare because clinicians do zip right past important drug interactions because they get so many sort of false alerts but I mean my sense is that they are paying attention and they are working to figure out how to begin to integrate drug-drug gene interactions whether they're thinking about sort of a pure tool that's just pharmacogenetics when there's not a drug interaction I'm not sure of but I mean I put it forward as a model not that those three are the answer to our solution. So recognizing as Dick mentioned that we need to enhance training particularly of potentially clinical pharmacologists I'm just wondering if any of you have thought about pushing these resources to the clinical pharmacists in the community so they're sort of outside the EMR, EHR network and they're always obviously interfacing with patients and could provide CDS guided recommendations for drugs as well so are any of you in that space? I guess three of them. I'll let's just. Yeah so one of the things that Geisinger has done that's been very innovative is they've actually developed a community pharmacist management model and so this was really taken on to address some commonly occurring problems like diabetes and hypertension, medication adherence, polypharmacy, et cetera and we found that by empowering the community pharmacist training the community pharmacist to interact with the patient we actually get much better outcomes than if we put that in the hands of the patient of the physician just because they have more time they interact with the patient more frequently it just works better so we're looking to expand that model initially into management familial hypercholesterolemia but also to use it for medication review when we have individuals with variants in hypertrophic cardiomyopathy, long QT to say are there medications on there that they probably shouldn't be on because they have one of these conditions and our ultimate goal is to create an educational program for all of our community pharmacists and to be able to move, we're developing a taking advantage of the patient facing genomic test report that we developed previously that's paired with the provider report in the pharmacogenomic space we're gonna have patient, provider and pharmacist reports designed by those end users to get that information out and hopefully tie it all together so that's something that we're investing quite heavily in. Kristen, then Lynn and then I'm sorry I don't know your name. So I think it's, I agree it's a really important and huge issue and I think we just truly mentioned our precision medicine conference and it was a huge focus because of our attendees and it was primarily pharmacists, the majority of them were community based either from a health system or community based pharmacists and so I think we're really at a point where we really, there is action, it is happening in the community pharmacy kind of external to our conversations and I think there's an opportunity for us to bring those two groups together because the community pharmacist is kind of placed in this practice and model where they are already engaged in these types of services through different models such as medication therapy management and they're very interested in becoming involved and I think learning and partnering with those groups and also at the same time equipping them to be able to make evidence based decisions and recommendations is a huge component and where there's a lot of push versus pull in this area. So it seems like there should be some community pharmacists around the table, Lynn. Yes. No, let go of Lynn first and then you. And I am not a community pharmacist but would love to have Kristen advise us and others. At Mission, we do have our own retail community pharmacies. There's about eight of them throughout this rural region and in terms of scale up, they're all very interested in learning about pharmacogenetics and being able to interact with the patient and also interact with the physician in this triangle. Me not being a clinical pharmacist or a community pharmacist, I didn't realize that the dispensing software that is used in the community pharmacy doesn't interface with our EMR and I said, well, what do you mean? We're all on the same EMR and they're like, well, we can't, we have to go into another system. And so the clinical decision support that we're building in our EMR, I recognize now, is not available in the dispensing software and we're starting to hear about companies who are going to the retail pharmacies with dispensing software information about pharmacogenetics and before that gets out of hand not to undermine our work and our field would love to understand the efforts that might be going on to help train in the retail pharmacy setting, but also have it interfaced in some way with the other work that is going on. So we're not duplicating efforts in two different systems. Hi, I'm Michelle Impequette. I'm from Canada, I'm from University of Toronto. Anyways, I'm part of a project, research project. It's a small pilot project which is called PROM, Pharmacist as Personalized Medicine Experts. So what our goal was to train 25 pharmacists in the community and to have them actually do the ordering of the genetic tests and to look at both the physician perception as well as the patient perception and how pharmacists were able to incorporate this into their day-to-day life. So we put a call out for the pharmacists and within less than a week we had 143 volunteers of which we had to choose 25 and we did an extensive online and in-person training program, probably about three to four hours online and two days in-person training. The pharmacists became much more confident and competent and they felt empowered to be able to do this. So this is, they started recruiting patients into their study about a year ago and we have about over 125 patients that the pharmacists recruited into the study and did their genetic tests and interpreted it. And overall the response was overwhelming and the patient satisfaction with their treatment plans went dramatically from about 20% satisfaction to over 60% satisfaction. There was several of the patients wrote in and said, how this is the best thing that ever happened to me. We had a lot of the physicians also once they got used to having the pharmacists doing the tests started actually really incorporating this as well too. So yes, I agree that so we are finishing up the study at the end of June and we are hoping to publish it at some point and but we'd like to do a larger scale study obviously because this was, we only had a grant for $50,000 from the Canadian Foundation Pharmacy and it was called an innovative grant. There was not much we could do with $50,000 and right now we're trying to find more. Thank you. Marilyn, you have the last word. Nice. So I think the idea of engaging in the clinical pharmacists is a really good one. They are the clinicians that catch things like drug-drug interaction. So for me, anytime I've been prescribed something that there is some sort of warning or something related to my hypertension meds, my doctor knows that I'm on the hypertension meds. They prescribe them. It's when I go to the retail pharmacy that the pharmacist goes, hold on, I need to talk to you for a minute.  I actually think you should take this instead of that because of this contraindication. So I think they are the front line with patients for those drug-drug interactions. And a second point, I've talked to a few of the pharmacists at Geisinger, especially thinking about the retail pharmacy and how I use a retail pharmacy and so they don't have my EHR. But the one data element that they always have that matches my EHR, and this is the only kind of data field that I can tell that is always the same. And every time I go to the clinic or the pharmacy they ask about it is allergies, drug allergies. If we can figure out how to put the pharmacogenetic tests of relevance for that patient into drug allergies, it won't be missed by their physician, their nurse or their pharmacist because it comes up every time. I hope your name is also the same. Well, I meant clinical data elements, not PHI. So thank you all for a great hour of discussion on this. So first thanks for the speakers, let's give them a round of applause. And now in the five minutes between now and lunch, we're gonna take a photograph, where is it outside that door so wear your sunglasses. And one other message from Teji is that speakers for the next session should either bring their jump drives up there so the presentations can be loaded before you speak. Thank you.