Okay, I'll get started. And first, I would hope my council member, colleagues, agreed that the program has shown a lot of successes. And this was very clear in the very first morning session before the break. In some areas are quite unique to the program. And I guess our discussion will be, is it better to build on those strengths and uniqueness and to partner with other programs for other areas or continue on a very global, very broad program as it seems to be. We saw presentations on many areas. One thing that I particularly didn't hear much, but I would like your opinion on, is on implementation science. I think we heard pieces of it, but we didn't see, or I didn't clearly see an emphasis on that with expertise on that area. And perhaps on psychological implications, even though we saw many presentations or discussions on ELSI related to this, which by the way, has been a great success. I mean, not only from the papers, but also from the actual actions that have been done. Besides that, in my area, I would say, it's important to consider what goes in the EHR, in the electronic health record, because that is the portion that the patients will control and will have access to legally. So if it gets in there, they can take it out of there and disseminate with whoever they want. So you got to be cognizant of what goes in there and of the limitations. Do you want a VCFI, do you want a BAM file, what do you want in there, and the interpretation so that the patients can take out to do what is called the blue button in our area, which is the button that the patient is supposed to click and get all the information that is inside their electronic health records. So those were my observations that escaped the general discussion, and again, emphasize that PCORI and other institutions are doing a lot of work in building cohorts that could be potentially used in combination with yours as well. I think that the emphasis on the clinical aspect on the implementation at the clinic is unique and is very important. Good. There are actually a couple of people here that we've really not heard much from who are on council and I'm going to ask maybe if we could get some comments from them. So I was gonna start with Lon. Lon, could you give us some reactions and thoughts about how you think the day went as Lon disappeared? Okay. All right. I don't know if Steve's or he's not. Okay. Another person I had on that who I hope hasn't disappeared is Wiley. Is Wiley Burke still here? There she is, yeah. Wiley, would you be willing to give us some reflections and thoughts about what you heard about today as a? So I just want to say I think it's been a great discussion, a really interesting discussion covering a lot of territory. From my particular perspective and LC perspective focused on a few issues that are very important to Caesar, I would say first and foremost, I think we have to take very seriously this issue of how we justify clinical sequencing and to whom we are justifying it and for what reasons. I'm following up a bit on what we heard from Naomi which I think was a really important reminder that we have resource challenged healthcare systems and we are proposing a very expensive new technology to be pushed into that healthcare system and it really doesn't matter whether we're being asked to adhere to new standards versus how other technologies sneaked in 10 or 20 years ago. The reality is to paraphrase something I've heard from a colleague, if we can't figure out how to use genome sequencing technologies in ways that are cost effective and generate better healthcare outcomes, they're not, those technologies not gonna be part of the healthcare system that any of us has access to. They're not gonna be part of the benefits that our healthcare payers will pay for. So we have to take that very seriously and I think that means we have to take a big breath before we start really pushing the idea of the variety of personal and social benefits that might come from genomic information and take very, very seriously the enormous potential costs from cascade effects. Particularly cascade effects related to VUS and VUS that are variously interpreted by different laboratories. Is it possibly pathogenic? Maybe not, one lab says it is, one lab says it isn't. All of the costs that flow from that are the kinds of things that our healthcare system can't afford and won't be interested in affording and put the enormous potential benefits of the technology at risk. The one other comment that comes up for me in terms of my work is somewhat related. We all agree that we want more diversity in the participants in our research. We also want more diversity in our workforce and we simply have to think hard and work hard to accomplish both those goals. In terms of involving participants in research, diversity means better efforts at involving populations that have traditionally been underserved and traditionally have less access to healthcare and less ability to pay co-payments or other costs that are involved. As we've heard very vividly from various people's testimony, what my personal experience working in particular with tribal communities on genomic research suggests to me is that we have to move beyond general and sort of rosy pictures about what this technology can do and be very concrete and specific about what we think are the real and tangible benefits from pursuing this kind of research and what the timeline is gonna be as well obviously as respecting very carefully our potential partners' interest in sharing with us control of the research process. And I think Cesar is incredibly well positioned to think hard about both those challenges because we have such a diversity of expertise at the table and because we're really committed to clinical utility ultimately. Thanks. Are there other comments? I mean, I had a couple of reflections I'd like to make partly in the spirit of being very provocative, which is I'm gonna set up a completely strong man that I don't really believe is the case. But it's a caricature, I think to some extent of how we in this field are viewed by people who are not in this field. And the caricature is that we are true believers who have already accepted that this technology is extremely important and is gonna have a tremendous impact on health and on well-being and on medicine and that it is extremely cool technology and that we are winers about people being unwilling to pay for it. And that caricature that I've just described like a lot of caricatures has certain elements of truth in it. But I think it also represents, I think in some sense, a perception challenge that we have to overcome if this field is gonna really move forward. Bob, did you wanna make a comment? Yeah, I was just gonna follow up on bullet three about regulatory issues and I think the people at NHGRI or people who know NHGRI well know this, but we have a division of policy communications and education of which I'm a small part. And so if you do in the process of working through these things come across regulatory issues, please come to us, let us know about them. We can't guarantee we can do something about them, but at least we wanna hear about them. I need to show on the mic, okay. So one comment about the regulatory issues, I think there was a question about what our thinking was in that space. And we did this debate whether or not to make it a formal part of the session topics. And the reason why we didn't is because it's not because it's not important. We recognize that there is a lot of active, the active discussion actually between the FDA, for example, and some of the other NHGRI programs. We I think chose to think of the agenda topics as sort of research topics focused for the next five to 10 years. I think the regulatory developments are important and we will certainly need to be abreast of them and keep our eyes open in terms of thinking about new opportunities. But the I think research angle of it, a research angle of our meeting today, I think is what drove more our decision not to have it as a formal topic, but we of course recognize that we need to talk with our partners in the FDA and be aware of policies going forward. So I see Terry making a note here. But there were others. I was just gonna say, but to the degree that CSER can identify issues that would help those agencies make regulatory decisions in what they've referred to as regulatory science, that's probably a useful thing for us to do and to incorporate. Deb and then Pamela. So I'd like to respond to what the point that Bob made. And I think you're provocative, but I think absolutely correct, although I wouldn't say everybody in this room is a convert and I won't point to those that I know are not. But I think that's why research on the value, the total package of what it means to do an exomergenome for a patient, the cost of the testing, the cascade, the value proposition, how it changes the quality of the patient's life. If there's a treatable diagnosis, all the uncertainty that VUSs may create, I think understanding that within a system or a trial would be extremely helpful, especially as we're moving toward healthcare reform and different payment models that unless something is valuable, it's not going to be paid for and improves the quality of care and the cost effectiveness of care. So I don't know if there are creative ways to think about funding that would do that, but I think that's extremely important. So we move from being the true believers to either having the evidence or having to relook at our belief of this is the be all and end all. Pamela Sankar, University of Pennsylvania. I just want to reiterate something that I think has been said a lot, but maybe it needs to be repeated. Sorry? Back up. Back up, all right. How's that? Move a little closer. A little closer, that's good. Is that good? That's good. I'm working on it. I'm working on it. Actually, I'm shrinking. I'm not just happy. How's that? I don't know. How's that? How's that? Good. All right, don't let me move, Deb. Okay, I want to repeat something that has been said several times, but I think it bears repeating, and that is that this is a very important moment for CSER and NHGRI to take seriously the need to integrate a broader diversity of populations, which I know we've talked about. My point is specifically that what we need to think about are the innovations, the practical innovations and requirements by which that is going to happen. This is a group that has a lot of goodwill, and I believe most of everybody sitting here really would like to see that happen. That's also been the case for, as long as I've been around NHGRI, and I think that what needs to happen is it can no longer be a desire in an RFA or in a program that it would be good if people integrated more diverse populations. It needs to be a requirement on which people are evaluated, which it hasn't been up till now, and I think that a lot of work needs to go into thinking how we're actually gonna put that into RFA language or whatever language it needs to be integrated in, because if we don't, in five years, we're all gonna be standing here having the same conversation again. I have Heidi and then Bob and Debbie. So I just wanna comment back to Bob, your question about, we at United States all embrace this, and I think one thing that's happened is the technology has allowed our sequencing, clinical sequencing space to swing from a very narrow focus in single gene sequencing to this complete other end of the spectrum where we're looking at everything and we're looking to return secondary findings and now this question of the cascade of what happens on that end comes up. And at some level, you can liken it to the radiology world where screening CAT scans and screening MRIs were put into place and clearly people have backed off from those approaches recognizing the cascade effect, as Naomi pointed out, but one wouldn't argue or we certainly wouldn't argue that CAT scans aren't very useful in a diagnostic setting as our MRIs, as our X-rays. And I think we need to swing that pendulum back to the middle somewhere and say there are settings, lots of them, in which diagnostic sequencing today is incredibly beneficial, but we need a little more guidance and thoughtfulness to what it is we're returning and we as a community, a Caesar and the broader community need to get together and figure out what's appropriate to return in a clinical setting versus what really is still in the research realm and we need to study it a little more and figure out penetrance and risk and things like that before we're ready to give it back. And so I don't want us to get burdened with the criticism because we've swung too far to the right and not recognize that somewhere in the middle is a pretty clinically useful tool and I think we have to, but we as a community need to get back there and I don't think we're quite there yet. Bob and then Steven and then Debbie, is that right? Oh, and then Robert. Just to, I think it's been implied today, but then question and answer to Deborah's point about trying hard to document this, that to not, to start paying attention more to the comorbidities as opposed to the primary diagnosis. So the primary diagnosis, you may not be able to change, but you may be able to at least document changes in comorbidities, not only with the patient, but with the family. Steve Jaffee from UPenn. The summary here of the sessions was a great summary and captured many, most of the questions that were raised over the course of the day, but I think it was very much at the level of the trees or the individual questions as opposed to like, what are the big picture forest level questions that if there's going to be a follow on program to seize or what are those big questions that need to be answered? And I hope we can spend some time talking about really the big questions that really ought to organize the initiatives going forward. Just to throw some things that I've heard that I think are at that level, to go back to point Deborah was making a few minutes ago, but that others have addressed as well, what is the value for money, for effort, for all of that of sequencing? And related to that, I think what's the value of a more broad versus a more parsimonious or targeted approach to the use of the technology? How do we best implement sequencing into clinical arena so that it actually fits into clinical workflow and to clinical practice? Those are the sorts of big picture questions that I think might be the things to organize a program around. And under that, there'll be lots of specific questions to address, but I think focusing on what are the big organizing principles is probably to me the most important thing that I hope will come out of this meeting. That is what I was hoping would come out of this end of the day session, so that's great. So that's a perfect setting. And I think that several people have mentioned precision medicine and actually Cesar is a test bed for precision medicine because it is applying that you're talking about the diversity of problems that are being looked at by clinical sequencing. This is what you would look at by clinical sequencing as Robert showed, there was just one area that wasn't covered in Cesar. And everyone is looking at implementation, is looking at cost, looking at effectiveness. I mean, this is an inherent part. So I think it's really good to look at all of those and the broadness of what we're doing and if it should become even broader. I don't know, but it certainly needs to become more diverse. The one thing I'd like to bring up too is the variance of unknown significance. We have large scale genomic profiling capabilities now that perhaps if brought along with Cesar to look at these variants, prioritize variants of unknown significance would push them. I mean, years ago, people used to push sequencing centers to sequence the really important areas. Turned out to be really hard, but they learned a lot about how to do well. And maybe this is a good way to push some of the large scale functional centers that are emerging to look at variants of unknown significance and function on a large scale. They won't want to do it because it's too useful. But, and of course, no, no, because utility sometimes means that you have to actually look at one variant rather than looking at 10,000 variants that you could do really well, but they may not be 10,000 variants that are really important. So I think that sort of tying some of these things would be really useful. I just wanted to bring that up. Robert, then Ken, then Barbara. Yeah, I'm just trying to respond to your visionary winers idea. I think if you've spent a lot of time in the Cesar meetings, you know that there's also a lot of really enthusiastic skeptics in theirs. I'm trying to find the opposite of that, but which is part, I think, of the process that these slides are not quite capturing. The process whereby LC is cross cutting and integrated, the process whereby this iterative patient facing standpoint is balanced by the E in exploratory on the technical and biological side, the process where there's this implementation science emphasis for things that are ripening in terms of their clinical practice and the process of exploring new biological and technical endeavors. And I don't know why it's worked, honestly, because it's a lot of stuff going on at once. But I have to say, I think it's worked in Cesar and it's allowed us to make contributions that are greater than the sum of the individual parts. So that's really just building on Steve's, also Steve's idea of the big themes. And the one thing I would emphasize in those slides that isn't quite written down there is the dynamics of these cross cutting processes. So you've got Ken was next, and then we have Ian and Deborah. Sure, I'll follow up also on Dr. Shafi's encouragement to think on a larger scale. I'm Ken Offit from New York. I'm here this week in Washington playing the referee role, not the player role, because I'm on the CAP and we'll be here for the next two days. I guess it's really more cheerleader than referee role, right? Get my sports right. And then go over to the BSC for the NCI. So I'm saying this only to be positive. And I agree with what was just said. I think that you all have done a remarkable job at taking us through the wilderness. I would just comment though from the large scale side with 4,000 adults 1,000 individuals on your chart and cancer centers that are not the eminent cancer centers necessarily that are members of the consortium, but some of the other larger cancer centers will be publishing papers in the next months with thousands of germline data, with tumor data that on the cancer side, just looking forward, we're talking about tens of thousands of germline, large sequence data on capture from cancer centers at St. Jude's Memorial and Chicago and other of the centers that aren't necessarily represented, although the centers here are the leaders. And I think we've come a great way in looking at the germline from the sort of toxic, effluent waste of tumor normal sequencing to the treasure that we know that it is. And you're seeing at cancer centers this looking to Caesar to set the stage forward as you have done, as we are all struggling on a regular basis with annotation. We've regressed as you know in the cancer centers from exomes to panels and we will probably not go back to exomes and genomes for just exactly the reasons that were discussed here today with Cascade. This is the future. And I think that within the Caesar 2.0 is a wonderful opportunity of scale to test some of the hypotheses that are discussed here today in a captured phenotypic milieu, if you will. So I guess that's really the major comment. But I would just reiterate my view, again playing the referee role, I will say again probably over the next couple of days that what an extraordinary role all of you have played as I think it was said earlier today, setting sort of the higher conscience for all of us in this field, struggling with all of the topics that have been discussed today. I've got Barbara, Ian, Deborah, Sharon, Jim. Great, thanks Bob. So Barbara Kanig, UCSF. So I've been also wanting to return the conversation again maybe to some of the big conceptual issues following what Stephen Jofi said. And Bob, I appreciate your putting the whining, the whining straw man up there. But I think one of the other things that we need to do at this particular point in time is use this as an opportunity to rethink the role of the embedded LC component of a project like Caesar and to really think about that moving forward and how to build it in. And I was just spending, today I just spent, as I was listening, I was trying to make a list of the things that were important to think about moving forward. And one of the issues is that LC issues don't, they tend to be, some of them may be novel, many of them are recurring, whether they're solvable or not is an issue. But often they deepen over time or they constantly transform and need to be addressed in different ways. So the conceptual issues that I kept hearing over and over again was the conceptual issue of how the research clinical care divide has been troubled by this technology and especially the sort of regulatory formulas that we'd relied on are no longer working so we have to rethink those. We've started doing that a little bit in Caesar, I think it could be very much a part of Caesar 2.0. And one way to think about a piece of that I think is the issue of, well, managing the uncertainty of all of this is a piece of it. But the other issue is how much to move toward individualized care as opposed to guidelines. And then if we do move toward guideline driven clinical interactions, how do we, how much do we leave up to individual choice in terms of individuals making decisions? We did a lot of research about that in Caesar, at the beginning of Caesar, but then how do we now moving forward think about setting defaults in the system for larger populations? And then I guess my final point, Steven Jaffe earlier said that it was important to remember that one of the things that Caesar has done that perhaps is different than some of the other more purely empirical NIH consortia like this is the issue of focusing on some of the normative and conceptual work. So I'm hoping we'll also be able to really continue doing that. And that's particularly important in terms of going back to the goal that Pamela just talked about, the issue of how we're actually going to make sure that we do a good job of thinking about some of the issues of integrating social determinants of health into the genomic analyses. And I think we still need to do a lot more thinking about how to have conceptual clarity in the research designs that we create. And also conceptual, well, and then we also need a lot of very strong and careful work about how to get the right patients into the databases. So just a few thoughts about to reflect on. So I think I'll be brief because I think it was sort of echoed in what Ken and Barbara just said, but to me the forest or beyond the forest is the uncertainty. That's one of the big issues that echoes through all of these subtopics that we've been dealing with. And I think when I think of Caesar, I think of it as contributing to clearing up some of that uncertainty or defining what those uncertainties are and varying classification. But Caesar's strength to me is it's having its finger on the pulse of the patients and the clinicians and being able to translate that uncertainty in an effective way. We're never gonna get rid of the uncertainty completely. We know on one end of the spectrum of Mendelian disorders, that's a little clearer, but there are Mendelian disorders that are non-penetrant and there are modifiers and getting all the way to the ultimate possibility for this technology, which is using it in health, where most of the uncertainty will lie. I don't think we'll ever get rid of the uncertainty. We have to strive towards minimizing it, but I think where Caesar's strength is understanding those uncertainties and looking at how best to translate that to the populations that we're caring for, whether it be the patients, their families, the clinicians, the payers, whoever it should be, and I think that's where our strength and uniqueness really lie. Over next. So I don't know if I'm getting too much in the weeds, but it seems that Caesar has done exome and genome sequencing and so you have access to a cohort that's larger than any of the individual studies and I think it might be interesting to see if Caesar could look at whether or not people with questions about people with a gene mutation in a particular gene known to cause a disease, except some of them don't have symptoms and some of them do, to do pathway analysis around that genetic disorder to see what are the modifiers and by doing this with Caesar's data and combining across studies to maybe look at things that aren't exactly what was being studied, you may inform the million person genome project and the kinds of issues that might come up as you try to look across the different cohorts and stuff. So I mean it probably wouldn't take that much funding to do these kinds of analyses because they're all in silico if you could identify the cohorts and then ask whether they have symptoms or not and then you'd have two cohorts to study for the pathway analysis. Sharon, I had you next. Thanks for keeping track. I just want to make, I guess, three brief comments trying to do the big picture of you. First of all, I'd like to really echo what Heidi said. I know Heidi and I wind up being on a lot of committees together but I thought her points that I think we've gotten lost in the weeds of the fact that we're three years into what might be a really good diagnostic test for many indications should not be overlooked. We are really at the beginning of knowing what indications it's good for and so sometimes we think, okay, that question's answered when, I mean we've been looking at lipid profiles for decades figuring out the best way to do that. The second is that I don't think we are all believers. I think there are some fundamental questions. Like at my heart, I'm not convinced that incidental finding, reporting is a good thing and I don't know that it's a question that we've really answered given all of the downstream testing and our lack of knowledge of the attributable risk of those mutations, et cetera. So I do think there's some very big questions around genome scale testing that CSER is very well situated to answer both in further longitudinal follow up of our original cohorts but also in really thinking creatively about different ways to report exomer genome scale methodology. I mean the ACMG guidelines are just a guideline. We're actually not required to follow them and I think there are still many unanswered questions about the best way to do genome scale testing in a clinical scenario. So my comments or echo a lot of what Sharon and Heidi said. I would just add that I think we all agree that looking at some aspect of clinical utility is vital. Something that Katrina Armstrong and I have been talking about here is that saying okay, let's check the clinical utility of genome sequencing, it's a little like saying let's examine the clinical utility of the MRI. Well that doesn't make any sense, right? It has clinical utility in certain contexts and it has none in other contexts and what we still have to do in our field is we have to figure out in what clinical scenarios this really makes a difference in patients and document that and I think we also, I'm hoping that with a CSER 2.0 if it materializes that we wouldn't be tied to having to do whole genome or whole exome sequencing. In other words, comparing when a whole exome approach adds something or doesn't add something to panels would make a lot of sense. As we just heard from Ken Offit, people are not doing whole exome or whole genome sequencing looking for germline mutations and they shouldn't be because all of our experience shows that it really adds nothing. Those are the kind of things we need to find out but we can't take a gauzy approach to saying, well, let's just look at whole genome sequencing. I think we need to be smarter about it than that. Gail. So similar to some of my fellow co-PIs, I mean, I think when we went into CSER 1, there was a very broad, like get the genome to the clinic, find out when it works, when it doesn't work, what the obstacles are, overcome those obstacles and try and measure outcomes and it was very broad and as I reviewed the publications that we've had to date, two thirds of the publications from my site which includes the workgroup papers, of course, were never anticipated. They're not aims of the grant. They're problems that came up that we put together a group of very smart people across this consortium to address and were able to come up with thoughts about or solutions to and I think this field is moving so fast we can't anticipate what all the questions are going to be. So we need a big framework of questions but we need nimbleness as well and we do need that core of really talented people who are thinking hard about this. John. Jonathan Berg, UNC Chapel Hill. I'm gonna try to answer what Dan wrote and asked us to do early, which is to try to point out where CSER is different and distinct and has its own sort of place in this ecosystem of genomic medicine and I think that it's because we complete the link between the patient, their clinical provider, the lab and then back and I think we complete that circuit. I think it gives us the opportunity to do novel work with communication to the patient of what the testing can accomplish, what it can't accomplish, what are the benefits and the risks, all those aspects of it. It allows us to look at some questions about how the clinical laboratories and the physicians have to work together in a sense, collaborating on the analysis to figure out what is the relevant variant here, how that information then gets communicated back via the provider to the patient and it can be a medical geneticist, it can be a primary care physician and really exploring a lot of those things gives us a chance to really make an impact in terms of the communication analysis and then incorporation into clinical care and I think that's where Cesar hits the sweet spot and is really critical. Thanks Bruce Korr from UAB. A few things that occurred to me just listening to the conversation today. One is we heard a very powerful description of the power of networking among patients and I guess I would also add networking among our colleagues in medicine. One of the things for example that we heard about the DOPA responsive dystonia story that is really not such an obscure neurologic disorder but I don't doubt that it fails to get diagnosed a lot of the time and one of the things that struck me as we look at exome or genome sequencing is every once in a while you make a diagnosis that you kind of kick yourself afterwards and say why didn't I think of that and the answer is usually you're at the end of a long line of people who didn't think of it and so it's occurred to me that we ought to be trying to push the experience that we have with some of these disorders back into the clinics and for example search for in clinics of people with dystonia just for one case in point for individuals who should be tested not maybe have their genome sequence but tested for conditions that are either ultra rare or just somewhat rare that we come to realize are explaining some of the phenotypes that had prompted the diagnostic odyssey so short circuiting that odyssey may not only occur by sequencing all these people but also by improving the communication to neurologists and radiologists and all the other specialties who see these patients so they think of these diagnoses more quickly than they currently do and not just by publishing them because I agree that people are buried under a pile of unread manuscripts so coming up with clever ways to push differential diagnosis closer to the point of care I think would be one lesson that the CSER and other genome sequencing projects may be able to impart. A second point is that I think I've heard a lot about trying to interpret individual variants in terms of their phenotypic significance and I certainly appreciate the complexity of that but it also occurs to me that they don't act in isolation and do we need to be thinking of constellations of variants and I don't mean by that thousands I mean two or three even that maybe the reason why you don't get the expected phenotype from one particular variant is that it happens to conspire with one or two other variants that are sitting there and so we need to think in terms of combinations I think not in terms of individual variants. A third point gets to what I guess is the glass have full view of the cascade issue which is maybe it's not such a wise thing to see a variant and then embark on a new odyssey looking for a phenotype in somebody who isn't obviously affected. On the other hand though, it'd be nice to think that there's a way that the genomic information can live somewhere in the medical record so that maybe even years later when a person appears with a new symptom and somebody's trying to figure out why it's there that the genome sequence is still at their disposal to help inform the differential diagnosis and although I don't think that is good I think that actually will ultimately decrease costs for that individual but it needs to be done in the context of their phenotype not just in isolation as a consequence of the variant and I guess the fourth thing is that with all the concerns about costs it seems to me that the cost of actually sequencing continues to be on a downward trajectory. The cost of interpretation a lot more complicated and the cost of counseling complicated too and I guess the question is how can we squeeze the costs out of some of those other aspects as people are trying to do squeezing the costs out of sequencing. For example, is the paradigm of the physician and or counselor in a room with a patient talking about one thing at a time which might be argued to be the gold standard of how counseling is done. Is that the way of the future? Are there ways of doing this more quickly and efficiently and at much lower cost using tools other than face-to-face interaction for each individual who needs counseling? So looking at new ways of imparting, interpreting information I think is another opportunity. Levi and then I'm going to actually call on former and current members of councils and ask Rex to make some comments and also Dan but Levi I think you are next. Thanks, yeah so just thinking through again the allergy to genomics in genome medicine that's actually out there. Actually it's both in the basic science and the clinical side and how to approach that from a healthy skepticism or even if I believe something I'm open to changing my beliefs. And I guess one way to think about it is so we've gotten a customer using the term genomic medicine in various contexts and maybe we should eradicate that term and just talk about medicine. And medicine and I think imaging has been a great analogy because of course when you do an abdominal CAT scan and somebody has a ribocytral pain you don't just do a scan of the gallbladder you do a scan of the entire abdomen and often you throw that away. So I think to a certain extent genomics is a complex technology like imaging as Heidi said that has a lot of applications but in the end we don't necessarily want to make the same mistakes that were made and actually they weren't mistakes at the time because it was just kind of a luxury of just having imaging proliferate to the point where we're so over saturated and people were just reflectively sending all kinds of images and we have to now kind of scale that back because the costs were insurmountable. So if we were able to actually be smarter and be able to say, all right, well, we picked the questions or the diseases or context or what have you and if in the end what we end up having is a mature clinical management pathway or a maturing clinical management pathway where we actually understand where genomics is useful and where it's not and whether it's an exome or whether it's something smaller and we just sort of say that's our goal. Our goal is just to figure that out. It's not to sort of say, oh, everybody's gonna have their genome sequence or it's genomic medicine which implies that we're starting with the premise that the genomics is of paramount importance. We're just trying to say, this is obviously an exciting tool but we recognize that it may well be that in the majority of medicine we don't use it or at least we don't use it right away until well down the pathway, but that's okay. I mean, I think it'd be a victory if we ended up with, if we saw that the way that clinical pathways emerged was that in many cases it was obvious in the decision tree when and how to use genomics but recognizing that in many cases it's not gonna be useful, that would be okay. And that would be sort of, I think it would still be a huge contribution to society if we took that approach. Rick, would you be willing to step up? Yeah, I'm Rick Chisholm from Northwestern. I guess my thoughts fall into a few categories. One, I think, thinking about this whole idea of, and I'm gonna use the word genomic medicine although you can ignore the genomic part if you want, because, and it's come up several times today, one of the things that Caesar is doing very well I think is generating evidence for the value of genomic medicine. It's doing it in case by case basis but that's probably about the best way you can do it is a case by case basis. And I worry a little bit about how well that scales but I think when we're always asked, well what's the evidence that genomic medicine works, I think we can point to a lot of the things that Caesar's learning and saying there is evidence here that genomic medicine is really working. So I think just keeping the development of that evidence is an important part of what we need to be doing going forward because when I talk to people, so I guess I'm a true believer, but when I talk to people, one of my colleagues in general internal medicine at Northwestern, they're not all true believers and they're skeptical and what they keep saying is we don't have enough evidence and so I think I'm focused on the evidence piece and again I think I've said Caesar does that really well. The other area where I've thought about it and I guess this sort of falls into what Jonathan was just addressing which is sort of what's unique but where does it overlap with the other programs that NHGRI has? Of course as somebody that comes from Emerge, I'm constantly thinking about how this might overlap with Emerge. There certainly are some things that overlap and just the last set of discussion is turned my mind in a curious way. I think it might be fair to say and my Emerge colleagues that are here could agree or disagree with this but I think Emerge has had sequence envy of Caesar and continues to have sequence envy of Caesar because in Emerge 3 right now at least it looks like for the near future we're gonna be doing a panel. So I was really struck by hearing that Ken said that on the cancer side they've regressed to doing panel so maybe I should give up my sequence envy but I think to think about the implementation and a real opportunity to compare the sort of whole sequence approach that Caesar is taking with sort of the more panel approach that Emerge is taking I think provides a really nice opportunity to think about experiments just an experiment, how do those two compare going forward? We're both implementing, we're both putting stuff back in electronic health records. I think there's a nice overlap the groups that are doing electronic health records implementation on both have already collaborated really well. I think we've seen similar kinds of collaboration on the LC side but I'm struck about this difference between sequence and panel so maybe that's something we should think about embracing the similarities here but also taking advantages of the differences going forward so I guess those would be my main thoughts. Dan? I really have very little to add to what's been said today and what Jonathan and Rex have just said. I think the, as I've said, sort of inside conversations through the day today and before, if you'd asked me three or six months ago it was not very clear to me what Caesar would add to what Emerge was going to do. Now that we're sort of starting to put our foot in the water, not just our toes, with respect to sequencing in Emerge it's totally clear to me that the things that have to be done in order to interpret and then deliver back to patients the information that we're gonna generate with our panel in Emerge and that you have generated with various panels across Caesar needs a huge amount of work and the Emerge centers are just not equipped or set up or have the bandwidth to do all the things that need to be done. And you've heard through the day about specific experiments that Caesar is doing and should be doing in the future so I think that I've come away from this day with an absolute conviction that there is a role for a big role for Caesar too in still understanding the relationship between an individual patient, a sequence variant and the healthcare system. And I really resonate with what Heidi said as well that this technology is so new that everybody in this room sees all the problems. And I think we, this is human nature. I think people do this all the time. I get asked about a ridden of things and I sort of take for granted that everybody knows what I'm talking about whereas in fact, most people are just not familiar with the potential that this technology can play in the care of real life patients. And so you have to not lose sight of that by drowning yourself or we have to not lose sight of that by drowning ourselves in the potential problems as we go forward in this technology that changes every six months. So clearly, there is a role for exome sequencing, for genome sequencing in people with difficult to diagnose, diagnostic odysses. Deborah and I have been exchanging lists through the day today of diseases that are often overlooked until they get to the 12th doctor and we have a list of somewhere between six and 12 diseases so far. And so those are the kinds of things that we need to emphasize perhaps not to NHGRI staff or to counsel but to the broader community as you sort of start to say where does this technology fit in? Not to oversell but not to undersell as well. I'm gonna turn to Sarah but I also would like two other council members who haven't commented yet at the end of the day. Amy and Shanita if you'd be willing to make some comments after Sarah. Sarah Skull and Baylor College of Medicine. So I just wanted to speak a little bit to the genetic counseling aspects and hopefully answer somewhat your questions regarding the psychological aspect of it. So I think one thing that we've learned a lot of our initial work has been done on informed consent through the project that was led by Barbara Bernhardt. One of the things we learned in that project was that once we started returning results that taught us something about how we should do the informed consent. And so I think now our next steps is really moving into looking at the return of results process. But downscale I think one thing we really need to look at is the actual patient understanding of this information. So we have our perception of how well the patients are taking in this information but I think it's really important for us to actually ask them what they are taking from this, what they're going home and telling their family members, what they're going home and telling their doctors. And so I think we're in a good place to do that. And also speaking to, as we do all of this I think we streamline each of these steps and I do think that they're because of the lackage of genetic counselors we are gonna have to look at ways where maybe genetic counselors aren't doing all of these pieces and so things like virtual genetic counseling have come up but I think that we need to really figure out which of these areas genetic counselors are most important, where their role is significant and where some areas where some adjustments can be made but I don't think it's really until we streamline each of these steps that we can determine that and what should go into things like a virtual genetic counseling. But I think going back to the communication piece that Cesar really is in a unique position I will say in the genetic counseling community we are probably the largest group of counselors who have given back these types of results. There's some clinics that still haven't touched this test and even with the informed consent information that the genetic counseling community was really hungry to hear what we had to say about it. I think the same will happen for return of results so that's just a little bit about the genetic counseling piece. Amy and Shanita and then Barbara after them, okay? Is that all right, thanks. Yeah, so first of all, I think this has been a really, really fantastic day. I can say similar to, I guess, I forgot who just made that comment. But maybe Dan made the comment but similar to Dan I would say that before this meeting as a Cesar grantee I was like, yes there should definitely be a Cesar 2.0 but as a council member I was kind of like, I'm not 100% sure and I think the question that kept getting floated by is what is Cesar's sort of unique contribution and what do they add to all the other consortium and I think after today I'm fully convinced that there's a real role for Cesar 2.0 and that there is a unique contribution that we have been making and that we continue to make and I think that really centers around the generation of evidence. And it's really interesting because first of all, as somebody who does the project three, the LC component of it, a couple of observations. One is that this has been a really unique and fantastic opportunity to embed LC within the larger context and I've worked on several embedded projects. I think this is one of the most, probably the most successful one in terms of how it's gone and in terms of the contribution that that part of the project has made. And it's also been a little bit of an expanded view of traditional LC research and it's kind of moved some of us, or at least me, over more into the outcomes side of LC which has been a neat experience and also very valuable I think for the work that we do in terms of thinking conceptually and normatively about some of these issues. My personal experience has been when we started the project, I really thought that the whole point of the project was to figure out A, can we do it? Can we actually integrate genomics into the clinical context from a technical perspective and B, what are the risks involved? And I think there was a lot of concern about psychosocial risks and things like that and we focused a lot on that and as we move towards a CSER 2.0, I think I'm at least shifting to yes, we can do it and sort of the question now is how do we do it better? And then from the outcomes perspective, I think there's, we've gotten a lot of data about risk and it hasn't actually shown that there's tremendous risk although that's of course a more nuanced answer than I just gave. But really the main question now moving forward is what is the value and really a focus on value? What's the utility and for whom and in what circumstances? And I think that's a really interesting, challenging and important question that needs to be answered. I think we're starting to answer it now in the projects but I think we're just sort of, I would say in the hypothesis generating phase and I think that we really need to start to move that forward. So I think it's a really exciting time actually and look forward to what we continue to do. Sure, thank you. So I would just echo the positive comments that have already been made about the significant progress the consortium has made in terms of several issues, one of which is developing models of informed consent under different clinical scenarios. I think that's been really important and is a really useful model that can be applied more broadly within the genetics and genomics research. I also think CSER has been really effective in terms of developing and implementing a model for conducting embedded ELSI research. I think that the way in which it's been addressed and the way in which the group has worked effectively to ensure that ELSI issues are incorporated and executed and studied appropriately is really important and very useful. I also think that CSER has been really effective at identifying key psychosocial and behavioral outcomes as part of the ELSI portfolio research and I think the group has worked very effectively to integrate these outcomes across the consortium and having seen consortium that haven't worked really well, I think that's sort of these issues. I think it's a really testament to the synergy and cohesiveness of the group that's really important. I'm really pleased to see the data and the on recruitment and retention of ethnic and racial minorities. I'm really excited and that this issue which was raised at the last meeting has been addressing a very forthcoming and direct way. I look forward to seeing what will happen next in terms of actually learning from the lessons within the consortium as well in the general field of minority recruitment and retention in other settings and how this consortium will think about and develop strategies to actually develop some concrete solutions that have the potential to increase the diversity of the sample populations. One of the things that I haven't heard mentioned a whole lot is what happens with these results in families in terms of how are they discussed with family members and what understandings and misunderstandings do family members have about them and in particular also with regards to the children and adolescents what happens with the results for them and what support if any may parents need as they begin to discuss some of these results with families. I bring this up because I was reading a transcript from one of our return of result sessions in which the mother said to her 18 year old daughter sit up and pay attention. This concerns you as the daughter's holding the copy of the report and she says what am I supposed to do with this and it occurred to me what is she supposed to do with this and do we know what happens with these things in families and how is this information kept and stored and shared with the PCP and what is the role in fact of the primary care provider for acting on results both relating to the underlying diagnosis and also to some of the secondary findings particularly with regard to pharmacogenetic findings if those are returned and also carrier status findings. So Gail Henderson again and I'm gonna be real short. I loved what Chinita said. I almost didn't get up because a lot of what I wanna say reflects things you just said but I wanted to introduce a little historical perspective because the LC program went through a bit of a crisis maybe five years ago and it wasn't quite clear where it was going and one of the things that's happened during this time has been embedding LC projects in a number of different consortium and to a certain extent I think this is redefined and rejuvenated what you all think certainly what most of you in this room think LC means it isn't actually it's not the LC of my grandmother actually not it isn't but I think it's I think it's very appropriate that now psychosocial measures behavioral outcomes I would say health economics as well I know it's not the PC but I would say that all is under an umbrella of thinking about social implications and I also wanna say one other thing and that is I've always thought of LC as thinking broadly about justice as well as thinking about autonomy and beneficence and I am most thrilled by today's it seems to me train that's now I think firmly left the track with an engine of genomic necessity that means that now people who are non-Europeans are gonna constitute a lot of the patient populations who are recruited into studies it just has to happen that's really just and it's scientifically valid it's scientific imperative so I think that some of the things that I've had most admired the LC program that Wiley runs and others as well about justice with health disparities it's going to come out of if there is an LC 2.0 and I think that's just brilliant so thank you that's the train leaving the station not leaving the track I hope Oh, oh, whatever Hi, Greg Fioro, main Dartmouth family medicine residency and I apologize I've been fairly reluctant to get up and say anything because I was sort of an insider and then I'm squarely an outsider but I think a number of things through the day for me were quite striking the first is where NHGRI and the genomics community is now in comparison to where it was 10 years ago with regard to coming to grips with some of the clinical implications of the technology as they emerged and it's really quite remarkable where I think it was if Joe's not still here at a Nichepeg meeting about a decade ago I attended someone came up to me after I gave a talk for NHGRI and said you people you're just like the big ship going through the ocean and creating all these dead fish and you don't deal with a dead fish it was a really funny analogy but I think we're dealing now with the dead fish basically the technology moving forward and creating a clinical challenge in its wake in a structured way which is fantastic that's great so the other part of me says we still have a long way to go to move the primary care community with what I've heard today in several respects one is that the evidence it sounds like Caesar is generating is powerful and anecdotal regarding rare disease to some extent more common conditions but largely still in the realm of rare conditions and until the corner is turned to deal with common complex conditions that really motivate primary care is day-to-day living I think it's gonna be hard to garner their attention one of the issues that struck me is Caesar it seems like is powered well and equipped well with the really brilliant folks that are involved to answer certain types of questions about sequencing but not well-powered in terms of the number of people that have been sequenced and who have good phenotypic information to answer really other really critical questions and so some way to sort of intersect some of the large cohorts that are being assembled and maybe already exist in some of each of your other's other programs where there is the power to answer some of those questions with some of the great work that's going on in Caesar would seem to me to make a lot of sense the other issue that was striking to me was that we made it through an entire basically two hours of discussion of analytic validity clinical validity and clinical utility and until Bob Nussbaum mentioned positive predictive value nobody had used a term that the standard evidence-based medicine world of internal medicine and family medicine usually thinks about in terms of describing the results of tests and so I think there is still an issue of a language gap between the genomics community and an understanding gap between the genomics community and some of the other areas of medicine that are pushed to population health perspectives on a day-to-day basis and I know my family medicine colleagues every day are thinking along the lines of managing the population health of their panels and that's really in the common complex disease world we care about positive predictive values numbers needed to treat, numbers needed to screen and those metrics and so I would encourage the Caesar investigators to try to involve folks from their primary care well I know Bob has done a lot of work Robert Green has done a lot of work in that domain and it's a challenge to get primary care folks to attend and provide input but I think really to have this grow beyond the genomics and perhaps some of the folks in special areas that are interested in genomics at this point in time you're really gonna need to engage those folks and speak in their language so anyway it's been a great day, thanks. Thanks, Brie. Hi, I'm Milder Chow at Stanford and also as a member of the advisory panel I just wanted to throw in my two cents as well I appreciate what Heidi and I think Dan said about sort of the not throwing the baby out with the bathwater but it also struck me today that the Caesar program is still quite exploratory and so I think another part of that not throwing the baby out with the bathwater is we can't assume that we already we shouldn't assume that we know more than we know and that we actually are ready to do this at a clinical stage and at a clinical level we did mention analytic validity but I think there are a lot of issues that were brought up that sort of make it seem like we've assumed that we've got we know what clinical grade sequencing is supposed to be and I don't think I think we're sort of jumping head with a lot of those issues and I think with the exploration that we still need to do we need to pay attention to the gaps such as the racial and ethnic diversity we can't assume that we know a lot there and so I would just urge that to sort of think about what it means to explore and not just to explore under the lamp posts we need to broaden the light out and look in places where we haven't looked before that also includes the LC parts where and I think that that means thinking about what the limitations of an embedded bottle of LC is because that kind of focuses things under the lamp posts I think we need to look beyond that. Brad? Yeah Brad, Ozenberger, Washoo Genome Institute. Wow, what a big a lot of big a lot of big topics discussed today but from a strategic planning standpoint I'm wondering how helpful we've been to NHGRI staff and as you one way to cope with that might be to kind of break this down, break it up tackle some of these big issues, maybe one by one but I just want to advocate and others have said this too that a big part of the success I think of CSER has been the interdisciplinary nature of project one plus project two plus project three and I want to advocate that that's something that I hope will continue. Everything I think was a great day, a lot of good ideas what we were asked to consider is the in the few minutes that are left would be impressions regarding a consortium versus more independent projects and considerations about somewhat format as well as content. So the consortium format seems to be highly effective in bringing together groups of people with different perspectives but I think the smaller awards like the R1 that could almost be like the big cooperative groups RFAs for projects that relate to what the consortium is doing and maybe using their data to do other kinds of analyses it is still useful. I don't know they have to be totally separate RFAs but they could be related to CSER's work and build on that in different ways. One of the aspects is how does the consortium change over time or are you in it or not in it and so that's an aspect to continuing with the consortium model. Yeah, I just to follow up on that I think that our consortium model has been surprisingly at least to me successful but I do think and I've argued for this within CSER but I'd like to put it out there for the larger group. I think we could be far more open to other partners from academics, other types of grants, other type of industry partners, industry partners in the pre-competitive space through FNIH and other mechanisms than we have been in the past and we could even further increase our influence by doing that. There are a lot of mechanisms for doing that that we can discuss offline but I think that that's one way we could concretely further multiply the impact of CSER in a subsequent iteration. So I think as we move to closing up being aware of the time I'm gonna echo Brad's comment that you have given I think as quite a challenge in terms of processing and thinking about this and I did and I will appreciate that his advice which is something I've been thinking about for most of the day is how to break it down and where are the things to go and that's certainly something that I think we're gonna be doing a lot of over the next I don't know weeks as we discuss this but I think one of the things that I'll take away is there is a lot of discussion both in terms of really getting to the specific of that interface between the patient and the clinician that we've been talking about but then there's also been a lot of talk about the learning health system and improving that and developing that and I think some of those are gonna start to be axes where some of this is gonna perhaps be ways to break it down and it's gonna be interesting and also just to thank all of the speakers and reactors and moderators for doing such a wonderful job and I'll just add to that that I'm very thankful for everyone's feedback today I think we've heard a lot of ways that current CSER could sort of look at our consortium and make it better and we've also heard a lot of ways in which CSER can sort of look outwards and make ourselves more relevant and really more strategically aligned with what's needed in genomic medicine I do wanna thank also the planning committee and the people who helped with the logistics those people who tweeted thank you very much and then especially all of you here today I do wanna remind people that we will have a workshop report that includes the feedback that we've heard we definitely want to make good use of the feedback we've heard today and make that information available to the community I'd like to turn it now over to Eric Green for any closing remarks


Welcome back, everyone. We're here live in San Francisco for the Red Hat Summit. This is Silicon Angles theCUBE, our flagship program. We go out through the events, and extract a similar from the noise. Now we're going to get some experts to talk about the cloud and government NERMA meta, cloud engineer, strategic innovations group at Booz Allen, and Jared Cottrell, engineer at Booz Allen. Welcome to theCUBE. Thank you. So guys, talk about obviously government 2.0 has been discussed going back, it's been played out, but the reality is that the government has been a adopter of all new technology. I interviewed some guys back in the day on who built the first big data platform from scratch, the squirrel founders, and back then there was no Hadoop, so they had to build their own. So the government's always pushing the envelope on stuff, and it has to be turnkey. So I got to first ask you guys, Jared, we'll start with you. Cloud adoption in the government, where are we? Share with the folks who aren't paying attention to the government space. Are they deep in the cloud? Is it just a throwaway? Is it kicking the tires? So it depends, I know people don't like to hear that, but it really does. So I think there's a lot of organizations in the government. It's a big group of people, and I think we're finding they're all at different maturity levels. Some are seriously still just in the strategy part of their cycle. Some are doing proof of concepts. Some are actually implementing cloud, public, private, hybrid, what have you. So we're in the mix of all that. Other folks are even farther than that. They're really gonna get cloud management, cloud broker, and that's really where we're starting to see a lot of interest, a lot of growth, and it really just depends on where they're at and their budget cycles, and what risk we're willing to take. So what's the big driver? Obviously the cloud has economic benefits. We've seen Amazon, it's really been a forcing function for many, it's been that tide that's been coming in and taking some territory down, certainly in the enterprise, and I'm not sure about the government, but it really forces the people to say, okay, cloud has some benefits, but you can't just plug it into a public cloud because there are some security issues, other legacy issues, the government certainly has a lot of legacy. So how do you guys talk about that and how do you guys identify the disruptors in the emerging sector, how do you balance it too? So some folks think virtualization is in the clouds. That's the first thing is helping them understand what cloud really is, right? And I think a lot of the drivers end up being around money, obviously, but what comes with that is the automation, right? So they have a lot of backend manual processes that we're trying to fix, and them as well. And I think cloud really benefits a lot of that, whether it's public or private, right? I think in general, another big driver, and I think this is where cloud management brokering comes in, is there's a new workforce out there. There's new buying patterns, there's new consuming patterns, a lot like us on mobile phones now, it's just changed our lives. I think that's starting to get into the enterprise and we're gonna see a lot of procurements around that kind of thing. So two things, the trends of pay consumerization is happening and with a cost pressure. But also, is there a development focus? Do they say, hey, we have developers, do we wanna add on to, what's the developer impact in the government sector? I think it's even a lot like we zone, right? We develop applications, we have developers that work with clients who do different things. They want them to focus like a data scientist, right? You don't want a data scientist working on infrastructure, right? Your application goes down, they should not be spending their time working with Amazon, building the servers and all that. So I think that's where a lot of the cloud comes in, the platform as a service, especially some of the open shit stuff that my friend Norm here is working on. Absolutely, right? It's about the efficiencies and getting it quick and having them focus on what their role is. Before we talk about some of the procurement disruptions because certainly Amazon's lawsuit against Intel was very telling, we covered that on theCUBE. Norm, I want to first go into the tech side, talk about some of the, what's going on under the hood? Right, it's a lot of interesting new dynamics around automation and orchestration. It seemed to be the hot areas. Could you give us an update on where you think that market is? Absolutely, so last year we won the Red Hat Innovation Award for combining JBoss rules engine with JBPM and Fuse to automate the underlying infrastructure as a service. And since that point, I mean four or five years ago when we started doing this, elasticity orchestration was still very manual and no one was at that point. Now, putting business rules in, automating workflows is just accelerating tremendously. The space is really ripe for an open standard spaced view of how to create application stacks, create environments that are standardized and being able to be deployed in multiple environments at the same time. So interoperability, migration, ease of deployment of code is definitely where things are going right now. Underneath the hood, there's a huge race right now between different containerization platforms, OpenShift, Docker, there's Cloud Foundry, all these other V apps, things like that. And it's just, it's a very exciting time actually because finally we're getting to the point where the gap between your developers and the infrastructure and your production code is becoming smaller and smaller. And really we're starting to actually see the benefits of where Cloud is supposed to be. Talk about the customer use case, let's walk through something. So I'm a customer, I have a lot of Red Hat, I've been working with Red Hat, Red Hat has a lot of presence, I might have some NetApp drives and some EMC drives, a lot of legacy stuff. Now, so boom, I got to go to Cloud. Cloud is enticing, it's intoxicating, it gets your attention at many levels. But then the reality is, oh, how do I get there? So it's like, I want to service the Red Hat component. I have hardware, maybe there's different configurations on the software side, assuming SDN and all that stuff gets taken care of. But I'm a CIO, I'm a practitioner, I got to get this to market. What are some of the critical challenges and opportunities and technical points that they have to think about? To go from, oh, I have Red Hat, maybe some vendor servicing it, maybe Booz Allen servicing it, maybe another vendor is doing the services. But I got Red Hat, what do I go to the Cloud? Right, so now Red Hat has lots of different options in terms of Cloud service providers. There's Amazon, there's Microsoft Azure, there's the recent announcement of Google Cloud Platform being able to put your Red Hat subscriptions there. So there's lots of places where you can put those RELL licenses or entitlements, if you will. However, I think what we're really seeing is our customers want to put something on top of that first. And that's the Cloud Brokering Platform. So what we're building out and what we're seeing our customers demand is a way for their internal customers, their developers, their operations teams to be able to provision their systems to whatever Cloud Platform, whether it's internal, external, legacy, PaaS, SaaS services, and be able to understand the exact costs for each of those options and make the best decision based on the needs at that time. That over time will allow their internal customers to migrate and start adopting Greenfield projects in new Cloud environment. Okay, I got to ask you the reality of OpenStack. Okay, a lot of people saying, I love OpenStack, I can look under the hood, Lego blocks, composite applications, all that stuff goes on and on. It's a great conversation. However, the reality is OpenStack is emerging. A lot of folks in the use case I mentioned like OpenStack because they can look under the hood. They're not necessarily up to date on where it is in its life cycle. So where are we with OpenStack? How viable is OpenStack today? And where are we on that vision? Because it is appealing to those folks who have these environments. They see OpenStack as kind of like Amazon, but I could get some customization. So can you share your opinion on that? Yeah, sure. So a couple of years ago, we started looking at OpenStack and no one was really talking about it. It was still very new, very beta, if you will, software. A lot of our customers weren't even looking at it. We were just starting to look at it. And there was still some shakiness about what direction the industry is going. CloudStack, OpenStack, what flavors. All of a sudden in the last, I'd say about four or five months, it is just blown up. I think that's because, and this is just response from the market, OpenStack is finally at the point where it's production ready, it's vetted. Our customers are more familiar with it. And I don't think- There's production deployment examples out there. There's absolutely, but you know, the bigger the customer base, the bigger the production area, the better, right? And so I think we're at this turning point right now. 2014 is definitely going to be year of OpenStack in that sense. We see a lot of our clients coming up to us asking for it, asking for information about it, figuring out what the support models are for it, what flavors work for their environment. So, I can say that honestly- Certainly Red Hat's got a huge opportunity with OpenStores. I mean, I have a lot of people trying to, you know, block them into the wall, if you will, to use our NASCAR example. Everyone's got their wheels on their cars, they're speeding down in the race right now. Jared, I want to get your perspective on it. Is Red Hat at risk?  What do they need to do to win them? I think their model works how it is, right? They support the community. They become the big name just to add to normal. So a lot of our clients, you know, they get a little shaky when you say OpenStores, but with a name like Red Hat behind it, let's not discredit that, right? Now there's a trusted name, there's support behind the product. I think that perspective, especially in the federal government, we're going to see a lot of adoption. We saw that with RHEL, what they did with Linux. I think we're going to see that same thing with OpenStack. To me, that was the, that's the biggest hope to get over. We're going to start to see OpenStack on the approved product list of all these agencies. They can't use it unless it's on that list. So we're really moving in the right direction. Well, I mean, I always get this question all the time, hey, is Red Hat blowing it in OpenStack? And I say, you know, if you look at the contributor list, not like in the last place, they're up in the top, if not number one. So that one, they're contributing a lot to OpenStack. Two, their brand value inside enterprise and government is massive. I mean, it's not like they're a bit player at all. I mean, they're pretty solid. So if you look at the competition, they're jockeying for that brand value. They don't actually have the same as Red Hat. So Red Hat has a huge opportunity right now, I think to be the leader of the community. Do you think that they're doing that right now? Being the leader, could they be doing more? Cause they're kind of a humble company. I mean, they don't do a lot of grandstanding, but are they at a good spot, guys? And what, if not, or what could they do better? Yeah, so I think this is where Booz Allen, as a, you know, a system integrator and a partner comes into play. We not only look at open source technologies and what Red Hat provides, but put together the use cases and the architectures and deployment models that will work for our clients. And really vet it and put a big check mark on it, saying yes, this will work for you. And I think, yes, Red Hat is kind of humble, but they are a recognized name within the open source industry. They basically represent open source to anyone on the street that doesn't know what open source is. And in the long run, they have a very good reputation in terms of supporting new innovations and new technologies and their own communities first. And I think that's a really powerful message that we can bring to our clients for long-term stability and innovation in the future. Terry, talk about the procurement cycle. We mentioned earlier, kind of teased out the Amazon IBM thing, which is kind of when the losses are on the bid. But to be more general, cloud has changed the cycles of procurement, right? It changes the implementation time to value. So the old days, when we were, I was just getting into the business.  Now you're seeing acceleration. What does a current deployment cycles look like? What, give us a taste of, what is it like, I mean? So the biggest challenge right now is, especially in the federal government, is how do you buy that stuff, right? Because it's not a typical, you know, paper use using the cloud. They don't have credit cards. You know, that's really the big hurdle. Not a lot of shadow IT in the government, is there? And there's, yeah, absolutely. Well, that's one of the drivers for CIO to broker cloud to their organizations, right? But the key thing is around, you know, how do we, how do they buy it? So that makes the procurement cycle a lot longer.  Those procurements just keep getting delayed. I think you guys could be the shadow IT organization for the government. Because they have to bring it into the house. So, you know, I think, like most big procurements, because what we're seeing is these large agencies, these are really big procurements. These are three, four, five, 600 million dollar procurements. It's going to take a couple of years to get those out. A lot of engineering as well. It's the requirements, right? How do we, all the different cloud providers, you know, license a little bit different. Their cost model is a little bit different. So until those become some sort of standard, I think we're still looking at another one or two years until we see really big adoption. Well, you know. Because of that. There's a huge process. The Obamacare website going down certainly puts a face to the average consumer out there who might not be in the weeds. So what do you guys say to the mountain, I want to get into that particular use case, but that kind of makes the government look like they don't know what they're doing in terms of technology. So how do you guys help the government kind of get faster and more relevant there? Not necessarily that website, but in general, for the tech. What do you guys do as a firm? So a lot of it is just, you know, by nature of being a consulting firm, we have a lot of clients, a lot of lessons learned. So we try and share that, right? We try and replicate how other folks are doing it. So we do have commercial practice. So we share a lot of that. We help push them towards some of these new procurement models. Granted, you can't change the government procurement model, but there are ways that you can work around and do different kinds of approaches. So a lot of it is just, as a firm, again, sharing lessons learned, helping them drive to different ways to procure. The technology is not always the hard part. It's there. We know it works to some extent. So Padma. How you integrate it? How you put the pieces together? So Padma Warrior was saying on her keynote this morning from Cisco that with SAS exposes things like these little scabs and like a website example. Highlights it versus, you know, if it's a software out of the box, but that's kind of an interesting anecdote. But more importantly, she said people would be buying outcomes, guaranteeing SLA around outcomes. What do you think about that?  Is that reality? Is that fiction? Is that soon to be 10 years out? Or is that just vision? What's your take on that? Well, internally within the government, SLA's tend to be a lot trickier because the business drivers are a lot different. On the incentives are a lot different. It's not profit that drives things and reducing costs. It's oftentimes ensuring services and maintaining mission criticality. So we see cloud as, and procurement of cloud services as a way to kind of reintroduce good SLA's back into the procurement models. And as more and more of these systems that we integrate with become services, those procurement models will reflect that. And those inherently come with those service level agreements. So hopefully this will be a big driver to ensure good consistent agreements between internal customers and the providers of those services. So I think who owns SLA will be key to. Literally all of our clients, all of our major government clients that they're asking themselves, they're either going to become a broker themselves to these different cloud providers. The provider owns SLA. Or they're going to go to a third party maybe like Buzan or someone else where they go and just buy those services. We own the SLA and we're responsible for that. So I think you're going to see those two paths sort of come together at some point. Guys, great stuff here. Love talking about what's in the front lines and the trenches around the cloud, certainly in the government sector. We don't get a lot of that down the cube, especially private, but their business to government is getting a consumerization trend hitting them right in the face. So that's great. But I want to ask a final question to summarize out for the folks watching. What's this show all about? I mean, obviously you got IBM here for 10 years. Cisco just showing up for the first time. Open Source on its, some argue sixth generation, fourth generation. The commercialization of open source is not stopping. That train is like barreling down the tracks. Why is this show? It's so important right now in this point in time. I think for us, especially from a cloud perspective, if you look at what cloud, what the promise is around being open, being flexible, different kinds of services, being able to move around it, without open source, cloud's probably not going to be successful as the least as what it's promised to be. I think that's what this show is all about to be. I see it as a platform. This show is about seeing where the industry will be in terms of seeing the communities, what they're working on, what problems and features they're working on, and seeing where our clients will be, and really being able to engage with the direct people that innovate at the edge. And that is so crucial. Although a lot of other places, other conferences, you don't get that opportunity. And that's what this show is about. You know, so we were talking about Cisco earlier, a software engineering driving the bus here, you know, and it really is about the innovation around DevOps. And you're seeing Internet of Things being impacted by software engineering, software architecture with reversalization completely changes the game. I mean, this is like, and that wasn't like that 10 years ago. You know, you wrote your apps, you deployed it, you stacked up some gear, now everyone's pretty much a software guy. You know, another big word I think we'll start seeing over the next year is no ops. Right? So the automation is going to be. A lot of companies have a lot of those guys. Is that a title or is that a trend? Just no ops, right? All automated, right? The digital enterprise, we will. He's a no-op. He's a DBA. Actually, that's going to be a really hot job. We actually interviewed the guy from the San Francisco Giants, the CIO here in San Francisco, great interview, and they all get World Series rings. And the favorite quote was, even the DBAs get a World Series ring. Everyone in the company gets a World Series ring. So, you know, we see that being huge. Guys, thanks for coming on theCUBE. Really appreciate it. We're stuck on time here. Red Hat, really the bell weather of Intel. We had Cisco on. Software, DevOps, Cloud is really making a real dent, Open Source is driving it all. Thanks so much for your comments. We'll be right back after this short break. Thanks. Thank you.