So, the next talk is from Gene Passamani, who is a long history of doing lots of stuff at NHLBI, NHGRI, physician education, and I'm just going to talk about physician education programs. Well, good morning. And for those of you who I haven't met, I spent two decades at NHLBI mostly designing and running clinical trials and trying to keep an extramural division together, a place where we could barely agree where to go to lunch together, much less anything else. So, Eric, I feel your pain. The second two decades I spent in a community hospital as an executive, a director of cardiology initially and as a chief medical officer later, so I know the enemy, which is the practice community who are very, very different from Fruit Street in Boston or Monument Street in Baltimore. I've dealt with a thousand of them and they are a different bunch. I think it's appropriate that I come at the end of this meeting. Physician education is way, way down on Eric's strategic plan. It's way, way to the right. Lots of things have to happen before that can be done very well, although it seems like that's where we want to end up. That's where the rubber meets the road, where patients derive benefit from all your hard work. There are a couple of light motifs in this presentation I'm going to make and my hope is that I can stimulate some discussion because we're going to have to grapple with this sooner or later and sooner is better. One is that specialties march through that strategic plan at different tempos. Infectious disease is really into this business now, I'm told. In fact, I have it from an expert in infectious disease that infectious disease fellows no longer do Gramstans. They do genomes of bacteria to identify the organism and then treat it appropriately. So the ID folks are really into this in a big way and further along than I think most oncology seems like another one that's out in front as well. So all doctors are not equal in terms of physician education. You've got to sort of temper your approach by where they are. The other part of that is professional associations are really the way to get at practicing docs and they have an ability to do both general literacy which I think is really important to kind of at least have the sense of what's going on in genomics and also they carefully draft guidelines which Paul quite appropriately criticized but some of them work pretty well and I would assert that they're the best or the worst except for all the rest and sometimes they really work. I just recently rotated off a professional education committee for the American Heart Association and I have some data from them. They have a professional education center and for the fourth quarter of 2011 50,000 doctors went to that site. Half of them cardiologists, another bunch of neurologists for stroke, family practitioners and emergency doctors. So they're doing something right and they have access to doctors. So I think we really have to think hard about professional associations in trying to support physician education. Also timing for professional associations is really important. If they put something together for education of physicians and it's too early, they irritate their physicians and nothing happens. If they put it up too late, you've got an access problem because you've got something that works that doctors aren't using and so they really try hard to time that just right. I actually pitched the notion of education in genomics with this group and that was their one concern. They didn't want to get into the game too early but they also didn't want to get in too late. So anyway, that one I think is very important in terms of interacting with these professional folks. I'm not going to touch on training of students, residents or fellows today. You know more about that than I do certainly and I think our speaker who didn't make it today was going to talk about that and probably knows a lot about it. I hope when we finish that we'll have good discussion about opportunities because I think we have to start this now because of what Eric said yesterday. This thing is really beginning to move and move very rapidly. So just a few comments on the opportunity and you all know their stuff. Their rapid advances in your science, astonishing reductions in cost and turnaround time and increased public interest and I think as Pearl said expectations are high and I think they're probably way too high. This is a long-term slog for us I think and patients aren't going to see this immediately. Seems like pharmacogenomics is approaching the clinical horizon. You've got good work going on in clinical decision support and medical leadership is very good particularly at your centers. I think there's a genuine interest and apprehension in practicing doctors and I'll tell you an anecdote we put together with NHGRI lecture series at Suburban Hospital for Clinicians and at the start, the initial lecture in that David Valley gave a wonderful talk and the clinician sitting next to me at the end of it looked at me and said, I don't understand anything. What's a SNP? Get that recommendation on your LinkedIn site there. I think it had to do with the reception rather than the delivery. I guess the other thing you're all aware of is a successful and really informative New England Journal of Medicine series on genomics. Those who read it probably know more than this clinician but in fairness to clinicians they knock it out hard every day and they're not reading everything in the medical literature that's for sure and yet they're the folks that are passing out this care to people that matter. The barriers, largely innocent of genetics and genomics, multiple categories of physicians with different interest and needs. I think we really need to keep that in mind and genomics transcends specialty areas so it stretches across a whole different bunch of folks, some who are really interested in this and sophisticated, others who aren't and it seems to me it's really important that we keep that in mind. As I mentioned, professional associations worry about getting ahead of their members and this can cause an executive director his job if he's not too careful about that. We've got lots of people interested in education, no means of fund it and it's difficult to monitor advances over so broad a piece of science and it seems to me that's a really big problem and one that we have to deal with. A minute on physician statistics, there are 850,000 physicians licensed to practice in this country. 624,000 of them do mostly clinical care and of those 209,000 are really primary care physicians. That is they deal with the general run of what wanders into a doctor's office. I also wanted to mention there's an estimated shortage coming up in the next 15 years, 150,000 by some experts. So we've got a real problem there as well. The other piece about the general practitioners is you all know that they slug it out in the office every day and it's really tough. Every 15 minutes a patient with three systems down, that's really very difficult. So a few thoughts on the way forward. I guess we could do nothing and let the traditional means of education sort of do their bit. It seems to me that's not very satisfying and I think it's really irresponsible. I think the first thing is I think this has already been said in place is to set up a process for review and consensus development. What matters, what doesn't, how big a deal is it, how accurate is it? And it seems to me that we ought to try for genomics literacy across the whole spectrum of practicing physicians.  They don't need to get into all the details but they really ought to have a sense of what's important in this field. Seems like pharmacogenomics is going to be pretty close to what most physicians deal with since all of them use drugs. And I mentioned to some of you, it seems to me that doctors who are in practice respond to cases, that's where they live. And so I wondered whether it would be useful for us to collect a vignette, a series of vignettes about where genomics matters today in patients. And I think you all probably have some examples of that and I would hope we could collect that and use it. I think we also have to rank order physician specialties and associations by the proximity of the science to the clinical horizon and the likely number of patients involved. That is, we ought to really focus on those where we have something really good to sell and that it involves a lot of people. I think we have to interact with these professional societies such that when a clinical science matures, that guidelines are prepared and delivered just in time. I think physicians are not ready to accept that this is going to be important someday. They need to have something that's important tomorrow. I think public education has to go on in parallel because patients and doctors have to see the same things. And I thought that New England Journal series was very good. I put this up just to sort of show you at least a partial list of these associations and the number of members they have. And they range from the biggest, which is the College of Physicians, a very important part, the Academy of Pediatrics. And then smaller associations, and I'm sure I left one out. I think I did leave out the Infectious Disease Society of America. But these are the folks that I think we're going to have to strike out and form some sort of communication relationship with now so that when we're at the point where we really need to deliver some of this stuff, we've got the relationship. And I threw in these societies, which are big and not necessarily physician societies alone. They're very large organizations which have a different collection of practicing physicians, and I think they're all really important. So that concludes what I had to say. I was hoping that we would get discussion from this, and I'll be happy to answer your questions or listen to your comments. Good. So, comments? I, okay. David, Howard. Can you go back one side? Yeah, so you don't have American College of Medical Genetics and Genomics, which recently renamed itself to reflect its interest in playing a major role and translating all of this into medicine. And it's tiny, and the experience of the American College of Medical Genetics and Genomics is the number of physicians entering our specialty is declining, but the number of master's degree genetic counselors is increasing nicely, although can't possibly meet the need. And the number of PhD laboratory trained people is increasing. So the laboratory component, the genetic counseling component, are growing, but need to grow faster. The physician component is declining in the face of increased need. So I think all of these education discussions need to be much broader in terms of physicians, PhDs, nurses, PAs, PharmDs, all the groups that need more genomic medicine education in order to address these needs, because it's just not gonna come from current physician workforce. Thank you, I didn't include them, I know they're there and I know they're very, very well aware of what's going on here. Other comments? Okay, so Howard, I think was next. So I'm wondering in the context of physician education, I'm wondering whether we're trying too hard because the radiology community didn't try to make generalists into biophysicists and we don't need to make generalists into molecular biologists or geneticists even. And so certainly knowing what a SNP is would be useful. But there are a lot of elements that can remain in the black box and they're just fine with that. It's more the usage part of it that's important. Most generalists don't care how Billy Rubin's measured, they wanna use it. So there are nuances that we get really jumpy about. But at the end of the day, I think if we maybe calibrated the need a little bit better, we might have more inroads, especially with initial stuff. Now, ACMG needs something different from the American Academy of Pediatrics or some of these others. But I think in some ways maybe need to lower our goals to something more achievable. I think that's a good point. It does seem to me that that's a part of the point that all physicians aren't created equal in this. I mean, podiatrists are probably not much interested in it. But internists who deal with subspecialties of medicine I think are very interested in it and they really ought to know the pharmacogenomics pretty well it seems to me. But I take your point. Other comments? There's a long speakers list. I'm watching it, don't worry. Okay, couple of quick comments. First having done this kind of education for about 15 years, recently the American College of Physicians has reached out to the American Society of Human Genetics, American College of Medical Genetics and a couple of other groups and said they're interested. And that's the first thing I found that's really, really helpful. It's now on their radar. And John Tuquer, who is their former executive director is actually charged with that. So I think there's a huge opportunity there for us to reach out to them. The second you probably noticed or you may not have in Joanne Armstrong's presentation yesterday that Aetna has a focus on genetics education and actually has a budget for it. And I think we really need to figure out how to partner with them because they can actually require the physicians in their system to do some Aetna education every year. And I believe they reach one in 10 US physicians. So huge thing. And then my last comment is that there was a very successful program that the American College of Medical Genetics did probably 10 years ago with endocrinology. And what they did is they switched the paradigm a little bit in that they made the endocrinologist a patient and took them through genetic testing. So they really saw the power of genetic testing and the effect that it could have on them and then took that back. And a lot of endocrinologists started using genetic testing who hadn't before. So that's a model I think we could replicate in many professions. We could all have 23 and me done on ourselves. Well, yeah, but I mean when they really, what we did is at their meeting we set up where they had had pre-testing and then they had short genetic counseling sessions and it was just eye-opening. I was one of the counselors there and talked to like 10 of these endocrinologists and they didn't know what a genetic test was. But it really became powerful when you made it something that would apply to them and their family and then use that in relation to their patients. Just curious what genetic testing did they have done? Like just a couple of candidate genes? MEN2 and some other things that have an endocrinology basis. So we had a whole work but we picked like six conditions Did any of them have anything? Actually, yes. Oh, okay. Say no more. Yeah, yeah, yeah, yeah, yeah. Physician-client privileges but I do think that's a powerful and you can even do it in an abstract where you set up the whole process but I think that would be something that would be really worth thinking about doing with some of these other groups because they just don't put genetics on their radar and this got it on their radar. Michael, Josh, Pearl, Jeff. Everybody wants to say something. So Michael. Wait, I have to say something. So I just spoke at the American. I know, I know, I just took over the mic. So I just spoke at the American College of Physicians. They invited us back for a second year. It was actually an ethics session but all on genetics, genomics. And I'm invited to speak at the American Geriatric Society later this fall and I still haven't figured out what they want me to say. But so I think we do, I bet there are lots of people in the room who get those types of invitations and there is interest out there. Okay, go ahead, Michael. Oh, and also I just have to say. In response to Dr. Leverett, I was just on the American Board of Medical Genetics and actually the numbers of clinical geneticists and so forth is kind of leveled out. We're no longer declining in our numbers of enrollees in the program. So that's good news for us. I was gonna say that the drivers of physician education are CME credits, board certification or maintenance examination content and patient questions in the room. So until those three things happen, you're gonna find a lot of physicians not interested. And the other comment would be that the new way to deliver physician education is in short 10 to 20 minute interactive, web-based CME credits. So people that are thinking about education and getting out to groups of practitioners, no matter doctors or others, that's one thing to really think about in your design. And those are really good ones for cases, like short cases. Right. Okay. Murray then Josh. At the Marshall Clinic, we've done just exactly that. So we have now mandatory computer-based training in pain management that includes a module on pharmacogenetics and it's really quite nice. And so our physicians, all of them now know about you know, SIP 2D6 and variants of that. So it's really nice. It's worked quite well. I'm not sure if it's proprietary. I don't think it is, but I could ask. Sounds like a nice model. Okay, great. So I was just gonna say, I'll raise our hands again. The, so I think this is a real demonstration of a case where there's going to be, there's a fire hydrant, it's just going to get worse. And it's one of those real opportunities as a representative of the informatics community for I think us to engage as meaningful use comes on board, more and more EMRs are out there that we can deliver that information just in time. And if you could, and this is what we're trying to do with some of our prospective genotyping efforts and trying to test some of that in front of physicians, what kind of language looks good and how do they respond with progressive enabling them to kind of click and learn more just kind of as they need it. But we're never gonna build support and educate around everything if we can support and educate around the concepts they need, then maybe we can do more and more just in time stuff with informatics. I wouldn't sell specialist and some specialist short. Some of them are very smart and very able. And if you put it in front of them and it seems like it's useful and they'll lick it up, that's no, I don't think there's any question about that. And in fact, we actually have a link off of our advisor rounds to 2019 where you can see all 60 articles that we've curated around Clopidogrel if they want to read it. That's great. Knowledge map for genome education. I know that medical school and resident education wasn't part of your talk, but is there any evidence that the younger doctors are trained up adequately and you know, well I know not a chance.  And on one side, I mean, there's a lot of concern that they're not being taught physical exams. And what are they being taught instead? And I know just anecdotally, just going into the wards and seeing a person with shock and I thought, oh, I know how to deal with this, volume, volume, volume, impressors. And immediately went off into what calcium channel blocker might be a mutation. And meanwhile, you know, the patient's dying of hypotension. So my concern is, I mean, is there any data that the new wave can potentially educate our geezers or is everybody in the same bad boat? I guess I'll take, I'll say a piece here. I do think we've run the risk of having sort of an intergenerational tussle here, but my goodness, surgery is really suffering today. And the reason is that they're limited to 80 hours in the hospital. And I've seen residents in their second year who I'm told don't know how to tie knots. And so what happens to those residents is they go out and make the mistakes while they're in practice that they should have made while they're in a more controlled setting. Anybody else know about what's going on in medical schools? So a couple comments. First of all, the online tools are great, but just a word of caution that our place, there was a current calculation of the total hours required to be spent on online exams of various things. Responsible conduct of research, HIPAA, blah, blah, blah, blah, blah. And it was estimated to take about one week of eight hours a day to complete the current online requirements. So adding more may not be received favorably. That's not to say there's still not a niche to fill. That we've changed our curriculum. We're trying to infuse a genetic thinking into the education of medical students. It's too early for us to know whether or not we've achieved any kind of success. And if you think about changing the curriculum as an experiment, it's a very hard thing to gauge the success of an experiment because the controls are very difficult to come by. So it may be in the category of an experiment that you just get all the opinions in one room and try to make your best choice and go forth that way. The other thing I'll say though is it's clear that medical education has different challenges at different phases of the career. So you have the medical students, then you have the house staff and the house staff do a lot of educating of the medical students in the third and fourth years. If you don't educate the house officers, anything you do to the medical students is likely to get beaten out of them in years three and four. And then you have to educate the practicing physicians and your academic colleagues. So I think it may be important to think about different strategies for different levels of medical education and in a way, I think they have to be taken on almost simultaneously so that you, because the educational experience goes on for such a long period of time. Thank you, David. It seems to me that this machine is going to deliver wonderful results and we don't know exactly where. And we're gonna be in tough shape if we're not ready to deal with them. Two quick comments. One relative to Andy's anecdote about the endocrinologist, Susanne Haga, who could not be here today, has published a paper that has shown after surveying hundreds of primary care physicians that those that actually had genetic testing on themselves were 10 times more likely to order a genetic test for their patients. So that's a provocative piece of data that could be channeled constructively into some educational programs. We could offer it to our medical students. It's being done. Okay, I have a second comment. Do you wanna? So let me, so for the second year this year, we engaged in an exercise at Vanderbilt, stolen from the Stanford experience. So I make no apologies for that. We managed to get it into the curriculum. They managed to get it as a summer optional course where we offer 23 and me testing to the medical school class. And we go through a long sort of pre-test exercise where we discuss the ethical downsides and what an odds ratio of 1.2 actually means. And then they go and pick up their kits and spit. I don't think anyone's had their dog spit yet, but who knows. And then we have a session where we deliver the results to them and Josh actually organizes a survey that they fill out. And the survey is sort of, what was your odds ratio for developing type two diabetes? What was your odds ratio for this and that? And then a whole session on ancestry and a whole session on rare things that 23 and me looks at. And would you do this again? Do you feel like you're smarter? Those kinds of things. But I think it comes back to the point that you made that when you do this on yourself and you get the results, it's sort of, oh, isn't that, there is sort of a resonance of some kind that I think Suzanne would have told us about. And I've certainly felt when I looked at my own website. So I think that's a tool. And one of the things that I hope I'll be able to do is interrogate the medical students who had it last year and who are now coming into fourth year and say, well, you know, do you remember that session at all? Do you remember anything about it? Does it make you think differently about a patient when you're sitting on the ward? So those are the kinds of things that I'd like to hear. And Josh participates when he's not traveling, which is not a criticism because I travel more in delivering that kind of information to the medical students. It seems like that might also populate our ranks with people who want to do this.  So your other comment. My other comment is that I think this guidelines topic with all the caveats that Paul raised this morning needs to be an agenda item for our next meeting when we engage these professional organizations because my bias is that it's the equivalent of having primary care physicians review a genetics grant. I mean, how are, just in relative, it came up in the eGap discussion yesterday about pharmacogenetics. I mean, the people that are making the guidelines are probably not in this room. It would be my guess, except for the CPIC group, which is sort of an interesting way to approach it. And maybe we should even think about how CPIC could be a resource for the professional organizations instead of having them reinvent the wheel. I'm sure that most of them realize that they need help and we might be able to give it to them. Okay, there are lots of hands still. Can you sit? Behind you. So, okay. Oh, sorry, the Air Force goes first. This Air Force goes first. And then you, sorry. It's just Cecilia. I just had a quick comment about the National Coalition for Health Professional Education and Genetics.  And in terms of our enrollment strategy for our study, I made the decision to enroll people who are part of the healthcare team because, although as physicians in theory, we're the leader of that team, I wanted to try to start that genomic literacy effort for all of our providers. So in our system, that's nurse practitioners, physician assistants, nurses, of course, and then our medical technicians as well because they do have increasingly independent roles in terms of the way that they interact with patients. Thanks. And pharmacists, very important. Sorry, if I could just throw one thing in here real quickly. So I surveyed. Okay, go ahead. So in a lot of our life, I created an educational management system and so I pulled up 400 concepts related to genetics and survey, 10 years of our curriculum and it actually has increased. 22 lectures, about hours of the curriculum in 2002 and 147 in 2010. Wow, thank you. Welcome to an informatics rich world. Right, and on the informatics note, so clinical informatics was approved as a board certified subspecialty last September. The first exams will be in 2013. So a few places are designing fellowships and rotations for the residents and so this is your chance to make sure that those rotations and fellowships include exposure to genomics, not just to Microsoft Excel and SQL databases. So the clinical genomics is a subspecialty in? Clinical informatics. In pathology. In clinical informatics. In pathology. And there's also biomedical informatics of specialty in preventive medicine. Yeah, that's right. Okay, other comments? I saw lots of hands, but they all just disappeared. Okay, yes. So two things. One is that one of the other ways to go about this is actually to get the fellows as specialists to take a little bit more interest in genetics. So ACMG has worked on, instead of going through a genetics residency or fellowship is actually have like a one year course of somebody's a cardiologist and wants to specifically look at cardiovascular genetics, have a program like that. And so you don't need to train everybody to be a general geneticist. You just need to train them in a specialty that's applicable to them and then hopefully they'll be the leaders as going out. The other thing that doing student education, one thing that I've started to do is how to teach the students how to look at uncertainty. And so these variants of unknown significance, these are common, we get them all the time, but how do you get the students to think about how do you critically analyze information? And this is Apple to the regular life too.  Do I watch it? Do I repeat it? Do I treat it right away? But this is a good lifelong lesson that if you can teach them how to look at uncertainty, how to evaluate it and how to react to it that it's not just Apple genetics or genomics, but everything else as well. David, one more. Just to put in a plug, this summer will be the 53rd annual short course in experimental and medical genetics at the Jackson Labs and Bar Harbor. It's a two week course, 53 lectures and eight workshops, lots of prominent speakers. It's really pegged at just the right level for fellows, PhD students and young faculty who are in various specialties but want to sort of get a two week intensive experience in genetics, I highly recommend it. Thank you, David. Okay, I think we'll stop this part of the discussion. The next discussion is from me and so you're gonna have to bear with me while I transfer the slides that I've been working on all morning onto a stick. I have to remember where I put them first. No, I don't know. When is it coming? Will you do that? Yes, please do something to take up the dead air. Okay, in terms of sort of an alternate thing, we're constantly looking from the IRB world in terms of how do you present probability, et cetera, and looking at recent literature on how numeracy in the U.S. It was something like 50% of the public cannot tell you if 10 out of 1,000 is 1% or 10%. And then they went to medical students and they were not much better. So I think where we're talking about even beginning to talk about relative versus absolute risk, we're talking just percentages. I mean, so it's really scary that we're gonna end on a positive note here. Yeah, it sounds horrible. The answer is for Eric to fix K through 12 education. We established that yesterday. So rather than watch Dan struggle his way through this, just a quick question for folks. We will be talking about locations for meetings in that, but any thoughts about this particular hotel? Is this a positive? If we come to Chicago, would you rather be here, even though it tends to be a bit more expensive? Or we can go to an outlying hotel, but you have to take a shuttle to get to it in that. Any strong feelings? Okay, so no strong feelings? Yes, the O'Hara Hilton is my favorite place in the whole world for a meeting. Well, there you are. It's my least favorite. I guess. The St. Croix vault is pretty nice too. Thank you, Mary. Thank you, Mary.


So my name is Nathan Scott, I work for Red Hat for the last couple of years. I have a keen interest in performance analysis, in particularly in what we call system level performance analysis, but I'll talk about different kinds of performance analysis in a minute. I work with the performance tools group at Red Hat. We look after tools like PCP, which I'll be talking about today, and other tools like Valgrind and SystemTap and a few other tools of that ilk. So a lot of people don't know much about PCP. It's kind of a, I might like to think of it as a hidden gem of the open source landscape. It's been around for many, many years, over 20 years. We've had PCP and over 11 or 12 years or so it's been an open source project, started out as a proprietary system from SGI. So in my talk today, I'm going to talk a little bit about PCP itself because a lot of people need an introduction to it, what it's for, how it's put together, and some of the basics so that I can then go into talking about containers and what that means for PCP and how we're attempting to marry the two concepts because they don't particularly fit well together with traditional system level tools like PCP. Also in this talk I want to talk about kernel instrumentation for containers and the metrics that are available to you if you have an interest in performance analysis and you're using containers. So I'm going to talk about the concepts in the kernel that back containers and how they fit together and what that means from a performance analysis point of view. So right up the top level let's talk about what PCP actually is. PCP is a toolkit for performance analysis. It follows the UNIX design philosophy of having many small tools that do things working together rather than a single big thing that does everything. Like I mentioned before, it's for system level performance analysis. So these are tools of the ilk of ganglia, collectee, that kind of tool and less of tools like profiling tools like oprofile or perf, things like that. It's a different category of tool. So we're in the system level analysis gang. In PCP we have focus on being able to do both live analysis of a system while it's running and while it's failing being able to analyze what's happening and also be able to use the same tools that we use for live analysis but also for doing historical analysis. Being able to compare what's happening right now on a system to what was happening yesterday or two weeks ago. PCP is inherently extensible in several different directions and we have these core concepts of monitors and collectors in PCP. I'll describe what that division is, but it's extensible in both those ways. I'll talk about that on the next slide. But last point I want to talk about in terms of PCP overview is that it's a fundamentally distributed system. So baked into PCP is the ability to analyze multiple systems, multiple computers at once that are forming a larger complex system. So if you have a web application, for example, that has a database service and storage, the application, that's a firewall in front, those are all forming a complex system together and they all need to be analyzed together. It's difficult if you have to analyze different systems separately to try to get a good idea of the overall performance of the system. A little picture is just trying to show that we have the ability to focus on what's happening right now under that sort of light pyramid. We can look backwards in time and we also try to aid future analysis work like capacity planning, things like that. We don't really do that directly in PCP, but we have the ability to store data so that you can project forward using other tools. So this is the architecture diagram for the live mode of PCP. If you install PCP and it's running on your system, these are some of the fundamental processes or components that go into the typical deployment of PCP. So on the left-hand side, the green side, we have what we would call the collector system. That involves a central daemon which pretty much runs the show. It acts as a multiplexer, so that's the PMCD daemon, Performance Metrics Collection daemon. And the client tools or monitoring tools that want to report performance data will typically connect to that daemon and ask for the data that they're interested in and then report that. So there's a very clear separation in PCP between the components that extract data from whatever domain that they're interested in and the tools that are going to report it or record it or take some action based on that data. So it's fundamentally... The collector architecture is fundamentally pluggable. Like in the diagram here, we have application plugins that might know about performance data for an application. Like response times, how many things are happening per second within the application. How many requests are happening per second. You might have an instrumented mail queue. You might have a database that's instrumented. Always the kernel is highly instrumented. All of this data is available on any collector system and you can plug in new pieces of your own. So if you have metrics in your own domain that you're interested in, you can also make those available through PCP for the monitoring tools. And there's many, many monitoring tools. So here on the blue side, we have just a few examples of some of the commonly used ones. But there are many, many different tools for a part of PCP. So the PM logger process is the one that is used to record performance data. So you can record any performance data. And that is then available for replay in the other client tools with the same sort of interfaces as the live mode. There's a tool for displaying strip charts. The PM chart tool lets you plot graphs of your choosing for arbitrary metrics. There's a tool called PMIE that lets you make decisions. I can feed a bunch of rules about performance and it will analyze those rules, evaluate them on some interval that you ask it to and then take actions based on those evaluations. So this is kind of the high-level concepts of PCP. Later on in the talk, we want to look into how we can make this kind of architecture work in the container world. You probably see fairly quickly that these system-level tools don't necessarily fit so well into the container world because they're designed for a single machine exporting data about that one machine. Whereas with containers, we might have many containers, all of which are pretending to be smaller, very lightweight versions of machines, and they all have their own metrics as well that we want to export through this system. So I'll be coming back to that later how we've started to tackle that. So the last slide in terms of introducing PCP concepts. One of the core concepts of PCP is this idea of a performance metric, which is basically anything that you can measure on your system. So anything that you can measure, we need to represent it in some way. In PCP, for every metric that you might make available, there's a series of pieces of metadata that need to go along with that performance metric that define what it is for the monitoring tools to be able to interpret it. So every metric is represented in the performance metric namespace. So in my example here, I've got an example command, PMINFO, which is just another one of the client or reporting tools. It's looking at a particular metric here, which is the number of read operations across all devices. So that metric has the name disk.dev.read. So that metric name is metadata that's associated with the metric. And in this example, I'm just asking for the metric descriptor, which is the information about the metrics and the values. That's what the fetch does, and the minus TT asks for the various forms of help text associated with that metric. So we can see all of the metadata that's associated with this particular metric here, and that gives us a good example to see the kinds of things that must be associated with every metric. So we need to know its data type. So this particular metric is exported from the kernel as a 32-bit unsigned integer. I'll come back to the Indom concept in a minute. The semantics of disk.dev.read are that it's a monotonically increasing counter. There are several different types of metrics that are exportable. So we have counters that always increase. So we have what we call in PCP instantaneous metrics. They're called gauges in some other systems, but their metrics, whose values may change in arbitrary directions every time you sample them. And they would be things like the number of users that are logged into the system, things like that. We also have the units. For this particular metric, the units are just that it's a count. Other metrics, you might have units that are things like it might be measured in kilobytes or it might be measured in milliseconds, nanoseconds. That's more metadata that needs to be associated with every metric so that the client tools can correctly interpret and report that particular metric. You can optionally associate help text with every metric. And with all the kernel metrics, we go to great lengths to provide help text to explain what every metric the kernel's exports is as best we can. And if you're writing new components that are plugging in metrics, you get to add the help text that's relevant for your component that you're adding. And finally, we have, of course, values that are associated with every metric. So a metric might be a singleton value like that example I gave before, the number of users that are logged in. That's just a single number for any one system. But you could also have a set of values like with this example, which is the read operations across all devices. Well, the set of values is going to be expanded across all the devices in the system. So in my example here, I have two devices, the STA and STB devices, and separate values associated with each of those. But they share all the other metadata. So we don't need to repeat that metadata for every instance. So that's a high-level overview of PCP for anyone who hasn't come across it before. Those are the core concepts. PCP is a huge project. It's been running for many years. We're talking hundreds of thousands of lines of code. So that's just a very brief overview of things. I want to take a segue now and talk about containers for a while. So even if you're not really interested in PCP, hopefully you can get something out of this section of the talk where I talk about the instrumentation that's available that the kernel maintains for the concepts that sit behind containers. So the kernel doesn't actually know anything about containers, as you may know. It has other concepts which containers are built upon. So containers is a user space concept, not a kernel concept. So in the kernel we have... Was that five minutes, John? Yeah. Okay. So we have C groups and namespaces in the kernel. And those are the two concepts that all of containers are built on. So C groups are a way of associating groups of processes and controlling them together. There's a set of statistics associated with each C group that is created within the kernel, and things like Docker will create C groups for each container. Associated with each C group are all these metrics that are available to you to analyze what's happening within that particular container. And there's several different subsystems. So there's a BlockIO C group, there's a CPU accounting C group, there's a memory C group, and there's one or two or three other C groups as well, but these are the core ones that have statistics available at a controller level, which a C group level, which ultimately becomes a container level. And these are some of the sorts of things that we can see at a C group level. So for each of those things, you get IOPS and throughput per device, IOs that have been issued from within a C group, which is ultimately from a container, and service and wait time. They can be aggregate or per device, so you can see for each device that is IOs submitted to or from within a container, you can see which devices it's written to. Those are exported by the kernel in a way that's relatively user unfriendly, so they have device major and minor numbers, so user space tools then get to decode that and try and present that in a useful way. So that's one of the things we do in PCP. Because I'm running out of time, I'll skip through. So I'll skip through some of those metrics. If you're interested in those things, the place to look for them is at the C group mount points. Usually these days that is slash this FSC group and then the name of the container and the naming of the C group, sorry, is dependent on the system you're using. So if you use Docker, it will name its C groups a certain way. If you use other systems, they get named differently. And that's one of the problems which we have to deal with in PCP as we try to map containers to the kernel concepts below them. The other concept the kernel has is this idea of namespaces, which if you know anything about containers, you'll have come across this, and they influence the behaviour of the processes that are running within a container. For example, if you're in a shell inside a container and you cat-procnet-dev, which is where the network device statistics come from, like the IO traffic for each network device, that looks different on the inside of a container from the host that's running the container as it does to other containers. So as you can see, these are all problems that are going to have to be dealt with in tools like PCP that are system-level tools. So I'll skip through that. I've talked about that. And namespaces are inherited across fork and clone and processes within a container share a common view using this namespace concept. So I'll talk now the last couple of slides, the last slide or two about the work that we've been doing. Oops, that went back. So I'll just use the same example again. So some of the goals that we wanted to use to tackle in PCP was to allow the tools to start targeting individual containers. So typically, historically, you've been looking at a full system. Now we want to be able to focus in on individual containers. So in my example before, I did a PMInfo requesting values for all of the network devices in the system. We want to be able to say, well, not for the whole system, but just for what's happening within that container, a named container tell me the traffic that's been generated from that container rather than from the entire system. So we've added... Actually, I'll talk about what we've added in the next slide, but effectively, we want to be able to say, PMInfo fetch values just for a container named crank and the network values, and that should come up with the network interfaces from that container rather than all of the interfaces on the system. Another important goal that we wanted to tackle was to not have to require people to install PCP and have the PMCD daemon running inside every container. That's kind of the obvious solution to this because then PMCD has the correct namespaces when it's running, but that requires a lot more software installation and software maintenance within inside the... every individual container that you might want to manage. So we want to be able to have either a PMCD running on the host, on which the containers are running, or a privileged container which can run PMCD and is able to switch into and out of the namespaces of the other containers. I talked a little bit about the device major miner thing, so there's one of our little side goals is to just make it easier to do analysis of containers and solve little problems like that for you. So you get the names like SDA, STB, rather than device major miner. And be able to do final goal data reduction is one of the key goals. So we're trying to look at just individually named containers rather than all of the containers on the system all at once. And that affects things like the process set. So if you're running a tool like top, if you run it on the base system, you're going to see all of the processes. If you just want to see the processes that are running in a container, we can use PCP now to do that kind of thing. So just restrict it to the set of processes that are running in a container and which ones within that container are generating CPU load or whatever load you're interested in. The last one. So I'll probably have to skip over most of this slide, but you can probably get the gist of where we were going from the last slide. The critical concept that we've been introducing recently is this ability to switch namespaces and provide information about containers of interest throughout the infrastructure that is PCP. So that starts from the client tools like PMInfo here. You might be connecting to a particular host and you're saying, on that host, I'm interested in container named Crank and I'm interested in the network metrics. So there were a series of wire protocol extensions to allow a container identified to be passed over to the remote system and then PMCD on that remote system is then able to say, you're interested in this container and looks on its systems, figures out what C groups or what other things that maps to which namespaces are associated with that container and can allow the collector system to switch in and out of that namespace as it needs to so that it reports the correct network devices for that container. Excuse me. So a quick look at the status that we're up to. The next PCP release will include this code in and this will be within the next couple of weeks. This is the first release that will have any of these concepts in it. There's a lot of work still going on within PCP to just improve this and make it slicker as we go forward and I expect as we go down the track we'll be wanting to write new monitor tools that are specific to containers so they know about C groups in particular and are able to like tools like VMSTAT, IOSSTAT and other tools like that today we'll be building tools that are container specific and so report the activity within the containers. So I didn't really get much time to talk about the details of those kernel metrics that are available for C groups but they're actually not the same as the metrics that are available for the raw host so you can't just run VMSTAT on a container and expect it to report something useful about that container. There's whole new tools that will need to be written. And that's the small links about PCP and then I'm done. Sorry for going out of time. We have time for tapping one question. Make it a good one. Thank you. No problem. Thank you.


Just okay hold on a second. I've always wanted to do this That's so cool All right, sorry this this stage is amazing as many as amazing To start off I I want to thank Mike and Jameson like it's a real privilege to be able to have all you guys's attention Thank you I Really appreciate you guys and I appreciate the opportunity to be here so today, I'm going to talk about the Frankenstein framework and Where I see kind of a potential future of the web and some changes in the ecosystem that are Really interesting But first we have to talk about the history of the web For my purposes today. I've broken up the web into three separate arrows the document web the hybrid web and the application web With the document web era things were primarily rendered on the server So you made a request to a server it grabbed some sort of static file it rendered that static file Or if the server was particularly smart You made a request to the server it did some dynamic stuff. It sends you back a response Libraries in this era augmented the platform using non-standard monkey patching. So I'm talking Mootools prototype They had really excellent API's but if anyone remembers trying to use like Mootools and jQuery on the same site It was a real nightmare like jQuery added this bougie no-conflict thing where you could like call no conflict and get back a variable But for the most part they didn't they didn't play nice together Libraries in this era leverage plug-in libraries to share code so If you wanted to write jQuery code You had to go to jQuery.com and go to their plug-in library and they had like a Star rating system and 99 out of a hundred times the plug-in was dead I do you guys remember like having to install these really ghetto plug-ins. It was awful It's anything with Mootools you go to Mootools.net and their plug-in system came like way late in the game. You were You were installing these plugins from this weird PHP plug-in repo. It was At the time it was really hot, but but looking back it was a step and Globals were used as a code sharing mechanism people used to at job interviews Test if you understood the immediately invoke function expression like that was really important to understand scoping right So we had no module systems. No way of sharing code We had globals and people talked over globals. There was a lot of clever ways to talk over globals Then the web had sort of an awkward teenage phase at least I call it an awkward teenage phase Sort of the puberty of the web if you will There was the hybrid web things were a single-page app. There was still server-side rendered and they came down with a ton of JavaScript and We called this progressive enhancement We took the document web and we tried to bring it forward into the application web But we couldn't really do it because we were supporting these old browsers, right? So you wanted to use this cool new engaging in experience But you couldn't because you were falling back to an older browser So we called this progressive enhancement. We were sending down gobs of JavaScript and We were bringing the page to the future if you will People were starting to get more serious about JavaScript This is when Ajax started to get popular you could make requests back to the server and grab data It was like this amazing thing and then you could render it into the client-side templates Then the web grew up to the application web and this is kind of where we are today Though I'm going to argue that we're a little bit further ahead than the application web Applications are rendered completely on the client. There are still some remnants of the document web For example, isn't it kind of weird that we send down an HTML document and then build up the whole thing with JavaScript? Like that's sort of a historical artifact more than anything else Applications or JavaScript frameworks in this era provided application architecture So we're talking like backbone ember angular etc. And we were augmenting application structure What are the frustrations of doing a polyglot framework based on this history? There's no good code sharing story So no one can agree on a module system like the module system warfare has been a nightmare If you remember backbone had like AMD support for like two days and then they took it out and some people support common JS and some people are like let's do UMD which is like Just I mean it works, but it's it's a it's a horrific hack There's just no good code sharing story Frameworks require utmost buy-in So my point here is that if you wrote angular code You were writing to and yours module system and if you wrote ember code You were writing to ember and if you wrote backbone code you were writing to backbone So you were coupled to the framework itself.  There was no standardized issue tracking But things are brewing. I think a polyglot framework is possible today The code sharing story is solved the days of common JS versus AMD versus UMD versus embers weird thing versus sprout cores weird thing Etc. They're gone. We can now depend on the ES6 module format. So we have a common format to share code across these different applications and different libraries The platform itself has changed We now have evergreen browsers so historically we had to augment the web in a serious way We had to like bring the web forward to the future Which is still true, but we now have a different story to do that We no longer abstract the web behind a library or framework We polyfill the web to bring it forward into the future the benefit of polyfills is that if somebody polyfills a promise Multiple libraries can consume that polyfill browser rather than all having to write to jQuery's promises or cues promises The ecosystem itself has changed People take this for granted, but we haven't agreed upon code publishing story which is significant github Fundamentally changed our industry which which people don't consider very often. They just take it This like let's use github, but but we now have a blessed issue tracker We have a blessed way of knowing whether projects that are alive and forking a project NPM officially opens their doors to the browser. This gives us proper versioning semantics for using vendor code We don't have to go download some library put it into like a bougie script tag or worse like some grunt task That's going to concatenate them. We have package manager and Frameworks are compositions of smaller libraries. I don't know how many of you follow Existing like like frameworks probably most of you follow all of them, but The future of these JavaScript frameworks are all compositions of smaller libraries like ember is a composition of smaller libraries Angular 2 is a composition of smaller libraries react itself as a library backbones a library So this means we can start to polyglot the best parts of these frameworks together if if we so choose Which brings me to the Frankenstein the Frankenstein is one attempt at Polyglotting all of these different frameworks into one thing But first we have to bring the past We have to bring the future to the past with our mad developer lab Which brings me to my tooling which today we're going to be using webpack with webpack we have a custom a a Feature called custom loaders and you can think of custom loaders as sort of like a pipe like you take Mario You put him through the pipe and on the other side. He's a different Mario. It's like a bigger Mario, right?  Sorry Anyways, you guys are you guys okay? All right. All right, like it's like tense out there. Geez stressing me out what a Anyways, you take these your code you take the code from the future if you will ES6 code you put it through this pipe You have the code of today. It's really rad So let's do a quick example of setting up webpack Unfortunately, you guys don't have github But I do if any of you want to follow along. It's just on my github repeat repo We I have this whole tutorial that we're going to go through today So the first thing you do is you install webpack server? Yeah, well, so you make a configuration file. This is what webpack looks for great Great great fetching start So you have this entry This is where webpack is going to go look for your code and then the file that it's going to build is this file called Frankenstein So we got to go make that source Frankenstein file and for now. I'm just going to console log. It's alive So there's this really cool feature with webpack dev server That brings it forward to the application web completely and it'll actually generate a HTML document for you you don't have to like make a stub HTML doc and put your script tag in there So we're now running through webpack. We're console logging. It's alive. We've got our build tool set up This isn't a webpack talk. I think there is a webpack talk though And I highly recommend going to it because webpack is an amazing tool yesterday nice Sorry, I I've been so busy With a certain with another comrades So let's bring to my next tool Babel JS, which gives us ES6 that module format I was telling you about like this blessed module format We have ES6 tools which what we're focused on today is the module system So this gives us important export syntax. We no longer have to write to common JS or AMD, etc We can write ES6 modules. So let's augment our webpack example really quick We're gonna MPM install the Babel loader augment our configuration What this does is it tells webpack everything that's in source Folder that ends with JS run it through Babel Let's make a new folder. I'm sorry. I keep opening these tabs in the tiny so we can use ES6 class syntax here and We export default Frankenstein. So this is how we're gonna export our classes Or our files and inside our Frankenstein. We're gonna import Frankenstein from monsters Frankenstein And then we'll new up our Frankenstein And we're we're alive from ES6. So now we've got The ability to write ES6 modules today and the great thing about webpack is it's flexible enough It's kind of like JavaScript and that it's flexible enough that we can band-aid the current situation we're in So JavaScript itself is the language is awesome Because it's so flexible that we can do all sorts of crazy things to make it more usable for applications so finally We can get to the interesting part. Let's start compositing these frameworks into a polyglot framework Our subjects today are going to be react because it's a wonderful view layer Angular first dependency injector actually in your to first dependency injector and flat iron for routing because it's a very beautiful agnostic router So let's get started with Angular 2 Unfortunately, the stuff isn't out yet. So you have to install these for now. We have to install these pre-versions We install a particular version of Di Again webpack is such a flexible tool that even though Angular 2 is currently implemented with tracer We can get tracer running with webpack using another loader. So we have to install this imports loader So we're telling webpack here when you encounter the tracer runtime install globally the window object because normally you don't have access to that in these modules So we now have a new DI. Let's go ahead and Have DI and instantiate this guy This is how you make a new jet injector in Angular DI You just import it and instantiate it and then you can ask the injector to Create a new object for you So what's really cool about this is we can now use a dependency injector to stub out our modules at test time I don't know if any of you guys have played with Angular 2's DI But it's actually really neat because it uses the actual classes as tokens So it works based on annotations which they're currently involved with TC 39 and standardizing. So hopefully we can We can get that standardized So we don't have to use JavaScript the way that we have to today for annotations because it's kind of gross but Let's go ahead and make sure this is still constructing I gotta restart webpack. I'm sorry and it's alive So we're no longer creating our objects. We have Angular DI creating our objects The cool thing about Angular DI is we're not particularly coupled to the I framework the code can still be run without DI As you can tell with this Frankenstein class We had nothing to do with DI in there, which historically Historically we had to register it with Angular dependency injector But now we can use the classes self as a token for the injector. We don't have to Register it so what's next? React.js By the way, I this whole building the Frankenstein thing I was super stoked about this so I want you guys to appreciate it Thank you This one's easy we install react No configuration needed We can make a new Frankenstein component So since we're using DI, we're gonna be using these factory functions, which Will allow DI to create pass back a component class For a time's sake I'm gonna copy it and paste it and explain it to you. So we import react We import annotations and inject from DI We import Frankenstein and Then this is how we have to do annotations today as opposed to an annotation syntax We tell it to annotate the Frankenstein components and then we inject a new Frankenstein model So then we're gonna print. I am monster dot name But we need to augment our model to have a name So let's go ahead and crack open source monsters Frankenstein, which is our model Nice nice. I try to use them on these things so it looks smart and stuff, but Never goes well now we have to render We have to render this bad boy. So the cool thing is is we can still use react or Angular DI To create our new component So we import react we ask react to render the app into the document body which web active server super rad and We'll give us a document body and now we're rendering react with any or two dependency injector So we have something to wire our code together that allows us to decouple it properly during test time So we can grab Frankenstein and we can new it up ourselves We can grab our component even and knew it up ourselves without the DI Or we can use DI to instantiate things Which we're now on to see I have to go back to the slides because I really want you to see this We're adding the leg of flat iron Flat iron is an awesome library It has a goal being isomorphic we're working on node and the browser for our purposes today we're going to be using this tool called director which is a Really a framework agnostic router which facilitates a lot of the polyglot behavior So again, we have to install it Let's make a new router file So we import director see the cool thing about webpack is even though directors actually a common JS module We can import it with ES6 syntax We grab annotate and inject from DI We grab react we grab our Frankenstein component This is what our old Frankenstein app used to look like right like that main file We used to be rendering but now we're going to render it from a route So we have our router we create our router and in the home route We call react dot render with that component which was injected by angular DI So it's injected up here and it's named here so we could rename this to something else if we wanted And we render to the document body Then our main we're going to do the same thing here. We're going to import router We're just initialize that router. So we see nothing here because we're not on a home route Once we send it over to a home route. We see our app So now we have a framework agnostic router Which is really kind of one of the keys of a polyglot framework because if we want it in a different route to render with A different view layer. It's very straightforward you can see In our router We're calling react our render on the home route But there's no reason in another route. We couldn't call reactive render or angular bootstrap, etc. We can sort of Separate decouple our routes from our rendering layer Which is what people do on a server side level if you think about it when you make a request to a server You some routes could be mapped to different backends and render different ways And that's one of the real powers of routing. So decoupling your router is very useful in a lot of ways And the final leg of our frigate sign framework is polyfills I talked about no longer having to write to libraries to abstract the browser We can now bring the browser forward and all right to the browser platform itself If you've looked at the fetch polyfill It's essentially a nice API for XML HTTP requests. I hate that it's called XML HTTP requests Except for what I'm talking to people who don't know tech because I feel like super super bad a like Saying excellent. It's just sounds a little smart But yeah, so the way that we import the way that we install polyfills is at the beginning of our app We just import it what WG? Fetch and I didn't solve that we do you fetch insults on mouse What WG fetch is a library from github that basically gives us a nicer promised based API for fetching making a synchronous request And you'll see what that looks like in a little bit So again, we import react we import annotate and the concept called inject lazy what inject lazy does is in Angular 2 It lets us it gives us a factory function that we can call to then create an instance of our model So you'll see we get create monster here as opposed to the actual monster instance Angular 2di is really awesome. I really enjoy it So far it's it's handled every case in terms of creating objects that I've needed If any of you have used react this should look really familiar We have a good initial state which just maintains a loaded and then we're gonna make a call out This is that fetch API. We just installed so we just use it and once the browser's all support fetch We take out our polyfill and we're in business Maybe has some subtle changes because certain things aren't standardized yet and we call that a prolly fill I'm not sure which fetch is do you know that it's a polyfill okay, so We make a request out to github which returns response We then return the json object of that response Responses an object that contains like get tax get htl get json, etc Then we call set state, but we create a monster, which is our model factory if you will and In render if it's loaded if it's loading We've just paint loading if it's loaded we paint the monster, but we need to augment our monster really quick To actually take out that to Leverage that response. I know there's a lot of Weird di language in this angular team is like infamous for using really smart words We have a transient scope, which you can think about it's like sort of a sort of a Short-lived object every time it's created. It doesn't get cashed by the dependency injector it gets created every time So every time there's a Frankenstein that's nude up it gets nude up just for that instance It's like the monogamous object So I think I should be like patented or something else good So when you create a Frankenstein take its response set some properties on the object really straightforward So now we're hitting github with the fetch polyfill you have that really charming photo of myself up there and Some stuff from from the internet right we've augmented the platform But we can make this more interesting really easily With the agnostic router We can go in here. We can add a username in our component. We can just paint This stop props a username We're gonna use ES6 template strings so we can interpolate my name the username Then if we go to somebody else's name We get their github account So we have a full stack here, and if you're using react I do recommend flux etc, but That brings me to my next point Is this approach viable?  What's going on here? You're you're effectively taking ownership of your code in a big way where normally you could pass that off to framework authors So I'm not sure this approach is viable, but it is adaptable We have the ability to render different views with different routes We have the ability to swap out a router if we'd like we have the ability to change out our modeling framework for different views Etc. We can lazily load code on routes, and we're not coupled to any particular framework So I do think it's very interesting in that way My goal today was not to sell you on a particular polyglot But to show you that things are happening in the ecosystem between github browsers Standardization etc that you no longer have to think of yourself as a particular framework author You're not an angular developer. You're not a react developer your web developer thinking in terms of frameworks is going to limit your mindset and Your your personal growth and it's going to hurt the future of the web I I believe that people who decide to think outside of framework boundaries are going to influence the web as standardization in a significant way and we really really need that for the future of Humans that's the beauty of the web is that any of us can contribute to the platform in a significant way and Thinking in terms of a framework limits you significantly So thank you guys so much. It's been a privilege to have your attention and thank you Mike I really appreciate the opportunity Long live the open web. Thank you guys