The reason that brought me to AOC was just because I realized that I wanted a change, so I wanted to find someone who was more of an expert in the field that they could help me and give me their knowledge so that I could gain that confidence. Usually I'd go up to bars, I had a very tight knit social group, so we would go out to bars and the bars were great and I'd usually just talk with my friends so there were times I wouldn't approach any new people unless they were brought into the social group by other people or just they were part of the social group already. I mean it was really difficult with my job, it was hard to meet people and just because my job takes me out of town and I'm isolated and there are no women, there's not no women, there's very few men as well, so it's difficult to meet someone in Calgary and if I go out to work to still make a date or make a second or third or fourth date just because I'm out of town all the time. There were times I'd go to bar and not talk to a single new person and I was certainly able to conquer the fears that I had. You're going to learn things that you've never heard of before, you're going to be able to witness them and experience them and that's really going to help solidify your understanding of where you can be and where you were, so it's really a risk that's worth taking. I know that the guys that you meet are going to be really awesome, you're going to have a lot of fun, you're going to experience things and you're going to do things that you've never thought you could do or you would do and you know that that's going to just make you a lot better and you're going to go out and you're going to have a great time, you're going to have some laughs, you're going to make some stories and at the end of the day when you're done I think you're really going to be able to look back on it and say that I'm glad I did this as opposed to not wondering maybe I should have, maybe I shouldn't have. I wouldn't be too concerned about the money because I think there's always a way, this is something you really want, which is I think something you should really try to get in your life and you can try to squeeze some pennies or really try to just convince those guys to make a good plan for you because this is something that would really help you and I think the experience is much stronger or much more effective than the payment. It's really going to open your eyes up to the world around you and any reservations that you might have I think you should really just embrace the change, embrace that on comfort because it's really going to help you in the long run. It certainly opened my eyes to what I was feeling, what I was experiencing before bootcamp and to what I can experience and what I can enjoy, how my life can be now. So I think it's really worth it to look into the program and to embrace taking the program.


process within the prepayment process and therefore when we audit the prepayment cycle, we should be auditing the prepaid expenses as well. The same kind of controls that are going to be in place, so this is another area of course. As we construct the audit process, there will be overlap when we consider these different types of accounts. We need to consider where that overlap will be so that we can do the enough testing that we need, but a minimal amount of testing in order to make the audit process efficient. Then we have the substantive procedures. We're going to say related to prepaid insurance in this case, so we have the substantive procedures, we have the inherent risk, the detection risk, then we have the inherent risk and the control risk, then setting the detection risk, then considering the substantive procedures, the actual testing items that we would be doing within the audit. Test of details of prepaid insurance accounts. Step one is to get a schedule of the prepaid assurance account. So obviously we're going to need a schedule. We're going to not want to know when the payments are going to be made. We're probably also going to need the policy. So those are the two things we're typically going to need if we get the GL account or the trial balance. If we get the trial balance, we're going to look at prepaid. We're going to say we need the GL account now. Most of the time, the company will be posting their payments to the GL account of prepaid insurance. Then they should be, if they're a publicly traded company, doing the month end reconciliation, and then we should just have to basically review that. So if you have questions about that, you want to go to financial accounting, think about the year end adjusting entry. If you're talking about publicly traded companies, they should have completed the adjusting entry, which means they would be posting their items when they make the payment to insurance to prepaid expenses and asset, not a liability. Then at the end of a time period, typically a month, they would make the adjustment to adjust the amount that has expired and the amount that has not been expired, the amount that remains in prepaid expense, the amount that is not. Now, if you're auditing a very small company or doing a review or a compilation or something like that, quite possible that they didn't do the period in adjustment. And you may actually have to look at that period in adjustment, which is basically just an adjusting entry, or it could be possible that instead of posting the transactions to prepaid insurance, they're posting it to a prepaid to insurance expense. And in that case, of course, you would just basically do the reverse of the adjusting entry. Instead of adjusting it out of prepaid insurance, you would be saying, okay, now prepaid expenses overstated, you'd have to reduce that. Again, publicly traded company, that adjusting entry you would think had been taken place, and we just need to basically test to see if it was done properly. Assertions related to prepaid insurance includes valuation, calculate portion of the policy that is not expired and portion that is. So with regard to valuation, we're going to get the policy, we're going to say, okay, how much time period does this policy cover? And then we're going to consider how much of the policy has expired and how much of the policy has not been expired. So the assumption here, we have the policy policy has been paid for, we can see that the policy has been paid for, then we consider the policy that has been paid for, how much of it has expired, how much of it has not expired, the amount that has not expired then should be still reported as prepaid insurance, the amount that has expired should be expense rights and obligations, confirm policy beneficiary. So we want to make sure that we know who the beneficiary is of the policy, inquire the insurance broker. So it's useful to be able to talk to the insurance broker because the insurance broker is outside of the organization. So it gives us a little bit more confirmation or assurance, a little bit better evidence that we can compile to give us assurance about prepaid insurance, existence and completeness. So with the assertion of existence and completeness, confirm policy with the insurance broker. So again, we could talk to the insurance broker about that outside of the company, bit more assurance, examine supporting source documents, so we can consider the source documents as well. And then we have classification. So classification assertion determine the correctness of classifications to accounts. We want to make sure that the accounts are being properly classified or the amounts are being properly classified to the proper classes.


The Director of Pediatric Ethics at Mercy Hospital in Kansas City, and today, John, I don't know what you're talking about. Genomics. Genomics, okay. And what happens when genomics leads to a false positive result? Exactly. Thank you very much. Thank you. And thanks. It's great to be back and always humbling to hear so many great talks. It's an astounding record of success here at the McLean Center. Those of you who live here may think that this kind of stuff happens everywhere else. It doesn't. It's an amazing place. My talk will really be focusing on this. We have a solution, but now we're trying to figure out the problem.  No conflicts, but we'll be discussing off-label use of things that the FDA calls devices, but I'll explain that in a little more detail soon. So in 2010, my hospital in Kansas City, Children's Mercy, started a Center for Pediatric Genomic Medicine. And the next year, the NHGRI had a call for proposals to look at ethical issues in newborn genomics. Like all good ethicists, if grant money was available, I became very interested in the topic. We had a director with an audacious idea, Stephen Kingsmore, whose idea was to sequence genes in 20 hours, whole genomes, provide interpretations to doctors within 48 hours for sick babies in the NICU, and try to show that such results would actually improve the care in real time of sick babies in the NICU. At the time, whole genomes usually took about three months to go through this process. The problem was that sequencing was getting easier. In fact, there were machines that could churn out whole genomes in 12 or 18 hours if you had enough money to buy them. But interpretation is pretty hard. Leslie Beeseker wrote in a paper about opportunities and challenges. You get the result, but it's overwhelming because the variants you see range from those that are extremely likely to cause disease to those that are nearly certain to be benign and everything in between. So how to speed up interpretation? We did what most places have started to do, which was to develop our own proprietary software or informatic devices or interpretive pipelines. These things have lots of names, and we, like everybody else, gave them cute acronyms. As far as had sort of a viking theme, S-Saga, Runes and Viking, none are validated for FDA approved because all are constantly changing and being updated with every new interpretive discovery or report. So how does it work? What does S-Saga do? Software by which clinicians can describe the signs or symptoms that the baby has, and the software matches those symptoms with previously reported candidate genes. So clinicians would go to a software interface that looked like this. You'd enter a new patient. You'd get drop-down menus for cardiovascular, endo, ID, constitutional, metabolic, et cetera. You'd put in the signs and symptoms that the baby had, and the software would spit out a list of genes that had been previously associated with that. Then the whole genome would be sequenced, but only those genes that S-Saga said were likely to be associated with the baby's symptoms would be analyzed and this shortened the time necessary for analysis and also, importantly, reduced the likelihood of incidental findings like BRCA or genes associated with Alzheimer's because although we had the sequence, we wouldn't be looking at it. It would, so to speak, end up on the cutting room floor. Then to figure out whether the variants identified were pathogenic, we had another software program, RUNES, Rapid Understanding of Nucleotide Variant Effect Software. Good, huh? Took these variants and assigned them to existing American College of Medical Genetics categories for pathogenicity. It did this by running each of the variants through dozens of existing databases which are used to categorize things according to this framework that's probably familiar to many of you. Category one previously reported to cause the disorder novel or expected to cause the disorder novel may or may not be causative or sometimes called a variant of unknown significance novel unlikely to cause the disease and the docs would get complicated reports like this where at some point on the report it put it into that category. This is a category one likely to cause the disease. And then Viking would do variant integration and knowledge interpretation which integrates Saga and RUNES, prioritizes variants and generates a report for the doctor in understandable lay language about whether this process led to what was called a molecular diagnosis, a genetic finding likely to explain the baby's symptoms. So taken together, these pipelines suggest whether genetic variants are likely to be associated with the patient's symptoms. Does this without the need for much human thought or interpretation? That is, once the doc filled out the Saga list of signs and symptoms, the rest was pretty much automated. And it led to lots of molecular diagnoses, not just at our place but at most places, people using similarly off-label and proprietary interpretive pipelines that are all a little different report high rates of molecular diagnoses in a variety of different populations. So all these populations are what I would call highly enriched. They're all populations of people thought to have some sort of genetic disease. But these are four recent papers, somebody said 40%, somebody said 57%, that was us, 58%. And that, I would say, is very cool science. Wouldn't you agree? Yeah, yeah. In fact, Stephen Kingsmore got into the Guinness Book of World Records for the fastest interpretation of a human genome. How cool is that? Lots of people thought this was the way of the future. This guy started the Precision Medicine Initiative. Francis Collins, the director of the NIH said, DNA sequencing will lead to a medical landscape in which each baby's genome is sequenced and that information is used to shape a lifetime and personalize strategies for disease prevention, detection, and treatment. So what could go wrong? I think the problem is with this term molecular diagnosis because the question is what does it mean to find a gene that's associated with a pathologic finding, especially because association, as we know, is not causation. Here's how Yang, one of the first papers looking at this, published in JAMA just about three years ago, defined molecular diagnosis. A whole exome sequencing case was classified as molecularly diagnosed if the pathogenic, so the result is in the definition of pathogenic or likely pathogenic variants, were detected in Mendelian disease genes that overlapped with described phenotypes of the patient. So a gene that had been previously reported, if it's a novel variation that overlaps, counts as a molecular diagnosis. But do these molecular diagnoses mean anything? Do they lead to beneficial changes in treatment? It turns out that there are very few reports of beneficial changes in treatment and most of those reports are not in peer-reviewed medical literature. In fact, they appear in places like Time Magazine. Genetic screening saved this baby's life. Time Magazine reported this was actually a case from our place. Six months old, that abnormal development was in the PICU on event, got molecular diagnosis that, according to time, led to beneficial treatment. The variant was associated with a defect in citrate transport. He was started on citrate and reported by time to be doing better. There was no follow-up in a peer-reviewed journal. If this is true, this would be a huge breakthrough and certainly worth reporting. But it's now three years later and there's no peer-reviewed report. So that was part of what this NHGRI proposal was meant to stimulate, to get people to start doing more rigorous studies. And we designed a study to try to figure out whether these tests were actually beneficial, whether they were harmful, or whether they were just irrelevant. Here's what we did. We had neonates who the doctors thought might have a genetic syndrome. We randomized them either to current standard of care, whatever that was, we didn't define it, or current standard of care plus the whole genome. So some of the babies got the genome, the whole genome, some of them didn't. And the main study question was, if you get a molecular diagnosis, the doctors and the parents perceive it as helpful, harmful, or cute, but irrelevant. Here were the preliminary results. It was a level for NICU. We only enrolled babies less than four months of age. We randomized 65 patients, about half got the standard tests, half got rapid genomes. Note that there were five crossovers because even at the beginning doctors had some trouble with equipoise, and if babies were randomized to standard care and the doctors really, really thought they needed a whole genome, they could cross over to the whole genome. What they found was, what we found, with whole genome there were more molecular diagnoses, 41 versus 15%, and they were made quicker, 16 days versus 65 days.  That is, if you're doing a test that's being done faster, the fact that it gets done faster is not a surprising finding. And in 57%, we got a diagnosis. The doctors felt that the diagnosis was clinically actionable in 37%, but here's what I want you to focus on. Of those 37%, about half of them, the action was what we called palliative care guidance, which essentially was a recommendation for palliative care in the published report in Lancet. Genetic diagnosis that confers a dire prognosis can empower early discussions with the infant's parents about palliative care to minimize suffering, and palliative care was given to 29% of infants with a genetic diagnosis, and I had this made up about half of the cases that were reported to be clinically actionable, 17 out of 37%. So let's look at one case just to see what I see as the problem with this sort of thing. This was children's mercy hospital case 545, a baby with bilateral chylous effusions of VSD and unusual facies. He was enrolled in the study and found to have a gene associated with Noonan syndrome. For those who like this genomic language, there are the details, and was reported as having a molecular diagnosis of Noonan's. The diagnosis was made at 69 days of age, palliative care was initiated, and he died at 88 days of age. So what is Noonan's syndrome? Features are short stature, congenital heart defects, developmental delay. A lot of babies have congenital heart disease and some mild intellectual disability, not in every case but in some cases. The heart disease is treatable and usually the developmental disabilities are treated with early intervention and individualized education strategies. There's many genes associated with Noonan's, although again these are associations and causation is difficult to prove, but there are no population studies to show how often these genes occur in asymptomatic or people without clinical features of Noonan's. So it's hard to know whether these genetic findings are true positives or false positives. So it seems there's many genetic variants. Each may or may not be diagnostic. The disease itself is not fatal. The molecular diagnosis may be a false positive, but even if it's a true positive, it doesn't justify withdrawal of life support. So some key questions here. Did they really withdraw life support based on the molecular diagnosis of Noonan's? If so, I'd say that's really bad clinical care. If not, then it's very bad science to report it as a clinically actionable finding and the way it was reported, we just don't know which is which. And this is a common feature of every report you read or that I've read about molecular diagnoses. There are exaggerated claims of clinical utility, no scientific standards in making these claims, little documentation of follow-up or outcomes. And it seems that peer-reviewed journals lower their standards in their awe and wonder over molecular diagnoses, or it may be that the papers are sent to other experts. That would be other genomicists who have an interest in showing that the work that they're doing is valuable. One more cautionary tale because this is one example where, unlike most tests that come back, we actually know what the prevalence of genomic variants is in the general population. Crab A disease is a neurodegenerative disease. On-setting infancy usually leads to death before age two and according to the Mayo Clinic website and many other sources affects about one in 100,000 live births. Remember that one in 100,000 number. So New York State and many other states, including now Illinois, started a statewide screening program in 2006. It was a very rigorous program. They first measured the enzyme activity, which was the traditional test and then they added for specimens with low enzyme genomic sequencing of 17 genes associated with Crab A and infants with low enzyme and a confirmatory molecular diagnosis were then sent for clinical confirmatory testing, which included a detailed history, physical exam by a neurologist, repeat of the enzyme activity and the genotype MRIs, LPs, and nerve conduction studies. So this was an extremely conservative approach to diagnosis. What were their results? This was as of about a year ago. They'd tested 2 million babies, 99.9% were negative, but 620 had low enzyme activity. That's one in about 3,000 babies. Oops. Of those, about half had a known or potentially pathogenic mutation in the gene. That's a molecular diagnosis. That's one in about 6,000 babies. And those were further evaluated. About two-thirds were thought to be at low risk, but about a third were, or no risk, about a third were low, moderate, or high risk. If you put those together, that's one in 14,000, so about six times what was thought to be the population prevalence before this was done. And of the 14 highest risk babies who were sent for confirmatory exams, only five had an exam consistent with Crab A. Four underwent a stem cell transplant, which is the only known treatment. Two survived with severe developmental delays and two died within three months after the transplant. But most importantly here, nine, or about 70%, remain asymptomatic and follow-up is now between one and nine years. So 70% of these tests were false positive and we found that the population prevalence of low enzyme and confirmatory molecular diagnosis is much, much higher than we thought, although most not associated with the development of the disease. So given these numbers, what would happen if somebody did a test for the gene of a sick baby in the NICU and found they had the genomic variant associated with Crab A? How many parents would be counseled to withdraw life support? How many would agree to a bone marrow transplant? And how many of the tests are false positive? In Crab A, we now know 70%. For all the other genomic findings we're getting, we have no idea. Quick real-world example, even though my time's up as part of this study, we also did a bunch of interviews with parents. We talked to one parent who had a set of twins. Both were normal until 11 months, then one started losing developmental skills and at 13 months had seizures. And after an eight-month diagnostic odyssey was diagnosed with Crab A at nearly two years of age. Both twins are still alive at ten years of age and the twin with Crab A seems to have leveled out developmentally, that is still impaired, but not getting worse. And the parents said nothing they read or heard from any doctor, any website, or any support group was accurate as to what the clinical course would be. And the natural history for the kid with Crab A was very similar to the natural history of kids who get bone marrow transplants and there's a claim that the bone marrow transplant was successful. It's illustrated by this study that was published in neurology just a few months ago, developmental outcomes after cord blood transplantation for Crab A, cognitive development below the third percentile and gross motor skills not developing at all. What seems like claims of success mirror the natural history of untreated disease. So, what are future needs here? I think we need standardization of the pipelines for interpreting variants, more careful reports of clinical utility, tougher peer review, and all in the service of learning more about population prevalence of genomic variants, penetrance, expressivity, natural history of disease or non-disease in the full range of patients with particular molecular findings. In spite of this, our doctors love whole genome sequencing. They're no longer an equipoise. In fact, we stopped the study because they're unwilling to enroll people to get them randomized and instead it's a complicated slide that basically shows the total enrolled in the study has dropped to zero in part because we're now offering this test clinically so you don't have to enroll in a study to get it. The key question may be where are we in the hype cycle? It seems that we're now here at the peak of inflated expectations. I think we're headed to here because we'll soon be able to measure just how many babies died because of inaccurate tests or inappropriate decisions or inaccurate but misleading tests. Thanks. That's children's mercy. It looks a little like Disneyland. Isn't that nice? John, this is great work and I think what you're doing is really pointing out what happens with new technologies in general. I don't think that genetics is although it's offering a lot, I think new technologies in general are approached like this. I think that applies to many. Obviously there are lots of conflicts of interest of people who are involved with this. One is just that's their field and they want it to be good. There's also the idea that these tests are sold for money and that's a profit center in hospitals. How do you see us making this a better process based on what you've found? It seems only by a careful study and the integrity of researchers to accurately report their results and report them with more rigor. As you say, it's not a unique story to genomics. Adam Sifu here has the best book on medical reversals which would be situations where we've gone from the peak of inflated expectations, everything from pulmonary artery catheters to antiarrhythmics to postmenopausal hormone therapy, all of which have similar things, but that eventually through careful and courageous research are eventually shown either to be useless or the hope would be after the expose. You do find the situations where it's most useful and then tailor the treatment or the test for those situations. Thank you. I really appreciate your talk. Tell everybody who you are. I'm Ben Brown. I'm one of the current fellows. I'm an OBGYN and a family planning fellow here. I agree with you wholeheartedly about the rush to adopt new technology. I was just having a conversation earlier about robotics in surgery. We could say that about many new technologies and I take your point very much. In particular, the correlation is not causation aspect of it. I want to push back a little bit on the primary ethical argument you make or at least the one that you highlight in your title which is the concern about death and are we having unreasonable deaths, unwarranted deaths for pediatric patients. I actually want to, I don't know if she would agree with me, but I'm going to borrow a little bit from Dr. Ross's framework of pediatric ethics and say while the certain knowledge of uncertainty is a valid part of decision making. And so I think that perhaps our biggest problem is the way that physicians are understanding these tests, the way that they are incorporating them into the clinical picture and the way in which they are counseling, not the fact that this is actually being incorporated potentially into the counseling and in fact that someone might choose to withdraw life support as a result of it as long as it is understood in the proper context of its uncertainty that that to me strikes me as an appropriate respect for the constrained authority of the parent in deciding what is going to be the ongoing course for that child's care in particular because my assumption based on the cases that you've described here is that these tests are being done for patients who do have obvious and significant clinical findings. What are your thoughts on that?  It's how the test is used but if what we learn from doing studies is that it's used in ways like this yeah there would be two solutions one would be use it better that could be either figuring out where it shouldn't be used at all or as you suggest figure out how to better convey both to the doctors and the parents the uncertainties associated with it so that may be an answer to Dan's question I mean to the extent that we could do that we would avoid many of these harms it's difficult to do though Thank you Thank you John Excellent as usual I'd have a comment and a question the comment is as was mentioned earlier this is something that is not new about genetics Jerome Lejeune 50 years ago when he described in trisomy 21 causing Down syndrome predicted just this kind of thing and I would tell you those of us and he was right and those of us who work with disabilities we're still in the trough of disillusionment there's no upward slope of enlightenment Individual with Down syndrome now live a full life and yet we're still screening for it and spending tons of money over it and so the question I have for you is if it's moved to a clinical who's paying for that? Are the citizens of Missouri paying for that if they're on Medicaid? That's a moral outrage when we can't pay for other things and we're paying for this which is causing more death that's terrible I guess the question is if you have reported I mean multiple reports in peer reviewed journals showing that it's useful many of the reports show that it's not just making diagnoses but they're clinically actionable if the doctors who are caring for the babies believe that it's useful and want to order the tests and the parents want the test as well where would you locate the moral outrage? Paying for it but if everybody wants it and we live in a democracy you just pointed out what they're wanting you're just actually wrong I'm not sure how to harness my moral outrage John it's even more complex because we're we're something like Crab Bays was put in newborn screening panels under the assumption that we would find these children and convert a lethal disease which was a rapidly progressive dismyelination with no responsiveness in the first six months of life to a kid who walked and talked and what they did is change it to a slowly progressive complex disorder at the same time to parents who had the condition that was much better than being minimally responsive again to Peter's point motivated by parents who had a kid with Crab Aay to get it done had hope but the individuals who had those genetic variants outnumbered by more than tenfold they were told you can never know when you will become the patient and waiting for the rapidly progressive type and their children and I know that you and I this yesterday Mike is developmental pediatrician who in Illinois has to evaluate all the kids to do that detailed neurologic evaluation so sees all the false positives whose parents are coming back. Let me just tell you one story though that relates to all these points and then I know we have to stop I gave this talk at Vanderbilt or a longer more complex version and at the end of it to your point that it's substituting another disease and these kids are not going to get up and walk one of the people who asked a question was a pediatric neurologist and he said now the problem is we're just not transplanting them soon enough if we transplanted them sooner it would be completely curative and I said well then you're going to have to transplant a lot of kids who never would have developed disease and he said no we're not we suggest you have to transplanting them in utero because many of them are born with the symptoms


Photon University is not exactly a university as the educational university. Photon University, this is very specific to Photon Interactive, built for Photon's internal training requirements. I am Giridhi, I take care of the learning management system of Photon. Photon is an omnichannel service provider, basically into IT services for all devices. The goal of Photon right now with respect to learning is to increase the knowledge capital of the existing user base. To increase the knowledge capital would also mean to record the learning data and also in other words to capture the learning data. It also need to develop a lot of content. Only with a lot of content we'll be able to, within the LMS and only then we'll be able to capture the learning data. Thereby we will have a clear understanding of what our knowledge base is for the organization. Most small and medium enterprises will have a similar scenario where the knowledge capital or the repository of knowledge of its employees or resources is not exactly captured. Unless it is there in a system like in LMS, it is not possible to capture it exactly to the rate. Only when there is an assessment or content that is there and there are some courses available upon completion, the organization will be able to know that these are the set of skill sets at least at the level of learning the employees have which can be utilized later. This is a very typical problem or a challenge that we have from even from the education institution side we just discussed. It's creating a content is quality content is always a trouble. It involves a lot of things. In the traditional method of content development in an online environment you'll have to go through a lot of sequential steps. You'll have to collect the data and you'll have to speak to the SME and put it in the right sequences storyboard it illustrated if it requires you'll have to go through the creation of audio, video, editing, audio editing and a lot of that. Only then you'll be able to create a quality content which can be produced in the course. To all of this you need the SME's time and a good amount of SME's time but in all practicality to have an SME's time in a set of like hours and most SME small, medium enterprises it's really hard to do that and we get little to no time of the SME's and it's really hard to create a content that addresses the learning requirement and it's even more hard to produce a learning content on time as it is required to meet the learning requirement. It's really hard. Hence the solution is switched from content creation to content moderation and curation. Instead of creating the content right now as in usually as a training team we go to the SME's and we ask for the subject information they provide us the information then we restructure it, write the objectives and go on with the creation of the content and in fast-paced agile and learning requirements such as photons requirement is as such it's by the time we create and complete the course that requirement is already over. So as a solution we are rather than focusing on creating the content we empower the SME's to create the content themselves. It is that I've been also stuck in this section for a long time as the doctor explained we'll have to create quality content the video should be really good the quality should be really good only then the users will give any attention if at all but it's not exactly that is also my understanding as long as the content is structured right it doesn't have to be really good in terms of graphics it's still fine, it works if the content has its structure right tell them what you tell them tell them and tell them what you told them if it isn't the right sequence these basic things are in place the users will still be able to make use of it and they are making use of it now always a high rich graphic design graphic rich content is always not required it is still okay even if there is not much content to it as in the graphics is there before we switch to this methodology we developed probably just 5 courses in 3 years 5 technical courses in an online environment these are the only 5 courses we were able to create in 3 years of time which we were able to make use of everything else are many a number of attempts it fell flat because most often we were not able to meet it create the learning materials itself and most often even before we could reach the complete the learning material the need is already passed most of the users who needed that learning material has already acquired that information from some other way and it's always in vain but after we switched to this method we were able to create 60 courses in a span of just 2 months so this helped us a lot what influence the changes we absence of rigid and formal structure of course format and choice of delivery methods content aggregation was accepted so as I said if there is a content that is available on YouTube or any other channel from LinkedIn to any other places we readily accepted with the citation and we also credited the authors who were even if it is not creation if they had just curated the content or just collected it it was still accepted and we gave credits to the author for doing the same so this is how it's exactly worked out little bits of pieces of content doesn't matter if it has to be an exhaustive course even if it is part of a course it still is accepted as it is it could be in the form of a self shot video a podcast, a word document or even in our case a directly a code sample it doesn't matter as long as the content is usable it will accept it so this way we will be able to make instead of in a place where the creation of content takes a lot of time and always goes out of even before the learning time is required we were not able to meet that now we are able to do that by taking the content directly from the SMEs in their preferable formats and we make use we connect this individual learning content with the help of the competence framework so let's say a user creates a learning content for some security-related course and it does not have to be completely available all the content is not required to be completely available part of the content even if it's there it still is fine we will be able to make sense of it by mapping it to the competence framework so little bit of each of this content we receive from so many different sources from so many different SMEs we will put it together in this framework and we will be able to make sense of it the competence framework will act as a guide and will engage the users and the training team it will be for the training team it will be the place where for the list of competence is listed we will know exactly for what competencies the content is available and for what other competencies the data is not available and to focus on that area for SMEs it gives an incentive for them to start working on areas if they have already information if they are the experts they will be able to come and provide information in that area for users they have the content and they will be able to make use of the content that is already available within our LMS which turn it records into our system as a competency that they have also acquired at least at the learning level so from the competency level that is acquired from the system as the user clears each of the assessment as per the learning materials our PMO team can make use of it which otherwise may not be possible because learning data earlier is not exactly captured it is not up to date in this method it will be up to date any new technology or skill that a user may have learned the PMO team will have the up to date information as to at least at learning level what is the stamp and based on that they will be able to assign it to projects with the appropriate skills for example blockchain is something that is really new and not many resources have that skill but if some user had already created that already knows that technology and that SME created some part of it lectures, videos, podcasts and it's there as a content and some more users made use of it and that sets as a base so with little effort from the training team and from no content to some content and the knowledge base is also increased so that is how the shift from content creation to content moderation and then setting up the competency pool and to the PMO and making use of that collective learning exercise into making it use becomes useful these are some of the key metrics that we focus on the total number of courses completed by users average time spent these are metrics that we collect we build this custom tool a plugin which collects this information for us for us to focus to start with and this metrics on how we calculate these numbers will differ as the system matures the challenges that we have currently is that the competencies are not or it's currently tied directly to the course this works perfectly well in educational institution setup because for the competencies for a course when it is listed it would also always have related content to that competence that is listed here but in setups such as our organization it's not always that the content will be available even though the competency is required at least at a learning level the content is not readily available as a course so there is an option there is definitely an option in the model to attach an evidence an external evidence setting that I have competency for iOS development or iOS security but when a user submits that any reviewer anyone, any person who has a competency review access will receive it the problem here right now is there are many departments from front end to back end and many technologies JavaScript to database .NET and a lot more various on that so irrespective of the department right now anyone who submits an evidence stating that I have iOS application security competency everyone with that review access or that role will receive that notification and that looks very cumbersome and also the learning plan mapping also doesn't look very promising looking at this scenario so this is an area that we are working on currently and hope we also find some answers from other users and also from the core team if there is a possibility where we can assign competency review to a very particular SME or a person who can give the authentication yes this person is competent or not to be able to do that and we have customized our foreign university and this is based on model and these are some of the screens we also build a custom mandatory plugin I'm not sure if it's very clear okay there is four days left around this area so that would be like it can be set at course level and also it can be customized to in a way where number of based on the very specific list of users the course due date can be extended or retrieved that is possible and we also have gamified the system there is definitely a point system for every action almost for logging into a system to enrolling to a course to completing an assessment to completing a course every action is monitored and those actions are awarded in the form of points and there is a lead report where the user can the user is listed in the ascending order and the details of the points are also listed here and with the along with the points the user achieve those points number of courses the users are completed points received and badges, certificates and competencies all of it is listed there and this is this is the course catalogue page is also very customized and we also have something very different here it has two types there is small bytes technically speaking in modal everything is a course small bytes is also a course course is just a course the reason why we have separated this both is that as said before we accept content in very little bits of pieces also which is not enough to form as a entire course for example an IOS application security course there will be a lot of aspects of it it may need about 50 different topics to be covered and a user can provide only two or three of those content we still accept that and give a heading to that and we create a course with that information and that will be a small byte as in user can check into that course and just read that information when we have enough information out of the small bytes we will convert that into a formal course with the proper structure with assessment and everything probably a certification and setting the right parameters and completion criteria everything will be set the quick bytes that's sorry the small bytes is just reading material every bit of information will be just there so this is how we also we instead of waiting for to create a really good content and never have been actually to achieve it the content is definitely there and from there we grow on from small bytes to a course a full-fledged course this is a enrollment page customized and this is the actual course page a forum learning activities quizzes this is the custom dashboard that we are talking about they are 5k matrixes and and also one of the problems that we face is that even before we start with the content and from rightly from many of the presenters who also it is hard to to get the users to the learning portal the usability is very low it's not probably the issue of content it's not only the issue of content at least many users refrain from or decide to use the content or not even before they log into the portal and an active and effective PR is required to get the users user system it is a major mindset for the users and how they view the portal as if the portal is cool enough to use they'll definitely make use of it and more than whatever the customization we do irrespective of the good looking the UI is it mostly comes the most important part is how we set the image of the portal as that along with the ecosystem of it the SMEs should have a way for them to benefit there should be a system where the SMEs for putting in the effort should have some benefit out of it for the learners they should have some benefit out of it yes definitely they have it's a given but it has to be explicit and marketable only when we market it right we'll be able to get the right expected output even before we start working on the customization thank you


All right, so hi everyone. My name is Mohamed. I work at Vex Hosts and I've got Matt from Stack HBC joining me today. So today we're here to talk to you guys about Magnum. So can I get maybe a show of hands of who's used Magnum in OpenStack in the past? Okay, we got a lot, wow, that's cool. Now can I get a show of hands how many of you have been frustrated by Magnum? Okay, wow, that's the same group. So that's why both me and Matt got equally frustrated and we figured there's probably time for a new Magnum driver. I'm sure all of you guys have spent a fair bit of time messing about heat stacks that are getting stuck and doing like all sorts of weird things to finally get things unstuck. So most of you guys will know about Magnum, so I'll go very quickly for those that don't. Magnum is an OpenStack project to allow you to deploy what it was initially called container orchestration engines. We lived in a world where Kubernetes wasn't the only thing and there was Docker Swarm and Meizos and all these other things, but nowadays Magnum is really for Kubernetes mostly. Though it would be nice if it supported other things, but for now that's mainly what it's for. The way that it works is you build a cluster template and when you build a cluster template that defines the state of the cluster, usually there's an image attached to it, the version of Kubernetes, maybe some features like if you have monitoring enabled or for example, if you have things like an ingress that you'd like enabled and then Magnum goes ahead and does that and there's just a Magnum API that receives the calls and there's a Magnum conductor that pretty much orchestrates calls to the heat stack. So heat stack, so it uses heat to provision resources and as we know, heat project is not getting a lot of love and maybe the room probably doesn't have a lot of love to it with all the wounds that we have from it. And it was also deployed in a non-standard way. So before it was relying on the hypercube image, which was something that was like the Kubernetes community was really all about at the time. And then eventually running the hypercube image or running Kubelet containerized became something that the community was against. So you can containerize API server, controller manager, scheduler, but the Kubelet was really meant to be running unconstrained directly on the host. But the community kind of stuck to that. So even Rancher, which was one of the last people to stick to the hypercube idea, have even dropped it. So you kind of had the challenge of trying to figure out how you get your Kubelet images built out. It was also a whole collection of batch scripts. So there's a lot of batch scripts that are all cobbled up together that were running with software deployments inside of heat. And then hopefully somehow all of those got together and got the cluster to run. And upgrades almost were never worked. It was kind of too risky to actually do the upgrade. You would probably rather just not do the upgrade at all and get a new cluster. And it was also getting difficult to maintain and slow to add new Kubernetes versions, because obviously Kubernetes moves so fast. And apparently it's a project where it's still on the one dot something release, but it had so many API breakages. I don't know. That's another discussion to have. But obviously it moves fast and so it's hard to keep those up to date. And so yeah, cluster API is to the rescue, so. Yeah, so can we do better than heat and bash and hypercube? And the answer is yes, and the answer is cluster API. So what is cluster, has anyone here used cluster API outside of Magnum before? Yeah, a few people.  So, and it's not just open stack that it supports. So you can use the same tooling to deploy on public cloud bare metal open stack. But the really cool thing about it is it uses all the standard tooling underneath. So that was maintained by upstream Kubernetes. So things like Kube-ADM and the core cluster API controllers are all shared as well. So what that means is we as an open stack project, we only have to worry about how we make network, some machines and things like that in open stack. And that really gives us a lot of power to share code with other projects in upstream and gives us a lot less to maintain. It also means that the auto healing and the auto scaling can be implemented using provider agnostic controllers, which is really cool. Again, a whole bunch of code we don't have to maintain anymore. So what you need, obviously to do this, is you need a Kubernetes cluster first, which cluster API refers to as a management cluster. And this cluster runs all the controllers that constantly reconciling the machines in open stack and things like that. And then your workload clusters you can manage with CRUD operations on those resources. And what this gives us is we can get rid of heat, which is a big plus for us. Cluster API manages the open stack resources directly using the open stack APIs. We get to use standard tooling, like I said, so we don't have to build our own hypercube images anymore. We don't have to use all these brittle bash scripts anymore. We get to maintain less code. Upgrading clusters actually works really nicely. Yay. And there's a really active upstream community around all of these components, which decreases some of the burden on the Magnum Core team. And these components are used in lots of other projects as well. So things like OpenShift and Gardner, Rancher and Asimuth. So I'm gonna hand back over to Mohamed to start talking about how we went about developing these drivers. Yeah, and as a quick side note, I think everybody here, if we have this much Magnum users, if you are at the keynote and you notice my coworker, Guillermo, actually upgraded a Magnum cluster on stage, I don't think anyone would dare do that with the current Magnum code. Yeah, definitely not. I know some places that have disabled it by policy. So, right, so the whole point of Magnum is there's multiple drivers. The reason why there's multiple drivers is like we said, it's a container orchestration engine. So there was a mesos driver and things like that. So we figured, hey, why don't we just add a new driver that supports cluster API? And so essentially the code to detect what driver to use depends on the COE field. In this case, that's Kubernetes. And then the image operating system and the machine type. So machine type is virtualized or bare metal. And then the image OS is kind of where we can fork the road because Magnum uses Fedora Core OS and with cluster API, you can run both Flatcar or Ubuntu, which we're all very comfortable with running clusters. So all the existing drivers, like I said, are heat-based and we figured we would implement a new driver that speak cluster API. So the idea is that since if we implement the driver, all it really has to do is, and this is really oversimplifying it, is that it receives the request from Magnum, translates that to, you know, a custom resource for the cluster API and applies that to the management cluster and then just kind of orchestrates and reconciles the status inside Magnum to the status inside the cluster API. And so essentially, Magnum is almost becoming a translation layer for the cluster API. But why would we do that, right? Why would you just not run cluster API for everyone? And the reason is, I mean, OpenStack has a huge community, is a huge set of standardized APIs. And if we didn't do the driver, then every person that raises hand here could not use that out of the box. I'm sure there's people using Terraform to deploy clusters. There's people using Ansible to deploy clusters. There are people using CLI to deploy clusters. If we drop that API, we would have to re-implement new clients, new Terraform providers, new Ansible modules, and that's just too much work. So the idea is we wanted to leverage that existing Magnum driver so that Horizon still works the same way. All the users that are used to using Magnum on a day-to-day basis, well, they don't see the difference. They're just like, wow, it just works a lot better now. And that's the thing we want to do at the end that is fully transparent in the background. And so I'll start by kind of giving a thing, because this was really early in development. So the reason why both me and Matt are up here is actually StackHPC and VEXOS. They're two separate organizations, but we've kind of realized that we were kind of doing the same thing. And so we're here to kind of talk about the two different drivers. Essentially, they both use cluster API at the back end. And ideally, we'd like to go and converge completely because we've recently just started realizing, wow, we're pretty much doing the same thing with some few key differences that Matt will highlight on his side. But I'll talk about the VEXOS driver. So the VEXOS driver lives on GitHub. It's fully open source. We got a very small shell script called Stack.sh. You put that in an empty VM and run that and you'll actually get a Dev Stack with the Magnum Cluster API implemented. I really recommend you run that in an empty, clean VM because it is very intrusive. It's going to install Kubernetes. It's going to play with your networking. So don't run it on your laptop or you'll have a bad time. And the focus, the way that we've kind of approached and we've taken is in the cluster API project, there's been kind of the idea or this concept of managed apologies.  And it made it a little bit harder to manage all those manifests. And the community realized that nobody wants to really do that. People just want to say, I want a Kubernetes cluster with this much control plane nodes and a couple of different node pool groups or node groups. And this is the health checks I want and you want to just have one manifest you apply and then you get the cluster at the end of the day. So we adopted this cluster class feature. So we actually have a cluster class that is versioned with the version of the cluster API driver that we install on the system. And then we use that. So when you create a cluster, we create a cluster resource. We point it to the cluster class that we create when we install it and we point things like replicas. We translate node groups and things like that. We have the drivers running in production with a few of our customers and the things that it supports right now because obviously we're kind of building a new driver. So there's maybe a lot of things that were implemented in the past that might not be there. So we've been kind of working on getting the feature set up to sync. And so what we support is we can do clusters that are boot from volume VMs. So if for some reason you have an environment where you only do volume, sender volumes for your VMs, you can use that. We support node groups. So in Magnum, there's a feature of node groups where you can have a set of nodes that have maybe a different flavor or maybe you have a different flavor because you want a bunch of nodes that have GPUs or anything like that. So we support that. We also support resizing clusters. So if you wanna scale up or scale down, you can do that. And also auto scaling is also available. So if you enable it, you can give it a min and a max. And it'll run an auto scaler outside of your cluster and pretty much scale up or down depending on your needs. And what's nice is that's fully all upstream. It's the cluster API auto scaler. So once again, it's all shared code. We also support upgrade clusters. So you guys saw that at the keynote. It does a full rolling upgrade. But once again, the nice thing about this is we don't actually have to write the upgrade code. We just have to tell the cluster API to upgrade the cluster for us. And that's the part that is really well tested. Also cluster auto healing, something we've added, but essentially it's just piggybacking on the cluster API machine health check features that's already there. And then what we do is we pre-install the Cinder CSI, which allows you to get persistent volume claims that are pointing out with your actual cluster. And what we also do is we install storage classes that match the volume types that you have in your cloud so that the user can just hit the ground running if you have an SSD type that'll just show up as a storage class in there. We install the Minola CSI as well if Minola is inside the cloud and obviously the Cloud Controller Manager so you can get load balancer type resources and things like that all working out of the box. And there's kind of small things here and there. For example, Kubernetes auditing can be enabled as well. And so we try and match as much labels as we can. We also have a neat little thing, which is we actually allow deployment of a fully isolated cluster. So Magnum dials out to the control plane of, so sorry, Magnum created clusters before they dial out to the control plane to kind of report their status. But with the cluster API, the whole point of how it's built is that the cluster API needs to dial out to the cluster. And we have scenarios where the control plane where we're running cluster API might not have reachability to the clusters where the user's creating. It could be a fully isolated network for security reasons. So what we've done is we've actually built a service called the Magnum cluster API proxy and it's very simple. What it does is it sits on the same nodes as your control plane nodes. And I'm sure if we've all used OpenStack, everyone's always done a IP net NS exec QDHCP and jump into a network. So essentially, HA proxy has a feature where you can send traffic into a specific namespace. So we've used that feature and pretty much watched for the namespaces and we kind of reconciled that with an endpoint slice which transports into endpoints. And then we have really a Kubernetes service in the management API that talks directly through that HA proxy, through the namespace all the way to the actual system which means it allows the cluster API to kind of be clueless and not know that it can't technically reach that cluster and the reachability is going through the HA proxy that we have set up. We got a couple other tools. So we have a tool that ships all the necessary images to build a cluster to registry. So maybe if anyone has ever used the container infra prefix and played a whack-a-mole with figuring out what images they have to have in the registry. We just have a tool, you just run it, point it to your registry and use this crane to push everything up there. We also have a preloaded image. So if you wanna download six gigs into your cluster and have all the images in there, you can also do that. And then we have also a tool to allow building images. So the images that are used by cluster API, same ones we use, same ones that StackHBC uses are actually upstream images. So there's a Kubernetes project called image builder and essentially that allows you to build the images that you need to launch cluster API systems. And so rather than having a very long read me file with all the things you need to do, we just have a small wrapper script where you just call it, give it the version of Kubernetes you want and it'll check out the repo and actually start up the whole packer build process and get it all done for you. The features that we have planned in the future, so we'd like to add support for OpenID Connect. So if you wanna use OpenID Connect for authentication with your environment, we wanna integrate alternative CNIs and what's interesting about that is really by integrate alternative CNIs really we need to work with the upstream community, the cluster API side to make it so that it supports things outside Calico and then we can do that, that rests on our side. And then we'd like to have support for FlatCar Linux which actually already exists in the cluster API. So it's more of testing it, validating it in R driver. So that's kind of the gist of R driver. Now I'll let Matt kind of talk and talk about the differences. So yeah, we also have started developing a driver. So StackHPC, we've been using cluster API in one of our other projects called Asimuth. We've been using cluster API in that for well over two years and back then the cluster class that Vex host are using was considered extremely experimental. So we aren't using that. We instead started off using Helm to stamp out cluster API resources. And we've got a very well tested, used in production battle tested Helm chart that we use to stamp these resources out. And it supports all the, it's got a lot of intelligence in it to support most of the things that cluster class now supports. And we probably will look to start to transition over at some point. But the nice thing of course about using a Helm chart is operators, if they want to do something different at their site, they can just use a different Helm chart that supports that. As long as it supports the same set of values that the Magnum driver is providing, you can have your own opinionated Helm chart. The other thing we did is we're doing all our development in Garrett and using the, and trying to get into the upstream code base. So we've been putting a big focus on improving the Tempest tests for Magnum. I don't know if anyone's ever tried to submit a Magnum patch, but basically there was no test in the gate to deploy, to test if a cluster deployed successfully. So we've been trying to fix that. And the things we've got working are all the kind of stuff that Mohamed was talking about before, like create, update, delete, resize, upgrade, node groups, CNIs, other open stack integrations like this CSI. We also already support the Kubernetes dashboard and a monitoring stack and configuring some of your networking. We, as of last week, we also support autoscaling. I forgot to move it into the working. But there's a few more bits to do. So there's a bit of funniness around cleaning up load balances made by the OCCM that we need to sort out before we can enable ingress properly, which we'd like to do. We're taking the opportunity to review the set of labels that's supported and work out what we actually really want to support. So there's more labels we will support. So the thing with our driver is, like Mohamed said, it requires, cluster API requires suitable images to be available. So we use the image builder project to build them, and both us and Vexhost actually have public buckets where you can pull images if you want them. And then you just need to provide the kube config for your CAPI management cluster via the Magnum configuration file. And we have a reference deployment, actually, for a cluster API management cluster. And we've also been working on the DevStack integration, like Mohamed said, so we can easily bring these things up for development. So how do we, if you want to use these, how do you do it? So I think Mohamed was going to, do you want to run through the prereqs? So there's a couple of things that are shared for both drivers, right? Because we're using cluster API, there's certain number of things that have to be in place, whichever thing you want to use. You need to have a Kubernetes cluster running somewhere. It can be as complex as you want it, or it could be a simple, small, kind, or K3S, or something small like that on a single node. But it's just have to talk Kubernetes. You need to get the cluster API controllers. So the cluster API, usually there's four components to it. Three are generally related to the control plane and bootstrapping and the actual orchestration of it. And then you have the infrastructure provider. So obviously, if you want to stamp out OpenStack, cluster is on OpenStack, you also need the OpenStack infrastructure provider. And also, as in the case of the StackHPC driver, you also need to have the add-on provider running on your cluster in order to be able to get the clusters going. That's kind of the first thing that you need to have, which is that management cluster that we talked about. The second stage is the Kubernetes images. So you're kind of free to do whatever you like. You want to download the StackHPC images. You want to download our images. And if you don't trust us and you want to build your own, you can also go to ImageBuilder and build your own images. So they're all pretty much the same images, just built at different places, but using the same exact code to be built. And then for the Magnum configuration, essentially you need to have the kube config for your management cluster that we just talked about available for Magnum so that it can talk to it over the API. And you just need to be able to create cluster templates that point to say that it's using the cluster API driver. And so in this case, you already have to do that when you are using the Kawara OS drivers and whatnot. And in the case of the StackHPC driver, if you have some specific overrides or things that you want different for your site, then you can kind of do that and make those changes directly in those home charts so they can be kind of permanently in there for all the clusters that are going to be going up. So at StackHPC, our stack is typically Kyobi Color Ansible. So we've been working on how to get Magnum available using cluster API in clouds that are deployed using Kyobi and or Color. So we've been pre-building images for Asimuth, and we're also sharing those images with our OpenStack deployments, which use Magnum, which use the Magnum driver. We have a Helm chart that's set as the default Helm chart in our driver that's tested in CI and battle tested in Asimuth. But you can, like Mohammed said, you can swap your own Helm chart in if you want. You can fork ours and modify it, or you can have a completely new one. And then we have a stack that uses Ansible Playbooks to deploy and provision to deploy and upgrade the Capital Cluster API management cluster. So the way we do that is just we use Terraform to stamp out a K3S node. And in some cases, that's sufficient, and we just whack the cluster API controllers on to that, and that's your management cluster. But we also have a way to create an HA management cluster, which basically, that K3S node becomes a cluster API management cluster that spawns a cluster user using cluster API, using exactly the same Helm charts that Magnum's using. And then we transfer the management to itself. And then we can get rid of the K3S cluster if we want at that point. And that includes monitoring and alerting, so we can do things like we've done some stuff with Kube state metrics so that we can look at the status of the cluster API resources and produce alerts and dashboards so we can see what's going on with our clusters. And then to pull it all together, we have some playbooks in. We have some custom code in Kyobi and Coloransible at the moment to make sure Magnum gets the Kube config coming through to it. That'll be upstream very soon. And then we also create the Magnum templates that we need pointing to the images that we've uploaded. And then the users are good to go. So on the Vexo side, it's kind of a little bit easier. So we get kind of lucky because we deploy OpenStack on top of Kubernetes. So we have a giant Kubernetes cluster that's already sitting there waiting for us to put whatever we want on it. And that was for us, one of the big drivers why we saw the cluster API is a very easy thing to just jump into because we already have a Kubernetes cluster that's running all of our OpenStack services. So if you haven't, I went for a little talk. Rico did a little talk this morning about atmosphere, which is pretty much a tool that allows you to deploy OpenStack on top of Kubernetes. So it's a fully end-to-end. It starts with deploying Ceph and then deploys Kubernetes and integrates that Kubernetes cluster with the Ceph. And then once it increased that, we use Helm charts to deploy OpenStack. And as part of that deployment, when it reaches the Magnum stage, it actually installs the cluster API for you automatically and prepares everything, creates the cluster templates, uploads the images for you. So essentially, just by having an atmosphere installation, you have everything all good to go and ready to install Kubernetes clusters out of the box. So essentially, that's the way that we usually have kind of recommended for people to use it. Now it's still possible if you are using a different deployment tool to pretty much do the same thing as what Matt mentioned. So essentially, if you're running Magnum anywhere, you just add the driver to the virtual environment that's running there, or whatever way you're shipping those images and point it to a Kube config that can talk to where the cluster API lives. And it'll actually just be able to work and do everything on its own. The only thing obviously is you'll also have to install the cluster API proxy component if you want single, fully isolated clusters. But that's something, for example, that we already include out of the box in atmospheres. So that runs as a daemon set where your Neutron DHCP services run. And so that way, we can just get that to run all very easily and simply in there. So I think we're getting close to our time. So we actually got to power through this. We got to power through this. So when you get a cluster up, for it to become ready, you need to have a CNI.  The way that we opted to do it is really to use a cluster resource set feature that allows you to apply arbitrary manifests into the cluster. And so we do that to install the CNI. We do that to install the CSIs. And we do that to install the Cloud Controller Manager. But obviously, nobody really wants to just have manifests anymore. It's complicated and not clean. There's better ways to do this. There's a very early cluster API add-on provider for Helm so that it can use Helm to automatically apply and reconcile charts. But Matt also can talk about some more ways that StackHPC has kind of thought to approach this. So we've had an add-on provider that we developed and for a while that's been able to install Helm charts onto cluster API clusters. And actually, we've been working with the upstream guys on the cluster API Helm provider. So what this does is just allow us to specify a bunch of Helm charts that we want to go onto the cluster with some values. We can template those values out from the cluster API resources as well, actually. So if the cluster API cluster creates a network, we can get access to that network ID, feed it into the OCCM configuration, like really nice and easily. But what we're pushing towards now, especially in Asimuth, but we also want it in Magnum, is this idea of fully managed add-ons where you use a continuous delivery tool like Argo to manage the add-ons on the tenant clusters. What this does is gives you things like being able to report health of the add-ons, which is really nice. And if you enable it, you can also auto-heal the add-ons. So if a customer does something like delete the CNI pods off the cluster, or delete the CNI demon set off the cluster, Argo will just put it back. And you get less, yeah, less spunkiness going on. So, but what we needed to do was a way, what we needed was a way to integrate this with cluster API. So we've been extending our add-on provider so that it automatically adds, it watches for cluster API clusters and automatically adds them as targets in Argo CD. And then when you define your add-ons using our CRDs, it will automatically generate Argo CD applications for those and they're targeting the correct cluster and they'll just manage the add-ons on the cluster. And this can all be used outside of Magnum as well, just with regular cluster API. Like I said, it gets used in our other project, but we're, yeah, so that's what we're pushing towards is this idea of fully managed add-ons. So, and then, yeah, just a quick thing about the future. Yeah, so the future plans, if you've noticed, there's a lot that we're doing in parallel. And what we're trying to do is start up a conversation. We already started to talk a lot more with Stack HPC and seeing how we can converge those drivers to ideally just have one so we don't have duplicated efforts and seeing if there's a way that we can share at least a common piece of code. But really, the main goal of this is really what we wanted is to kind of show what's happening with the cluster API and the Magnum world. Here, if other people are kind of interested, get more people involved in this conversation, because, you know, when we were setting up this talk, I thought, well, how funny would it be if somebody pops up in here in the room and says, well, we already also wrote a cluster API driver? But yeah, we want to have the conversation open so people are all on the same page. If you see me or Matt around, feel free to kind of talk. And there's also a forum session about Kubernetes and OpenStack where I'm sure this topic will probably also come up. Tomorrow afternoon, is it? Yeah. So, you know, be on that. And yeah, I think. So yeah, I mean, that's supposed to do in a bunch of cool stuff. We quite like the isolated cluster thing, so we might, we've got our eyes on sharing that maybe. And yeah, so the main, our main kind of push at the moment is just to try and get something in upstream magnum so that anybody who wants to use this has it there by default without having to build from like a Gerrit patch set or add extra things in and build their own images. So that's what we're pushing for. So anyone who feels comfortable testing and reviewing patches would be greatly appreciated. Yeah. So if you're interested in that come and see me, I can point you at the Gerrit patch set for our driver. Or Mohammed can get you pointed at his driver. Perfect. Great. Well, thank you so much, everyone. And happy to answer any questions if anyone has any. If we've got any time for questions. Yeah, I don't know. Do we have time or? I'm looking at the back to see if we have time or not. Maybe one more one question. All right, maybe we have one question. I always say, it takes one person to ask a question and everybody starts asking questions. But I guess maybe we- We obviously covered it all. We did cover it at all. Well, we'll be around if you guys have any questions directly or personally. But thank you all so much. Thank you very much.


There we go. Alright. Awesome. So, welcome everyone. I'm David Fogt. I'm a researcher and PhD student at the Technical University of Berg Academy, Freiburg, which is quite a small university next to Dresden, a small mining university, but that doesn't have to say anything, right? Okay, so the title of the talk is Behavioral Generation for Interactive Virtual Humans, which is some way of fancy title, but what you try actually to do is we have a cave environment. In this cave environment, I want to animate the virtual characters to react to human motions. You want them to be responsive, reactive. You want people to go in there and interact with a virtual agent. And to boost your animation for a virtual character, what people usually do is imitation learning. And in doing so, you create a system that allows the virtual character to observe a human being and then adopt this motion onto its own physiology, right? But over the last couple of years, this methodology slowly changed towards new direction because if you have classic imitation learning, you only focus on a single agent. So in order to create responsive virtual characters, they need to somehow adopt the behavior or at least have a look at the behavior of two persons simultaneously and how they interact with each other. So the question is only what to imitate and when to imitate, but also what are the intrinsic details of the interaction? What makes the interaction so special? So what are spatial constraints? What are spatial relationships? What are temporal coherences? And our approach on this is what we call more or less interaction learning. So we're trying to create methods that allow virtual characters as well as other synthetic artifacts such as human robots or robotic arms to engage in two person interactions. And what we do for that is we basically have a system composed of three steps. So the first step is somewhat a task demonstration.  We need to record those interactions. And based on that, we then compute something what we call an interaction model that captures those spatial relationships, that captures those intrinsic fine details. And then in the end, you can have like a robot or a virtual character be able to interact with the human. And what I'm going to focus on is how Blender can be used to realize all those steps. Obviously, we need to do some sort of extension. We have to implement something more. That's not going to work out of the box. And throughout the talk, I wanted to discuss each of those boxes in more detail. So let's start with the first one. The first one is a task demonstration, meaning that we record two person interactions. But as you know, there are so many motion capture systems out there. So our requirement basically was to create something that is, that can be used for multiple systems. And I guess most of us are using like a Kinect one or a Kinect 360 or something. But in our lab, we also have an optical ART tracking system, which is a marker based motion capture system. And what we do is we, well, the system has the drawback is only can record one person at a time. But I wanted to do interaction. So somehow I need to spit the motion captures you. And what we ended up with is we had each person where eight markers in total, but you can't really record the whole body motion with all the intrinsic details without using some sort of inverse kinematics to compute the missing joint angles.  And it allows you to reconstruct a character's posture based on only eight markers or seven markers, depending on how many you actually have. But you can record eight markers and recompute the whole character posture in the end. And the system works as follows. You basically have some sort of middleware, which encapsulates a unified motion capture data stream. And you can use, like, two connect one cameras. It's basically based on the SDK and merging joint positions in the end. Or you can use one connect 360, or you can use an optical ART tracking system. But you also have, in this, the biovision motion capture system, which is a marker less motion capture system, that can be also used as well. Okay. But what we do is basically you record all those two-person interactions one at a time. So one demonstration per interaction. That's it. And we store everything in an SQL database. And in the end, you can retrieve those motion either live or by using a small shared library. It's a shared object loading blender dynamically during runtime, so you can access your data. Okay. So now we are getting more into the scientific part, the math part of the system, which is our interaction model. So the requirements for the interaction model are it needs to somehow capture spatial constraints, spatial relationships. If you imagine yourself high-fiving a virtual agent, you're for sure will never going to do the same interaction over and over again. Your hand will be at a different place, different position, different speed, different velocities. You have to compensate for that in order to have the virtual character respond to those interactions. So we have a lot of constraints on the interaction model. And what we realized in the past is human motion is intrinsically based on low-dimension manifolds, meaning you have quite a lot of redundant joints. If I move my hand for example, my elbow will move as well. So there is some sort of redundancy. And you can strip off the redundant information by doing dimensionality reduction. So what we do is we have eight markers per person in the end. And those eight markers are a 24-dimensional space, three Cartesian coordinates in the end. We recompute the joint angle rotation and the marker rotations with the mercilimatic solver, so we have only 24 dimensions in the end. But 24 dimensions, that's such a big amount of data. What you can do is you can apply dimensionality reduction to compute a more compact, more robust low-dimensional space. In the end, you have a small space in which you have 95% of the information with only four-dimensional or three-dimensional. This is lovely in the end. So in this low-dimensional space, each point corresponds to a posture. And the interesting thing is that the same principles of distance apply. So if a posture is similar in high- dimensional space, the points in low-dimensional space will be quite near. Meaning if you have two similar postures obtained by the user, you will see them as two close points in low-dimensional space. And this is what we see here. So this is the low-dimensional space of the dude wearing the orange t-shirt. And all the interactions start at the up-straight position. So all the interactions start at one point in low-dimensional space. But from there, they differ quite drastically. Okay. So a trajectory in low-dimensional space corresponds to a motion and the point to a posture. But we can harness this further. In the end, we want to somehow learn a compact representation of the interaction. We want to use the model to classify during one time in which interaction is the user. What is he actually doing right now? What are the semantics of the interaction? And this is where it becomes mathy. What we can do is we can compute a kernel density estimate. But the key idea here is that we compute keyframes. Those keyframes are based on the Gaussian mixture model. But in the end, you have several keyframes. And those trajectories are the interactions. So and the numbers are the keyframes. So if the user redos an interaction, it will start moving somehow and you will see a new point in low-dimensional space appear, the current user posture. So this new posture will be somehow, when the users start over here, between keyframe six and seven. So what is the system supposed to do now? Since the posture is similar to different interactions, like this one, for example. This one can be part of a handshake. It could be part of a high-five. We don't really know. But what you can do is you can look at the history. What has the user done in the past? So is he quite fast moving up to like a high-five or is he like more weighting here? And this can be seen if you compare those keyframes in the past. So if you're now at keyframe, let's say, what's that, 15 or something? If you're like this keyframe, for example, and you see, okay, the past keyframe has been made. So it clearly has to be this interaction. Same goes for the five. If you have the five as a current keyframe in an interaction, and you see the previous one is three and nine, then you know, okay, it has to be this particular interaction. Okay. So this knowledge, this sequence of keyframes has to be somehow computed. And for that, you can use a hidden Markov model. So the hidden Markov model now captures the sequences of keyframes in low-dimensional space. That's pretty much it. So it allows you, during runtime, to somehow extract the context of the interaction. It allows you to say, okay, the user is now doing this interaction. You can be based on the posterior state, probably, is quite sure about that. Okay. But still, you want to animate the character in an ongoing interaction. So if you know now if the user is in this particular interaction, that's good, but we still don't know in which sequence, in which part of the interaction. So we need somehow to figure out not only the interaction, but at which frame in the interaction is the user currently in. And still, we need to somehow optimize the character response. So if I have recorded this high-five interaction, and now during runtime I'm doing this, I am required for the virtual agent to be responsive for that motion as well. So he needs to adopt his motion to that. And adopting those motions can be realized with something called an interaction mesh. An interaction mesh is basically a spatial relationship observing topology. It's a net. So you can compute this by using telonide triangulation, or you can use our context-dependent interaction measures. It doesn't really matter. In the end, you have a net. And this net is deformed. So if we read with the interaction with the virtual character, the right-hand side, which is the virtual character, and the left-hand side is controlled by the user, the mesh will be different. It will be deformed. So this deformation can be optimized. Okay. So this should be it from a mathematical point of view without any equations whatsoever. What you need to know is basically this is a plug-in route in MATLAB, purely MATLAB. MATLAB allows us for really, really fast prototyping. There are a lot of algorithms out there coming from Sikrov and Gankasa and IFA. And we wanted to explore those algorithms. We wanted to implement them. And the best way to do so is to go for MATLAB, because it allows you for really, really fast prototyping. But in the end, you start prototyping and prototyping and prototyping. You have a lot of class, a lot of dependencies.  And what we did is basically recreated MATLAB bindings for MATLAB, MATLAB bindings for Blender. So we can use the power, or the computation power of MATLAB in Blender via a unified interface. And this is also, once again, a shared object that has been loaded during runtime. Okay. So, so far, so good. So we talked about demonstrating those two person interactions. We talked about how to compute the model. So, meaning the interaction model composed of the context-dependent thing, the hidden mark of model, and the interaction meshes. We have one interaction mesh per frame. So there are thousands of them. We need to figure out during runtime which one is it actually. And this brings us to the question of what actually happens during runtime. So, during runtime, so we capture the user's motion once again. And as I said before, we project the user's motion into a low-dimensional space, and we realize, okay, what are the active keyframes? Compute the posterior state probability of the hidden mark of model. Hidden mark of model tells us, okay, this is obviously this interaction. So, and this interaction needs to be analyzed further. And in the end, we have a pointer to a data set, a motion capture frame from the initial recording that resembles the current situation, temporal-wise, context-wise, and spatial-wise. So, this is, in the end, the posture of the virtual character that will be optimized to the new situation. And the problem here is this has to be done at 30 frames per second. So, there's a lot of computation going on in the background. But now we know what the posture of the user will be and the posture of the virtual character. Since, to each motion capture frame, there are character postures for the user, as well as the character. Since we recorded two-person interactions simultaneously. So, how can we actually interact with the agent? We now have a model. We have a process of how to compute ongoing interactions. We have a model of classifying those interactions. So, we wanted to create a system that allows you to interact with the virtual character in different scenarios. Meaning that you have, for example, normal laptop computer, a big wall, or you have a head mounted display, or you have a cave. Regardless of what you're doing, you can still use the same interface. And this is possible due to something what we call a character post-streaming library. It's a fancy word for broadcasting the final character postures in the end. So, as you see, we haven't used Blender for visualization at this point at all. And we will not use it. Blender is purely there for managing data, managing time, which is very important, and managing the broadcast and the connections to all our different servers. So, now we are in Blender doing runtime. We require the user to do an interaction, do something, compute the interaction model, and get the final character posture in Blender. Now the inverse kinematic server comes into play, solve, since we are only optimizing those eight markers. Again, we reconstruct the final posture. And this posture is then streamed to whatever output device you want. And I thought it would be interesting to show you our cave environment rather than a normal display or a head mounted display. But in the end, it's the same. So, how can we stream this data to a cave environment? Well, our cave environment is quite big because we have 24 back projection televisions and one ground projection for the inner floor. So, if you add up all those full HD TVs, including the ground projection, you add up about 50 megapixels. And this is a huge amount of information you have to calculate. And we do this by using quite decent hardware, I guess. There are more computers running in the back. There's at least one more for motion capture, our ART server, or my PC, if you're using like a normal Kinect. And Blender is once again the central part. So, Blender runs on my computer. The interaction model can be a supercomputer. It can be my computer as well depending on how long you want to wait for the model to be computed. But once it is computed, you can just copy the binary and that's it. Okay. And we've got the motion capture server and Blender is basically managing in between. So, right now, we are streaming into our cave environment, which basically means broadcasting at 30 frames per second and 96 degree of freedom motion capture data stream, all the joint angle for the virtual character, times two since we have two characters. We have an agent for the user, the user agent and the virtual character you are interacting with. So, what does it actually look like in the end? Okay. What you see next are two scenarios. The first scenario is more fighting scenario, I guess. And the second one is more casual high five shaking hands and stuff. So, we have the cave environment up and running. The interaction model is once again computed and I wear the fancy motion capture suit or the small markers. And then you can start interacting. And depending on how I move, depending how fast I move, depending on where I move, the virtual character will respond. And in this example, we have, I think, six or seven interactions in total. And you can basically decide in which interaction you want to do. So, there's no real semantics going on. You can do whatever you want to. The biggest drawback we have at this point is the latency. We have a latency about 350 milliseconds, which is not fast enough for real-time applications. We are running at at least 25 frames per second, depending on the inverse kinematics server, since the inverse kinematics server accounts for at least 45 percent of the information, 45 percent of the whole computation time. Okay. But what is going to be interesting next is the user is high-fiving the virtual agent again. And in depending on where he is moving, the virtual character will respond. So, if he's high-fiving the virtual agent a bit higher or a bit lower, the virtual character can still respond to those interactions. And this is due to the fact that we use the interaction mesh at frame level. So, the interaction itself, so meaning it is a high-five or it is a handshake, this is depending on the hit mark-off model. But the optimization of the final posture is purely dependent on the interaction mesh. Okay. To be honest, I'm not a computer graphics guy. I'm a roboticist. So, we have also robotics lab. And our robotics lab is quite new actually. What we bought recently is one of those UO5 robotic arms. And interestingly, you can use the same motion capture framework. You can use the same blender instance. You can use the same interaction model in the end. The only thing you have to actually change is the inverse kinematic server and the streaming capabilities. So, meaning that in the end, you're not streaming to a virtual environment, you're streaming it to a robot. Sure, there's more involved in that. You have to compute joint velocities. You have to compute joint torques. You have some impedance control. You have torque control. There's way more going on. But in the end, the final posture is computed completely the same way. Okay. Interestingly, you can interact with the robot the same way as with the user. So, once again, the user's motion is captured. Those are the interactions you can do with the robot. So, now we have some sort of semantics. First, you place the object on a stand, assemble a tube, part of the tube frame, and then assemble the final frame. And those interactions can be redone with the robot as well. We can operate the robot faster, but sometimes you have, I don't know, glitches in the inverse kinematic solver. And imagine yourself having a big robot and throwing around the big box. So, let's play it safe. Okay. So, depending on where I am moving, the robot will move as well. But interestingly, despite not having explicit knowledge of the scene, the robot can still recover from false interaction attempts. So, if I'm going to grab this object and the hidden mark of model says, oh, no, this is not this interaction. This is that interaction. And the robot will move somehow weird. I can just redo the interaction. That's it. Trigger the hidden mark of model once again. And so, in doing so, the robot can recover from those false interaction attempts. And I can continue assembling my things. Which is quite a boring example here, but anyhow. Okay. What you're going to see next is the interaction mesh once again. So, this is the key frame. And depending on where I move, the robot will follow and continue its motion. And this is, I think, quite cool. Because usually you have a robot in a term taking scenario. I do something. The robot does something. I do something. The robot does something. What you have here now is a continuous optimization.