I wish to welcome the group to the annual lecture series this year on COVID-19. We are so delighted that our speaker today will be Dr. Angie Wall, who Dr. Ross and I have known for quite a while over the years. Dr. Wall is an MD, PhD, fellow of the American College of Surgeons, who is an abdominal transplant surgeon and a bioethicist at Baylor University Medical Center in Dallas, Texas. Dr. Wall is the vice chair of research at the Baylor Simmons Transplant Institute and chair of the section of surgical professional development. Her clinical practice focuses on living and deceased donor abdominal transplants, including uterine transplants. Dr. Wall's medical training began at St. Louis University School of Medicine, where she received her MD and her PhD in healthcare ethics. She completed her general surgery residency at Vanderbilt University and her abdominal transplant surgery fellowship at Stanford. Dr. Wall's research focuses on transplant policy, ethics on ethical and clinical questions in uterine transplantation, on the ethical expansion of donation after circulatory death and perceptions of wait-listed patients regarding different types of donors. Dr. Wall published a book. This was a few years back entitled Ethics for International Medicine, a practical guide for aid workers in developing countries. The book was published in 2012. The book uses a series of cases to provide medical aid workers with a method for identifying, analyzing and resolving ethical issues within the context of international medicine. Dr. Wall has also published many, many papers, including a paper entitled, here I quote, two cases of pregnancy following uterine transplant and ethical analysis. That paper was published in 2020 and one of the co-authors was one of our former McLean Center fellows and faculty, Dr. Giuliana Testa, who works also at Baylor in Dallas. Today, the title of Dr. Wall's talk will be when a scarce resource becomes more scarce. The central role of ethics in managing the impact of COVID-19 on transplantation. Well, I'm just delighted to welcome Angie Wall back to the University of Chicago again. Dr. Wall. Thanks so much, Mark. Thank you. Thank you so much for the invitation to speak to you all today. Thanks to Dr. Ross, Dr. Angelos, Dr. Siegler for letting me be one of the participants here in the ongoing discussions about ethics in COVID and all the different places where COVID touches clinical ethics. As Dr. Siegler said today, what I'm going to talk about is the impact of COVID on transplantation and how ethics does truly play a central role in the way that we have had to manage transplantation in the setting of a pandemic. So let me go ahead and get started, and tell you, number one, I have no disclosures. Maybe one day I'll be famous enough to have a disclosure. I do have a few objectives. First off, I hope that you'll be able to describe the effect of COVID-19 related limitations on resources, limitations and resources on organ transplantation specifically. Second, I hope that you're able to apply the principles of triage to organ allocation decisions during the COVID-19 pandemic, and finally, be able to discuss how local factors have to be accounted for in organ allocation decisions during a pandemic. And so before I get into the talk about these issues, one thing that I do want to say to set the stage is that I am currently in Texas. And in Texas, this is from the New York Times from yesterday. We have had kind of two waves of two peaks of COVID that were pretty substantial, one back in July of 2020, and then a second very significant one over the winter here. And then we sort of dipped back down, and then we had another peak in March and we have been declining in case numbers since. In addition, what you can see, and what I think is very interesting whenever you look at these New York Times graphs, is that you can look at not only what the reported cases are, but the hospitalizations and deaths. And one of the things I do want to point out is that even though the two peaks look very different on the reported cases, the hospitalizations and the deaths are actually not as significantly different between the two major peaks that we had. And so as I talk about COVID transplant and triage, I will talk about COVID sort of in the past tense a little bit, because we are currently in a not a peak, and we do not have the same kind of challenges that we have had with the peak. But I recognize that we may be going back up again, especially with Texas opening up. So the first thing that I want to talk about is organs, not about COVID. And I hope that you will all agree with the fact that solid organs are a scarce resource. They are just by the numbers, we have over 100,000 patients on the waiting list for solid organ transplantation. We do about 30,000 transplants per year from deceased donors. And this is all comers of organs. We do another about 5,000, 5,000, 6,000 living donor transplants per year, mostly kidney but liver, liver and lung also account for some of these transplants. And then we add about 50,000 patients per year to our wait list. So you can do the math. There are more people waiting for organs than we have organs available. Now, my opinion that I'm just going to put out there is that just in the just in the face of scarce resources of organs of scarce resources allocation is somewhat easy, somewhat easy, not 100% easy. You have two pieces that go into allocation. You have your hospital or your transplant center, which makes a decision about which patients to list for transplantation. Then when you have patients on the waiting list, you have in the US, we have a national allocation system that runs an algorithm for each donated organ. And then the list comes up with patients in an order. So it's actually it's we have one scarce resource and we have a way to deal with the scarce resource. It's not necessarily super straightforward, but it's relatively easy. And for those of you who are not super familiar with transplantation, I'm just going to draw your attention to what we what what the governing principles of organ allocation through the final rule, the legal principles for organ allocation or guidelines for organ allocation in the United States. And what we have what we have recently been talking about a lot is the geographic disparities. And the fact that we cannot have geographic differences in organ transplantation or organ allocation to the extent that we can we can possibly not have those not have those disparities. That's been the real focus of changes in organ allocation. But what COVID has done is had us focus on the first three parts of the notafinal rule, which is that organ organ trans organ allocation needs to be based on sound medical judgment. We need to have the best use of donated organs that we can have. And then we have to preserve the ability of transplant programs to decline donor offers. And hopefully throughout this, throughout this talk, what you'll recognize is that while we have this national allocation system, we have to have center level decisions that are based on sound medical judgment, who is really going to do do the best with what we have available with this potential donor organ. How are we going to make the best use out of the donated organs that we do have available to us? And then finally, if we are not able to transplant a patient because of COVID related hospital reasons, how do we not have transplant centers penalized for that situation? So, so that's sort of the background on organ allocation and how organ allocation works. And the next fact that I think is important to point out for this particular talk. And again, I hope that you all don't argue too much with this is that COVID-19 has limited our hospital resources and other resources needed for transplantation well beyond organs. We are limited in supplies, we're limited in space, limited in support and limited in staff. The four S's of limitations of COVID-19. And in the next few slides, what I'm going to describe is some of the limitations that we and I expect many of you have had with the COVID pandemic. So starting with supplies, we have been limited in mass gowns, gloves and 95s, all the things you need to perform an operation and you need for patient care. We are, we have been limited at different points in the pandemic on medications, specifically on medications like paralytics and pain and sedating medications for ICU patients. As many of you are familiar with a COVID patient who goes on a ventilator is often on that ventilator for two, three weeks. So it is very resource intensive and very medication intensive to keep somebody on event for that amount of time. Oxygen has been a limitation in some hospitals, ventilators themselves have been limitations. And one thing that is central to some of our decision making and transplantation is especially at the start of the pandemic, we were very, very limited with our ability to test for COVID. We all have had issues with limitations in space, limitations specifically in types of beds, ICU beds, limitations in OR rooms because in some places OR rooms were having to be utilized for ICU space or the ventilators used in operating rooms were having to be allocated to ICU patients and so those OR rooms were basically no longer not functional. Also not every operating room is COVID safe, meaning the airflow in the operating rooms at least in our center had to be adjusted and we had to create COVID specific operating rooms in one part of our operating theater. So we were limited in what patients we could operate on who were either patients under investigation for COVID or who were COVID positive or are COVID positive. So there are specific space limitations associated with COVID and from a transplant standpoint not only do we have to think about our own space limitations, we have to think about space limitations for deceased donor transplants in the donor hospital as well as in the recipient hospital. Support is another thing. So in transplantation one of the main support structures we have to have is transportation. We have to have the ability to move organs around the country because of the way that allocation works. Sometimes we have to move teams around the country because of the way that allocation works and one of the things that happened with transportation in the US is that we've had less commercial flight paths because not as many people were have been flying because of COVID and then also more people are utilizing charter planes for travel if they have the option of doing so and so that limits our ability to use charter flights for organ travel. We've also been, we've had to be creative about our organ procurement organization support with coordination services because we've tried to limit the risk of our support staff outside of the hospital, not bring as many people along, utilize local resources, etc. And then the other piece of support that we need for transplantation is donor referrals and as you all can imagine when you are a bedside nurse in an ICU dealing with a bunch of patients and being overwhelmed with COVID the last thing you think about is calling your organ procurement organization and referring a potential patient for donation. Also active COVID at least in most places and especially at the start of the pandemic is an absolute rule out for organ donation because we just didn't know how COVID would affect the organs in our recipients. So donor referrals that support that we need for transplantation has been limited with the pandemic. Finally this one this one's pretty straightforward limited anesthesiologist intensivist. Even transplant surgeons there are there were some transplant centers where multiple transplant surgeons were became infected with COVID because you know they had exposures on with donor travel and so forth and when you're down a few surgeons then it's really hard to keep up your volume of transplantation. So when we talk about transplant we talk about COVID what we have to do is we have to sort of combine these two contexts and the way that I think about combining these two contexts is number one we have a limited supply of organs and that limited supply of organs becomes more limited in a pandemic because of our resource constraints. Number two our allocation system is national and it's based on standardized criteria but each hospital donor and recipient is affected by COVID in a different way with different constraints and local resource constraints are not at all a part of the national allocation system. So our allocation continues by our current algorithms but each center has to make individual decisions about which patients to transplant based on their local context. And then third the other issue that I think is very important for transplantation is that there there was a lot of talk about what surgeries you know stopping uh stopping non-essential surgeries so that you could increase hospital capacity resources etc and the American College of Surgeons came out with a set of tiered guidelines about where surgeries fall into elective to urgent to emergent and um and transplantation organ transplantation in general was categorized under what's called tier 3b which means unhealthy patient high acuity surgery and the guidance on tier 3b procedures is that they should not be postponed so transplant operations as a whole were not constrained and there was no guidance of you know this transplant operation is urgent this transplant operation is not they were all 3b um in consideration from uh from kind of our organizational guidelines. So the question that I think we had to that we have had to address over and over again is how do we decide who to transplant when there is no change in allocation um and all transplant operations are treated equally in terms of urgency at least from the uh from hospital and national guidance. And the way that I kind of conceptualize um the underlying ethics of how we make these decisions about allocation at a center level is that we lean on some general benchmarks and so many of you are probably familiar with this paper the fair allocation of scarce medical resources in the time of COVID-19 published by um Emmanuel and colleagues in the New England Journal um this lays out several guidelines for considerations for allocating scarce resources. I think we have to specify those guidelines to the context of transplantation and that's something that we did early on in the pandemic um in this paper on utilizing this ethical framework for rationing absolutely scarce healthcare resources in transplant allocation decisions. And then the final thing and actually something that Dr. Siebler talked about earlier is that we need to bring in some lessons learned from other contexts and I would specifically point to the context of global health and the fact that when you are in um in a resource limited country you truly have resource limitations and you have to make decisions based on those limitations. And so I actually draw a lot of my thought process for allocation decisions and transplantation and COVID from some of the work that I did back in graduate school for my dissertation on um on resource other resource limited settings. So I'm going to go through each of the ethical guidelines presented in Emmanuel and colleagues paper and talk about how they relate to transplantation. So let's start with the first principle of maximizing benefits which everyone is familiar with. The strength of maximizing benefits is that it focuses on patients that are likely to have the best outcome. The weakness of this of using this principle is that oftentimes the sickest patients are not the ones who are going to have the best outcomes and so we end up not prioritizing those patients who are in most need of whatever given resource you're trying to allocate. Now the question that um when we when we talk about maximizing benefits and transplantation we have to ask who will benefit the most from transplant and how do we really define what what sort of benefit we're going for. And so when we think about um when we think about transplant and benefit is it the person who's the most likely to die without the transplant um or is it the person who is the most likely to live the longest with the transplant? Is it the sickest or is it the one who's going to do the best? And currently at least from our national allocation standpoint our prioritization standpoint the answer to that question is that we currently prioritize the sickest patients when we have a way of tiering who is the sickest patient and then if we don't have a way of tiering who the sickest patient is we default back to those who have been waiting the longest. However in all of our allocation systems at least I can speak for abdominal transplantation we do have some secondary considerations in prioritization for patients who are likely to do well after transplant in for example pediatric liver transplant candidates or younger potential kidney transplant candidates. So um the question that we have had to answer kind of throughout the different peaks and troughs in the pandemic is which of our transplant patients are going to benefit from transplantation with the added risks uncertainties and resource constraints of the pandemic and I'm going to talk about some case examples later that utilize this as one of the underlying principles that we that we considered. The second principle of allocating scarce resources is to treat people equally. The pro uh the pro for this uh for this principle is that it encourages the just distribution of resources to people with the same claim on the resources. The con is that it's really hard to determine objectively who is equal and from the transplantation standpoint one of the things that um that we have that we struggled with especially early in the pandemic is the question of should we put categorical limitations on certain populations of transplant patients and hopefully these questions sort of help illustrate that the kind of distributive just this question that we struggled with. So our current allocation system is designed for equal treatment but it's inadequate in the setting of COVID. What we do is we list patients and then we allocate organs. Now one of the questions some of the questions that we had were for example should we have a lower limit of meld for working up transplant patients so should we have kind of a minimal level of sickness where we think that the that these patients really do need to get worked up right now for liver transplant listed right now for liver transplant versus waiting until we get over a wave of COVID. Should we stop living donors? Should we um should we categorically stop that because we think that you know the risks are too much for the living donor for the recipient? Should we should we focus on only in patients so patients who are already utilizing resources? Should we only transplant that category of patients for heart transplantation or for liver or lung transplantation? Should we stop all kidney transplants because those patients have the alternative of dialysis? Are there categories of individuals that we would typically allocate organs to that we should categorically not allocate organs to in the setting of the pandemic? That's where I think the questions of treating people equally came up and were a big struggle for us and have have been a big struggle for us in both shutting down parts of transplantation and then opening up parts of transplantation. The third um the third principle of allocating scarce resources is to promote or reward instrumental value and what this says is that it prioritizes healthcare workers or research participants or frontline workers who have contributed into the system or who will contribute into the system prioritizes resources to go to those individuals that they get sick prioritizes vaccination to those individuals because they've bought into the system. The issue with promoting or rewarding instrumental value is that one some people would argue that it is baseline unfair to bring a value judgment of social worth to certain groups.  What do we do with teachers? What do we do with grocery store workers? Are these considered frontline workers just the same as our police officers as our firefighters as our nurses and so I think that you get down a slippery slip of um of trying to figure out who is valuable and who is not when you talk about promoting and rewarding instrumental value. Now when I think about it from the transplant side I think there is a very unique consideration that we have to have and what that is is we conceptualized instrumental value in how it relates to deceased organ donors. So deceased organ donors I hope you can uh you will agree with the statement they have an instrumental value. They are a deceased donor can save multiple lives but a deceased donor also is taking up an ICU bed that could be used for a patient who for example has COVID and needs a ventilator and needs an ICU. So I think that we have to think about the instrumental value of deceased donors and how they impact the entire healthcare system and I do think that the fact that you can save multiple lives with a single deceased donor is an important consideration for resource and ICU utilization but it's a very hard concept because your deceased donor is in one hospital using an ICU bed that might be needed by a patient who's in an emergency room in that hospital. The patients who are going to benefit from from the transplant may be in ICUs in other hospitals and so seeing the link of where the deceased donor has instrumental value and not having the reflexive we need to take this person off of the ventilator right now let's forget about organ donation because we have somebody else who is waiting for it I think that's really really hard and that is a very difficult decision to make. Now the other thing about deceased donors and something that I think people kind of you may it's kind of a secondary thought sometimes when you're thinking about organ donation and utilization is that if we have a deceased donor and let's say you have a deceased donor who the heart is going to a patient who's in the ICU in that hospital and so you you know you're gonna you're you're gonna have a deceased donor they're gonna go to the operating room they're gonna the the um generation operation is going to happen that heart is going to go into another patient who's in the ICU that heart patient is going to do great and come out of the ICU you now have two available ICU beds this is a great situation um the other consideration is that that donor probably also has several organs that are that that can be utilized and so if we are going to accept or utilize one organ from a deceased donor should we try to um should we try to place all of the organs should we try to should we try to maximize the benefit or the value of each donor um and then the other the other important point um and this is more for the specific transplant audience but one of the things is yes donors deceased donors have instrumental value but um but we can do our part to minimize the donor work up and to accelerate uh getting to the operating room so that uh so that we do try to minimize the uh the constraints on ICU resources um so that's sort of that's sort of how we how how we in our in our paper on um on COVID transplant triage thought about instrumental value now the final principle for um for fair resource allocation is uh is giving priority to the worst off so if you've addressed all the other um all the other uh scarce or all of the other allocation principles and it comes down to you know two patients whichever one is the worst off maybe they should be uh should be given prioritization based on this principle the pro of this is that it favors the person most in need of the particular resource but the con is that it can and often does conflict with the ethical value number one which is maximizing benefits and so those two do have to be specified and balanced when you think about who we should prioritize the one who's going to do the best the one who's going to who is the worst off um the difficulty I think in a pandemic is comparing across different patient types and one of the frustrations that we felt specifically um again at the start of the pandemic was that all of their resource allocation conceptualization and discussions were talking about how do we allocate these scarce resources to these sick COVID patients and what we often forgot to bring into the discussion is that other patients are sick and other patients may have an equal claim to these resources and so you know in transplant we have patients with decompensated end organ disease that I would argue have an equal claim to ICU resources and care as compared to our patients who are sick with COVID and so we have to think we have to figure out how how these different patients with different claims on the same resource can um or can kind of be compared apples to oranges to determine who's the worst off or who's going to do the best um the other the other point that I think is very important to make is that the worst off transplant patients may already be utilizing significant resources and their only road to recovery is through transplantation so if you have a patient with fulminant hepatic failure who's in the ICU they're either going down the road of they're going to die from their liver failure and they're going to utilize ICU resources until they die or they can get a transplant and if they do well from transplant then they have a potential of getting out the ICU out of the ICU and recovering and so we do have to think about the the fact that the path to recovery the path out of the ICU for many of our patients is through transplantation and the final thing and this is outside of what Emmanuel and colleagues talked about but I really think that this is an absolutely essential element to think about when you're thinking about allocating scarce resources in the setting of transplantation and COVID is that we have to use our local context to to define hard limitations and this really comes from work that again that I had done on resource limited settings we are used to working with essentially unlimited resources other than organs and transplantation we always have an operating room available we always have OR supplies we always have an ICU bed that was never a question until COVID we always had blood and blood products COVID took away unlimited resources and has to and continues to take away these unlimited resources in different pockets of you know the different resources that we have and so what I what I think is essential for you to take away from this talk is that while the decisions that we make about allocation have to be made with an ethical framework they also have to include this box of practical limitations on what resources are available and the way that I think about this about using these hard limitations is actually something I have termed transplant Tetris so let's play a little bit of transplant Tetris and talk about different scenarios so I'm going to put a few cases out there talk about our thought process and then I'm going to open it up for questions so let's start with living donor kidney transplantation living donor kidney transplantation needs operating room stuff anesthetic meds and we need COVID tests for both donors and recipients we need some space a lot of space we need two ward beds a few days and two operating rooms we felt like one of the things we needed from a support standpoint for living donor kidney transplantation is a safe surgery pathway and I'll talk about that in a second and then we also needed staff for two operating rooms for our for our patients on the ward and we need two to three surgeons to do a living donor kidney transplant we use two in the donor one in the one in the recipient so it's very resource intensive from a space and staff standpoint not so much in a in a hospital length of stay standpoint so our decision for living donor kidney transplant was to shut it down early when we didn't know what was going to happen with the first with the with the peak in in COVID cases we shut it down early for two very specific well at least two very specific reasons number one COVID testing was not available and we didn't think it was it it was safe to bring in a donor and a recipient for what could for a surgery that could be done you know in a couple months or whatever without without ensuring that they didn't have asymptomatic COVID because we knew that patients had post-operative complications even with asymptomatic COVID and we were worried that that we would that this was a very risky proposal to bring them in without being able to get COVID testing the second thing is something that we have termed a safe surgery pathway so what our hospital um what our hospital did is we have we actually created two hospitals we have a tower that is sort of all COVID ICU stacked on top of each other and um that hot that tower is uh was actually uh part of the Ebola pathway when when we had the uh the scare with Ebola so there was a direct pathway from the ER to these ICUs and then we had specific floors for our patients who didn't require ICU care so we tried to sort of cohort our COVID patients in certain areas and then for our safe surgery pathway we brought patients into um an outpatient clinic for um for COVID testing within 48 hours of their surgery to make sure that they didn't have they didn't have COVID if they tested negative we had them quarantine for those two days and then we brought them through a different pathway into the hospital so that so that we minimize the uh the risk of exposure for um for uh our surgical patients and so until we we could ensure to the best of our ability a safe surgery pathway we didn't want to pursue living donor kidney transplantation the next the next one and I've kind of talked about this uh is the brain dead donor so when we think about brain dead donors we have to think about we need OR gear we need COVID testing for donors and recipients just like with kidney transplantation um the uh we need space for both the donor and for the recipient or the recipients and remember often donors are in different hospitals than recipients so you need space in multiple places in order for a deceased donor to uh be for deceased donor organs to be utilized we also um needed support specifically for livers charter flights um and then OPO support for coordinators and even local surgeons who could do organ donor procedures so that we minimize the travel risk uh that went along uh with our team going out for livers and so we had to we had to lean on each other as transplant centers for um for support especially um again in the early stages of the pandemic where we were we were particularly focused on um on risk and minimizing exposure as much as possible we also need staff donor or staff recipient or staff ICU staff etc um so with brain dead donors we continued uh as our as our organ procurement organization offered uh they continued to um to work up and accept brain dead donors uh as I had said earlier the the numbers tended to drop each time we had a peak because of the um because of the fact that there was less there was less uh kind of thought process that went along with referrals but um but we did continue brain dead donors we were we were a little bit more selective um uh with which donors we would accept so that when we accepted a donor we were we kind of knew that that this donor would have um great organs and that we were that we were putting resources into uh what would be a successful case um donation after circulatory death donors I think are also an interesting case to bring up they are very similar to brain dead donors in general in the need for supplies space and support but the one thing that's important to note on on donation after circulatory death donors is that you actually need either an ICU attending an ICU nurse or um or a combination of both to be available to be able to sit in an operating room or in the ICU at bedside to provide comfort measures and to be available to pronounce the patient at the time of cessation of circulatory function um and as you all know there is there were there have been many times where there has been so much um uh pressure on the staff to you know they're they're squeezed with the ICU trying to take care of so many patients that that it's unrealistic to ask an ICU physician to walk off of the ICU for an hour an hour and a half to sit with a potential donor and um and be there for pronunciation so I think that that that was one of the biggest factors that that has affected donation after circulatory death um from a resource uh from a resource standpoint. Disease donor kidney transplantation I think is an interesting case because you you know from a supply standpoint it's just like everything else OR gear anesthetic meds COVID tests um not a lot of space and OR a couple days in the ward and that's pretty much it um the support is kind of unique you have to have transportation for the kidneys so when you get into flight consistent distance kidneys you have to you know some centers are affected more than others because there are not a lot of flights that go into for example Jackson Mississippi um whereas Dallas probably we weren't we weren't overly affected by um by the transportation limitations and then we need staff we need an anesthesiologist to do our cases we need our ward nurses it takes one surgeon to do a kidney transplant um we had early early on in the pandemic um we had been we we kept questioning should we keep doing kidney transplant should we not because we you know thinking about it like patients can wait on dialysis they can wait for a kidney transplant but they're kind of two factors that went into thinking about well maybe we should continue kidney transplant one is that the patients um uh the organs would otherwise go to waste so if you're already doing a disease donor and those kidneys are are coming out with everything else then they either go in the trash or they go in they go into a person and so while kidneys are while while a kidney transplant in general is not an emergent procedure the in order to utilize the kidneys from a specific donor it actually is urgent it has to happen now or it's not going to happen and so that is one thing is making the best use of the organs that we have available so we did feel like if we had the space in the capacity we should continue with disease donor kidney transplants um we did focus on utilizing standard criteria organs because that meant less risk of complications and longer hospital stays um but we did continue on with disease donor kidney transplant the other thing that um that has grown into our our conversation about kidney transplant is yes they have the alternative of dialysis but the problem is that dialysis centers are very high risk for the um for the spread of COVID and so we were we we have become more aggressive in trying to get our patients transplanted because they're alternative of continuing dialysis especially the ones who are on hemodialysis who go to a center three times a week they are at very high risk of contracting COVID um and so if we can transplant them with a good organ and get them out of the hospital in three days the risk uh associated with with getting COVID in the hospital for those three days is probably a lot less than spending six more months on dialysis so that that's sort of something else that comes into the calculation of thinking about when we should pull the trigger on doing um on doing disease donor kidney transplantation now the final um the final case that I want to talk about or case example that I want to talk about is disease donor liver transplantation so it is much more resource intensive in the way of supplies and specifically for for liver transplantation most patients require blood and blood products and as many of you know blood and blood products have become critically limited at various times throughout the pandemic and so one of the main considerations we had with disease donor liver transplantation is do we have the what what blood and blood products do we have available for um the compatible blood type for our for our potential recipient space we for a straightforward liver transplant you're looking at about a day in the ICU and three days on the floor so it's not it's not um a huge space limiting situation but the sicker the patient or the patient who's already in the ICU pre is probably going to spend more time in the ICU and more time on the ward so that is definitely a consideration when you're thinking about which of um which of the potential uh transplant liver transplant patients we should transplant um support is uh again in the way of transportation for either the donor team or for the liver and typically this requires a charter flight so again limitations and charter flights can uh could become an issue uh with uh with doing deceased donor liver transplantation although again we we have been very lucky with not having not not being limited by uh by transportation issues um at least to this point in the pandemic and then it it requires staff anesthesia at least one anesthesia provider um our ICU nurses up front for the initial hospital stay and then obviously ward nurses um for the day to follow and then two surgeons uh in our center are um are necessary one for doing the donor one for one for doing the recipient so we need we need a couple of couple of surgeons as well available to do that so liver transplant um has been interesting because they're kind of two pathways you can think about one pathway is that you have patients who are super sick in the ICU who are utilizing resources already and they're like I said earlier they're either going to get a transplant get out of the ICU or they're going to die because of their significant liver disease and so when we think about those patients if we get offered a great organ for that patient and we have all of the things necessary for um for doing the transplant and specifically in that situation if we have blood available because we know the sicker liver transplant patients are harder to do and they're going to need they're probably going to require more in the way of blood and blood products than a more straightforward liver transplant patient if we have the supplies available that's that's one that we we would absolutely pull the trigger on now the other case that we um that that came up multiple times um was that as the pandemic kind of waxed and waned in different parts of Texas and Oklahoma which is our our region for um for organ allocation we um we saw some centers would be if a center would get overwhelmed with COVID patients they wouldn't be able to accept a liver at a given time and so what would happen is that the that the liver allocation would actually drop down a really really good liver to a patient who's sort of mid midpoint on the list who's pretty straightforward and we thought would have you know this short ICU stay short hospital stay and so our take on that was that this patient needs a liver transplant we had them listed for a liver transplant if we have the resources available and we are not going to put a strain on our hospital system we should take this liver for this patient because they will probably never get an offer like this again and they will benefit from from the liver transplant so that that was sort of um another one of the considerations that we had as uh as we were being allocated livers uh for deceased donor liver transplant so I am going to finish up so that we have some time for questions with this table um and this is out of the American Journal of Transplantation article and what what what we did is we brought up some questions and considerations about resource availability and triage considerations for the allocation of uh transplant organs and for determining who to transplant so you know from the donor hospital and organ procurement organization standpoint we asked what are the limitations there are they limited by blood products ICUs ventilators etc what are the limitations at the recipient center again blood products ICUs ventilators all that all that stuff what are what are your what are your limitations at the different places where your donor is located and where your recipient is located the next question we had is what patients are safe to transplant given the limitations is it none do we literally have no resources to transplant if it's none we don't transplant patients is it the sickest who are already utilizing the resources like we talked about with fulminant hepatic failure patients is it only the most urgent you know the patients who are sick they're at home but they're not gonna they're they're they're gonna need a transplant before this pandemic is over or is it like the last case I talked about a moderately urgent patient who will not put a strain on our resources so we think we can get them in and out of the hospital relatively quickly with a good organ and they will benefit from from that transplant and then finally what resources does the institution and the opo have to minimize risks and this was this this was definitely an issue more at the start of the pandemic specifically with our ability to test both donors and recipients for COVID we also wanted to make sure that we had isolation room capacity for our recipients because they were they were getting immunosuppression medications we wanted to make sure that the recipient hospital had enough personal protective equipment so that so that you know for example our nurses could gown and glove and mask every time they went in to see our transplant recipients so that we did not increase the risk of nosocomial spread of COVID so those are just kind of some of the some of the considerations we had or we suggested in in our allocation decisions for transplantation and so what I'm going to do now is I am going to conclude and see if you all have any questions comments etc and I think Laney is in charge yes first I just want to say thank you very much Andrea on behalf of everyone that was an incredibly interesting talk the um the first comment that we received was from Dr. John Fung who's the the chief here of the section of transplant surgery as well as co-director of our transplant institute and he made the comment that um one interesting development during the COVID pandemic is the enhancement in cooperation between transplant centers in terms of procuring for distant teams in other words a local transplant team would procure and he then which you also mentioned in your talk but he went on to say it was increasingly available encouraged by UNOS and the Association of OPOs and paved the way for the current changes in geographic distribution with increasing organ sharing and I just wanted to know if you wanted to comment on the increasing organ sharing absolutely so this this is a very important point so something that happened at the same time as the COVID pandemic began was the uh there was a change in liver allocation and what that change did is instead of allocation um in your own uh within your own organ procurement organization or donor service area uh allocation was in concentric circles or it's called a QT circle so 500 miles is sort of the the radius around the donor hospital where allocation happens and then it's by severity of disease and so what happened is that that the new allocation system has increased the number of flight consistent distance donors and it's increased where those donors are coming from so for us instead of interacting with four OPOs we're now interacting with about 11 11 to 13 OPOs within that 500 mile radius and it also means that we're interacting with a lot of other transplant centers and because of COVID at the same time exactly what Dr. Funtz said we are um we are utilizing the local resources and asking other surgeons to procure for us and one thing that I hope as a positive comes out of the COVID pandemic is that we continue to do that because I think that you know ultimately we should be able to trust each other to procure organs we can decrease the costs and the risks associated with travel if we can continue if we can continue doing this great um a question that I had that you didn't address at all was just about um vaccination prioritization for your transplant recipients and was wondering if you wanted to comment on that absolutely so um I am going to advocate for my patients right so I do think that um I do think that transplant patients um we have seen are you know they're at increased they're they're at risk of getting COVID infections and of um and of getting sick from COVID um there's actually not I mean they're the literature is kind of all over the place and so there's not a great argument to say that transplant patients get significantly sicker because of their immunocompromised state and they may actually be a little bit protected from that secondary immune response um uh that that causes like the terrible lung injury with COVID but one of one of the places where I think we absolutely need to advocate for our for our transplant patients is the end stage renal disease patients so there is clear evidence that end stage renal disease patients who are going who go to dialysis centers for their dialysis are at high risk of contracting COVID because they they're exposure they're they get exposed to a lot of other people frequently um and so and that's something that we have advocated for and that we have actually concentrated on so our dialysis centers uh in Texas offer offer the vaccine. Wow that's that's great. Got a comment from Janice Benson who says Percy tells you what an amazing talk which I agree with and then he writes you have a recommendation for our advocacy on these current regulations of transplant organ allocation and changes both diversity and due to COVID restricted resources like shouldn't make changes given that we're in the era of COVID. So I I would one thing changing the allocation system is like moving I don't know like an elephant I think is probably the the best way I don't I don't know how many of you uh we used to describe our um some of trying to make changes in our VA rotation in general surgery as you either try to push the elephant or you just ride the elephant um and so I I kind of think about the national organ allocation system as the elephant it takes a lot of time to make changes and when you make changes they're permanent until you can take a lot of time to make a lot of other changes and so I would not advocate for changing the current allocation system based on the based on factors that go along with the pandemic. What I would advocate for and what UNOS has done that I think is very important is that they have added a um so when you say no to an organ you have to give a code you say either it's donor factors it's recipient factors uh organ transportation distance there there are all these different things one of the codes that they've added is COVID related recipient reasons or COVID related transplant center reasons and so you can turn down an organ and say that my transplant center cannot accept this organ because of COVID related transplant center reasons or recipient and so I think that that addition to the allocation system to be able to account for the local effects of COVID is really important and it doesn't ding the transplant center to say that you're turning down organs for you know a reason that uh you know that that's less acceptable or whatever so um I think that that's where we should focus the change in allocation um so so given though that we are in the the COVID pandemic era um you made a comment about the standard criteria organs is there a reason to sort of not then that there'll be a greater discard rate of the non-standard the extended criteria donors just because of the possibility of longer stays and so did we see an increase in discard rate of organs so I cannot answer that with evidence from like a published paper but what I can what I can say is that in discussions with our organ procurement organization our local organ procurement organization they were less willing and put less resources into extended criteria donors and the reason there were many reasons why but one reason is that an extended criteria donor typically requires more time in the ICU and more testing pre-donation and so they're very resource intensive and the number of organs that you that you end up utilizing from an extended criteria donor are low and so I think where I think that our organ procurement organizations when they said okay you know we're we're not getting as many referrals we really have to concentrate on getting you know on these donors where we are going to get multiple organs have you know save multiple lives etc so I think that it was sort of a mindset change in our OPO and also recognizing that from the transplant center standpoint like if I take a kidney and my patient has DGF and then they're in the hospital for like five six seven days they're using dialysis and then they then they leave and they're still on dialysis that's a big risk because now they're on dialysis and they're immunosuppressed so I think that we kind of we kind of all kind of constrained what we were willing to take great Pringle Miller asks his transplant during COVID revealed differences in the allocation of organs from a race ethnicity standpoint Pringle so good to hear from you number one thank you for being here um you know I don't know I will say that there is evidence to show that that is already a problem in transplantation in general um there are disparities in the listing of patients um from uh from a race ethnicity standpoint and there are also disparities in transplantation um specifically liver transplantation from a gender standpoint that has not changed with COVID but I don't know if it has gotten worse with COVID the only fact I can add is that there was a 25 decreasing kidney living donors in 2020 and the greatest decrease was in black living donors and we could talk about lots of reasons for the difficulty and access issues that absolutely Dr. Ishmael asked the question how important was reason regionalization for success in the pandemic time regionalization um I mean I every so every transplant center was affected differently from you know from the reports that we've seen so for example even even if I just talk about Texas we in the the hospitals you know in in our area versus in Houston Dallas got we had a very significant first peak but we at least at our center we really prepared very well and so even with the first peak we never hit a hospital capacity where we could not support urgent operations and transplant was part of that urgent operation in Houston they got hit harder with the first peak they got to the point where hospitals had no capacity for ICUs and so they could not do liver transplantation at certain hospitals at certain times like there were you could clearly go through an allocation you know you would get a liver offer and you could see like the Houston hospitals can't can't can't do livers today or they can't do kidneys today so regionalization was really important um from that standpoint different transplant centers could and couldn't accept organs at different times and then also I think that it affected when donor when particular donor hospitals were kind of more or less kind of willing to work with OPOs to to to manage donors so it definitely I mean New York shut down Washington shut down for a fair amount of time we never shut down in Texas but Houston definitely had had a heart and we still haven't shut down um Houston definitely had a harder time than we did even um when you look at the effects of the pandemic back to john fun put in another comment he says it was interesting that when we look back on the national transplant activities is the rapid recovery shortly after the stay at home mandates so talking about your two different um surges and how you addressed it yeah yeah and and lanie you had commented earlier before we started the actual number of deceased donors this year is like a record it's very close it's it's a really high number and so while yes during the during the peaks we had we we had about a 40 percent decline in donors the rebound was pretty astounding like when when we would get into a trough the the number of donor offers and the number of transplants that we did would pick right back up so we saw kind of peaks and troughs in uh in transplant donor availability as well uh peter angeles writes angie thank you for an excellent talk covid has taught all doctors to think more about allocating scarce resources and we have often wanted to transplant surgeons have been dealing with these issues for decades are their recommendations that you have for the rest of us and how to address the provider distress caused by not being able to give all of our patients everything that we may want this is a tough one so i the first thing peter thank you for the question and good to see you um the uh the you know one of the things that we do in transplant and that i think we do very well is we take the allocation decision or the prioritization decision out of the hands of individual transplant surgeons and individual centers and so yes we have the choice of who we list but we don't have the choice of who gets priority and i think when you when you do that you take less of the guilt and less of the um the the difficult decision making about who to transplant um out of the out of the process and so one of the things one of the lessons learned that we had is that we and i'm sure that many other centers did this we created triage committees um here at baler and what the triage committees did is they were a group of two or three um two or three physicians uh apps or nurses who um who if we got to the point of having to allocate our scarce resources meaning if we had questions about who we should put on the ventilator who we should give priority for for icu that those decisions would go to a triage committee and they would be they would be presented in a way that we use some standardized criteria like sofa scores and so forth and so we took one of the lessons of transplant which is trying not to have the bedside physician have to make this allocation decision and put it on somebody who can look at it a little bit more objectively and create standardized criteria do you ever have to use that we did not we never had to use it we um we put it together uh actually at the level of the dallas medical society so all hospitals in dallas actually bought into this and then um we had we created the triage teams and uh we we had like a plan for when we would have to implement it uh we got close in the first set um but it ended up that we that that our hospital was so nimble that we were able to build more icu's in more isolation rooms before it ever got to the point of having to uh having to triage so great i'm very glad that we did not have to use it i think a lot of hospitals were developing those and are also glad here in churago from the hospital medicine standpoint one approach we took was to try to keep hospitalists on our transplant services clean meaning we kept these providers off of our many covid services sometimes these were pregnant providers or providers with other medical issues risk factors that kept them from caring for covid patients we also had hospitals who were on covid units cared for covid patients rotated off of full two weeks before they were able to care for transplant patients so well we did so we did a couple so one we have a transplant floor in our hospital so and our our covid icu's were in a different building so our post op icu and our transplant floor we attempted to not have patients under investigation or covid positive patients now as you all know there are patients who are asymptomatic with covid who come into the hospital and then like three days later you find out they have covid so it's not like you can absolutely keep your patients away from from covid you know from from the risk of covid so our nurses on the floor were only caring for our transplant patients on the floor we did ask that they wear gowns gloves etc like full contact precautions for for the covid patient or for the for the transplant patients just as if they were caring for covid patients um we from a provider standpoint what we did we did a couple things so one the transplant surgeons we decided that we would we would be happy to take any transplant patients whether medicine or transplant to our service to offload our medical providers so we would we were happy to be the admitting service for you know 10 years post-liver transplant patients 10 years post kidney transplant whatever so we offered that number one that gave our nephrologists and our hepatologists the opportunity to help out on the medicine side and even to do the some of them did like covid icu rotations or would were like word covid providers so we did that um and then the second thing we did from a surgery standpoint is that we you know transplant surgeons are general surgeons as well so we we built a call pool of all of the specialty surgeons who were also general surgeons and we took the acute care surgery call as well as offered to take trauma call but it never happened um so that our trauma icu surgeons could completely focus on the icu so we were so we basically offloaded other services to allow them to take care of ic of covid patients and we took care of our transplant patients and our general surgery patients an anonymous attendee asks will there be any changes to policies and organ donation for patients who have had covid 19 there's no there is no policy that says you cannot use an organ from a patient with covid 19 there is a general practice that uh that most centers will not use a an organ from a patient with active covid 19 now there are places that are starting to open up to the idea of using for example asymptomatic patients who have had like let's say a traumatic injury they come in they get their covid testing because everybody gets tested when they go through the er and it turns out they never had symptoms but they're covid positive those those individuals almost certainly are not by remake from covid and there's almost there there's a very very low risk that you would pass covid on with uh with an organ transplant a solid organ transplant from that donor so we as a center are willing to consider those donors on a case by case basis so i think that policy i think that policy at least from it from a you know standpoint is that you have to test donors but they do not specify if you if you have to um if you can accept them or not well that has to be an interesting consent process with the patients and it might be made easier if everybody were vaccinated right i mean might be more willing to accept such an organ if i know i've already had my second dose i have i i've accepted one organ from a donor with a positive covid test um and i called the patient myself and explained the situation and said look this is up to you i'm not i'm not pushing you one way or another um but this was a scenario in which this was a perfect organ this like there's no way this patient would have ever been offered this organ if it was not because this this donor had a positive covid test we had enough history to say that they likely had covid a month prior and that they um and that they were still just asymptomatic shedding so the chances of virumia were very low but um but i had a very honest discussion that said we have no idea what number one what your risk is of getting covid and number two if covid having had covid has any effect on the organs and the guy ended up he ended up saying taking it and he's doing great you know and of one the happy anecdotes we like to hear you know yeah but i do think lani to your point you have to have a very kind of open and honest discussion and talk about the unknowns because it is what it is so um we did have someone wanting to know about the cost costs of organizing a private flight for organs and i guess i would just ask increasingly used or the same with all the new allocation rules um so i can tell you off the top of my head since we did a study of our trap our transportation and cost pre and post um the change in allocation we fly for about 50 percent of our donors and that was pre and post so it's right around 50 percent um the uh the distance that we fly has not changed so the median travel distance is 160 miles both pre and post um and uh the costs have gone up um but the costs have gone up in a very not in how you would expect so for example the cost of flights are the cost of flights there are there are a lot of variables that go into it contracts etc um there are certain we have certain um flights that are less expensive for example if we fly out with our opo to like our opo is very big and so el paso is about 500 miles away if we fly out to el paso with our opo on a private jet um and we procure kidney and livers or liver and kidneys um those three organs fly back on the plane with us the flight costs get split between the three recipient centers so that works when you're working within your organ procurement organization with the change in allocation there is less likelihood that we are working within our organ procurement organization and so flight costs go up because you pay the full full price of the flight um the charter flights have certainly probably will continue to increase in cost because the demand is increasing um but we i mean it's hard it's kind of hard to parse that out our contracts haven't changed at least up to this point great so um elizabeth murphy had been the one who talked about the hospital medicine and keeping the quote service clean i just want to know she wrote back to you saying that your response was really wonderful how much other physicians chip in has correlated strongly with provider burnout uh which is a really important issue during the covid era so i just wanted to repeat it for all of us to hear um at this point i don't have any other questions in the chat or the q and a so if anyone has any otherwise i'm just going to turn to you mark and see if you have any final words or we should end up 112 three minutes early whoa are there any additional questions no nothing's coming in mark so um i i wanted to just tell angie how delighted we all are to have you back in the midwest not not in st louis but up in chicago and uh and how excited we are to to hear from you and the talk was extraordinary and um uh and and the the questions and and responses were wonderful um i i was i was moved by uh john fung's first question and and your response to that um so it's it's wonderful to welcome you back and we look forward to uh to continuing to work with you thank you so much well thank you thank you thank you thank you this has been fantastic it is just such a pleasure to have the opportunity to uh to interact with the mclean center and i you know i have um well i didn't do a fellowship there i like to think of myself at least as uh as being an honorary member uh and um i i love every opportunity i have to uh to talk with you all so thank you so much we consider you in the same category well thank you that means a lot well everyone thank you so much for your time for your attendance and um and i'll uh sign back on to talk with the fellows in about 15 minutes great thank you very much andy thank you very much thank you bye bye thanks lean


Thanks people for joining us today about discussing how we try to help about the CPU power management so Let me present myself Silvan Bosa, I'm the senior software engine. I'm a senior senior software engineer from Red Hat working on working on Nova since the last decade. I'm also the open-stack Nova PTL and previously the last decade I was also an operator Hello, so my name is Roné Ribot. I'm a software engineer I'm a quite new contributor and I discovered Linux and the Floss community in 1995 and previously I worked as a solution architect mostly on cloud and DevOps activities So let me explain how this story started. This story started because previously on the last Open-infra summit that was in Berlin someone from the BBC presented something that they were using called SkaFound for getting power measurements from their clouds and when I saw it, I thought it would be nice if we could help At least the operators about that. So That's why we that's why we do it Also by the way if you remember about last year The electricity bills Became more expensive. So I think it was nice to help out to have less power about that And also we are all facing an enormous challenge about Global warming. So that was one also of the motivation to work on that topic. So after the summit When we discussed it was for the Antelope PTG So the Nova community started to discuss about it What we did is that for example we had two topics for discussing about that We are having for example one topic for discussing how to reduce the power consumption and also another one about power management measurement and what we could reduce So first about auto reducing the power management Maybe you know about that but for example for oops So basically Sorry about that So what you what you can do with a Linux for reducing the power? So basically you have multiple ways to do it So for example for active or idle processes what you can do with rel is that you can change What we name the CPU profiles by using something named tuned So you have multiple CPU profiles. So for example you can use a specific a specific profile like Let's say power save or balanced or a new tune profile That's named CPU person in perverse safe. That was created last I think on the last month What you could do then it's nice because then you can modify the c-stats It basically modify the c-stats sorry like using C3 which is good for active processes like DPDK You can also ask the kernel to stop the CPU cores. That's something that the Linux kernel supports Or you can also ask to reduce the CPU frequency by using CPU Freq if the OS supports it The only problem about that is that maybe you know about that already But it's not automatically. It's not automatically So what we did is that when we saw about that Then we wonder how to automatically doing that with Nova That was what we did last cycle on enthalpy So what we did is that we created new config options that were saying for example if you want Nova to support the power management then you can say true and then you have two strategies first one is saying that you can offline a CPU core or Instead of offline in the CPU core the other strategy is saying I would like to modify the CPU governor Just to be clear a CPU governor is about modifying the frequency for the CPU. So as you see that's an example You can say in that example that you would like to stop CPUs in between 2 and 17 If none of the instances are using them that being said that means that you need to make sure That those CPUs are dedicated for instances and are not shared in between instances. So that only works for pin CPUs So basically, oh, does it work? So this is demo time Up it will be working So in that example, I have a house with eight CPUs All of them are active are currently active if I'm modifying the Nova Conf in that example I want to only use two CPUs to be dedicated then I'm basically modifying the config options When I will be restarting the Nova CPU That's what I'm currently doing I'm using DevStack here Then if I'm looking back at the CPUs as you can see both of the physical CPUs Are no stopped no If I'm creating a new flavor that's asking for a dedicated CPU in that case I'm here in that example. I'm asking for one dedicated Dedicated PCPU then when I'm creating the instance No, this is active just before it becomes active Then basically Nova automatically Online the CPU in that example given the instance was using CPU 6 it's basically It's basically online and when I'm deleting the VM or when I'm stopping the VM Then basically the CPU is going back offline So that's it for the demo Okay, but basically how does this work? This is pretty simple We use ccfs So basically with Linux you have a specific ccfs file file system and Here for example each of the CPUs is a directory From ccfs. So what the Libre driver is doing is That when restarting the Nova compute We look at all the dedicated CPUs and then we ask ccfs We directly call ccfs to either modify The file for online or to modify the file for cpu frag scaling governor if by the way The US supports it again. No, no not all the US Support scaling governor, but for for those who do then you have a problem you have that and basically When then you want to start an instance or you want to create an instance Basically, we're basically getting the CPU from that instance and again We're calling ccfs to either online the cpu or modify the governor Know that you have that I Personally think that you have some quick wins You have some cool queens because given the current scheduler I think it's nice to basically pack the instances to specific hosts So basically the power of that host would be better You have multiple ways to pack your instances Either you use a cpu white The surface affinity white multiplayer or you can leave your user to ask to use the same host filter by the host parameter What you can also do if you want to pack your instances after they're created is that of course We can migrate your instances to move them to exact to the exact same host Or you can use what sure if you want to do it quite automatically that said Nothing is done automatically by the scheduler About some scheduling decisions related to the power measurement Nothing is done automatically. So basically you need to do this by yourself As you understood There are some limitations though with the new feature the first one. I Think you already understood it This is only about dedicated CPUs This is not about shared CPUs because basically we don't exactly know whether the shared CPU is used or not and by how many instances So this is hardly the whole for public clouds But for the other clouds that basically use dedicated dedicated CPUs They can use that of course If you have restricted CPUs that doesn't that are reserved For I would say other processes, but Nova Nova won't be able to do this right because those are reserved processes I was told I was talking for example about DPDK That's one of the examples. Nova won't be able to modify the CPU of those other reserved CPUs As a reminder as well That's not only the CPUs That have power consumption and like GPUs or nicks Can have also large power measurement. So I think it's probably Something we could be trying to see later Which will be about How to help that but for the moment this is only about CPUs also If you really want to pack your instances on some specific hosts Then you may you may have some power cons more power consumptions of the top of the rack switches because of course you're basically sending All your instances to the same hosts and basically the nicks Would be having more usage So it would be a problem with the switches maybe more power consumption basically need to modify we need to monitor that and eventually I Guess most of you are wondering What actually is the difference between an idle CPU or an active CPU because actually that depends on the instances on the workloads if you prefer and That's why I think it's also important instead of just trying to like Stop the CPU or start the CPU to measure it in terms of what and that's why we're moving to the next section Okay, so what we could improve on measurements. In fact, we looked at the BBC use cases and The use cases is great, but they are measuring the consumption on the overall VMs. So we thought that we could improve it by giving the power consumption of processes inside the guest VMs So the idea and the goal could be to enhance the dashboards with more granular view and also maybe to have dashboards more oriented application oriented and So users could act on So what is scoff on so scoff under is the underlining tool used by the BBC folks for various cases It's a monitoring agent Dedicated to energy consumption metrics The author of the tool is someone called Ben Whappety and it's developed by the community It's under Apache license Apache v2 license and I would like to make a strong disclaimer here because this is a community tool without product utilization or support by red Hat So how scoff on there is working so scoff on when you start an instance of scoff on it's defines server CPU topology It gathers rappel data. So rappel data It's data providing by the Nintel technology to estimate CPU and run power consumptions And in other in another hand, it gathers system processes based on this collected data Scaffold is doing some calculation to provide the consumption from the host down to the Processes so you will have the power consumption of host CPU scores and down to each individual processes and When you have that you can export this data using Various common format so it could be primitive use which is the most famous one and Jason Reman a lot of formats and combining with a tool like grafana You could do nice dashboard nice dashboards like you can see on the on the right So I would we expose and configure scoff on to have the guest VM data So on the left you have the standard configuration So you are just running an instance of scoff on the compute host and scoff on is providing so the Processes consumptions and you will have to look at the QMU processes to have the consumption for your VMs and Moving to the right. This is the configuration with QMU exporter. So you are first you need to run scoff on with QMU exporter on the compute host and what it does it will scoff on will extract the data on for each each QMU processes into one one directory and Scaffold will try to extrapolate the the power consumption for this for these VMs Into the form of a rappel device. So basically, this is a kind of a trick to emulate or fake Rappel device for guest VMs guest VMs Once once you have done that you need to share the the metric the This data with either of a virtio fs or virtino virtio 9p from your host to the guest VMs and Finally on the on each guest VMs you will have to run scoff on with the dash dash VM option to collect this data and then At that moment you can use whatever exporter you want and it will give the Power consumption data of the processes for the guest VMs and that's it So a quick glance of what is virtio fs and what we have implemented in Nova to support that So virtio fs is a shared file system It allows to share three directories and directory tree in between your host and your your guest VMs and the benefits of using virtio fs. It's It's really It has the same semantics and performance as a local file system. So that's good and to use that via libvirt you need to Configure that into the XML files Libvirt XML files and that's the reason why we had to implement something because the XML file is managed by Nova You cannot modify it by hand and that's what's the purpose of a feature we introduce Initially to support many shares So This features requires a QMU file the 0 and libvirt 6.2 and really important the instances Should use share memory. So it means that on these instances you will have to either Configure file back memory or huge pages to use the features The status of this feature is still in in dev because we have to fix some Some issues with Manila, but that should learn hopefully in this cycle So What about the scaffold feature? So first I would like to say we name it like this for convenience because it was easier for us to share about it But the idea was to share Power data. So this is kind of tool agnostic and basically we leverage what we have done with virtio fs And What I just mentioned and the idea was to share also a local file a local share on your computer host To the guest VMs To enable the features you have two steps. The first steps is to enable on the compute node So this is done by using the share local FS parameter and you have to supply The directory you want to share and a tag that will be used by the guest VMs and On the instance you you add to enable an over to enable that we're using hardware power matrix Parameter and it can be done either via an extra spec or an image property Time for the demo so Quick demo just to kind of proof of concept So basically it runs on the already configured compute host and so I have created a new flavor Passing the hardware share local FS parameter. So now I will resize VM stop and resize a VM to use the the best new flavors with the parameter to enable Let's say the scaffold feature So when it's done we can I will start the demo instance So if we just have a quick look at the Nova compute logs, we can see we have a section in that file system section defining the shares and If we connect to the guest VMs You can see that You can look at the metadata of the guest VMs using a curl command so that will list The fact that we have a share and the tag calls staff on so we could use this tag to mount the share so that's done here and If we look at into the share we can see that we have intel rappel data and Then if because we have the good data we can start scaffold and the scaffold is currently trying to process the data and we can you can see that we associate power consumptions with With the processes so for the moment. This is just a POC This is a as well not to merge so it might change in the future because I have some comments to change some stuff and also don't Don't really look at the data that was more or less fake data Just for purpose of showing it first Okay, what's about the limitation? So right now we have some limitation due to Virtaio FS and the churn memory and in fact the limitations also are from QMU because QMU as to Disables from features like migration or snapshots Because when you have a file system attached There are some Let's say more development to do around that. So as a result for us We had to disable some operations like resize migrates shelve live migrate and evacuate So this is quite annoying for the moment Because if you want to use that kind of operations You need to stop you need to unshare by stopping your VM An overall limitation as Siva mentioned We are just we just have the CPU and memory consumptions. So we are missing the GPU one even if might be introduced into Ska font soon and also more complex topic we don't have the devices like Knicks or FPGA and Maybe let another limitation. It's that rappel. It's like I said, it's an Intel technology So it's available on Intel and AMD so it means that that's not supported on IRM or RIS 5 CPUs and Finally, what are the next steps improvements and ideas we are on this topics so first QMU upstream as already tickets and they already works on Removing the limitations Operational operational limitation so we could do as well on Nova when it will be available So this is one of the access another possibility we have a colleague which is called Anthony Arrival and basically what on or simply what Anthony has done he he put into QMU the mechanism used by Ska font for calculating the power consumptions and So that would be available in into QMU if the patch the patch is accepted And at that moment it means that sharing will not be required anymore. So that could be a nice improvement Obsoleting what I what we have done so far and An over-alternative instead of using virtio FS. I mentioned that really quickly We could use an over protocol, which is virtio 9p and virtio 9p does not require short memory. So in the meantime, we could allow to We will not have to reduce the operational stuff. So If you are interested by that feature, maybe we will discuss that in the in tomorrow PTG sessions and We also have a tool called Kepler So Kepler is a tool that using machine learning to predict consumption. So maybe that's another way to To Let's say have better results and maybe to deal with all the devices and things like that. And by the way Maybe Kepler will be fully supported by Red Hat So could be an over-alternative Well, that's it. That's it for our presentation. Maybe you have questions. We are here for that. So no worries Any questions so yes It's a it's a compute specific config option So basically either you modify of course all your config of I mean all your modifier your computes But yeah, basically, it's per compute Second question is Correct. So basically what we do if you prefer is that For example, say you reserve some CPUs outside of Nova when you modify the config options by saying, okay Those are the reserve CPUs and I only want to take those Dedicated CPUs that we are basically using the CPU dedicated set that people are using That's the new config option name that what previously that were named. I think pin CPUs I can't remember exactly the right name, but that that was the previous name of the config option So basically what we do is that we just take those specific CPUs and then we basically stop them But for the other reserve CPUs by the way, I forgot about that But we also don't support CPU zero because in general CPU zero is used by the kernel and for other processes So you can't just try to modify CPU zero if you do that then you will have some problems But yeah, basically just about the physical CPUs that you can use for Nova You have to be careful to manage on the CPUs that are not used by other systems So this feature does not make checks any that these CPUs are in the machine slice or that are allocated for Nova Yeah, basically if you already support basic if I use Basically pin CPUs that's something that we already support like I can't remember for like a lot of years before Basically, what we do is that we we basically Offline them if you don't use them. That's basically what we do any other questions You talking about the offline online stuff, so what do you mean by bar metal Indeed, I mean say for huge pages or for stuff again, it's just about CPU pinning It's just about CPU meaning. So again, if you see you pinning basically will be offline What's the impact of the consumption to disable one core or not Very very good. I was pretty sure that someone was thinking for it So basically the measurement of that basically, it's a very good question So what I mean, I'm not seeing someone that that was trying to look at that I would say For the instances It would be good, but sometimes About the performance month what I would say is that from what I heard Because I'm just a developer, right? I'm not really testing it, but I mean surely I'm testing it But not not about measuring this Musing the power what I'm saying is that Yeah, basically, I would say For instances that that's nice in terms of the power measurement that you have for example if if you use that cool in general if you use for example NFV and then for example you have srUvfs With dpdk and something else Maybe you could try to use tune D What that's what I said in a specific slide for those other Processes like dpdk and that's why I told about the tune D profile because that's what we saw at Red Hat It also helps those those processes I Excellent question. I haven't thought about it yet Yeah, I mean in general, you know Nova doesn't I mean that's that's that's done by I mean that Will be done by Nova automatically that will be done by a deployment tool So so I would say that will depend on the employment tool about measuring the power good question The problem is that if if you basically stop the host and then you restart it Sometimes about the measurement of the power and the time it takes maybe it's maybe not that good. So maybe you should be Continue to just run it and try to have like like I said see states for the other processes I I think it's would be better. I guess And I think we are on time. So thanks. Thank you


Hey guys, it's Chilly, and today I'm here to do another YA Chronicles unboxing. So first look, this box seems pretty damn full. I'm already kind of like in love with just the colors of the YA Chronicles card. And then you also have their wonderful bookmarks. This is volume 17 to someone say fantasy. And I know I'm already going to like this because it's fantasy. So it's like amazing. As Trabaz, this is from Darker Shade of Magic series IV, Schwab. And it's amazing that you guys should definitely read this series. It's one of my all time favorites. And the last one comes out soon. So it's the perfect time to read them, do it. Then you've got two other beautiful pieces of artwork.  So reading up on the card, these guys are from Tales. Fantastic. And there's obviously quite an amazing artist. So you have the Princess and the Dragon and Robot Walk. So, damn. Hell to the YA year. I'm so excited because one ring to rule them all. That's right. You know exactly what this is and it is not good news. I need to go and find a volcano to throw this into. Dude, if you guys haven't heard of the behind the pages store, I don't know what you guys are doing, but it is an amazing artwork store. Like just look at this stunning bookmark. It is actually one of the most beautiful bookmarks I've ever had. Plus it's a Raven Cycle bookmark. I mean, if you guys haven't read that, like what are you doing? Ah, yes, yes, yes, yes. This is the Penguin Teen Superproof. So this has like all different like inserts from the stories from these like authors coming up. These are books that are going to be coming out in 2017. If that didn't explain it with the title, but it's really exciting because there's so many new releases in this that I really, really want to read about and now I can have some snippets of them.  There's two books in here. I just like spat everywhere and the book, a shadow bright and burning by Jessica Close. And this is like on my anticipated book list. I am so excited. I need, I need to get out of my reading slump so I can read this book because I am so excited for this. Because the way a chronic is always says it better than me, you have a shadow burning bright, which is about Henrietta Howell. The girl thought to be the first female sorcerer. Thanks to a prophecy stating that a female sorcerer was the key to defeating the seven ancients. Henrietta is recruited to train with her majesty sorcerer in Victorian London, where she is to learn the skills necessary to complete the prophecy. Magic Fae Demons. This book has something for everyone. So you can see why I'm pretty excited about it because it's going to be fantastic. There was a lot more in that box than I thought it was going to be. And I'm so happy with it. Like, it's amazing. I feel like I'm going to have to go and take a time. I just took my appreciate everything inside of that. I will leave all the links to the way chronicles down below. I highly encourage that you guys check them out because they are amazing and the amount of effort they put into their boxes is outstanding. Thank you guys for watching. I will be back in another video soon. And until then, I hope you guys have a lovely bookish day and that great things happen to you. Hey guys, it's Tilly. You know that girl who isn't buying any books this year, who thought that with every month she goes by, she would get, you know, more books on her red side.


Like after that car accident, my career was supposed to be over.  Why is he coming back fighting Danny Garcia? He's crazy. He's just taking it for the payday.  Just like with the Oogas. Straight after that eye injury. Hey man, you want to tune up? I don't want to tune up. Give me Oogas. I got to get that third belt. People like, man, you shouldn't take that Oogas fight. He's just five man in pack now. Man, he looking good. Man, give me that fight. If it's going to be what it's going to be. Right. And as soon as I got Oogas, I watched him deal with what I had to do with him. And I already knew that fight, I was going to watch him.  I said, oh, he too comfortable. I said, he living American dream now. I said, I got him.


Hi everyone, Joe for Jaspy's casebreaks.com with another hanger box, the last one we got to do to unlock the next of Mako Football 2 box random team break. This is break 22. Big thanks to this group of people here. There's some teams right there. Let's roll it. Randomized names and teams. 2 out of 3, 5 times each. 1, 2, 3, 4, and 5. 5 times for the teams, 1, 2, 3, 4, 5. Alright, here's the first half of the list right here. Second half of the list right over here. Let's just print this out and rip up this box. And then we'll do another dice roll to see who's going to win those last 10 immaculate spots. And then coming up in a different video will be the 2 box break itself. And then we'll probably be able to run another 2 boxes later tonight. We've got time for it. Oh, look at that. Zach Wilson. There you go. That's pretty cool. I think those stars parallel are exclusive to this set. It's Andrew Roth. There you go, Andrew with the jets. Nice. This looks really sharp. I think he ended the season pretty strong, Zach Wilson. So hopefully you can build on that. I'm sure they'll add more pieces for him in the draft and all that good stuff. Let's see. I just want to see improvements every season. He doesn't have to suddenly light it up. I mean, if he does, great. But just like with Josh Allen, every year you saw something a little extra, something new with Josh Allen. Same with Lamar Jackson. All right, let's gather everybody's names, new dice, new list. Top 10 after nine, four and a five. One, two, three, four, five, six, seven, eight, and ninth and final time. So from 11 on down, no, no go. But thanks, folks, for giving this shot. Happy times, though, for the top 10. Mark is happy. So is Andrew, Andrew, Andrew, Sean Maddock, Andrew, Sean, Matthew, Andrew, and Kevin. Congrats to the top 10 here. We'll see you in the next video for the break. Jasper's Case Breaks dot com. Bye-bye.