Ladies and gentlemen, we're going to resume. I'm going to introduce our moderator, Tom Ricks, who's an ASU Fellow at New America, known to many of you as the writer of many important books on the US military, Fiasco, The Gamble, The Generals, just to name a few. A Pulitzer Prize-winning journalist, and I'll hand it over now to Tom to introduce Senator McCain. I can't think of a group that needs less of an introduction to someone than Senator McCain. You all know who he is. You know he's an American hero, and you know he's a fun guy to talk to. What's up? Senator McCain, welcome. It's a real honor to have you here, and we've had the parade of the circus come through. Now you and I are going to clean up after the elephant. There you go. You know what? There's a story about that, the guy that did that all these years, and somebody said to him, why do you keep doing this?  And he said, what, and leave showbiz? Anyway, that's the kind of way I feel being with you, because I should be questioning the expert rather than vice versa. I know nobody who understood the war, the conflict in Iraq, as well as Tom Ricks, and reported on it in depth and candor, and taught a lot of us a lot about it. So thank you, Tom. We're here at the end of a two-day conference on the future of war. There's been lots of talk about blurred lines between war and peace, about ambiguous situations, about no distinction being left between public and private, foreign, and domestic. Is the US military ready for this world? Well, first let me say, I'm not sure they're going to be ready for any conflict if sequestration continues. We had a hearing with all four of the Uniform Service Chiefs. All of them, obviously, at the top of their profession. And Senator Angus King of Maine asked them, said, if sequestration continues, we'll let's put the lives of the men and women under your command in danger. All four of them said, yes. My friends, I can sit here and Tom and I could spend the rest of this time we have together and identify waste and mismanagement and ways to save money in the Pentagon. But with the meat acts of sequestration, that's not the way to do it. And we are really impacting in so many ways that are not really visible right now. And one of them, by the way, that disturbs me, probably more than anything else, is that some very highly qualified, high quality men and women, both in senior non-commissioned officer ranks and mid-level officer ranks are leaving. They're just tired of it. Are you talking to people who are leaving? Oh, yeah. What are you hearing? They tell me we're tired of it. I don't know where I'm going next. We don't have the proper training. We can't operate. Pilots are flying two days a month to people down in a talk to down in Fort Bragg. They haven't had a real operational readiness exercise. And they can't remember. As you know, Tom, the first thing that happens when you cut the military is they cut the readiness, the operations, and maintenance. Those that are the easiest and yet have the most long-lasting effect. So first of all, right now, if we keep up sequestration, I'm not sure we are going to be able to defend the country given the enormous increase in threats since sequestration was enacted. But let's say that sequestration is not. Let's say we're able to reverse it. I think we're going through a period of adjustment to the new kinds of conflicts, the ideologies, the sectarian aspects of the conflicts, which you first saw back in 2001, 2002, 2003. And I think we need to have a far better understanding. What Henry Kissinger testified before the Armed Services Committee and what we are seeing is the breakdown of the old world order. The nation state is now breaking down throughout the Middle East. How do you make a military better prepared to adapt to that new world order or new world disorder? One thing that we have to understand is that when one group of people can inflict an attack on the United States as they did 9-11, and that changes everything the way we do business. Anybody who gets on an airliner knows that. Anybody who understands the need for security that has cost trillions to react to 9-11. And obviously, their objective is to attack the United States, not to defeat the United States, but to attack the United States. So I think we have a lot of adjusting to do to this new form of warfare. We have some very bright people in the military. Some of these young generals and colonels I've been incredibly impressed with. And they're sort of working through it at places like Fort Leavenworth and others. I'd just like to give you one example. After the attack, after we invaded Iraq, David Petraeus went back to Fort Leavenworth, as you know. And he spent two years with some of the brightest people he could find. And he came up with a strategy that worked, the surge. When we were losing, you chronicled that, as well as any other observer that I know. But he had developed a strategy that did succeed. And the US Army bitterly resented him for it. They resented him for it. He liked politicians. He talked to reporters. And he was successful in Iraq. In the army, that was three strikes. I'm not quite that cynical. But I do know that around him, he recruited a cadre of young people that looked up to him and admired him so much that he did leave behind in the United States Army a group of the Colonel McFarlane's, the originator of the N-bar awakening. And those people are still there. I would bet you the majority have left the army at this point. I'm afraid you may be right. I keep, because I keep on noticing it from emails, like, oh, they're working from McKinsey now and so on. I think you may be right. And one of the aspects of the military that you have seen and I have seen over my lifetime is there's really two ways up. One is the political path, and the other is the performance path. And quite often, the political path ends up in the positions of highest level. But so in short, a long answer to your question is we have a lot of adjusting to do. And we are in asymmetrical warfare, and we've got to understand that. Sir Michael Howard, himself a veteran of Anzio landings, wrote that adjustment, being adaptive, is the key to future victories. How do you get a military that is more adaptive? We had General Odi Erna here this morning, basically saying the US Army, as far as he could tell, did everything right in Iraq because the problem was that the rest of the government didn't show up. That's a mighty, unreflective position to take, I think, and not one to prepare as the Army for adaptation. Well, I think that, and you would agree, that he may have been referring to the fact that once George W. Bush had the courage to fire the Secretary of Defense, fire a whole lot of people who had us on the path to defeat, which you chronicled extremely well, and brought in Petraeus and this new group of young officers that they did succeed. And my friends, they did succeed. I can show you the statistics of the reduction in attacks, the Russian, all of the indicators of success, but it required leaving a residual force, a stabilizing force behind. It required that we didn't do it now. We are paying an incredibly heavy price for it. Do they have to be more adaptive? Yes. And I'd say in one area that you and I have discussed in the past where we've got to be a lot more adaptive, and that's in acquisition. I'm glad you brought that up, because I want to ask you a question about that. But I just mentioned there are people out in Silicon Valley that I've talked to that I've said, hey, why don't you come and join us and get into the business and help us with this new information warfare challenge that we are facing, the internet, the propaganda. And they said, we don't want to have to go through that labyrinth and that obstacle course that you call acquisition in order to get a government contract. What the hell, we make products and we put it out on the market and we make money. We screw around with coming back to Washington and try and figure out how the Pentagon works. Now, that's really alarming when you see what the great advantage America has in the world is we have the most innovative people on Earth. I was out, by the way, the other day at a place called Snapchak. Have you ever heard of Snapchak? There was not a person in that room over 25 years old, including the kid that was running it, which is only worth $4 billion or something like that. You know, it's staggering. And to think that that young man would want to come and do business in the Pentagon, I mean, that would be crazy. So we've got to fix the entire acquisition system and we have failed miserably. Do you see any quick fixes for it? Well, one of the things we need to do, among other things, and I don't want to get into weeds, but we've got to make the service chiefs more responsible, put them into the acquisition process. We have to do that. At a hearing a year ago or so, the chief of naval operations was one of the witnesses. And I said, look, the Gerald R. Ford was supposed to cost $10 billion. It now costs $12.4 billion. I said, do you know who's responsible? Do you know what his answer was? You know what his answer was. No. Somebody's got to start to be held responsible for God's sake. What could my state of Arizona do with $2.4 billion? At Ash Carter's confirmation hearing, I put up a list on the wall, not on the wall, a list at the hearing of $40 billion in programs that never got started, that never turned out a product. $40 billion. I mean, it's staggering when you think about it.  And then, of course, this is thrown back at me. And I don't have as good an answer as I want to have. So if I would argue my priorities, repeal sequestration, but fix acquisition, fix it. And it's bound to be able to. Do you know that there's never been an audit of the Pentagon? Do you know that? There's never been an audit. Do you know that the Pentagon can't tell us how many people are in their employ? Plus the contractors. Yesterday, Missy Cummings, who you may know, like yourself, a former Navy fighter pilot, and now Professor Rick Duke. Far better than I was, yes. And she knows a hell of a lot about drones. A very impressive talk. One of the things she said about drones is she's seen that the US Defense or District is not particularly interested because the profit margin is so small. They want to build big, expensive programs, not thousands of little drones. Is there anything you can do about that? Well, first of all, I think there's going to be more money in it because we may have seen the last manned fighter aircraft in the F-35. We will have manned aircraft for logistics for a whole lot of reasons. But I think we're going to see a technology that we use drones in. It's already been proven. When a plane can loiter for 12 hours, a drone can, and fire on a target, what do you need a pilot for? Now, we're going to need pilots, but certainly not for the role that the traditional fighter and attack pilot did. And so I think there's going to be a lot of money there. I was disappointed in the ideas that the Navy initially had on what kind of drones they want, but I think we'll try and fix that. By the way, a drone landed on an aircraft carrier and took off for the first time in history not that long ago. That's huge. It's almost like the Kitty Hawk. Are you going to encourage the Navy to go for a UCAS? Absolutely. Absolutely. And do you think they're pursuing it fast enough? I don't even think they have yet decided on really what they want in drones to tell you the truth, much less whether they're going in the right direction. And so what we really need to do is have much more congressional scrutiny, but force them through the Defense Authorization Bill to come up with certain results and models and certainly definition of missions. One of the problems right now is that they're not really clear on all the kind of missions they want drones to be. Today, my friends in Ukraine, the Russians are flying little drones.  This is already, you can see how it's changed the way we fight wars already dramatically, the drones are playing that incredible role. Is there anybody else in Congress interested in defense issues besides you? Lindsey Graham, of course, is. Oh, that's your mini me. There you go. My illegitimate son, Lindsey Graham, we've got some new members of the committee that I'm very excited about. Joni Irons' guard, Tom Cotton, Iraq and Afghanistan. Dan Sullivan, Marine Reserve is who's had command. We're starting to get some new members who have had recent Iraq and Afghanistan experience. And that's invaluable, especially on the personnel issues. They know what it's like. They know what it's like. And so I am encouraged by this new crop. And there has been a number of new ones in the House of Representatives as well. You don't have to be good at the military in order to have had military experience. Don't get me wrong. Sam Nunn was an example of a person who never served in the military, but was an outstanding chairman. I think you would agree. But it really helps to have some people on the committee and around you that say, wait a minute, you're asking now, by the way, you're asking now, aircraft carriers deploy for 10 months. That's too long. That's too long to send young men and women away from their families. They should not have to deploy for 10 months.  I won't touch that one. I want to turn to questions from the audience. But Peter, I just have one request. The clock here has gone screwy. I have no idea if it's correct or not. So I don't know how much time we have left. Could I just mention one thing while we're fixing that? One thing that I've tried to do, when Republicans took control of the United States Senate for the first time in, I don't know how many years, in the 1980 election was the Reagan sweep, John Tower became the chairman of the Center Armed Services Committee. And what he did was, working with Scoop Jackson in a bipartisan fashion, they had a series of hearings getting the best minds in America before the committee. That's been my role model. We've had Kissinger and Schultz and Albright and Scowcroft and Brzezinski and... Anybody under 80? Well, I'm partial. Anyway, you should have had some of the people from the panels here the last couple of days. It's been very dynamic. And by the way, I think more than half women. Is that correct? And we have, also by the way, our staff, we have a very... We have the largest number of women on the Center Armed Services Committee staff that's ever in history. But so because I've wanted these new senators and there are a lot, by the way, do you know that over 60 United States senators have been there six years or less? Of the 60s, Democrat senators that voted for Obamacare, 30 are left. So there's been a really significant... Those who are for term limits here should be happy. So we really have to try to focus. We'll never succeed, but we want a policy, as much as possible, a policy-driven budget rather than just take the Pentagon's numbers and see how we can cram it into different areas. So I'm really trying to work as hard as I can on that aspect of it. And I'll tell you, the members have all been really happy to hear from these people. Yes, they're old, but there's hundreds of years of experience. These are the people that won the Cold War. These are the people that may have let us, America's finest hour in many respects. So I've been doing that and I've been very happy with the results. Now comes the hard part.  I will not vote for a budget in the United States Senate that has sequestration in it. I can't do that to the men and women who are serving. How many people are with you in the Senate on that? I don't know. I do not know. I've never been very good at counting votes, but I do know that there is real unease out there about continuing the sequestration. And there's all these meetings that go on. I've been to too many of them. Well, if we cut here and if we do away with this loophole or close that, one person's loophole is another man's. You've seen it all time after time. Okay, we're gonna go to questions. People in the audience, hold up your hands. It's a little bit hard to see you back there in the way back, okay? Yes, please. No, the guy, I think there's a hand behind you there. Yes. Thank you. Senator Brian Mangan, I'm a retired Marine. The reason I wanted to bring this question up is about a year ago, I asked the former command of the Marine Corps, General Amos, if there was a way out of sequestration. And his answer was, we've got to cut entitlements. My question is, how do we continue to ensure that the service members who are serving in the military of all branches are able to continue to serve and not be subject to pay cuts when we have this acquisition process that, as you very well stated, is considerably broken? And how do we make sure that we balance those two? Thanks, sir. First of all, we have to obviously understand the fundamental and that is we're having all volunteer force. Second, we just had in the last couple of weeks, in fact, I just left a hearing the Personnel Subcommittee, the Armed Services Committee, where this panel of very well-respected experts made recommendations on pay, benefits, retirement, et cetera. If we're going to do that, we have to show, one, that any changes should be prospective and not apply to men and women who are already serving. But I have to tell you that healthcare and benefits are now up to 20% of the defense budget. In the words of former Secretary of Defense, Robert Gates, who was one of my really great, I think great secretaries we've had, these, quote, entitlements and benefits are, quote, eating us alive. So we're going to have to make some changes. They have to be prospective. But if we don't look out within, well, supposedly 10 years from now we're gonna go from 20 to 30% of the federal budget is going to be to entitlements and benefits. We just can't keep that up. And so this committee, or this commission that just reported out to Congress has some pretty good and innovative ideas. And one of them, by the way, is that every service member shouldn't have to wait to 20 years before they get retirement pay. That's a thought that we could keep people for 10, 12, 14, 16 years. And because right now 85% of the men and women who join the military do not get any retirement benefits. So that's kind of an idea I think we ought to kick around. But there's gonna have to be reforms to be made. I've asked Lindsey Graham to chair that subcommittee. He is a colonel in the Air Force Reserve, understands these issues as well or better than any member of the United States Senate. And he's often given to suicide missions as well. So I've asked him to do that. But we're gonna have to address this issue. And could I just mention one other thing? Maybe it's a little sarcastic. But I've had so many people come up to me and said, listen, when I joined the military, I was told I'd have tri-care for life. That's the healthcare. Really? Is that why somebody at 18 joins the military because of tri-care for life? Give me a break. It might be why they re-enlist, though. Yeah, good point. But I think every 18-year-old I've ever known, including this one, I joined because I wanted to get into the fight. That's so, anyway. Would you support a tax increase to help preserve the all-volunteer force? I think it would have to be what kind of tax increase it is. And what a lot of us hope we could do, which we haven't done since 1986, is some reforms of the tax code, which over time would give us more in the way of compensation. But right now, I think it would have to be a part of tax reform rather than just say we're gonna increase taxes. It's cowardly that we haven't done tax reform. We had a question on the way back there, right under the spotlight. Peter Singer with New America. Thanks for joining us, Senator. To kick off this forum, we did a survey of our network of experts that range from technologists to lawyers to Navy SEALs, Air Force officers. And we asked them, what do we get most wrong about the future of war? And so I thought I would pose this question to a Navy officer turned senator. I think what we got wrong most of the conflicts we've been in is that you can win militarily but if you don't have the followup, you are going to lose every single thing that you've gained through force of arms. Great example of that is Libya. After we got rid of Qaddafi, and it was the right thing to do, by the way, because he was in the gates of Benghazi and where he's going to kill off several thousand people, Lindsey Graham and Joe Liebham and I wrote a piece in the Wall Street Journal and said, you got to do this, this, this, and this. Ranging from civil society to securing the weapons caches to helping them with border enforcement, treating their wounded, they had 30,000 wounded in no way really of treating them and build a society that had never known it. Instead, my friends, anyone will tell you, we walked away, we walked away. The same thing with Iraq after the surge succeeded and we're gonna see the same movie in Afghanistan unless we, and I'm happy to see that they're talking about now changing that calendar-based withdrawal more to conditions-based withdrawal. So I would say the first lesson is you can succeed militarily but you better have some kind of a follow-up. I was with the Emir of Qatar this morning, very interesting young man. And the question I had to him was, Bashar Assad is as big an enemy to the Arab countries as ISIS is. And I said, well, what happens afterwards? What happens afterwards in Syria? How do we make any kind of order out of chaos there? And he said, his answer was, well, we're gonna have to use some of the previous regime's people. In your book, Chaos, you talked about one of the biggest mistakes that Bremmer made was de-certifying the entire Iraqi military. That was one of the fundamental errors we made. So I guess if we're gonna go into a conflict, obviously we're going to win. But the most important lesson is is after you succeed, you have got to stay and you've got to do the grunge work, the real hard work of building a society, particularly in countries that really never knew these things. And so I think that's the biggest lesson that I have learned from all of these conflicts that we have been in. And I just finally like to add, there is many of commentators that say, well, Americans are tired of war and they're not gonna be involved. Let the fill-in-the-blank do it. Iraqis, Syria, fill-in-the-blank Afghans, whatever the conflict that's going on, fine. But I am convinced that Mr. Baghdadi wasn't kiddin' when he left our camp Buka after spending four years there when he said, I'll see you in New York. They are intent on attacking us. So if I thought we could withdraw from these places and everything as terrible as it is, the human tragedies, today we find out that they captured about 100, 150 Christians and you know what they're gonna do with them, ISIS did. But setting that aside, we have to be engaged. But once we succeed and we can succeed, we had better have plans as to how to, when the shooting stops, as to how to build a society and many times that's a place where they have never known anything like what a government is. What approach would you like to see taken to defeat ISIL? We have to have boots on the ground. We have to have forward air controllers, special forces, intel, trainers. Right now, as you know, they're only at the battalion level, they've gotta be on the ground. We've got to develop some kind of strategy for Syria. We have to rebuild the Iraqi army and believe me, that is, my friends in the military here will tell you, when an army is beaten, it's a long time to reconstitute that army. And anybody who says that the Iraqi military alone can retake Mosul obviously hasn't been talking to the same people that I have. And understand one of the truisms about warfare is air power is a vital factor, but it's not the decisive factor. Air power alone does not win conflicts. And so we're gonna have to have so many things, Tom. Why is the authorization for the use of military force ISIL only when Bashar Assad has butchered 200 and some thousand of his people? Is it because, as some people are saying, well we don't wanna alienate the Iranians who are gonna be our new partners, who are on the march everywhere in the Middle East, they're in Iraq, Lebanon, Syria, Yemen, our success story Yemen, the JV is there. So we have to develop an overall strategy and those elements have to have not the 82nd Airborne, not massive American troops, but significant number of American troops to do the things that these people are incapable of doing. And I know of no living American who can tell me what the strategy is to succeed not only in Iraq but in Syria. We're almost out of time. I'm gonna ask you one last question. Unless there are any two handers, my friend Elliot Cohen says two handers, which is if you don't get to ask the question, you're gonna go home muttering to your spouse. Okay. Books.  He's a closet intellectual. He reads a lot of military history. I remember before one hearing it said an armed services hearing. We were talking about the Boer War and the book you were reading on it. Is what one book you've read lately that has really illuminated your view of military affairs? I like a lot history as you mentioned. I read a wonderful biography of Napoleon that's new. If you haven't read it, it is really worth a read. It'll give you an appreciation for the intellect and the capacity of this man that most of us think about Waterloo. It's amazing what he did. Also, this recent piece in The Atlantic was an in-depth study. The one by James Fallows? Absolutely. And you know, I just read an old book called There's Not to Reason Why about the Charge of the Light Brigade in Crimea in War.  You can't make it up how bad the British bungled the brave 600 into the valley of death. And I mean, it is a remarkable, it's an old book and I'll tell you who gave it to me was Sheldon Whitehouse who was also a student of history and a remarkable one as well. So that's about what I read Kagan's book, of course. I had to, so otherwise he wouldn't speak to me. And so I thought it was excellent. There's a lot of good thinking now about this challenge and the way things are going. And so I do read avidly. And I kind of just make it one plug very quickly and that is Ukraine. My friends, they're being slaughtered. Vladimir Putin is on the march. Mary Opal is next. Lindsay and I have predicted every move that Vladimir Putin has made because it was so predictable. But to not give these people defensive weapons is really, I think, a shameful chapter in our history. They want to defend themselves. And the rationale for not doing so well would provoke Vladimir Putin or they can't beat the Russians. My God, that is ridiculous to say something like that. The French shouldn't have come and helped us because nobody could beat the British, as you know. And we should never have helped the Afghans when Russia invaded because they couldn't beat Russia. We really need to help these people. And I'm worried about two things. One is the dismemberment of the country and the most important part of it. And the three things. Two, the collapse of their economy. And three, a reaction on the part of the Ukrainian people against their elected government, which seems to them to be failing. And that can lead to a course of events that I think could be very damaging in the long run. I met with the Lithuanian foreign minister this morning. My friends, the Baltics, are feeling enormous pressure, not to mention Moldova. And I'm not sure where Vladimir Putin stops, but if you look at it from Vladimir Putin's viewpoint, I think he's done pretty well at a relatively low cost. And I'm not sure that he has a grand plan, but I do believe he is ambitious to restore the Russian Empire. And I do believe that he's a great opportunist. If he can see taking Mary Opal and restoring the land bridge all the way to Crimea, if he can go across to Moldova and not have a cost, that's fine with him too. But already the pressure he's putting on the Baltics is really intense. And one of the ways of that is Russian television. Russian television is really, they're doing a great job. We're gonna have to catch up with things like Radio Free Europe and Voice of America. And that's one of the things that Corker and Menendez and I and some others are working on. I'm very worried about how far Vladimir Putin may go in his adventurism. And there's one school of thought that says his economy will sooner or later put a break on his activities. There's another school of thought that history shows when dictators have problems, domestically they strike out more. So these are very difficult times, my dear friends. And that's why I'm glad you're here and I'm glad you care. Well, we came to talk about the future of war and we wound up talking about Napoleon and Putin, which actually may have aspects of the future of war in it. Thank you very much. Please join me in thanking Senator McKay. Thank you, Senator. Well done. Thank you. So Harold Koh yesterday talked about the authorization of the use of military force and he said supersede sunset and silence. There is nothing that I can say that would supersede the richness of the last two days of conversation and really thought provoking ideas and critiques. So I'm gonna sunset this conference, but before the silence, I do want to thank the future of war team at New America. In fact, all of the New America team, our events team, our production team, all the people you've seen who've made this conference work and in particular our co-chairs, Peter Bergen and Dan Rothenberg and Bailey Cahall and David Sturman who really put all this together. So join me in a final round of applause and thank you.


This research suggests that the combination of carboplatin and tryptolide may be effective in treating melanoma. Tryptolide was found to inhibit the N-R pathway, which is important for DNA repair, and when combined with carboplatin it had a synergistic effect on cell viability, migration, invasion, and apoptosis. Additionally, this combination therapy was shown to reduce tumor growth in nude mice. This article was authored by Gungwang, Hongmingwa, Yenrun, and others.


Everyone thinks I'm not autistic because I don't get over-stimulated and I was wondering if there was a way to force it Is this a joke? Like I've seen some stuff around people saying like was it like Transabled wanting to transition into someone who is disabled or something, but is trans autism a thing? Say it seems seems to be pretty bizarre like What exactly is that trans autistic is valid? Sounds a bit strange to me. I don't particularly understand this idea of trans autism We might have to look it up actually Let's get get the old the old phone out.  There's an informational card Which seems to provide a better understanding of trans autism a trans autistic individual It refers to someone who want who wants to be or acquire a form of autism It cannot also be described someone who feels they should acquire autism Trans autistic people are not necessarily Oralistics they can be autistic individuals who want another form of autism is not how it works You can't you can't make yourself autistic. You were born autistic. I think like I Imagine at this post It kind of it kind of looks like some form of like a meme account Like someone that's making like a joke or making fun You have a lot of this stuff kind of on like reddit and Twitter because a lot of controversial Posts tend to be pushed like more into popularity. I don't actually know if this person is like Actually like wanting to become autistic when they're not Autistic or not. I do want to clarify that. I'm not saying that people who aren't diagnosed autistic shouldn't Like get a diagnosis and find out if there are autistic, but you can't like make yourself autistic You either are or you want they think you're born autistic. No, I think that the if someone is taking like the trans autism thing like very seriously Like the the the kind of the wording on that website that we came across said You know, they want to acquire autism Doesn't make any sense. I think they are just making fun of inclusivity in the disability activism queer feminist space Would be my guess the others might my guess as well as I'm I think that's a good guess and then this person is obviously like Taking it seriously


Welcome to our talk. Where's your money going? The beginner's got to monitoring measuring Kubernetes costs My name is Mark Poco. I'm a senior software engineer at Grafana Labs And what I'm I'm I'm quack also Grafana labs working both work out the platform team. Yes So before we start let's do a little bit of a story, right? Imagine a scenario you get your cloud bill Kubernetes cost doubles month over month and you're trying to figure out where did your work? Where did it come from? So you start with your bill and you look at it and no matter what you do It doesn't really help you figure out where that cost is coming from so the next thing you do you go to your cost explorer and see what other dimensions are available for you and Really don't come up with much and in this case. We're showing on What's it called the incidence types? You might be able to see where that cost is coming from but nothing about the cluster or the workload So you turn to your favorite The individualization tool and you look at things like CPU usage and memory utilization Look at this over time and still nothing is really standing out to you on what is driving your costs and this leaves you sad so The show of hands how many people have had this happen before all right. That's yeah everybody How many people are running kubestate metrics in your Kubernetes cluster? All right, so the good news is for those that are doing this You're gonna be able to walk away today with some prompt all queries that you could run and be able to visualize this and those that don't Hopefully this is a good incentive to maybe look into kubestate metrics. So what can you expect today? First thing we're gonna do is that there's a we're gonna try to show a couple approaches that we use like Rafauna labs to help Bridge the disconnect between your billing statement and the metrics that you're already collecting their Kubernetes cluster After that, we're gonna step through a couple of prom QL examples to help measure And in this particular case, we're gonna show CPU because that's usually the most costly And then finally we're gonna share a couple of lessons learned both in terms of setting this up and measuring it and also How we helped improve that cost and with that we turn it over to Juanjo So, um, yeah to start digging in into this we need to understand the nature of our spendings This is a really very simple formula as you can see Spending the amount of money that you pay for your resources on a particular period of time Usage being the amount of units the number of units of such resources and the rate is how much we're being charged for for each of these units This is important to to to group to get to group separate actually to be able to focus on the force as you put on each part of these factors Like in usage, we can see the things that for example amount of CPU cores amount of memory Traffic and such and it rate our dollars per each unit of these of these resources from the cloud providers and this then drives Different really separates how we need to focus or what are what are the work streams that you need to develop to be able to tackle this In the case of usage We need to be aware of our workload resizing This is the most common intuitive one when you think about improving your cost sense Usually think oh, I need to properly do a right right sizing of my resources That is how much I ask how much I use but there are also several other factors that are pretty important in driving your cost other one is auto scaling having proper auto scaling story both On Kubernetes like things like HPA and VPA, but also underlying how how much your Cluster can be elastic to actually create new nodes or destroy nodes that you don't need anymore And then another aspect that is highlighted there because is the aspect that will be actually touching doing the rest of the talk Is cluster been packing cluster been packing mean mean that you have a node How well you're able to actually pack all the posts in this node to have the delete the least slack resources in that node And as you can see then there is a different aspect like rate all together that is essentially You need to understand What is the kind of discounts that you can get from your cloud provider on aspects like committing for Certain amount of resources that you will be using over time But also things like for example the circuit CPU architectures that you're using is it in the AMD arm Are your workflows ready to actually run on this on these architectures and then things like for example spot VMs What in Ritchie? He did a really good approach to the to these spot VMs are kind of free Chaos engineering because these are kind of VMs that can disappear anytime you have it there They have a steep discount, but they can go without the notice. It may be just a minute now this so important then to focus here is How we separate this because also it impacts which teams will be in charge of each of these items that I'm highlighting there So let's let's try them to to focus the rest of attack the talk on cluster been packing What we see there essentially is trying to by the way, this is a Graphing Grafana as you can see this is a result of some query there Well, we see there over time is suspending like for example if we focus at 5 p.m During that hour from 5 p.m. To 6 p.m. We will spend in something like 1.4 Dollars in total the top box is CPU the bottom box is memory And this also is hinting us that we will narrow the rest of the talk to compute resources only because in 25 minutes It's what what what we can get so Again, this is a spend so this is dollar time dollar over time If we apply the formula here We see that actually this is spent is composed of this crit unit of resources in this case We are trying for example again that's 5 p.m. We're trying to graph something like four nodes For example, you have there the four nodes each node may have eight or 18 B.C.P And then on the top you have CPU and on the bottom memory, and you can see that the number is the same But again, it's important to understand that it's not continuous what you request especially when When you're running a Kubernetes cluster on top of cloud This will be discreet indeed the nodes that being created or destroyed as for example to the right of the graph This will be discreet amounts and then a different dimension is how much are you paying for this as we just saw At the end of the day what you will be paying for will be the volume that you're seeing there Essentially the integral let's say the sum of each of these bars over time over a period of time Let's see then Let's try to focus then because this is kind of a pretty low-level view, right? We are seeing there the nose we're trying to graph there the nose that we are driving from our cloud provider Let's see we need to understand then how Kubernetes workflows drive the creation of these units And this is key to understand how we're going to cope on with the cost on each of these aspects Then let's see what drives when it is cost Here we try to graph, you know a Poth there is just a Simple work service that is asking for one CPU and to be ground It's not you're running It's kind of pending until it can land into a node that you have It's asking one CPU to be gram and it happened to land it in a node. There is a CPU and 16 You've run you can start to sense that there's something not so good regarding this picture Until you actually put a replicas and you have this kind of unicorn awesome theoretical scenario where where you are Right, we are exactly fool feeling the capacity of the node Obviously, this is pretty theoretical and indeed mark we will address this when we show the how we tackle this We have another poll there and we have essentially the same scenario before Let's that's thinking we have Essentially if you if you read through the through this graph You will see that you have CPU at the node level, but CPU at The pot level or so usually when we use covenettes We actually only see the up part like if you sum all the resources that you are Your both are using you will be seeing the the aggregation of all these small boxes inside But we need to be able to actually separate both to measure both to be able to to to see The driver and the effect of that driver being what is inside and the fact is the actual nodes that being allocated So then we need to indeed again. We are in this talk. We are just focusing on compute So CPU memory of nodes and likewise for workloads As Mark mentioned If you have cube state metrics, you will have these metrics provided by you By the way, this is not meant to be a prom queue a class of course But just a quick introduction what you have there are like cube nodes status capacity It's indeed the name of the metric and this metric can have as you can see have several labels These labels can have several values the cardinality of this and then the the combination of the metric name And the label value will give you a single will give you a single point in time of data and this is called time series So just put to put an example you fix the cluster because you're taking care of you're actually watching this particular cluster You fix resource to be equal CPU for example You fix the node because you're watching this node and then you will get the number like 8 for example for that metric about for That time series about 8 because it has 8 b CPU Now we again this we are watching at the box that is hosting this box if we got a bot level You will see that we have a metric that is Q pod container resource request so it will have we are we are not actually putting there the label like pod and and Containers us for the sake of of space in the screen, but essentially we Essentially the same idea. You see that we have cluster resource node and namespace And then mark we will help us understand how we can bring some What is true? Thank you. Want to so all right again Bringing this all back is we're going to focus on a usage side because that's what drives your cost in your clusters We have two metrics now that we could do some work with and so let's step through we're going to three examples Of how to measure compute in your cluster the first one is how do we measure the cost of our nodes? So in pseudo code and promql We have this again. We're expressing that formula sum is equal to usage times rate and To measure your nodes you need to update your usage to get how much resources you requested So in this example, we have cool node status capacity and we're saying we want the resource of CPU and what's charted on the right is going to be how many CPUs for an entire cluster you have What's called how many CPUs are being requested to summed up? As the graph goes down that's kind of showing that there's fewer nodes as that graph goes up. It means that there's more nodes and Now that we know the number of cores of a cluster we could then figure out. What is it cost to run that cluster? We're going to cheat a little bit today for rate What we're going to do is we're going to go to a cloud provider, right? And there's three things that you really care about for cost of a CPU It's the region that you're in the instance type and whether or not it's on-demand machine or if it's a spot machine So in this example, what we're going to do is we're going to find out We're going to use the on-demand price because that's most likely you're running an on-demand machine so we're going to take the three cents per per hour for vcpu, and we're going to plug that into rate and charted what that looks like is over time how much we're spending per Per minute and we're going to come back to a minute and they their cloud provider charges you per hour But we're going to cast it to a minute because in this this query you could explore your data Right, this is something you could whatever tool you're using you could explore it And you can charge it over time But most likely if you have a lot of nodes a lot of clusters this query You're probably not going to go further than a week maybe a month if you're in a smaller cluster So we cast it to a minute and what we're going to do is we're going to create something called a recording rule in Prometheus And if you haven't used Recording rules before it's pretty much it's a job that runs periodically It's defined in the apple and what you do is you specify the the Metric name that you want to store this data into you give it the expression in this case It's that the example who had some by cluster and You attached to it a label in this case. We're saying resources CPU and so when this runs in Prometheus What's going to happen every minute this runs? It's going to take it's going to Calculate the cost per minute and it's going to store it in the record cluster cost per minute sum and With a resource label of CPU and this allows you eventually you could do memory you could do persistent volumes You could do object storage Realistically resource could be many different factors. So Once you have this running use it stores it over time. So that's how you measure nodes Most likely your nose isn't really, you know, you probably could get a sense of what you're spending you probably really care about What is actually costing what in this case? We're showing a little bit more of a realistic example where we're running pretty much the Prometheus community and One of the things that we do at least internal with the Grafana is we use name spaces to isolate our workloads So in this case, we have a Prometheus instance that running in likely high-value Bellamy mode It's bigger because it's most likely consuming a lot of memory and then we have a bunch of other little workloads So the question is how do you measure this? Like how do you measure that namespace because this is probably the most important aspect for your engineering teams? so Coming back. We're gonna alter that Equation just a little bit in this case. We're gonna do some by namespace and instead of the usage We're gonna look at requests, right? What is the workloads running in that namespace requesting? So From kubestate metrics, you have this metric kubepod container resource requests and again We're doing CPU because that's most likely the most expensive and If you look at the graph one of the things you might notice again, is that it's going up and down over time So nodes when you're measuring nodes, that's just gonna be new nodes added new nodes delete or new Nodes deleted in this case. It's most likely replicas You likely are using some type of horizontal pot auto scaling policy and so is the Number of cores goes down you likely have less replicas as it goes up you likely have more Once the newer version of Kubernetes comes out and you could have dynamic resource requests This is probably gonna change a little bit So again, we're gonna cheat here and for rate. We're gonna use that same threes about three cents Per hour, we're gonna convert it to a minute and This is the cost per namespace tried it over time and and this is a made-up example where we generated some fake data we have namespace a and namespace bay and You know, we could look to see how much is a cost per minute over time. So Similarly, this is useful. You could run this today and you could get some actual data On the cost of your namespace, but if you want to go over time, it's most likely not gonna work So I'm gonna fall back on a recording rule in this case We're gonna call it cluster underscore namespace cost per minute some and for the expression We're gonna use what we had before in our Explorer and we're gonna again add the resource Label of CPU. So again, if you want to do memory instead of it being the resource CPU you change that to memory find hourly price of memory do a little bit of conversion because Bites and gigabytes that they don't always match up. So This is two examples We have one more that I think is really important that lines up with almost every keynote speak about energy efficiency and so Both those exam almost all three examples are extremely unlikely to happen in in in in a production cluster most likely what you're gonna have is you're gonna have a lot of nodes over time and each of those nodes is gonna have a different level of What's it called? Workloads running on it and What we really care about from a platform perspective is figuring out how much of that extra space on that node Are we not using and the reason for that is that's I mean that you're paying for that entire node and If it's not being used you're wasting money, right and you're wasting energy and It's yeah, it's something important for us to track. So how do you measure that this one's probably the most complex? But you have to kind of follow me Instead of caring about the usage or the number of requests for Node or workload you care about you want to know how much capacity is on that node Minus how much your workloads are requesting, right? That's gonna be what drives or idle usage That's what's left over and then times it by the rate and that gives you how much you're spending on just idle resources. So It's a lot if you're not familiar promql But let's step through it right you have on the inside you have a sum by node of the capacity of the node minus sum by node of the Requests on that node and again, we're looking at CPU and what's charted here on the right is how much we're spending per minute on Resources that are not being used by anything. So and we convert it to 60 seconds because again You could do maybe a week of this 30 days six months is not gonna not gonna cut it. So Take a pause it's three promql queries that we've looked at if you're not familiar it could be quite a bit I would love to be able to spend more time and step through memory persistent volumes You're gonna just have to bear with me. Oh God ahead of myself Let's look at that in recording rule because this is pretty interesting with what we do We have cluster namespace cost per minute some and we have this query here right the same one we had before Notice though in a query. We don't sum up by namespace anywhere And there's a reason for that is idle resource isn't associated to a namespace. It's associated to a node We add an extra label called namespace under double underscore idle and we call that a virtual namespace Pretty much what we do is we associate all of the date all the spend that we're not Using anything on into a single namespace and we could do some pretty cool stuff with that. So Here's where I Showed a couple of things here Unfortunately can't step through the entire process but I do want to show you what we do internally in Grafana and the way that we measure this so If you put this all together right you draw the rest of that owl you have this pretty complex looking query We're what we're what we're what we're graphing here is how much what's the percentage of All of our spend that's being associated with idle resources and this is important for many reasons, right? This is money that we're spending To our cloud provider that isn't being used at all. So we have targets and goals with an organization of what we want to do with that and What's really cool about this is that every month or every week we get a report on how much we spend with an Organization by team and also as a platform we care about the idle resources So you might notice that there is a point where it kind of shoots up, right about mid-July July 14th We noticed that and We were able to react and we're able to get that cost under control so that we didn't have a surprise bill And so this is six months. We tracked this we could do it over a year 18 months and Yeah, so With everything right this has been an iterative process. It's taken a while I've been doing this, you know, we've been doing this for about a year just trying to get to the point where we're at The first main Like shortcoming of this approach at least with what we shared with you is that it only it only works for homogeneous clusters Most likely you're gonna have clusters with many different node types and in our case We run across many cloud providers. So that's that's a hard problem to solve So what we first did that worked very well was we averaged how much our CPU cost Across all of our cloud providers and instead of using that like three cents an hour. We use that that the Average for a month and use that in all of our recording rules. We did that for about six months And then from there we used open cost we deployed that to all of our clusters and instead of using that estimated number we're able to join a query and we're able to Get the actual cost of the node and it takes an account the region it takes an account if it's spot the type of instance and the interesting thing for us is that when we converted from the estimated to the actual numbers our Estimated was about 90% correct right and again, that's across three cloud providers if I heard that I wouldn't believe it That's it was really I couldn't believe when I saw the numbers But we wrote about this in our blog post and we adopted open cost So go check that out if you don't believe that The other thing right now is this only takes no account compute resources. So we could do CP. We do memory We do persistent volume Most likely at least within Grafano We use a lot of object storage which has a very different type of compute and we want to be able to associate to our teams And let them understand how much you're spending on that resource as well. So it's something that we've built internally and We're not quite sure what we're gonna do, but there will be something from that Now you might have noticed all my examples I used AWS for billing You know if you've used that before but I pulled the numbers from GCP Not all of your cloud providers are gonna give you an actual breakdown of CPU of memory So if you're in if your cloud provider doesn't give that to you, it's about 80 percent If they just give you the total cost of an instance, it's about 80 percent for CPU 20 percent memory So you could do some math on that to figure that one out and then finally Namespaces doesn't match up the teams. So from a platform perspective, this is really powerful for us to figure out how to resource But our engineers they they need to know like not just the namespace, but how much are they spending? So we have this kind of like magical Little metric that is part of our CI process That associates the namespace to teams so that when we build these queries After the recording rules We're able to join up namespace the team and we could then provide all of our teams a breakdown of their costs Yeah, just to add a bit on top of what Mark said Attribution here right being able to attribute so that teams actual engineering teams can take care of the costs that they own And imagine that underscore idle to which team it goes it goes to platform team So Interesting here also they're defining the ownership doing these these being able for teams to actually own Their own cost so they can actually do the what is called TCO for total cost of ownership force by the teams With this lesson element, let's see what lessons we learn on the our been packing journey Oops, maybe none or maybe we need to ask ourselves first a question Do we have a resource request for our workloads resource CPU? Memory yes ish at least the more relevant once few hands up, okay You cannot embark in this journey if you don't tackle this properly because the resource that you request for your pause Is what drives the scheduler to actually land those posts in a particular node that has that capacity available? And then the view that the scheduler has regarding the proper been packing the scalar together with the cluster of the scalar Needs to know how much you are requesting for the poll again We're talking about be packing not my sizing that is we're not talking here in this talk about how well you are using The resources that you request we are talking about how well we are being packing what you request for on the nodes that you have available so Here essentially the idea of this is to share the to share you some tools some approaches that Has helped us in GKE by default the cluster run with something that is called balance mode Balance mode. Let's say it's right to be gentle with the post trying to do a fair distribution of the post over the nodes As you have available if you switch that and this is a cluster wise setting to optimize utilization This is the opposite. It will try to actually do this been packing But again, if you don't have those requests is fine and instead of been packing you will have pot smashing, right? You need to be able to Tell the scheduler how much you're requesting for very useful for that for for us. I will give you some some numbers in a bit For the kind of patterns that That we have in Grafana we have essentially continuous deployment So we don't have a day where we deploy we have so many teams working where so essentially we're Constantly deploying rolling out new versions or or new settings for for the services that we run and this creates a lot of Fragmentation at the nose. So let's say that you have a point in time where you have an awesome been packing picture like 20 something percent as Mark was showing but then you do a rollout and then essentially you fragment all this know with all these gaps of resources So by allocating new nodes where the new posts are landing and what we found in a because of the way that we use the platform Is that in Giki at this is what this was taking too long? After all a rollout it may take two days to actually settle back to that idle percentage Call out and gay again to be able to measure that because as you're running blind All what what just mark show and this is a permitted measure So we are actually tracking that idle so we will able to react on on that and say okay This is taking too long a tool that helped us that is certainly not a silver bullet And you need to understand how the world so has many knobs is that the schedule or this scheduler will try indeed to Observe how well you are using those nodes has several strategies one of them is high node utilization And then it is fit observes that the node is being underutilized. It will evict the post to be able to kill the node But of course you need to understand the schedule Again, it's not a silver bullet you if we have some in this some Incidents because of this we have to really took this properly. This is which is on Giki. We are running on three main cloud providers AWS We replaced the cluster the autoscaler a Call out an important call out here in Google the scheduler and the cluster of the scalar are in the quarter plane So you in Giki you don't control it in AWS There's this difference the scheduler is in the control play, but you need to run your own cluster of the scalar So this is replaceable carpet. It is an open source project That actually it tends to replace the stock cluster of the scalar and it has so many knobs things like you can configure for example the Especially diversity of the underlying nodes to better match the shape of the post by shape I mean the ratio of memory divided by CPU much the shape of the post with the shape of the nose and have a better been packing story and We are really lucky Today like yes today. We published a blog post was by a couple teammates that were actually working in carpet Logan and Paula With which it has which has a pretty well detailed and some really interesting views on again on this on how car pattern Have helped helped us especially on the on our been packing story and then finally a call out for this white paper by Google status Kubernetes cost optimization and With that I think that we're not. Yeah. Thank you. Thank you for being here Thanks Sorry, fine. I forgot something numbers. I had promised some numbers We went for something like without touching anything on our cluster. We were hovering 40 45 percent of either resources Considered that you have 50 percent of either resources You are essentially paying for the double that you are actually using right your your nose are using half so we went for something like 45 ish to 22 25 percent of idle usage Questions question Too late, please use a microphone. I notice you were using requests for everything. Does that not Considered limits by itself. Is that the container request and then the limits are on top of that? So the main thing that the scheduler cares about is what you request, right? So if you have upper limits Like it could go up to that and that's fine But the main thing that we care about is tracking what our resources are requesting We've we've done some analysis comparing the two right looking at the maximum value between either the the limit the usage that they've had versus their quest and at least in our organization it's Really has been negligible. So we focus on requests at least you had to keep it simple So as we mentioned in this particular talk we're actually focusing on been packing so understanding how the scheduler is able to Be impact the post we are not focusing on resizing that would be a Completely different approach you have different metrics to try to see how well you are using the amount of Requests that the request that you are requesting. So but that's why we wanted to highlight the start of the talk when we talk about about Usage there is a different aspect of usage. This is how how much of your if I understand correctly your question would be How well you're using how the resources that you're requested at the top level The other question is about CPUs versus memory costs I'm not sure that's accurate. It's like let's say you reserve a note on a yes, babe It comes in for CPU and let's say 16 gives them you can't really break down that into Like separate cost.  So Yeah, and that's a great question. So I The way that we we approach this internally is we're not trying to get to accounting level accuracy to have it line up Exactly with our bill. We're just trying to have a rough approximation of each workload What it's requesting what those units of cost are and we sum that up over time and for us We're really looking for trends, right? We want to see over time for our workloads what are they requested and Calculate the cost of those requests Thanks. Thank you. Great talk Question is there a place that you publish it in dashboard that has this like different the queries Let so like open source or like internally Not yet. That's not that we're trying to figure out. So we we use Like json it internally to publish a lot of our dashboards a lot of our queries and whatnot So it's one of those things we're trying to figure out the right balance because most people probably aren't using json it but if it's something you're interested in you could join our slack channel and We're we will more than happy to share that. Yeah, we actually run short of time We wanted to create a repo with let's say already manifested because as Mark mentioned we're using json it sometimes that the It's kind of hard to get started with so yeah So we want to let's say to to to share the already manifested Recording rules, but we run out of time and that's why we created this resource essentially a channel there in our slack And we may actually publish in some of these as excerpts But yeah, it's in our plans to actually publish the full Reach out to a slack. Thank you. Thank you Probably have time for one more question anybody but that thank you


Are you starting to feel festive with the holidays and the end of the semester approaching? Let's take a minute to celebrate SJI's iSchool with a September, October, and November recap of news and blogs featuring news, career blog, Siri blog, iSuean blog, Mara blog, and the Kara blog. The news in the fall held many exciting stories to share, including scholarships, faculty, alumni, always making waves, and the COA election sweep. Let's read some student success stories because who doesn't love those? Please join me in congratulating Diana Irvay with the Graduate Equity Fellowship Award. She was selected for Academic Excellence and Commitment to Community, and these three students all received the CLA Scholarship for Minority Students in memory of Edna Yelland. Shout out to Monica Vega-Latona, Brandy Ville, and Myra Fentes. In addition, Madeline Dickerson received the Julia C. Blitzer Scholarship for an essay she submitted that landed her a chance to travel to the Ariel Fall Forum, and here she is pictured on the left at the forum. The faculty have been busy putting a lot of books on your wish list, including Dr. Anthony Bernay, who released an updated version of his influential book Transforming Young Adult Services to further address the core question, how should LIS envision its YA users? And Dr. Michael Stevens, much anticipated wholehearted librarianship is now available. The book explores leadership with heart in the field of librarianship. In addition, the Emerald Publishing's 26th Annual Literary Awards recognized Dr. Michelle Chen and Dr. Pat Franks with the prestigious Highly Commended Award, and alum Emily Kuhn was celebrated with an outstanding paper award. And an exciting iSchool alumni and student sweep at the CLA 2019 election. Congrats to Jayanti Alderman, president-elect, Jeanette Contreras, and Brandy Buenafe, both elected to the board, and to my refuentes as the student representative. Turning our attention to the career blog, this blog is a must-read for students and job seekers. And it was written by current iSchool student Greta Snyder. That's me. Here are some top reads for fall. Did you blow your last job interview? Here are some tips for how to turn it around. Find out about the CLA SJSU iSchool poster reception. Who was presenting, and what were the topics? Cozy up with emerging technology and take a look at library 2.0. And networking for novices. Don't worry, you've got this. But where to even start? Here are some great tips to begin. Next up is SJSU Center for Information Research and Innovation, or CERI. This blog is written by CERI coordinator Dr. Lululu. Two fall research highlights include Dr. Jeffrey Lu exploring the possibility of applying data mining in library settings and information services, and Dr. Kristen Redman's work as she shares part of her MLS grants to address challenges associated with Native American digital access and inclusion. And next up is the not to be missed iSchool blog. Every student must check out the iSchool blog to neck with others and hear about development opportunities. It is written by current iSchool student Havilah Steinman. iSchool director Dr. Sander Hirsch turns up for Twitter game to engage with students and LIS professionals through an iFLA continuing professional development and workplace learning live Twitter chat. Hear more about it and explore some new places for LIS professionals to work, such as global records and management for aquariums and zoos. Read about Mara's student, Laura Frolke's internship at Oregon State Archives, helping launch her career. And don't miss a day in the life with iSchool student Kim Edwards, who currently works at the Thomas Jefferson National Accelerator Facility. Turning to the Mara blog, it's written by student, Kenna Wilker. Some highlights include Arma Infocon, Kenna's first time experience, hear all about it. Here she is pictured left with Dr. Pat Franks. In addition, an interesting story on archiving at the Vatican, a project involving high school students and AI. And in case you missed it, October is American Archives Month, but find out how it was celebrated through this blog post. And Vakara, did you know that SJSU holds virtual world events as part of our virtual center for Archives and Records Administration, or Vakara? The blog is managed by Mara coordinator, Dr. Pat Franks. Read the recap of the virtual reality versus virtual world Smackdown that was held on October 15th. I mean, there were dinosaurs present, as you can see for this picture. So I think that makes it most read. And also check out the post event synopsis of a tour of a nonprofit commons library held in November. Thanks for watching and happy holidays. No matter how you holiday, hopefully you'll get a chance to reset, recharge and rejuvenate. Look forward to an end of the semester recap and stay posted on the news and blogs at ischool.sjsu.edu in the news and events.


So, hello everyone, Haralambos or maybe Babis from Nubifcus, and today we'll give you a short presentation about our project on providing a serverless framework for Unicarnals, based on Unicarnals. So, at first, a small introduction about serverless computing. As you may already know, it's an event-driven execution model which allows microservices bundling containers to deploy it in a stateless function, either on the cloud, either on the ads, and as I said, most of the microservices are bundling containers, which allow the easiest management of the orchestration of the workloads, and we also know that containers also solve the problem of software delivery. It's very easy to manage, and they are able to do that without creating any more performance overhead or more memory management or memory footprint, but as we know, containers are based on the operating system level virtualization, and this means that all the workloads, all the containers, serve the same kernel, and this is a very important issue, especially for isolation, and in order to provide a solution for that issue, what happens now is that we usually deploy the containers inside virtual machines, and this creates some side effects because as we know, VMs do not have the same instantiation time as containers and they increase the footprint of the workload. So we see the emergence of lightweight virtual machines and lightweight virtual machines monitors, like KMO Micro VM or Five Crackers, in order to provide a solution to create a sandbox execution environment for containers without minimizing the bootstimes and the memory footprint. So our goal is to build a serverless framework for the cloud and the ads, and we base our solution on unicarnals. So the goal to do that is, what we want to do with that is to minimize the instantiation time, and we do that with a very lightweight hypervisor that we built. We minimize the attack service using the unicarnal functions, we bungling the functions inside unicarnals, and we provide an orchestration compatibility with existing popular container runtimes. So at first, how should the lightweight hypervisor look in the case of serverless? And the key points for such a hypervisor is to provide as fast as possible cold bootstimes. It has to provide strong isolation, for example using hardware extensions as KVM does. We, an idea could also be to minimize the mode switches when we have VM exits so we can have faster IO input and output, and also we can also minimize the ABI in order to strengthen and provide a smaller attack surface. And all of that has to be also suitable for edge devices that are also coming in service at this time. So let's take as an example a generic virtual machine IO request, for example a network packet. So we have the application that makes the network request. We have the packet as we know will travel from the application to the kernel space inside the guest, then the kernel from the network stack, then the kernel then will trap. This will create a VM exit which will make the KVM to wake up and manage this trap. The KVM doesn't know what to do with this trap and it will give the control to the virtual machine monitor which will therefore forward the packets to the kernel again in order to send the network request. So we propose heads which is a hypervisor for the heads and our main, what we usually mainly do is that we try to minimize the mode switch between the host between the kernel and the user space. So instead of the packets travel from the guest to the host, then from the host to the user space and back to the kernel space, we simply move the virtual machine monitor inside the kernel. And of course we have to do that in a very minimal way in order to keep it as simple as possible and as small as possible. So in order to do that, we also use Unicernals as guests which are able to provide stronger isolation. So in case you are not familiar with Unicernals, Unicernals are specialized single other space images which are simply constructed using library operating systems. A library operating system is a type of architecture of operating systems that libyify every component of any operating system. For example, a driver, a user space, the user space management and all of these things. So what happens in the Unicernals is that the application is linked against these libraries and it forms an image that consists only with the application code, its configuration, any runtime that it might need from the application, any library and of course the operating system components that glue all these things together. So with that design, Unicernals are able to have a very, very small memory footprint and they can achieve very, very fast good times, even the same as containers or even faster and they can provide strong security in two levels. At first, they benefit from the isolation that is provided from the hypervisor and secondly they have a very, very minimal attack surface, a minimal attack surface. On the other hand, as you might already know, Unicernals provided a completely different execution environment which makes porting existing applications in libraries very difficult. Also, there is no support for hardware acceleration and since they are not containers or very, very lightweight VMs with only one function, they also create some challenges regarding their orchestration. So in the case of hardware acceleration, the challenging thing is that the existing frameworks are very, very big and it's not easy to port them and also it's very difficult to access any hardware acceleration devices like ZPUs and FPTAs which require for example a driver to be ported inside the Unicernal. So what we do is we decouple the function code from its hardware specific implementation with VXL and VXL, as you might have seen yesterday, is a simple library that consists of a static and static or user defined API that interacts with the application, a glue code, the plug-ins that we see for example here on the bottom that it's just a glue code for the hardware implementation and the main component which is the VXLRT which is a multiplexer for the requests that come from the application and they are mapped to the respective hardware implementation. So it's more like a VM remote execution API. So in the case of orchestration, as we know, most of the frameworks that already exist are tailored for containers and what we need to do is to integrate the Unicernals in that model. So we bundle the Unicernal binary with all its dependencies in a container image and later we unbundle the Unicernal binary and then we spawn it from the container image. We also build from scratch urnc which is a Cata container based runtime which is able to spawn Unicernals and at last we also need the interface that will interact with the serverless gateway in order to invoke or get any metrics from the function. And for example, as you might, if any of you is familiar with OpenFuzz, there is a function called watchdog and what we do is that we port this functionality inside the Unicernals and this snippet will then call the function that is bundled inside the Unicernal. In this diagram, we see what we want to achieve, we want to allow the user to have the option to deploy their functions either in a simple container, as a sandbox container or even as a Unicernal from a unified framework like Cata containers using Kubernetes. So to sum up, we built a lightweight serverless framework and we base it on Unicernals, stripping down the virtual machine monitor and moving it inside the Linux kernel. We use Unicernals to bundle the functions that the user wants to deploy. We provide hardware acceleration, it's as far as we know, there is no other hardware acceleration support for Unicernals right now. And we also enable the deployment of Unicernals on the edge. Both our projects, Heads and Uransi, which are the main components, are working progress. VXL is more stable and you can easily try it out whenever you want. So I also need to say that this project is partially funded from Serano and 5G, a complete, which are Horizon 2020 projects. Thank you for your time and we'll be very happy to answer any questions you might have. How stripped down Linux kernel? As far as you like, but it's still very much inside, same memory manager and so on. Do you still have the separation of user space and kernel space in that? Unicernals? Yeah, that's how you do it now. That would be a nice Unicernal if you can make it actually. This is an open issue right now, like how we can have Linux compatibility in Unicernals, for example. And it would be, I think there are some kind of projects that they're trying to achieve that. And then it's more like just as, let's say, a bit of more overhead because of the mode switch between user space and kernel space, which is, is it really necessary if you just have only one application? Do you really need that? That's the notion. Do you have any benchmarks of real-world applications, how they are comparing Unicernals and containers? Yes, there have been quite a lot of papers, for example, Unicraft and Solofival, like Rampran, Unicraft, OSV, these are the most popular, let's say, frameworks for Unicernals. And they are able to achieve much faster network I.O. and much faster boot times compared to containers, instantiation time. I don't have any numbers right now, but I can point you to these papers, if you want, harder access. Sorry? Yes, but that would be an option, but if we have a pass-through, then we limit how many gates we can have, right? Like, we cannot serve the same resources easily with pass-through, but that's also a case, but you put it in the right migration, not even. You are dependent on it. Yes, exactly. Thank you again for your old time.


Dr. Pomerance here from MRI Online, we're talking rare diseases today and Charcot-Marie Tooth disease, this enigmatic disorder that you learned about in medical school but thought you'd never see. Well, you will see it. It goes by the abbreviation CMT and there are some other synonyms that you have heard throughout your medical career. Hereditary motor and sensory neuropathy or HMSN, perineal muscular atrophy, distal hereditary motor neuropathy, D-H-M-N, hereditary sensory neuropathy or H-S-N or H-S-A-N, some debate whether that's included in the subset of nomenclature and distal spinal muscular atrophy or D-M-S-A. Now this is one of the few disorders that is named after three doctors who originally described the disease, Jean-Martin Charcot, Pierre-Marie and Howard Henry Tooth. Jean-Martin Charcot is often referred to as the father of neurology and you've heard his name on other disorders. Now in terms of genetics, the genetics center around the duplication of one major gene and that is PMP-22 gene, which you receive one from each parent by the way. In CMT-1 and its common subset, 1A, this is a demyelinating autosomal dominant disorder but it accounts for at least 50% of all the cases of Charcot-Marie Tooth. CMT-2 may involve more the center of the nerve, the axon, it's also autosomal dominant, CMT-4 recessive and these are far less common. This single gene duplication of PMP-22 accounts for at least 50% of the cases, however, there are over 100 gene mutations associated with Charcot-Marie Tooth syndrome that have been described. CMT, while an autosomal dominant disease, it's not just one disease. It has innumerable subtypes and each variant has in common involvement of the myelin sheath of the peripheral nerves. The disease is slowly progressive and variable and can arise without any hereto-famil history in some cases which makes it a bit confusing. CMT is the most common neuropathy in the lower extremities and the perineal and tibial nerves will often be the presenting areas for you and so looking at the lower extremity is going to be very important in these individuals. The symptoms usually present in adolescents. The symptoms are variably motor and sensory. They can be more towards the pure sensory, more towards the pure motor and they involve the hands and feet. The patients may experience foot drop, muscle atrophy, cramping, peristeges, nerve pain, clumsiness, sprain ankles falling and typically when you look at these patients who have progressed to a certain level you'll see something called a stork leg where the upper portion of the leg is nice and plump and then it tapers very abruptly for a long distance. Pescavus or a high arched foot, Pesplanus, a flat foot and then claw hand or hammer toe, a pearl. The longest nerves in the body are affected first so the branches of the sciatic nerve, the perineal nerve and the tibial nerve are commonly affected. There are a couple of other oddities that I wanted to share with you. One is known as NHPP. It's a hereditary neuropathy with a predilection to pressure palsies. Now unlike the other cases of Charcot-Marie-Tooth where there is a PMP22 gene duplication, this one is a gene deletion which makes it all the more interesting. The DMT3, formerly known as Dejurin-Sotus syndrome, some of you have heard this name previously as a subset of giant cell neuropathy with nerve swelling. We no longer use this term, Dejurin-Sotus has fallen out of favor. And then there are hereditary sensory and hereditary motor neuropathies which some people include in Charcot-Marie-Tooth and others divide them into disorders unto themselves. The diagnosis of CMT involves nerve conduction testing, biopsy of the serial nerve, DNA testing, also not shown here, assessing the other members of the family and sometimes imaging. Let's have a look at a real case on MR imaging. This is the 35-year-old man who presents with weakness and clumsiness and also knee pain. He's carrying a small anterior neofusion and what should strike you as you look at the stack of artery, vein, and nerve is the signal of the nerve. Now normally a nerve signal is going to be intermediate and confluent. If you have very high resolution then you may see subunits of the nerves. You might see little dots inside and they should fill that entire space. They should be symmetric. They should each be the same caliber. None of them should demonstrate any crimping or interruption or defects. What we see in this patient, let's look at the tibial nerve right here, is something that's a little bit gray on the outside but rather white on the inside. Let's blow it up. It is a T2-weighted image. Where is that confluent solid gray signal intensity? That's not gray, that's hyper intense in the center and also in the periphery. Only a thin shell of hyper intensity remains. Just to show you that it's not just one slice, it's contiguous. You never, ever see that confluent, beautiful, homogeneous gray signal intensity.  The sciatic nerve innervates the hamstrings. Let's demagnify or demystify this and look at our semi-membranosis muscle. It looks so much different than it's brethren on either side. That's because it's severely atrophic as a result of this demyelinating peripheral nerve syndrome. Not done yet, we're going to track our perineal nerve which as we go up doesn't quite meet the tibial nerve but it's on its way. Let's blow this up and make it bigger again and look at our perineal nerve. It's so bright within that it's almost cystic appearing. Now let's work our way down, signal, signal, signal, signal and the signal persists for quite a long distance. And remember, this is a disorder that likes to affect the longest nerves in the body. Another tip off to the diagnosis, although subtle, besides that semi-membranosis atrophy which is weird is this.  So this is one of our cases of Charcot Marie Tooth. Shall we look at another? This 28-year-old girl presented with leg numbness extending from the knee down so more of a sensory presentation. Let's look at the axial projection. I'll bring out my trusty pen and my MRI online yellow right there and we've got an artery nice and black with flow void in the middle. Then we've got a vein on top of that, it's a little bit larger. And then behind that we have the nerve. So let's take my yellow markers off for a minute and let's look at the nerve. It's got numerous components to it. What is this component? This very bright area. I'm going to put an arrow on it right now so the non-radiologist viewing this video can see it right there. That is cystic degeneration of the myelin of a nerve and also it's axon. Then we've got another finding and that is in the perineal nerve. Look at the bizarre appearance of the perineal nerve. In the prior case, even though I don't have my gray color to show you, the nerve should be a fairly homogeneous structure and occasionally with high resolution you'll see some of its internal subunits. I drew it in yellow but it's really more gray or dark gray. What is this high, fat-like signal intensity that wraps itself around the nerve? This is Schwann cell degeneration, so to speak. And then we look at the coronal projection now and we have an interesting finding that should have been the tip off to the diagnosis in the first place when we hone in on the area that the patient describes as being symptomatic, which was in the outside of the leg. We see that the lateral compartment is a bit atrophic. It has a little too much fat in it. And the fat for the non-radiologist again being the high signal intensity, because this is a fat-weighted T1 weighted image. Whereas the image on the right is a water-weighted image.  Here it's dark gray. And what do we see in the muscle? We see a muscle that is intensely bright all the way up and down in the lateral compartment. And this is an example of myoedema secondary to denervation. So this is a slightly different example from the former case, where you're seeing extensive muscular edema and slightly different manifestations of nerve cell degeneration in one of the variants of Charcot Marie Tooth.


So my beautiful people I am back with another out and ring video and this one's a little crazy to their showcase to you guys How you can get an additional 80 to 100 percent extra damage on Anything and this stacks with other damage buffs too and guess what guys it's unlimited Wow, how's it going guys? My name is DP journey if you do enjoy the video living a like really helps out And if you like what you see I want to see more be sure to subscribe Okay, so there's an item in this game, which you can apply to a weapon. It's an ash of war It's called the Royal Knights resolve this gives you on that next shot with that weapon It's applied to an additional 80 to 100 percent extra damage under a shot math skies. Leave me alone. Stop bullying me But either way so I apply I've got it here right here on my katana, which I showcase to you guys Obviously got your grace go down to ashes of war. You'll see it here Select it and you can see it's right there the Royal Knights resolve It's applied to my katana. So when I activate it my next attack gets Additional damage between a 70 80 100 percent I'm not sure but it's only for that one extra attack and the effect wears off between 8 and 12 seconds That's how long you've got to hit somebody with it. But guys.  If you don't in a way, I'll get this go for another one So if I do this and you'll see I get 731 damage now if I activate the ash of war Do the same to this coat? boom extra damage extra damage Nothing else there, but yeah extra damage and you get that for 8 to 12 seconds. It's unlimited Pair this with something like the comet azer pair this with the terror magicus How much damage buffs are you gonna get? Like I said, it's infinite guys because you can just literally Keep spamming it spam it when it runs out It's so simple. It's unreal. So yes an additional 80 to 100% extra damage on anything you use and I mean you could even put the weapon away so you can apply it and Even put the weapon away and use something else if you want to that's completely up to you You still have that 8 to 12 seconds. Okay, so how do you get this Royal Knights resolve? Well? It is a bit of a quest and I've seen a couple of other guides on how you get this thing But none of them are really clear and it is quite annoying and I understand you guys I know a lot of people are gonna be watching this video thinking finally someone actually showcases the exact location of how you get this and How you get to the area to get it. Well, it's quite simple So on your map you'll see an area or you probably know this area Could volcano manna now that shows this actual item is located within volcano manna But you can't actually get it. We are going to volcano manna initially You have to go here via an alternative route and to do this guys. You need to come down to Reyes Great Lucy are you gone? Oh, what's this place good people?  I get it so confused.   This is where you want to go to I will run through you on screen now with you live So you just don't get confused people now there will be a bus in the way which you will have to take out We should be fine people so once you load in turn back on yourself from the grace point and Come down these steps and go right. You'll see this massive wheel where you need to jump on to one of the platforms So wait for one to come up jump on to it and you are good From here once it gets to near the top run around the top and get on to a platform on the other side And just follow it to the very bottom and just just do that simple Well, I will say is guys you'll also need a stone saw key, so make sure you have one rolls So when you're at the bottom guys just jump off and you'll see one these giant mothers I don't know what these are called either, but they're just some Funky ass looking enemies in this game this one right here and the trick is guys You need her to pick you up in her belly or put you into her belly or whatever this contraption is and kill you So you may want to think about your ruins as well here because you won't be at well You probably can come back and get them or if you've got a smaller man tried a hundred and thirty five thousand didn't bother me I can get this back in literally two swings, but that's fine with me But yeah, you need her to pick you up put her into her belly Which is just one however look in place and you need to be killed via doing this and what this will do guys is This actually takes you to a secret area or a well Yeah, it is actually a secret area because I don't know if there's another way gain here Until you progress through it.  You can't actually get to we're going to Volcano manor above ground not until you progress via this place first then you unlock the path So follow the path I take on screen now guys jumping across We're jumping over all this lava and then you'll come up to here and they will have to drop down.  So yeah, you should be fine Okay, so just follow all the path I take on screen now Jumping across here.   So, you know what? I'm gonna take him out and I realized Yeah, that happens. So yeah, don't worry about that. So then come up the steps Grab whatever you drops. Yeah, black frame nice nice. We like that So come through here guys and you will get a boss fight Skud skin no, but it's quite easy to take out. So yeah, just take him out guys And once you are done activate the grace right behind him Then collect any collectibles in this small room if you want to and then write the grace guys There is a lift with a lever next to it. Simply go up in this lift Then guys from here follow what I go right around come round this platform Then guys jump down here and follow the route I take Let's keep coming around and I'll be careful here because these slugs are obviously black in your path or snails I don't know in a slugger snail without a shell. I think so.  You don't want to fall down here. So keep coming up keep coming up Let's come across here now from here guys. It's very basically straightforward as long as you've got That's done.  So do that Once you get through the door guys, you can kill this dude or run past him. It's up to you I think yeah, I decided to run past him And jump through this that will go through this door Then guys just run straightforward avoid all these ads avoid all these enemies don't worry about them And there's come down these steps Keep going forward guys and ignoring these as best you can Pull weapon though that flame whip when you can actually get that off him in the back right of this room There's gonna be a door which you have to go through So open the door. I don't know why it takes you like a week to open the door when you got all that power But either way And from here guys, you want to make your way up these stairs So yeah, go up these stairs and in the store right behind you You'll have to use your stone.  I died trying to jump I just missed a ledge absolute failure, but either way and just jump that worst walk down guys Be careful. Just keep walking down ignore any enemy shooting at you And once you get to the last crate you can try and jump across this platform like I did and failed But you won't die if you jump down from here anyway, so he gets up to you So I failed miserably but either way then guys just run straight up to that room right in front of you And there you have it guys right there on the floor next to this body Will be the Royal Knights resolve ash of war and there you have it people This is how you get an additional 80 to 100 percent Damage bonus and guys well if you follow the route I take now You actually come up to the door Which is probably if you've been to volcano manner the main house the main manner you probably noticed after doing all the quests There's actually one door that you can't open. It's this door, which you are it opens from the inside So yeah, and there we have it people Guys if you enjoyed the video leaving a like really helps out if you like what you see I want to see more out of ring be sure to subscribe and hopefully people I will see you on that next one