And he does he does a lot of moral high ground posturing. Yeah, I think all of that's there so we can talk through that. And then, you know, that's that's good classic stuff for them. And there's containment shown in him and I can point those out. And then we can chime in. Therefore, is it we're giving it seven minutes? Do we do we want to play through it and just go stop? And it might be easier, Scott, I think it's going to be easier. Yeah, to do what? Just just just play through it. And we go, OK, let's stop. Let's stop there and say a little bit. Oh, yeah, we're the crazy guys. Talk about amazing. Yeah, so we're all right. All right, you ready? Yeah, here we go. I'm Scott Rasmussen, a body language expert and analyst. I train law enforcement, the military and interrogation body language. I'm Mark Bowden. I'm an expert in human behavior and body language, help people all over the world stand out, win trust, gain credibility every time they speak, including some leaders of the G7. Hey, I'm Chase Hughes. I'm a behavioral expert, did 20 years in the US military. Now I teach interrogation and behavior profiling. I'm a trial consultant here in the United States. I'm Greg Hartley. I'm a former Army interrogator, interrogation instructor, resistance to interrogation structure. I've written a bunch of books on body language and behavior. And now I spend most of my time with corporate America and Wall Street. Today, we're going to talk about President Trump's speech dealing with the unrest in the US. It's all over the US. It's important to remember as we go through this, we're not political when we do this. And we call them as we see them right down the middle. And if it leans over this way, we're going to say how we think it leans over this way and we'll give you the reasons why we think it leans over this way. We're going to see, we're going to tell you why it leans over this way and give you the reasons why. So it's important to remember we're not political. We call them like we see them. I won't tell you exactly what we see and the way it looks to us. So right out of the gate, let's just go ahead and watch some of this. And then we'll get started. Greg, you had something. So guys, I worked years ago with the White House writers Gil with a few folks from there. A guy named Josh Gilder, who was a partner on a TV project I did. And they would say, Josh said the best moments of his life were Ronald Reagan as president, because he could deliver a line like no one else realized that all of these people are delivering scripted information. They may control content, but the writing and the actual speech is not something they've written. And we're going to see a lot of that. We're going to talk a lot about that in this delivery. You don't see the usual circus of body language that Trump does in this, and we'll talk through it. And Mark, I'm sure you see things that stand out and we'll point those out as we go through as well. Oh, yeah, that's some great writing here for sure. OK, all right, here we go. Thank you very much. My fellow Americans, my first and highest duty as president is to defend our great country and the American people. I swore an oath to uphold the laws of our nation. And that is exactly what I will do. All Americans were rightly sickened and revolted by the brutal death of George Floyd. My administration is fully committed that for George and his family, justice will be served. He will not have died in vain, but we cannot allow the righteous cries and peaceful protesters to be drown out by an angry mob, the biggest victims of the rioting, our peace-loving citizens in our poorest communities. And as their president, I will fight to keep them safe. I will fight to protect you. I am your president of law and order and an ally of all peaceful protesters. But in recent days, our nation has been gripped by professional anarchists, violent mobs, arsonists, looters, criminals, rioters, Antifa, and others. So what we're seeing here is one of Trump's most important speeches ever. So as he comes out, he's doing that classic presidential stoic walk. You don't see little jerky movements. He's not looking around. Nothing else happened. But he's walking toward that podium. When he gets there, we notice it's important to him because as he's holding both sides of the podium, not once, not one time during this whole speech, do both hands come off. You see one, you see them, they both come off at different times, but not never at the same time are both hands free. You see at the end where he says, whatever it is, it's over, you see his hands come up and sort of tap the top of the podium. But that sort of signifies that the speech is over. Usually his illustrators are huge. He's classic. People always ask us, I'm sure Trump is like a gift every day when you see him. And he is, because he's doing, he's always doing this piece, pinpointing this and he's doing that. It's got his elbows in. He's almost doing karate chops as he's talking. His illustrators are huge, but in this case, I believe he's been asked to reign that in, you know, kind of real anew. But because as he's gripping that podium, he's still illustrating, hitting those specific words and phrases. Illustrators are the things that we use to emphasize specific words and phrases like I did just then. And as we, as he does that, that's why that's where the squirming starts and the ducking and the squinching and the elbows coming forward and the, and the shoulders coming in because he's trying to do that. He's just not used to, to holding still and not using all of his, all of his illustrators. This is a great example of why absolutes don't work. If we, if we were watching him for absolutes, we would see that shoulder rise and think it meant something, but he's holding the, I agree with you. He's been coached. You'll notice later, I'll show you, I'll point out exactly where you can see the coaching. He's been coached to hold onto that, to that podium and you see his shoulder move because he wants to move his arm. And there's only one time that you get any real body language as a result of it because he's moving around so much, you can't miss it. And sorry to step on your Scott, but I wouldn't know. No, no, it's what we're doing. We're, we're telling what we think. So yeah, so that, that's, that's what, that's the way I see it as an overall, he's trying to see, he's attempting to stay controlled, thinking there's a pretty good job of, of not moving his hands out, but, but a couple of times, a couple of them get away from him. But that's because those are really important to him when he's illustrating those, you know, you've got really good symmetry there, which is usually a good signal that somebody has been told to be symmetrical, especially with Trump, because usually you get these OKL gestures would come in pretty early with him. So he's really buttoned down, has to be said, he can take, can take instruction quite well, did many, many series of the apprentice that he does a great job in the apprentice, you know, was instructed by Mark Burnett on that. So when it comes to taking instruction, he can take it. We will see him do a few OKLs, I think, a little bit later on, but, but he's well knuckled down at this point. Yeah. Chase, where you go? Pretty similar to you guys. I saw the, we see a deviation from his normal behavior here. And that's one of the reasons I think that, that we thought this, this video would be important here because it's so different and people ask us to do Trump all the time. But I think this one is great because it shows just that grabbing on to the podium, absolutely coach. And what we're seeing, I think, with the shoulders is resisting his, his normal impulse. And I saw a body language expert analyze this video already on, on YouTube and said the, the teeth exposure is a sign of him being really angry. And that is absolutely false. And I hope you guys agree with me. Oh, her percent. He does it all the time. And it's not angry. Yeah, you go back to his appearance on Oprah in like 1985 or his interview with Larry King in 1987 when he's a 30 something. And you can see the exact same behavior. It's never changed. And one thing you'll notice here is when we have outlier humans, they tend to have outlier behavior just like Bill Gates. So we'll see a lot of the strange behavior in humans that are, that are outliers. Yeah, Chase, I'll go back and I'm going to hit a couple of things you said, but also add a few others. If you go back and watch him as a young man, his body language is a bit of a circus. I always say he is an outlier. He is, he's an 800 pound gorilla in his own world. He's a creature of his own making. He's managed to create this bubble around himself and people copy his body language is the great indicator. But the way he moves his mouth, all of that stuff is very much Trump. It's a unique thing. The thing I noticed in here very early is he strikes a pose when he hits small high ground. He knows that those, that script is right when it talks about things like I took an oath, the American people, exactly what I will do will not have died in vain. And every one of those he strikes a pose a bit. Now, that's not because he's lying. That's not because that's because he's Donald Trump and he is keenly camera aware and because that's human nature. If either of us were delivering that same speech, there's a certain amount of pride and a certain amount of raise of your head is going to happen, uncovering his throat, all that's clear. It's even though he's coached, even though he's controlled, his body language is still leaking. And this is what I want you to pay attention to is it doesn't matter how good your coach is. You're still human. They're going to be a couple of times you're going to see things in here. I saw things in here. I wish I could not see, not because he was lying, but because of other reasons. I'll point out as we go through because humans leak information. So those dominant behaviors that will usually see him do at the rallies and he'll do them full blast, you know, which is that chinge up that he'll do in profile to get that classic Romanesque dominant coinage icon, essentially. Again, I think you're absolutely right.  Stay on board for this one. It's super important. OK, he does a good job of staying in script. Yeah. A number of state and local governments have failed to take necessary action to safeguard their residents. Innocent people have been savagely beaten like the young man in Dallas, Texas, who was left dying on the street or the woman in upstate New York viciously attacked by dangerous thugs. Small business owners have seen their dreams utterly destroyed. New York's finest have been hit in the face with bricks. Brave nurses who have battled the virus are afraid to leave their homes. A police precinct has been overrun here in the nation's capital, the Lincoln Memorial and the World War Two Memorial have been vandalized. One of our most historic churches was set ablaze. A federal officer in California and African American enforcement hero was shot and killed. These are not acts of peaceful protest. These are acts of domestic terror. The destruction of innocent life and the spilling of innocent blood is an offense to humanity and a crime against God. Chase, you got anything? Yeah, so what we're seeing here is the beginning of what's called a story arc. And if you're into hypnosis or you teach hypnosis, this is something that's called fractionation. Or if you are in some kind of an interrogation world in the military, this we might call this emotional fractionation. We'll start out with this is my oath. Here are some victims, but I promise you I'm going to fight for you. And then so this just a couple of little bumps there. And now we're just hitting the point where he's talking about the victims, the police department, the guy in Dallas who who who was killed and all the people who were affected by this. So now we're at the big fall of the story here. So this this would represent act two in a three act structure to where this is like the big crux of the book we're coming down. We'll we'll see how he decides to make this thing go back up again and and how we're able to finish on a good note and we leave the reader or the viewer satisfied towards the end. We see a little more of the teeth exposure here that's completely normal for anybody that's that's watching that's his normal behavior from probably the time he was 20. And we see some more podium gripping here. And one thing you'll notice is any other interview you see is town hall meetings, his rally. He's making eye contact with people in the audience nonstop. He's looking at the camera. He's looking straight ahead. Right now, if you go back and look at his eyes, it's from teleprompter to teleprompter. And the only time we see him look forward is just a couple of times when he is switching from one. He memorizes that final few words makes eye contact switches to the other teleprompter. Right. And I believe that's why it sounds almost it's I mean, maybe he's read it twice. I'm read the speech because he's going through that.  But I think that's I think that's from inside. I think that's actually bothering him. He's trying to get it done. So that that's what that's about. But yeah, I agree with you. Obviously he's not it. This isn't hasn't been rehearsed very well at all. But obviously he's read it. But he's got a problem going through it. Greg, what do you got? So I got a few things when he says righteous cries of peaceful protesters is the only time I see his arm not move but close to his ribcage.  We create exoskeletons and we feel stress. So we put our elbows tightly to our side. The only time I see it in the entire thing. Explain what an exoskeleton is. Yeah. So just meaning you're trying to protect those soft white underbelly organs of yours from attack. We're like I say, we're the only animal that walks around and shows our abdomen to the world. So we have to protect. We create an artificial skeleton or bony surfaces between our vital organs and the rest of the world. And you can see his elbow pull in, which is the only time I see it. He talks to one of my favorites that and this is that Trump condescend shine. It's the same one in residence. He's very condescending to these governors in that moment. If they can't hear it, shame on them because I can certainly hear it. His punctuation between when he's talking about a police precinct has been overrun here in Washington. He misses the punctuation. You can tell that it says a police precinct has been over run and here in Washington or but he just rambles through that as if he's saying here in Washington, this happened. These two pieces, the sentence doesn't work together. And then there last two are really powerful language written by someone. I am their president. I am your president. This is one of those things you could all the times we see Trump will often say he is divisive or all politicians are divisive in today's world. This is one of those attempts by his writer to say, I'm their president. I'm your president. I'm going to protect you collectively. It's good writing. Okay. What do you got, Mark? So it really is very, very good writing. Let me pick up on some of the classic images that are being used here, which have a contemporary meaning, but there's a very old resonance, cultural resonance. Let's start with savagery from savage, while this idea of a wild place, the idea of the thugs, the thugues, which were, to my understanding, a group from India who were rampant in battle and very difficult to handle. The vandals, a Scottish tribe that Hadrian had to build a wall in order to keep them out and keep them encroaching on the state of Rome. So there's this idea of the, the ordered state, what we might call the walled garden, the paradise being encroached upon by chaos, savage, wild chaos from outside. And then the idea of the crime against God, so it invokes a higher power in order that the repercussions of this are due to that, that, that, I guess, encroaching on the higher power and the higher order. So it's a big, big statement to make. Well, if you notice, he purses his lips at the end of that. And for me, what I think the lip-persing is, those are not words that Donald Trump typically uses. All right. Jason, you got anything? Yeah. So let me just add on to what Greg said, just for the the viewers, those arms coming in to the body, the humorous sticking in towards the torso. We've spent a lot of our, our history as humans running away from giant cats and one thing that stress and fear does to us, it makes our bodies protect arteries. So the this artery under here are are a brachial artery down here, squeezes in towards the body.  And it almost jumps in front of the carotid artery during that just I talked to a cancer surgeon a few months ago so that he's had to move that out of the way to get to the carotid and many surgeries because of the the patients. But there's one thing fear does it makes us protect arteries and we see we saw a good example that here. Excellent. All right. Let's move on. And a crime against God. America needs creation, not destruction, cooperation, not contempt, security, not anarchy. Healing, not hatred, justice, not chaos. This is our mission and we will succeed 100 percent. We will succeed. Our country always wins. That is why I am taking immediate presidential action to stop the violence and restore security and safety in America. I am mobilizing all available federal resources, civilian and military to stop the rioting and looting to end the destruction and arson and to protect the rights of law abiding Americans, including your Second Amendment rights. Therefore, the following measures are going into effect immediately. All right. Chase, what do you got? I think Mark's really going to expand on this from his perspective, but we see a lot of juxtapositional phrasing here, which makes people, we pay a little bit more attention and it's if we are, we're either A or B, you're for us or you're against us. So that's, it's a great way to win an argument and it's one of the techniques that's taught in law school for litigation and cross-examination or doing your closing arguments in the jury. That's fantastic and we still see him squirming against the instructions to grab onto that podium and don't let go, which I think is fantastic to watch. He's using words that are not his own. He's using more team focused pronouns here than ever before. It's usually I, me, my, all of these self-focused pronouns, but there's a lot of we here. Excellent. This is probably my favorite part of the whole thing because it looks like he's dancing in there. If somebody's going to grab that and put a track behind it, I know it because it really does look like something's happening there. Dance wise, but he's squishing out so long or something. Oh, man, it's all it's really it's hilarious. But that's when people say, I bet this is a gift. His body language. That's what that's what they're talking about for us. Anyway, we see things like that. He's trying his best to hang on that thing. So Greg, what do you got? Yeah, I got a lot on this one. He is leaking. Trump is leaking through all of the coaching. These are not nobody scripted these words. We will succeed 100 percent. Our country always wins. Those superlatives, that is Trump, Trump always uses superlatives. There's a grimace at one point when he's talking about restoring security and that there's an orphan here that. So when I say I don't like things I can see in body language, there's an orphan phrase in this entire speech that fits nowhere else. And that's a second amendment. Explain what an orphan phrase is. So if I were walking through this entire thing and I were building, you know, higher ground, pulling people together, all of that, all of this fits except for the Second Amendment piece. And I have a feeling this is an insistence that he puts it in here to say, we're not going to leave you out. We're not saying that you can't protect yourself, but we are going to protect you as well. My guess is that some Donald Trump language put back in here and it's tightly tied to the right, clearly, you know, the conservatives, although there are many gun owners in the U.S. who are not conservative. People seem to forget that. He is simply saying to citizens, we are going to take control, but we're not going to take your guns. That's an important piece for him to point out. And it is actually tightly tied to the message. We're not going to take away rights, but we're going to we are going to take over the streets because himself a law and order president. This is an interesting twist. I just feel like that Second Amendment is an orphaned statement in the rest of this entire message. And Mark, you probably have comments around that one, I would imagine. Yeah, so to chase this point to your point, here's a piece of classic rhetoric.  So he does a whole list of it's going to be this, not that ending with something around security, not anarchy. So this, not that you can start with the positive and go to the negative or start with the negative and go to the positive, but a great long list. And and what you get is an element of truism, which because it's rhetorical and because it's truisms, it feels really good. Just the rhythm of it already feels good. You don't even need to understand the linguistics. You get wound up in that kind of journey that that rhythm is taking you on. So again, I totally agree with you there, Greg. This is not Trump's usual speech pattern. That's a good rhetorical writer. Really good catch there with the 100 percent. I mean, that's a that's a measurable and no really great speechwriter would put in a measurable like that. That's a classic superlative of I'm I'm surprised he said 100 not 110 percent. All right. That's what we want. Your second amendment writes, therefore, the following measures are going into effect immediately. First, we are ending the riots and lawlessness that has spread throughout our country. We will end it now. Today, I have strongly recommended to every governor to deploy the National Guard in sufficient numbers that we dominate the streets. Mayors and governors must establish an overwhelming law enforcement presence until the violence has been quelled. If a city or state refuses to take the actions that are necessary to defend the life and property of their residents, then I will deploy the United States military and quickly solve the problem for them. I am also taking swift and decisive action to protect our great capital, Washington, D.C. What happened in this city last night was a total disgrace. As we speak, I am dispatching thousands and thousands of heavily armed soldiers, military personnel and law enforcement officers to stop the rioting, looting, vandalism, assaults and the wanton destruction of property. We are putting everybody on warning. Our seven o'clock curfew will be strictly enforced. Those who threaten innocent life and property will be arrested, detained and prosecuted to the fullest extent of the law. All right, Chase, where you got? Right here, we just saw him go through this juxtaposition list, which basically, you know, as we're walking through the juxtaposition, we're saying, OK, yep. Yes, I agree. I agree. I agree. I agree. So what he's doing is we're clicking back up that little emotional roller coaster and we see him go to the top here. So now we're going back up on the emotional plane here. And we're also seeing his use. Any time you see him using words when he's really making eye contact with the audience, you know, he's going off script. You know it. All those moments when we're here in phrases that are like 100 percent were thousands and thousands of military people coming out of the streets. That's not in there. And you can tell because he's making eye contact. He reverts to his natural behavior because those are his words. Excellent. I agree with you. And we know some of these things are when he's not hitting the origin is supposed to be there supposed to be emphasized. He does words like when he says, end it now. End it now. It just sounds odd. We should be saying, well, end it now. We should be saying that. That's why I don't think he's rehearsed it or read it very many times. I want to say again, once, maybe twice the whole time. A lot of these things when they should have emphasis on him, when I guess he's used to doing it with his illustrators, they don't it's not there. The power that this thing could have isn't there. We could really be punching. It just feels like he's going like this where he could really come around and get you a good Reagan, not Reagan. Right. This is. Yeah. Yeah. There's no Reagan. So for me, Chase, the thousands and thousands of sort of jump in the thousands and thousands, I think is a big piece. He starts to give you here comes the Moses down in the mountain. I'm going to quickly solve the problem for them with thousands and thousands of these. I strongly recommend, you know, he's walking through the things that he's going to do for us. And he does start Trump's leaking out here. You can start to see him. And you can also, I imagine, see the speech writers standing on the side, clenching, thinking, please stay on message. I'm sure his handlers are thinking because he is his own animal. Mark, where you got? Yes, a couple of points here that I picked up on. I'd be interested in other people's view. The violence has been quelled. I think what I saw there was that a bitter signal in his in his mouth. If you think if you think about sucking a lemon and you get this kind of bitterness there, it's that sense of this will be a bitter sting for people. This will be a nasty taste. And then I think I've got to breathe out on fullest extent of the law and and and a blowout through the nose. Again, my little my little puppy does that when she she wants you to know that that she could be pretty aggressive and nasty. She'll do a little like that just to tell you, just to tell you who you're dealing with. Does it happen around food time around food time around if you won't play with her? She'll like give you one of those apes do it. Often it's a full outbreath or a barker like that. So, yeah, exactly. I think I think we're seeing one of those from from Trump at that point. Again, this is this is leaking as well. I think he stays pretty close to on because of his hands. Someone is coaching very well. We'll see that leak here in a couple of minutes in this next piece, but he's pretty still on. His brain is starting to hop in and say, I need to punctuate here. It's what's made him Trump to this point, regardless of whether you like it or you don't like it. It's what made him Trump. So I agree with it and those anchors or he's on there. Somehow they've anchored that in here on him because he never does that. And as he second, please, please. No. So as he's as he's doing that, somehow they've anchored it in there because when he's trying, he knows not to come off there. So whoever's asked him not to do that, you can't tell him what to do. When they ask him not to do that, it's got to be somebody that he, you know, will listen to because he's not doing it. We covered Hillary and and Trump in Twitter and we were watching the same thing.  She would grip the podium and he would stand there and dance around with all of this stuff. Yeah, but it's coaching, coaching, coaching. Absolutely. Look, I mean, if in my view, if Trump is anything, he's a great entertainer, a great entertainer and he's been well trained in entertainment by one of the innovators of entertainment. The only reason we watch TV like we watch it now and some of the formats is because of Mark Burnett. And remember, he worked for years and years with Mark Burnett. And what you've got to see about entertainers is their illustrators will be big because they know it fills the screen and it keeps people watching them. And so when you stop Trump using those big, big illustrators, you kind of stop him doing what he does very, very well, which is attracting an audience. Whether you like him or not, you can't help but watch him. I think when you watch him in election and we can cut this out, but I think when you watch him in election, he does what I call the flaming bag of dog. You can fill in the rest for yourself.   And he's just a master of that kind of thing in its entertainment. Mark, I agree with you. It is 100 percent. And as far as the big illustrators go, remember, he's as he's trying to hang on, he's still getting them done, but that's why he's doing that squishing, squirming and ducking stuff. It looks like he's in dancing because he's trying to still illustrating, but he's not able to use his hands or his arms. So I'd like to find out who's the one that had that talk with him because that's that's probably the most potent person in the White House. Yeah, for sure. Maybe his wife said, you look silly. Keep hold on to that. I should go from that. This is of this terror to be on notice that you will face severe criminal penalties and lengthy sentences in jail. This includes Antifa and others who are leading instigators of this violence. One law and order. And that is what it is. One law. We have one beautiful law. And once that is restored and fully restored, we will help you. We will help your business and we will help your family. America is founded upon the rule of law. It is the foundation of our prosperity, our freedom and our very way of life. But where there is no law, there is no opportunity where there is no justice. There is no liberty where there is no safety. There is no future. We must never give in to anger or hatred. If malice or violence rains, then none of us is free. I take these actions today with firm resolve and with a true and passionate love for our country. By far our greatest days lie ahead. Thank you very much. And now I'm going to pay my respects to a very, very special place. Thank you very much. So, Mark, you want to go first? Yeah. Here's the most interesting piece for me. And that's in terms of the rhetoric that's happening here is there's an adaption here of the prayer of St. Francis, a piece of rhetoric there, where there is light, sorry, where there is darkness, let there be light. And instead, what the writer does here is where there is darkness, there will be more darkness. It's a it's a classic piece of rhetoric, which has been corrupted to some extent. The first part of it intrigues us because we've heard it before somewhere. And then the second part of it makes it even worse. So it ups the emotional stake of it. Quite clever there. Again, if you're writing a speech, you can often do nothing better than just steal an idea from something that culture already knows. Take a pop song that people know well, take a prayer that people already know really well and adapt it slightly. And there's always something about the rhythm and the cadence and some of the words that cause people to think, well, this must be good and this must be true simply because they've heard it before somewhere. I love that entire idea. And we can hear we're kind of closing up the speech here. And there's a threat at the end. If we don't do this, we're threat. And we hear him speed up and very quickly use the word terrorism or terrorists. I think that was in the script on purpose. I don't think he would have used that word.  But we do see him go off script a little bit here. But I think he's just adding words. And if there's one thing he's an expert at, it is adverbs. He is the adverb king of America. He is bigly is a bigly the adverb guy. And we see him do this here and towards the end of the video, we're seeing a perfect this this final knot. You know, we've sewn this thing up with perfect stitching and he's put a beautiful knot on there. And when somebody mentions the word artist, we think painter or a musician, but the person who wrote this speech, that is art. This is an artist. And just like what Mark Bowden said, there's a real popular book out there called Steel Like an Artist. And that's what we're seeing here. We're seeing a lot of those things borrowed or stolen from things in the past. The moment we encounter some kind of familiarity, we're more likely to believe something that we're hearing. So, Mark, Scott, you want to go next or? I'm good. I'm good. OK, so this for me, I'm going to get to this in a second. When people ask me, is there ever a time I wish I could not read body language? This is it. There's something very ugly in this. And I'll get to it in a second. But he goes off speech when he goes one law in order, one beautiful law in order. And he uses the Trump hand. He moves on in there and then he goes into the most disturbing part of this entire thing for me. And he says that's his brain punctuating. When he says, first of all, he says, no safety, no prosperity. He slams his hand down. First emphatic we've seen from him yet. Pushes his tongue out, then he moves into the next part of the speech. By far, our greatest days lie ahead. As he starts to do that, his hand comes up. Here's a great indicator he's been coached. He closes his fist and puts his hand back down. He's remembering his coaching. Our greatest days lie ahead. I wish I believed he believed that when I look at him, his chin drops. You see concern in his forehead. His lower lip draws in and his upper lip laps over. That is not joyful looking. That's not now. Here's the problem for me. I've been in the intelligence community. Chase, you've been there. We knew this much intelligence about what's going on in our world.  He knows everything that's going on. I feel very, very, very uncomfortable with where we're at as a nation when I see that body language when he's saying that. And then he ties up using that proper written language to pull back to I'm going to a holier place. I'm going over to St. John's and he wraps the speech up. I couldn't agree with you more. This is perfect, beautifully written. I just wish I could not read the body language I saw there. Maybe I'm wrong. We can be. But I see sorrow. That's not a good thing. Excellent. I agreed. I've got that written down in my notes. I was afraid to talk about it. I didn't I didn't know if we get kicked off YouTube. Maybe controversy to find out. We'll say we'll say controversy breeds popularity. Right. Yeah. Yeah. So what we thought around room one time and everybody kind of even 10 to 15 seconds really short concise. Tell you what you think is going on and what happened. So Chase, what we start with you? I just think this is a wonderful example. This video in particular because he doesn't do this very often and he is an untrusting guy. He's naturally suspicious of every human being that 100 yards away from him. And whoever it is that they got for him to trust enough to say you need to grab that fricking podium and keep your hands on it the whole time. Whoever that is, I would like to meet that person. Same here. So we see we see him fighting against his his training the whole time. We see him still injecting some of himself or or leaking. We have nonverbal and verbal leakage that that comes out there as well. I really enjoyed watching it because it showed who he was as a person. He's usually very honest, very honest and very brash and lacks a little bit of self control on the podium. And we saw something very different today. Mark, what do you get? Well, to chase this point, I don't know who's working with Trump at the moment. Usually when Trump is performing well, people start saying, where's Mark Burnett right now? Can we get him on the phone? Is he what town's he in? What city is he in? Mark can do an incredible job with people, including Trump. I do want to say that the speech writing on this is I think quite excellent. If you want to write great speeches or you want to know how they're done, I want to give a shout out on Simon here. With the UK government writing speeches and he's got a really good handle on on rhetoric. So I just want to push his book out there because it is a brilliant book on how to write great rhetoric. Will you say his name again? Because you kind of glitch a little bit. Yeah, sure. It's it's it's winning minds. It's by Simon Lancaster. Great book. OK, great. The first part is what really kind of bothered me. I didn't say anything about that at first. I don't think I did. I have a written my notes because when he comes out and he's so I can see him running that as he's walking toward the toward the party, we can see he's focused. Usually he's looking around. He's being a what's up, man? I'm the I'm the alpha, man. It's me. Yeah, it's me. He's not doing that. He's like coming straight for that. I was like, you know, this is going to be, of course, he has to do that because of the situation in his position, especially now. But that's the part that bothered me. And the words that he that his diction is the clearest on that he has no problems going through all the big the hoo-ha with him. Sometimes his diction, he's got great diction. It'll be long and one to 10. He's always a 10. But he's got some things he usually hovers around a six or seven. But the things were the words he hits a 10 on. Those are the things that bothered me because that's when he talks about the Second Amendment. That's when he talks about being the president for, you know, for everybody, you know, for both the nonviolent and the regular normal peaceful protesters. Those are the things that started bothering me from the first. So I started watching that and kind of gotten a little worm or not a wormhole, like whatever the rabbit hole or rat hole about what he was saying there. And that started kind of worrying me. So I think it was an extremely potent speech. I think it's one that's going to stick out for a long time because it was so different. Like Chase was saying at the very beginning of this. So that's my take on it. Greg, what do you got? Yeah, this is everyone wondered when Trump would speak. This is one of those times you can't simply sit quietly and let the country go through what we're going through. And under high duress, I'm sure he finally decided he had to say something. No matter what you say when you're president, it's not going to be received well by someone. That's just the nature of it. It's my archangel thing. It doesn't matter if you're satisfying 900 people if 100 are dissatisfied. Something is not going to work. So it's a real balancing act to say anything. At least he was smart enough to listen to who told him to hold this this podium. Otherwise, imagine what we would all be saying right now. We're having a very difficult time reading real body language past seeing that he is dancing around and trying to move. We can see that leaking on his face. I think really good scripting, really good delivery by Trump. First time I've ever seen him stay on script and stay with it. You can see the leaking start. Like I said, I imagine his handlers are going, oh my God, if it's Burnett or whoever. This is Trump, the product. It's a bad time in America. Let's hope that it satisfies and settles someone or at least a few someone's. And let's hope it pulls us together in some way. There's a lot of work to do. I agree. I agree 100 percent. So I think it's what really well and Greg, I think yours was the most important ending of all this. And I think we I think I think this has been a good decoding or analysis.


Hey, everyone. Welcome to Cube Day Singapore. Today, we are going to talk about handling billions of metrics with Prometheus and Thanos. I'm Ravi Hari. I work as a principal software engineer at Intuit. Hey, I'm Amit. And I work as a senior software agent at Intuit. At Intuit, we work on an AI-driven expert platform which drives the work for all the developers at Intuit and derives modern development experiences. And this platform is powered by Kubernetes Infrastructure. And these are some of the metrics that signify the importance of the platform. Our day-in-day job is to ensure the platform scales, as well as solve new use cases. Here is the quick agenda. We'll see the metrics evolution at scale with Prometheus and Thanos. And then we'll see different use cases with argoloads for metrics. And then how we leverage metrics for AI ops. And then how we are using metrics for Argo CD, metrics extension dashboard. And then how we push metrics for golden signals to even best via Kafka after. So it all started like this. We have close to 320 Kubernetes clusters at Intuit. And it's growing on a day-to-day basis. And then as Prometheus is a de facto standard for Kubernetes for the metrics monitoring and these HPA use cases, we thought of running Kubernetes Prometheus on top of Kubernetes with Prometheus operator. And we created a Prometheus HAPair, where one of them is primary and the other one is secondary. And then we install service monitors that scrapes the data from services and collects the metrics from the services and stores in Prometheus. So we used to store this data for 36 hours because of various use cases. And because of which, we had to run Prometheus with an EBS volume because it's a standalone instance, each one of them. And then we cannot store data in an object store with Prometheus as of that time. So what we did then was we were collecting the data and then we used a third-party tool for visualization. So we had to translate the Prometheus metrics into custom metric format that this third-party visualization tool supports. So we created additional add-ons like the storage adapter for Prometheus and then the S3 adapter that stores the data in the custom metric format in the S3 bucket, which then was leveraged by the custom visualization tool for us. And then the metrics started growing and the number of metrics that we had were overwhelming. And it was difficult for the developers to always look at these dashboards and figure out what's going wrong or stuff like that. So what we thought of is coming up with a few golden signals, leveraging this rate methodology, the metrics such as rate of request, error count, and then latencies along with CPU and memory utilization on the parts. We kind of looked into some few critical metrics for the parts of the applications. And then we had written our own custom Prometheus rules on top of the base metrics. And then we started shipping the data out into Event Bus because we had a separate pipeline that reads data from this Event Bus and processes the data and sends alerts for these golden signals. So we had to introduce a component called Prometheus Kafka adapter. We looked into the open source solution. We liked it, but it missed some of the features. So we had enhanced it. We'll cover that in the subsequent section. So we had to add up to golden signals, and we had to write the data out from the Prometheus remote right into Event Bus. This is an additional use case. And what happened was we started with some opening up 300 metrics per part, and eventually it went all the way up to 1,200 metrics per part that we had to scrape. Even that was not sufficient. In some cases, we went all the way up to 4,000 metrics per part that we had to scrape. The metrics use cases have been growing, and the rate of requests for the metrics has been growing. So at Intuit, we have seen that the rate of metrics in the last couple of years has been gone by 2x. So single instance of Prometheus HAPR was not sufficient to handle all this metrics. We had to scale Prometheus horizontally. And when we looked out for the solutions for it, we came across Thanos project, and Thanos can scale Prometheus horizontally. The way we have done this is we have run Thanos' sidecar along with Prometheus, and then this sidecar will write the data from Prometheus into Thanos' S3 bucket that is shown on the top. So what we did was we reduced the retention in the Prometheus instances. That kind of reduces the size of the Prometheus instances, but we are storing the data in S3 bucket and retrieving this data for up to 48 hours using Thanos' pods. So Thanos' query is front end, and Thanos' query is Thanos' store, which retrieves data from S3 bucket and sends it back to the users. So we had to scale Prometheus horizontally to handle all the additional metrics growth at Intuit. So this is how we started getting into Thanos. And once we have opened the doors with this, there are additional use cases that came out. So one of the challenges that we faced was when we scaled Prometheus horizontally, we started seeing HPR not working appropriately. The reason for that is when we have multiple shots of Prometheus, each shot of Prometheus was getting the data from a given namespace only to a set of pods, because the pod IPs is what we eventually derive from when we query from a service of that particular namespace. And then each pod is hashed to a particular Prometheus instance. So the data for the metrics is getting split, even though they all belong to the same set of namespace, the pods belong to the same namespace, the pod distribution among the Prometheus instances is getting divided. So when the HPA metrics were getting calculated via the Kubernetes API, the data that we are getting is not accurate because it's not aggregating across multiple Prometheus instances. In order to solve that, we had introduced another component of Thanos called Thanos Ruler. And Thanos Ruler is a component that queries Thanos query, which internally can retrieve data from Prometheus as well as it can retrieve data from SV bucket. And then what we had done is we had evaluated these rules in Thanos Ruler and we have not evaluating the Prometheus rules in Prometheus anymore, so that this central evaluation had solved us for the HPA use cases. Once the rules are evaluated, we write it back to Thanos receive component, which puts the metrics back to SV bucket, right? So this is how we solve the HPA issues. And then more and more use cases have been evolving at Intuit. So one of the use cases to use AA ops, AA has become a norm in the last couple of years and we had Intuit started using this project called Numa approach, which leverages metrics to solve for the anomaly detection with multiple applications. So one such application is Argo Rollouts. We'll talk about that in detail. The additional use case that we have come across is Argo CD. Argo CD started extending its functionality so that use cases such as config map and config map plugins or metric plugins and new dashboards for rollouts are all now integrated into Argo CD. One of such extension is metrics extension. So users don't have to go to different screens. Once they install Argo CD, they can view events, locks, metrics in the same dashboard. So Argo CD also started leveraging metrics from this Prometheus instance. So we had to call this Thanos query instance to retrieve the data. But if you see all of these different workloads are putting pressure on Prometheus and scaling Prometheus is costly. So this is something that is kind of burning the cost whereas now we started looking at how can we optimize the cost and not impact Prometheus with all these different use cases because Prometheus has this remote rate functionality. Can we leverage it in such a way that we can reduce the cost? So what we did was we had separation of concerns for each one of these use cases. For the regular use cases, we let the cluster admins and the developers take it off the regular query path, which queries the Thanos and then retrieve data from Prometheus because there are very few limited use cases for this, the TPS is less. Whereas where the TPS is high, for example, for the AOPS use case, we wanted to store data for eight days. And retrieving data for eight days means that the Thanos query parts were overwhelming and it reads a lot of data from the backend. So we had to add an additional component called query frontend that kind of chunks the data into the interval that you want. We have chunked it for every two hours so that the Thanos query parts are not exhausting. And it retrieves data back from the Thanos receive. So the remote right here would write data back to Thanos receive. And here we store the data for eight days in this use case for the AOPS. The primary reason we chose to store it for eight days here is because for any anomaly detection based on our study that we found, in order to get a 95% confidence on that anomaly, you need at least eight days worth of data with 30 second scrape interval of time. And other use cases for Argo CD here in order to show the live metrics because developers will be mostly interested in live metrics when they are looking at Argo CD, they wanted to see immediately how their deployment went and how are the metrics coming from the new deployment and other things. We didn't want to store for too long. So we stored in this instance of the Thanos receive only for six hours of data. So with these different use cases, we solved the problem by splitting into different pipelines but the source of all these metrics is to Prometheus and Thanos has helped us solve these problems. Let's look at one use case with Argo rollouts. So Argo rollouts is a replacement for deployment from the open source Kubernetes. So Kubernetes in the open source doesn't support blue, green or canary or progressive rollouts. So Argo rollouts is created to solve the problem and is made available in the open source, right? And then we leverage a metrics primarily to define when to promote deployment from canary to stable or shifted from blue to green and stuff like that, right? To make it stable, we use this analysis templates that run the analysis and qualify whether the new deployment or the new image that you have deployed is running successfully or not. A definition of analysis template looks like this and as you can see here, we are using a Prometheus query to query the data from Prometheus and ensure it meets a success condition, right? And the way it works internally is that you have a stable deployment and once you introduce a new image, it creates a canary replica set and that creates a canary pod. If let's say we have five pods running for this replica set and one pod shifts into the new replica set, that percentage of traffic gets shifted into the new replica set and at that time, we can run the analysis template which triggers an analysis run that queries data from Prometheus. Argo rollout supports other metric providers but at India we primarily use Prometheus as a backend to retrieve the metric data and once this analysis run is successful, we can then subsequently promote the rest of the pods and make the canary as stable. So let's look at a quick demo for this. So here you can see the rollout definition. In this rollout definition, we are seeing that the name of the rollout is rollouts demo and it is running in metrics analysis name space and there are five replicas and we use canary as a strategy and we have defined different weights. So initially 20% will shift and then run the analysis template that triggers an analysis run. If that passes through, then the rest of the traffic gets shifted after 10 seconds each time. We can also run this analysis template in the background so that at every step, this analysis template gets executed and we ensure it is up and running successfully. Let's look at the analysis template definition as we have seen earlier. We're calling this as a success rate. Here we are ensuring that this runs every 30 seconds but we can define the count and the failure limit other fields as required. We have just taken it as a base and simple thing for the demo as one but you can also put the time out on the Prometheus if it is not responding, you can time it out and retry after some certain time and stuff like that. So here we are querying for the CPU utilization data in this namespace for this particular app and we have defined the success criteria for that. Let's go ahead and quickly see that we have already created this rollout and we'll just go ahead and update the image with a newer image so that the canary process starts. So I've just zoomed in here for the easy view. So as you can see the newer image starts getting deployed and newer pods are getting provisioned. If you look at the analysis template, this is the analysis template that we have defined and you can see that a new analysis run got triggered and it is successful, that is why it started promoting the pods in that namespace. If you look at the value that we got from Prometheus for this query, it is 0.018 which is lesser than our target. So it is the success criteria passed and it started promoting and as you can see here, it started shifting other pods into the new canary and it'll make the canary stable. So this is the overall use case for leveraging metrics with our rollouts. Amit will cover the other use cases. Thanks Ravi. Yeah, I'll move forward with the next use case which is regarding AI ops with metrics. So at Intuit we use automated detection of discrepancies and issues something called anomaly score. So it's a valuable tool for developers to find out issues quicker than other traditional methods and also it reduces MTTD and MTTR. So in Intuit we use ARCO rollouts along with this anomaly score to automatically roll back changes if the quality of changes doesn't meet the required standards. So while implementing this we faced several challenges. So one of them was access to historical data and whenever this numerologic infrastructure was querying this data, we had to scale based on the load. So we had this in cluster Prometheus but it was already doing some multiple operations like scraping, compaction. It was also serving HPM metrics and it was remote writing to different things and also doing rule evaluation. So we didn't want to put another read heavy operation on Prometheus. So we came up with this initial design where as you can see we're pushing the data from Prometheus using Thanos site card to a S3 bucket in the highlighter sections and we created a dedicated pipeline for this numerologic infrastructure to query the AI ops data. So in the S3 bucket we are storing the data for eight days and we are using this query front end to efficiently fetch that data but there is some problem, right? Because we are storing all the data for eight days in S3 bucket but for AI ops we don't need all the metrics, right? There are only few metrics that we are interested in. Also this huge amount of data in S3 is incurring a lot of cost and the way TSDB blocks are designed whenever you are retrieving a particular metric it would load the entire TSDB block in the memory and it may contain some unnecessary metrics that you don't want. So we improved the design a bit and what we did was we replaced the store parts with Thanos receive and here what we are doing we are only writing the required metrics from Prometheus to the Thanos receive and we are keeping it there in its local TSDB and we are keeping it there for eight days. Since the number of metrics are less so we can afford that and whenever the numerologic infrastructure is squaring the data we are again going by the same path using Thanos query front end and Thanos query but this time we are fetching the data directly from Thanos receives EBS volumes so that is much faster and there is a significant cost reduction as well due to the volume of data that we have now. So I'll quickly show a demo on the performance of this new infrastructure so recorded one, let me play it. So as you can see at the top this is a query S3 instance that means that it is connected to the S3 based pipeline and I'll quickly show you the store that is connected to so it says it is connected to a Thanos store instance and we can quickly verify that using the IP. So it's a query in Thanos query S3 store instance and let's try to get some data for eight days. Yeah, as you can see it took around four seconds to fetch that data here, right? And so yeah, let's try to see the newer approach using receive will query the same data here. So before that we'll quickly see it is connected to to receive instance and we can verify that using this command. So yeah, let's try to fetch the same data here and we click on execute as you can see it only took around 300 milliseconds or 350 milliseconds to fetch the data, right? So there is a significant performance improvement on this so there was a demo on this. Let me move forward with the next use case. So as Ravi was talking about Argoscd live metrics, right? So we have this new extension is in Argoscd that allows the developers to customize their key metrics and show it in the Argoscd UI itself. So you can configure metrics like CPU memory and some HTTP error metrics and you can directly view it in the dashboard. So it also helps providing a unified view to the developers and identify any issues quicker. So in the diagram you can see that whenever there is a query from the Argoscd browser UI app it goes to the Argoscd server and there is extension service in place which communicates to different Argoscd extensions. Here that we are interested in the metrics extension so when the query goes to the metrics extension it tries to query the in cluster Prometheus to fetch the metrics. But here we face similar set of challenges as before like the in cluster Prometheus are already doing multiple stuff and we don't want to add another read heavy operation to it so we came up with a similar approach here. Yeah, so came up with a similar approach here. So in the highlighted section we created a dedicated read path for this Argoscd live metrics and as you can see we are remote writing the data from Prometheus only the required metrics to a separate Thanos receive TSDB instance. So whenever this Argoscd is querying this data it is going via separate read path and this path can scale independently and it doesn't affect the in cluster Prometheus. So this is how it looks when you configure the metrics tab in Argoscd. So you can see all the metrics in the same tab as your application. And coming to the next one so we wanted to send some golden signal metrics to Kafka. The need here was to send data to multiple Kafka topics and we wanted to support tag based filtering as well. The challenges were there were some open source solutions available but it doesn't support multiple topics. Also there was no graceful shutdown and connection handling. Also if you wanted to use the solutions we need to add extra remote writes to Prometheus and that would add an extra 25% memory overhead on Prometheus. So what we did was we came up with our custom Prometheus Kafka adapter which had a better connection handling and all those things. Also we added a multi-topic support that means we are able to send metrics to multiple topics based on something called a tag parser. So a tag parser is nothing but, there is a image as well. Okay, yeah. So the tag parser is nothing but a collection of labels and a matcher. So whenever a metric coming from Prometheus matches those tags here, right? So it has to match this foo equal to anything and bar equal to true and that metric will go to Kafka topic one. And anything which doesn't match any of the matches will be dropped. So I'll quickly show a demo on this. So let me play this. So we are using telegraph to generate some mock Prometheus metrics. As you can see there is metric one with this labels foo and bar. There is metric two with baz equal to true. So ideally metric one and metric two should go to Kafka topic one and two and this metric three should be dropped. So we are also creating some local Kafka instances using Docker compose. And this is the same configuration that I showed in the slide. So let's try to start all these components. Yeah, let's start the telegraph first to generate the mock metrics. And as you can see it started generating some mock data. We are also starting the Kafka as well. So let it start and then we can start this Kafka storage adapter which starts accepting the remote right request from telegraph. And as you can see there is all the metrics are going to the correct topic. Metrics one going to topic one. And similarly all the other metrics are going to the correct topic based on the configuration. Yeah, so that was the demo on this. And I'll hand over to Ravi to talk about metrics component in Twitter. So we have seen certain use cases that are critical for us. But in general there are many other components at Intuit that we leveraged metrics for. So here are some 15 add-ons that we use at a high level. Some of them are inbuilt and open source via kick approach. And some of them are consumed from the open source like Prometheus and Thanos, right? So we have tools like active monitor that does synthetic monitoring of the add-ons and self-heals if the add-on is in a bad state. And then AWS metrics to check the resource limits on the cloud provider and then self-heal if in case the limits are breached. And then we use skip state metrics and then we use metric server to fetch CBM memory data for the pods. And then we use alert manager to create alerts through GitOps. And then we use node startup monitor and cost monitor to keep a check on the cost and the startup time on the nodes and stuff like that. So there is a lot of depth at which we look at the metrics and process them for our use cases. Thank you. Any questions? Yeah, good. There is a mic in the aisle. Thanks for a very interesting talk. I just have a question. I believe that the sharing rule that will play a crucial role in determining how effective your scaling effort is. So I just want to ask how was it designed, configured, in order to adapt to the dynamic environment as communities in which the targets added and removed constantly at the very high speed? So how do you design the sharing rule to ensure that the workload will be distributed evenly among the prometheus processes? Yeah, thank you. So today, our sharding is based on a GitOps process. So we get alerts if the instance of prometheus or the shard one of the prometheus that we start with, if that reaches some on resources, we get alerted. And then we add another shard through a GitOps process that creates a new shard of prometheus. And both of them starts processing them. So essentially, if your workload environment spins up multiple pods very quickly and stuff like that, you might need to lower the threshold and scale it accordingly. That's the solution that we have gone with. But did you evaluate whether the workload distributed evenly, or is there any way you measure that? Do you mean in prometheus? Let's say if there's a skilled workload, can we detect that? We haven't looked into that use case because we haven't come across such an issue where one of the prometheus instances is getting skewed and stuff like that. We have not seen that yet. But it would be good to look into. We'll watch out and let you know if there is anything like that. OK, thank you. I saw in one of the other Intuit talks that you've been relying on Istio as well in your service mesh. Are you leveraging any of the generated Istio metrics for your use cases as well? And if so, can you elaborate more? Thank you. That's a great question. So we use Istio primarily for all our service mesh use cases. And we consume a lot of metrics from Istio generated ones. And we have a different scrape limit because Istio generally emits a lot of metrics. So the scrape limit on Istio pods is very high. And the golden signals that we were talking about, it is one of the primary use case for Istio because there are so many metrics that Istio generates. We came up with some set of five golden signals that we wanted metrics that Istio generates and then consume them via the event bus and alert the user based on anomaly and other things. Two more follow-ups. Are you using it for the rollout, your Argo rollout? And if so, what kind of Istio metrics that you use for that analysis? Yeah, so Istio is used in Argo rollouts. We use virtual service to kind of leverage canary deployments with Argo. And how do you maintain the balance between cardinality? Istio generated high cardinality metrics. Oh, we control it by the scrape limit in the Prometheus. Thank you. I think we are at time, but we are available outside. If any of you have questions, please reach out to us. Thank you. Thank you. Thank you.


ISMB ECCB Day 2 recap. Yesterday, I spent most of the day listening to talks about single cell analysis and networks. It started with a keynote by Dana Peer, who received the ISCB Innovator Award for her work on analysis of single cell data, especially pseudotrimed trajectories. She presented several interesting new methods, including cell rank 2, spectra and decipher. I then continued to NetBio, where I found a shortage of tool and resource presentations. However, the talks about Gemini, a method for integration of networks and Wraphit, a method for prediction of interaction networks, were the highlights for me. After that, I went to Bosch, hearing only part of the presentation about Spiles, a method for large-language model-based zero-shot information extraction, before rushing to the Elixir track, hearing a talk about string, new queries and functions. I then went to the poster session where poster A409 caught my eye. It showed how study buyers can cause the power law observed in many biological networks, thus explaining why it is so common.


Marines with India Company, 3rd Battalion, 24th Marine Regiment, perform projects around the farm of St. Louis native Kim DeWalter during Marine Week St. Louis. DeWalter suffers from multiple sclerosis, so the Marines came out to make her life a little easier. You know, we're putting up a chicken coop for her because her dog likes to eat the eggs and she sells the eggs, so I guess it's bad for business. We're putting in fences, a painter barn for her and doing all kinds of good stuff out here. Multiple sclerosis is an autoimmune disease that targets the central nervous system. It makes it hard for those diagnosed with it to live a normal life. But with the help of the Marines and the MS Society, DeWalter now has less to worry about. For Marine Week St. Louis, I'm Lance Corporal, Nick Neighbors.


Google Drive's 2020 update brought many changes to how we use its services. One such change is the ability to set an expiration date when sharing files. Rod here. Welcome to Burn to Learn. Until this update, setting an expiration date was a free option in all Google Drive accounts. Now this option is only available as a premium feature in G Suite accounts, the paid version of Google Drive. In the next 60 seconds, we will show you how to set an expiration date for your shared files. When your document is ready to be shared, head over to the upper right corner and click the Share button. In the dialog box that opens, add the names of people you want to share your document with and click Send. Now head back and click on the Share button. You'll see the names of the people you have shared the document with. To the right, locate the Share Options button. Just below the option Viewer, Commenter, and Editor. You'll see the Give Temporary Access option. Click here and set the expiration date for access to your document. Now save your changes. And there you have it. It's important to note that the expiration date will only work for viewers and commenters. Editor access does not expire unless removed manually. Additionally, this can also be achieved from within Google Drive by right clicking on any file and selecting the Share button. From there, follow the same process as above. If you have more questions on the Google Drive 2020 update, please leave them in the comments below. Also, take a second to subscribe to our channel and click on the bell. Burn to Learn will keep you up to date.


Hey there friends thanks for checking in Shacho 2022 and I'm at the Keltec booth and they are releasing at least in the second quarter the P-15. This is a subcompact 15 round magazine handgun with a 4 inch barrel so they're heading straight into the competition with the very small and lightweight double stacked 9 millimeter handguns. What makes this interesting is that it weighs 14 ounces unloaded. Check that out. Very thin grip and it does come with a 15 and an 11 round flat base plate mag. So with the 15 round mag you get three finger contact and you may notice right here it does have a grip safety. It also has a magazine disconnect. You're looking at a 5 pound trigger right there. It's just about a full reset. What is also very nice about this edition is that it is 425 MSRP and you will notice it does have night sights and a Tritium fiber optic front night sight. Alright so you have the night sight thing going. It's also optic ready so plates will be available to put optics on there. Some of the features it does have forward serrations. Really deep cut forward serrations and rear serrations. Standard Caltech grip with some checkering in there. I already mentioned the grip safety. Nice trigger. Very light. Extremely light. So you figure with 15 rounds you're gonna add another seven and a half ounces. You're looking at 21 ounces. The typical weight with a subcompact 9 millimeter double stack is like 25 26 so it is lighter. It is less expensive but it doesn't end here. Take a closer look at that. It does not end here because they also have what will be coming out is right here. An aluminum frame P15. You can see it has the other polymer grips but it has like a wood grain to it and this one does have an optic so it is optic ready. They will have plates available for your favorite optic but same same concept with an aluminum frame so it's a little bit heavier but aluminum is not heavy as it is so it doesn't feel much heavier like hardly at all actually. We'll take a look at the trigger once again. Right there very light trigger and a reset right there. Kel-Tec P15. Red dot ready. Night sights included. 15 and 11 round mag. 5 pound trigger. Low MSRP 425 on the polymer frame one. It's unknown right now. This is new nobody nobody has seen this. This one yet by the way. Nobody has seen it. You guys are the first to see the aluminum frame P15 but it does have a nice overall feel. We'll see when Kel-Tec actually gets it out. They say it's shipping in the second quarter. I'm hoping it will. I'm hoping to get my hands on one so I can take it to the range check it out and compare it with other handguns in the class. Let's see I'm going to hold these side by side see that they are they're very light you know 14 ounces is very light like that's like right around the micro 380 weights anyways but there they are the P15 and I do like that sight picture and I do like the fiber optic trading in front dot but offering quite a bit for not a lot of money they're getting it done so far 2022 we'll see when we can actually see these in our gun stores if you like videos like this please subscribe and share I always appreciate thumbs up button thanks for watching you guys be safe


Single molecule electronic biosensors have been developed to detect biomolecules with high precision and accuracy. They provide detailed information about the molecular interactions and reactions that occur between biomolecules, which cannot be obtained through traditional methods. The sensors can measure the electrical signal generated by a single biomolecule, allowing researchers to study the behavior of individual molecules. Additionally, these sensors can be used to monitor the optical signal produced when light interacts with a single biomolecule, enabling researchers to observe the molecule's response to light. This article was authored by Xinyue Chang, Yanihua, Song Zhao, and others.


It's disgusting.


August is hot and so are the savings at Appalachian Wireless, where all smartphones are 50% off. Even the 64, 128 and 256 gig models, two-year agreement required. Better service, bigger savings. That's today's Appalachian Wireless and East Kentucky Network Company. In Knot County Friday morning, six people were arrested in a drug roundup that targeted 14 alleged drug dealers, some with multiple counts of first-degree drug trafficking. 60-year-old Dwayne Blanton of Viper, 49-year-old Teresa Fields of Viper, 56-year-old Ricky Taylor of Red Fox, 27-year-old Dusty Shell of Red Fox, 44-year-old Scotty Church of Litcar, and 24-year-old Andrew Hall of Malley were all arrested by Kentucky State Police and Knot County Sheriff's Deputies and lodged in the Kentucky River Regional Jail in Hazard. It was great to have the Knot County Sheriff's Office working with us today. They provided the Sheriff himself come out today, and he had three deputies with him, and so it really just gave us more manpower to safely do this operation. We have an operations plan just in case we do run across problems, but we always do it the safest way. We always make sure we have marked cars, that our troopers are in uniform so they're readily identifiable, and we always have at least two serving the arrest warrants. According to a statement from Kentucky State Police, the arrest resulted from months-long investigations by KSP Street-level detectives. So all 14 indictments, they're all felony charges. Some of them had more than one count of trafficking, but they're all trafficking in the first degree, which means it's a felony drug, and most of these drugs were prescription medication and methamphetamine. Eight people with active arrest warrants from Friday's Roundup are still being sought by police. They include 57-year-old Marvious Sturtevant of Malley, 58-year-old Iva McIntosh of Malley. And E-Step 25 of Litcar, Kimberly Patton, 34 of Kite, Letitia Anberg, 32 of Malley, 47-year-old Gary Adams of Red Fox, 44-year-old John Adams of Red Fox, and 19-year-old Stephen McClain of Red Fox. Anyone with information on the whereabouts of those with outstanding warrants is asked to contact Kentucky State Police post-13 in Hazard at the number on your screen. This is important to us, you know, targeting drug traffickers is important. That really affects everybody in the community when we have that kind of issue. Reporting in Red Fox and in Hazard, Chris Anderson, EKB News.


Hey, and I'd like to discuss with all of you live migration of virtual machine over CXL memory. And the buff preamble is along the lines of I want to give sort of a high level overview through the slides and then, you know, at the end or whenever I'm interested in brainstorming ideas or discussing. And also, I'd like to point out that I'm just starting to tackle this problem. So I do not have all the answers yet. And another note is a project assumes CXL 3.0 spec devices with shared memory features. And I'm using VM as an example, but this concept could be applied to containers or processes. And so here's sort of the optimistic theoretical goal of the project. Comparing it to traditional live migration, it's basically done through four different steps. We have a pre copy section where we copy the memory over from the source to the hyper to the target hypervisor. And then we cross the VM and we do a secondary copy of, you know, pages that have been hit while the migration is happening. And then we unquest on the on the target hypervisor. The new migration, the goal is to actually use migrate pages between CXL notes and then pass the VM by reference through a hand. So we'd remove like the need to freeze memory, you know, the question second copy, because we're redirecting where we're taking page faults. And also aim to make a VM live migration as instantaneous as multitasking, where VM migration could occur in between allocated process or slices. So for example, the VCP is done with a slice on the source hypervisor, the next slice would be on the type target hypervisor.  And I just want to sort of introduce the basic topology that I'm referring to throughout this. We have a traditional CPU plus memory, new my node, and then we have a CXL device, which is seen as a separate compute less new my node, and those two are connected over a CXL switch. So first step of the migration is to decouple the VM memory from the compute. So we migrate the VM memory on to the compute less node through migrate pages. And then at this point, VM is live and executing and loads of stores from this VCP could be happening on your CXL device, or your traditional new memory while this is going on. And then when decoupling is complete, the VM CPU is executed on the traditional, you know, new my node CPU. But VM memory footprint is on the CPU less new my node. And so all the CPU, the CPU loads and stores are redirected to the CXL mem device. So at this point, the memory is physically stored on a CXL mem device through a switch. And we use like set memory mem policy or similar to prevent pages from migrating back out of the CXL mem device. And also at this point in time, the VM is live and executing. And then this is the second part of the topology. So we throw an additional hypervisor accessing the same CXL mem device where our VM is stored. So we'll call, you know, this new hypervisor, the target and the original one, the source hypervisor. And so then we the next step is the handoff. And this is probably the hairiest of them all. And we are the most dragons, no pun intended or intended. And so the, the step is to sort of recreate the compute portion of the VM on the target hypervisor and destroy it on the source. So in a perfect world, that will be as simple as a process context switch when your processor time slice on the target for the VCPU is done, you're stored in memory, and then the context is simply restored on the target hypervisor. This would require the hypervisors to have a way to cooperate on this shared memory. And there would be a need for some kind of in memory handoff ABI between the hypervisors to hand this VM off. And then the, you know, the final step would be to recouple the VM compute and VM memory. So, you know, at this point, the VM compute is and the memory are still split. What the execution is on the traditional numeral memory and CPU, but the load and stores are going to the CXL memory device. So we want to merge those back, and we start to migrate pages from the CXL memory device into onto the traditional CPU plus mem. And at this point, the VM is live migrating, I mean, live and executing. And then, you know, recouple is complete. So it's fully on the local traditional Luma notes of all your loads and stores are local to close memory.  And that's, that's sort of the gist of it. I've added some information to the slides for people that are interested in following or contributing or helping or have questions and open for discussion. And we can go back and forward through the slides, if there's a particular thing that people want to discuss, and I'm sure there, there will be, you know, many flies in the ointment. Hey, I just stated a comment. Oh, yeah, go ahead. Oh, sorry. Okay. This is John Groves from Icon. So in CXL 3.0, and forward, terrible allocations have a tag. And by default, they will surface as DAX devices that are findable by the tag. And I think the details of that aren't established yet. But so there's a really elegant way to migrate a VM. If you have homed that VM on the tagged capacity, which is the same thing as homing a, you know, putting a VM back by a DAX device. Because that DAX device can be surfaced at another node. So the VM you can actually pause it, you have to make sure the cache gets flushed. And then you can just restart the VM against the memory image on another node. VMs work this way. Processes and containers much harder because you can't, that I know of, back a process for a container by like a DAX instance. Yeah, perfect. That's good information. I'll actually, you know, a little bit on this. I, I wasn't aware of this. And so awesome suggestion. Cool. Yeah, I mean, so, yeah, you don't have the opportunity to not even move the memory on this. If it's, if you can back something like a VM by effectively DAX memory, if, but the, you know, processes and containers are still a charter. There is probably some performance implications involved with that. But I would agree that's probably a pretty clean mechanism to start with. Yeah, I mean, you'd be motivated. I'm sorry, one more thought. You'd be motivated not to have hardware cache coherency enabled in this case, because, because that's expensive and you're not really doing something that you need hardware cache coherency for, except at the handoff. So you would have to, you might have to do something that makes everybody really smile. Lots of right back invalidates in order to get everything to be guaranteed that everything from host one is flushed before host two starts. Yeah. And so this, yeah, this would be kind of the mechanism of sort of for like a better word, can in the VM and then rehydrating on the secondary host, right? That's why you wouldn't work for like processes or containers. We have a question in the room. So yeah, this is David, I was just wondering, like, what do you think about like past through devices that would be using VFA or whatsoever and pinning the pages such that you cannot migrate them? And how would that work with like, in your setup, or don't you just not care about like past through devices? Yeah, yeah, that's, that's the case. We serve about 200,000 customers, but we don't do forms of past through. So definitely it's not something that you'd be able to use if you're using past through. Thanks. The piece of this that we can't do can't do today, though, that was interesting. That was this idea of starting off in malloc based, normal, typical DDR, and then migrating to see Excel and then migrating back out. Do you see that as a primary use case? Or do you think things can just live in CXL all the time? In, of course, this is kind of everybody uses their own tools differently. But the sort of use case that I'm thinking of, these CXL devices are used primarily for migration in between hosts. There's a, you know, in my slides, I have extra slides at the bottom. There's a, let me just switch to that. So I have, you know, clustering and this sort of load balancing and, you know, that, I guess, that's what relates to your point. So if, for these slides, anybody that wants them, you can just go to nil-migration.org and scroll all the way to the bottom and you can download the slides. I'm sorry, did that, did that answer your question? I guess the question is more along the lines of, do you think that's a primary use case or, or, or you're just designing this based on, like, is CXLs only a transport for migration and not, not a place to pull your memory and have, have consistent, persistent access to it? Yeah, so like first use case is primary migration. Yes. And that's the, the case that I'm focusing on. The case where you persistently have VMs there, you know, which was, I was saying in the clustering or the load balancing, I feel that is a little bit of a more difficult and advanced problem than the live migration. So if you have the live migration work seamlessly, then that secondary use case scenario where these sit in CXL memory only or exclusively becomes much easier. Just a suggestion, probably instead of doing pre-copy cycle with a, essentially you demote every VM pages to CXL and then promote them back. So probably it would be more efficient to do a cycle of post-copy and then only the cold pages that remain is still not copied using the post-copy migration. Demote only them to CXL and then start moving the VCPUs from one host to another.  So you mean to basically do the migration over that work and then just synchronize that small piece over CXL? Yes. Okay. Yeah. I mean, that's definitely an option. What I'm trying to, what I'm sort of the goal is to have the VM running constantly and decrease the amount of steel that the customer experiences by doing live migration. So I think if we're going through CXL, the VM can be executed most of the time during this migration and if we're transferring things over the network, then that's, I guess, not necessarily true. Just now, regarding to the whether we can make CXL memory persistent for a guest, I was wondering whether it would be a very interesting use case. For example, if we have a memory tiering already for a bad metal. So we have DRAM, we have CXL always distance with all the no matter latency or throughput information we can make a decision. It means we have a mixture of memory types on bad metal. And actually, I think it may be to be interesting to have similar pattern for a guest so that we have a large chunk of CXL memory directly assigned to the guest without, I mean, permanently. And also some part of DRAM, maybe a few gigs just to put the kernel pages and important information for the guest. It looks similar to a bad metal in this case. And when we migrate, we don't really need to do new my balancing to do the extra step. Basically, what we need to do is to just deliver the CXL assigned memory to the other host. In the meantime, we just migrate a few gigabytes of the DRAM. I was wondering whether you have thought about that or maybe it would be a good idea or not. So this is, if I'm understanding correctly, with like CXL 2.0, where you would be unplugging or binding different logical devices to virtual ports. Do I understand? My question was whether we can just have part of the CXL memory being part of the guest memory permanently, because it's really, to me, it's really a similar model to when we have CXL memory to the bare metal, because we have some of the DRAM, some of CXL, and we have to make a decision. So just a comment on that, I think at S plus 23, there was a paper on doing exactly that. So you would predict how much cold memory your guest has, and you would give it like real DRAM and CXL memory and expose it as that to the guest. So the guest would be using auto-numa or whatsoever. Yeah, thanks. I think that just that it can avoid the Numa balancing for most of the memory. So it's probably good. So can I ask you a question? Yes, go ahead. And just sorry, everybody, I seem to have a lot of like buffering with the sound, so I am not hearing everything. So I might have to ask you to repeat. OK, so we've been actually working on the M life migration of what is aggregated memory, not CXL, but closely related IBM technology for the last year or so. And we have a working prototype and we've evaluated how it works, like from just in one direction. And it's actually quite astounding. The way you describe it should work. It will work this way. And then we've seen that the performance compared to traditional migrations, especially concerning the performance of the workload within the virtual machine is a great benefit. So CXL and this aggregation as a concept here is really a great benefit for this kind of tech technology. And we have extended QEMU on IBM technology to implement this. Yeah, and so my goal right now that there's no CXL 3.0 devices. So I want to get to a point where I can emulate a CXL 3.0 M device and attach to hypervisors and just try to get this basic migration going, which is between two instances. Well, maybe we can chat afterwards because you might be interested in looking at our setup. It's not the exact same. Yeah, absolutely. Please, my contact is in the slides. And yes, I would be glad to check that. Ask Andreas, is your solution have kernel components that are upstream or is it done without that? We don't modify the kernel at all. It's just custom firmware and then we extend the just QEMU. Since QEMU, you can pass a backing file for the memory and we expose that it's aggregated memory through a character device with that support. Also, yeah, we do have a kernel module to provide this character device. Yeah, so that is the only change to the kernel we make. And then you can on both sides of the migration use shared memory devices for the virtual machine memory and then the rest of the migration works basically off the box. This should actually be doable without modification to the kernel at all using shared memory devices in QEMU and QVM and you just you get back that by a remote node and bind everything to socket zero and have the memory on socket one if you wanted to increase building the change span to the model of the Excel device. For example, I know some other people have already done that model to Excel devices. I have a quick question and we brought this up on the list a little bit. I was curious, but we thought we're about and I know we have like two minutes left. So the path by copy we're moving from one host to another host and that means the virtual to physical address. Mappings are going to be different, at least in theory. There's no guarantee that that's the Excel memory is going to be mapped at the same physical address range. I was curious what your thoughts were on how you might accomplish that. Yeah, and that was the so I guess to kind of scale it back for just the basic test of having, you know, this running through QEMU you can have those addresses match. And like we talked on the mailing list, the devices can actually present the same mappings, but as Jonathan pointed out, is that you shouldn't rely on that and that you would need to make these sort of adjustments between the physical spaces. And I don't I haven't solved that problem yet. And my first pass would have been to just make those in QEMU match just so I can sort of validate this approach and then deal with these readjustments later. And I also need to dig a little bit deeper into the CXL spec to make sure I fully understand the issue. So sorry to disappoint Gregory, but I haven't figured that out yet. There's one more follow up question, which is for moving the actual context, the execution context of the VM from one host to the other. Was the plan to use typical TPP or whatever interconnect or were you going to use VXL memory to actually move that context as well? So my if this may or may not be possible, so but my goal is to be able to pass that that through the CXL device. Because I think that can be achieved, that secondary portion of, you know, clustering hypervisors or load balancing when you're running X amount of vCPUs from one VM on hypervisor A and X number of vCPUs on hypervisor B kind of becomes free. So that's my sort of preferred method. But at this point in time, I don't know yet whether that will be possible or not. And if not, then we'll have to go the other route. So linking these two things together, if you were to dump it all to CXL memory, you could also dump the existing page table contents to CXL memory along with the CXL based memory address whatever memory you're using. And then on the other side, you can go through and fix up the page tables as needed. And that would potentially solve the problem. Right. I mean, you could do the transport of the the context through any medium and do the exact same thing and going through the page tables would likely expensive. But that would be a first at depicting that general issue. Yeah. And that's for for pages, right? But not for sub-page allocations, like any kind of kernel metadata structures or anything like that. Yeah, that would be that. Yeah, that would be the I would say the hard part. Yeah. And that's that's kind of also what I'm trying to figure out is, you know, migrating the pages will probably be the easiest one. But what do you do with like sub-page allocations?  Or like it's going to be very difficult because I mean, these addresses could be anywhere. So that's that's the part that I am still struggling with figuring out whether this cooperation between the hypervisor happens on a page level or does it happen even on a smaller level? I think I'm over time. But is there any other last question? Well, give me a little bit more time because we had trouble getting you started. So we'll go to the two thirty five mark. One observation is I don't see you asking for different ways to identify CXL memories. Like it seems like the current and for this use case, is it a correct observation that you don't need CXL nodes and these kinds of things. You can figure it out from the information that's currently available in the kernel. There is an so my goal is to have the switchable CXL memory devices be their own nodes and then attached a separate sort of sub-tag in Sysifes just for the purposes of migration and figuring out your path of migration. So if I if I switch to this slide, this is in the extra slide. The way I'm thinking of is you would have multiple CXL memory switch devices connecting to various hypervisors and there would be some sort of application on top sitting that would map out this migration graph and determine which nodes you need to pass or if you want to quote unquote exit the main space and call up onto another rack, which node you have to go. I think I think that's the end of our session. I think we lost you at the end. But but yeah, I think thank you. And I think we'll move on to the next session now. Excellent. Thank you so much.