Hello everybody and welcome to our next talk. SETI is a senior software engineer at a big company who will introduce us to some principles why not to get woke up by emergency calls that are totally unnecessary at 3 o'clock in the morning. Please give a warm round of applause to SETI. Hey, I'm SETI and welcome to my talk Understanding alerting and how to come up with a good enough alerting strategy. So first up I'm SETI. I'm working at a big tech company and I'm sure you never guess which one. I do a lot of analog photography and while everyone else was baking bread in lockdown I was starting to train Brazilian Jiu Jitsu. So don't fight me. Okay, so today I'm going to talk a lot about alerts and incidents. So that's why we very quickly need to distinguish between what's an alert and what's an incident. So I'm not confusing anyone. So first up an alert is basically an event that is generated by your monitoring system. Every time your monitoring system thinks, oh that's interesting, let's get someone to look at it, that's an alert. But what you as the operator will receive is the incident. So an incident is basically always starting with an alert or most of the time starting by an alert. And then you just going to look at it and you work with it and you try to resolve it hopefully as fast as possible. So let's start with a question. Who thinks that incidents are a good thing? Let's show our fans who thinks incidents are a good thing. That's not that many. Interesting. Okay, to actually understand if incidents are a good thing I'm going to introduce you to something that's called the line below the line framework. So let's basically start out with our system architecture. So just in regular ordinary large complex systems designed it consists of several components. We have multiple servers running behind layers and layers of load balancer routers. We have databases everywhere and just our typical system. And that's what's actually creating value for our company. But in order to build and operate and run the system we need a little bit of supporting infrastructure around that. So we need stuff like our Git servers, our CI CD pipelines, our monitoring system. Even the gyra you use to track your work items that's all supporting infrastructure and someone needs to maintain that. And then we have lots of people interacting with those systems. We have the devs, we have the SREs, we have program managers doing whatever program managers do. But they all going to interact in some way or another with some parts of the system at least. And the worst thing about this is everyone got it very differently. So I mean just ask two different developers in your company to explain to you the systems design of the software. And I'm pretty sure you end up getting three different architectures. And that's true for everyone. So PMs got a very different view on your system than the devs do. The SREs got probably a very different understanding of how it works. And I mean we are all quite familiar with that, right? And that's because everyone got a very different mental model. And if we combine that we get this pretty complex looking diagram but it's actually quite simple. So we have our system that actually generates the value. We have more systems around that to support them. And then we have where the actual work is done with the people. So you can't really touch the system. You can't see the code run. You can't see your database actually inserting something to it. You have a mental model of how it works.  And that's what the green line is. So the green line is our line of representation. You're never going to see a server being deployed. But you know that it is deployed because you interact it with some system that deploys it.  But it's never going to be the entire system. No one understands the entire system. Everyone just got a tiny fraction of the system as a mental model.   But you interact with some sort of representation of your system.  So what are incidents? So incidents is the delta between how we think the system works and how it actually works. Or more precisely how it fails and how we think it might fail. And based on how we think it might fail, we design around that. So let me ask again, are incidents a good thing? So who thinks incidents are a good thing now? Yeah, that's a lot more. So yeah, incidents are in fact a good thing because we need them to update our mental models. If we don't have incidents generated by our system, it's very hard to keep track of how this system actually works and keep our mental models up to date. But are all incidents the same? I don't think so. And most of the incident management tools that are out there agree with me. And usually you've got these five severities of incidents. And they might be called a little bit different depending on the system you're using, but you have like those five broader categories. And they are categorized as an alert as a record, alert as a notification, alert as a page, alert as a page, and it's actually serious. And everything that could go wrong actually went wrong. And we would never want to end up in the SF0 category because that's just awful. And we can divide those five incident categories into we notify someone, these are lower two severities, or we actually call someone at 3 a.m. in the morning. And I hope no one really wants to get called at 3 a.m. It's just not nice. And you can imagine those severities as a SF4 is probably the same as in Recruiter reaching out to you in LinkedIn. It happens a lot. You never really look at them. But if you're actually looking for a new job, it might come in quite handy that you can go back to your records and see all those five recruiters reach out to me in the past two days. Maybe there's something to them. A SF3 is probably the same as receiving a message on Signal. You're not going to answer it right away. But you answer them once you get to it. And the SF2 is someone is calling you and you pick up and they say, okay, there's definitely something going on. And SF1, that's a whole different story. Like that's someone screaming at you on the phone saying, oh, God, this is so important. Something really, really bad happened. Well, we haven't really talked about the SF0 category yet. So I would argue that no automated system ever should create a SF0 automatically.  And if you want to have like a step up to that, if something really, really bad happened, then you have to manually raise either a SF0 incident or promote a SF1 to a SF0. And trust me, you don't want to be on SF0, because that basically means your C-level executives, they're already in cause with PR agencies, with the lawyers, and something really bad happened. So let's talk about how we actually can detect those incidents. How do we detect that something went wrong? So basically, we are doing this using service level objectives. And every time a service level objective is breached, we alert. Easy as that. But what's actually in service level objective? So if we look at the Google book and what Google wrote in their book on SRE, it says a service level objective is a target value or a range of values for a service level that is measured by a service level indicator. Okay, but what are service level indicators? Okay, same page, finally. Okay, service level indicator is a carefully defined quantitative measure of some aspect of the level of service that is provided. Still not much better. Let's make an example. I'm standing here, I do a presentation, and it would be great if I actually be here and do this presentation, right? So I'm checking if my heart rate is somewhere in between some values. So my heart rate is actually the service level indicator in this example. And the entire statement is the service level objective. And as long as that service level objective is true, I'm probably alive. I hope. Okay, but now that I get this notification from my Apple Watch and it says, oh, your heart rate is raised. It's probably above 175. It's a bad thing. I mean, I'm still here. I'm still speaking, so it can't be that bad, right? Because I'm doing a presentation and probably I'm quite nervous about it. So high heart rate is not that bad. It's just vital. It's just diagnostic data. And there's a rule of thumb that you never ever alert or call someone based on diagnostic data. And instead you alert on actual symptoms. I mean, if I trip over that wire here, fall down the stage, probably heart rate is still raised, right? But that wouldn't be so good for my talk. So another set of examples. Okay, we established that's not a good example. Different set of examples. Okay, I'm present. So I'm speaking, also very good. And I'm not speaking too fast or speaking too slow. So this looks like better SLOs. But I think they could still be improved a bit. And there's a certain art when you talk about site reliability engineering and it's coming up with actually good SLOs. And there is a general formula that shouldn't surprise anyone and that's you just divide the amount of good events by the amount of valid events and then multiply by 100. So you get an even percentage across all of your service level objectives. So all your SLIs are percentages and then you can alert on is the percentage in some range or another. But what does a good event and a valid event mean? Well, I mean, I think it's very dependent on the service you're providing and let's have a look at an example. Just as one example here, we are looking at an authentication and identity system. And for an identity system, it's quite valid to return HTTP 200, 401, 403, just general response codes that we would consider as good events because it would be nice if our identity system tells someone, yeah, you're not allowed to do that. And the good requests, the valid requests, they are all the requests that are in the range of 200, 400, 500. Because if there's something in the service goes wrong, we probably throw in 500 into the server error. That's still a valid event, but it's definitely not a good one. But in this example here, we just completely ignored the entire 300 range. And that's because we do not care about moved permanently. It's just no one cares about it. It's not an indicator if something good happened or something bad happened. So that's why we can safely ignore stuff like HTTP 301 moved permanently. But in another example, if we would look at a redirection service like Bitly or the HINY URL or something like that, then those 300 requests, they would be considered good events because that's basically the service we are providing. So, again, the good requests are all the 200 and 300 ones and the valid requests are basically everything else. But when we talk about service level objectives and service level indicators, there are like two distinct ways how we can come up with service level indicators. And for us engineers, it's probably more natural to come up with service centric SLIs. And service centric SLIs, they are usually stuff like, what's the queue depth of my message queue system? What's the object age in my cache? What's the cache hit ratio? What's the availability of the database subsystem? All those service oriented metrics. And they are certainly valid and good and we need them. But if we look at what product managers usually want, it's very different. They don't really care that much if my database is available. My customer certainly doesn't care if the object age in my cache is getting too old. What they want are customer centric SLIs. And that's what's also commonly known as end-to-end tests or end-to-end measurements. And they usually contain stuff like the error rate of an API, the availability overall for a service. And it's usually measured using outside in monitoring. And it's very, very important to have both types of SLIs. We need to have customer centric SLIs because they're probably much better suited for detecting that something on a regional level went wrong. Maybe my entire region for the cloud is down. It's usually catch by a customer centric SLI much quicker. But the customer centric SLI is usually not that good at detecting what part of the system failed, which server failed, which load balancer did something bad. And that's why we need to serve a centric SLIs because they look at the component level and then go from the component to the zone, to the region, to the entire world. Well, the customer centric one, they probably look at the entire world, our entire service, then look at the regions and they don't really get much deeper than that usually. Okay, so we covered how we can detect incidents, how we can detect stuff, and in which severities we alert. But how does the alert actually get to us engineers? How does an alert reaches me at 3 a.m.? How does one wakes me up at 3 a.m.? And that's what's commonly referred to as alert routing. And first of all, I'm going to show you how to not do it. You trust me, do yourself a favor, just don't use Grafana managed alerts. It's the worst type of alerts. And I have to excuse myself. I use Prometheus Grafana, PagerDuty here as examples, but you can just replace them by any tool you want to. It's very generic. So that's just on. Okay, better option would be, okay, let's just use Prometheus alert manager and alert manager notifies me on Slack. I mean, it works. I've seen a lot of systems work like that, but Slack can't really wake me up at 3 a.m. So what's usually done, and this is probably the most common type of alert routing that I've seen, is the alert manager decides, okay, it's a SF2, I'm going to send the alert to PagerDuty and PagerDuty is giving me a call. If it is not SF2, if it's something lower than that, just route it to Slack and someone will look at it. But there's a problem. I as an engineer, I have to watch on two different systems. And there's also quite a disconnect between who's actually working on that incident. I don't see it from the Slack status. I don't see it really in alert manager. I have to go to PagerDuty to see who's working on this. So an improvement to that is usually look something like this. So Prometheus always routes to PagerDuty and PagerDuty routes to Slack. By that, we can have annotated Slack messages that say, okay, I acknowledge the incident, I'm working on it, and it shows up in the Slack channel. But it's still not ideal because I have to calculate the SLA reports once a quarter. How do I do that? I don't want to go through Slack. I don't want to search 110 incidents on Slack. No one wants to do that. So PagerDuty is not much better. I just see on PagerDuty all I can see is, okay, I've sent 10 calls in the past 24 hours, and 235 calls in the past month. It's not going to help me much. So this looks a little bit more complicated. But it's actually not. It looks complicated, but it's a bit more centralized. So Prometheus always should route your alert to a central alertings ticket system. It could be your gyro that you're already using for tracking development work. It could be fancy incident management ticket system like FireHydrant. But the point is you have all your incidents on the central ticket system that you can query whenever you like. You can generate executive reports. You can generate quarterly SLA reports, all that kind of stuff right from the ticket system. And the ticket system makes the actual distinguish between Dustus Alert as in CEP4, is it just an alert as record? And no one will look at it anyway except something bad happened and you want to see if there's a CEP4 maybe lurking around somewhere, then people can go to FireHydrant directly. If it's a CEP3, yeah, we notify someone on Slack who cares. Okay, it's a CEP2, CEP1, CEP0, something like that. Route it to PagerDuty. PagerDuty wakes someone up. And when I, as the engineer, go on call and I get paged by PagerDuty, I know exactly where to go. I just go to my ticket system and pick the incident that's right up there. I don't have to rely on the SMS that PagerDuty sends me on those 130 characters where no incident description whatsoever fits. I can query that. I see my incident. I can work on it. And I maybe notice that's not an incident for my team. That's just not my service. That's a problem with the underlying storage or something else. And I can just reassign that ticket. And as soon as I reassign the ticket, PagerDuty just goes off and calls that person from the actual responsible team. So that's actually way better. And it gives you the ability to query. All right. I'm actually much faster than I expected. So that's 21 minutes. And I thought I couldn't make it in an hour. To my defense, I tried it yesterday and it took me 45 minutes. So I probably spoke very fast. Sorry about that. So just a quick recap, we talked about that incidents are actually a good thing. Don't be afraid of incidents. In my job as a resiliency engineer, I see so many people that are just horribly afraid of an incident. I don't know. To interview people on incidents, that's my day job. I interview people, okay, what went wrong in your incident? And they're already freaked out when I reach out to them. They just don't like incidents. It's very stressful for them, but don't. Incidents are a good thing. You can learn from them. Embrace them. Don't demonize them. Categorize your incidents carefully because trust me, because your disk is running out of memory. There was a very good talk last time on GPN from Momo who talked about that stuff. Yeah, just don't. Someone can look at this during business hours, but I don't want to be woken up at 3 a.m. Define carefully, define SLOs and refine them. Maybe on a monthly base, maybe on a quarterly base. Depends on how early you are with working in SLOs. Just refine them. Do monthly or quarterly refinements of your indicators? Look, if your indicators are still valid because that might happen. I worked with systems that out of nowhere we suddenly had a spike of expiring certificates. And we realized we need service level indicators that checks the expiry of X509 certificates. Review them, refine them and your grant. And last advice, just don't use Grafana manager alerts. I'm maybe biased because I had to work with them and just don't. Use alert manager. All right, thank you. You can find me on Twitter, on LinkedIn, GitHub, whatever. Just reach out to me and I guess it's time for Q&A now. If you have any questions, hand up. Yeah, I'm just wondering about the Grafana alerts. I'm still wondering why it's so horrible. Yeah, so Grafana managed alerts. I mean, they work. You can do everything you could do with alert manager alerts. My personal problem with them is you have to configure them in the UI. I hate it. Seriously, my arm hurts from using the mouse. I don't want to do that. I want to write the alarm definition as a YAML document. It's just easier. I can check it into Git. I can see if someone modifies it. I see exactly when and who and why I modified it. All right, answer your question. Awesome. So my question, my question is about the as a low based alerting. You have a lot of high traffic systems, but usually you also have systems that have way lower, or get way lower traffic. For example, you have some small management service where you create data for another high traffic service or something like that. Usually it doesn't work that well because if you get a single 500, your error rate is already at 100% if the service doesn't get much request. What would you recommend for alerting here? Good question. It's obviously much harder if you are in a low traffic environment. Maybe just increase traffic. No, sorry. We can follow up on this later with like a motto or something, but I don't have any solution of that top of mind. Do you have any examples on an alert 1 or 2 which would wake you up at 3 p.m. that you can talk about? I have to mentally check my NDAs. So the type of systems that I usually work with are cloud products, so it's hard to talk about them without revealing any PII. Stuff that woke me up at 3 a.m. where things like when an entire region went offline for no reason and we didn't really know what was going on. And the fiber finder 3,000 in yellow just dig the hole and ripped out an entire fiber cradle and everything went dark. And we lost the entire region. It was bad. It took a month to restore. So you said it's important to learn from incidents. Well, you mentioned that you call people. So what's your runbook learning start of incidents? So you do have postmortems, right? And one term that I hear a lot is blameless postmortems. You have to do blameless postmortems. But what blameless usually means is sanctionless.  I pulled the wrong cable. I applied the wrong configuration. I applied the wrong sub-mental model. It's still taking blame, but you're not facing any sanctions for that. So that's a very important step. So you have to be sanctionless. Don't punish anyone doing a mistake. It's just human error. And the way that we work is we follow up on incidents and we talk to people, hey, can you give us, all of you, what happened? What did you do to resolve it? And then we just compile that to a narrative and we try to teach that narrative to everyone involved and we try to find up themes that occurred during the incident and then we try to systematically eliminate those themes. So it's not that much of... We work with one incident and just focus on that one incident and we make sure that this exact incident never happens, but we try to work on the themes that occurred during the incident. We try to extract themes, we try to collate themes across multiple incidents and then figure out, okay, we have 10 incidents and all of them involved software stack X and Y, so we probably have to do something about X and Y. Does this help? I'm not sure if I answered it. Okay, thanks. Any more questions? Test incidents and test alerts? Could you describe them a bit? Yes, we do quarterly drills. I can't really elaborate more on them, but it's basically chaos monkey style. You just plug random...  I can't go into the detail, sorry. I was wondering, I'm working more with electrical engineering and some scientific stuff where you also have to monitor things like temperature, flow rate, things like that. I was wondering if a similar model of alerting would be useful there for long-running things or if you have any experience with that? I don't have any experience doing that, sorry. I work with clouds. I don't know much about electrical systems. Sorry. I guess you also need an IT service management concept or framework like I tell. Is it still valid with all these incident problem management, capacity management, asset management? Yes and no. I very much dislike the concept of ITIL. It's just way too bloated. I don't like it. Some concepts in ITIL are very good like the asset management part. Yeah, it's crucial to know the dependencies of a service. If my storage system goes down, I want to have an immediate graph of which services are dependent on my storage service so I can inform them so that I can inform my customers that depend on it. But if we stick to what ITIL specifically does, I think it's just way too bloated and way too formal and I think it's written by people who don't know how IT systems work. At least modern IT systems. It might be a good thing for the 80s, but we are not in the 80s anymore. Sorry. I hope I didn't hurt any feelings. Okay, yes, it sounds very good. Very open to that.  Can you think of other ways? Yeah, brownies. Or whiteboarding, whatever. Yeah, whiteboarding. A lot of brownback sessions like I don't know, probably more known as lunch and learn. You just have lunch together, teach your peers something that you know but you think they didn't know. Just go on lunch with your co-workers. Talk about how you think the system works. Go to your senior engineers. Go to your principal engineers. Let them explain the system to you. Another very good opportunity to learn is when a new hire starts in your team. And it's especially good if a junior new hire starts in your team because they don't know what you're talking about. So you have to explain everything very, very carefully to them and then you might end up learning new things that you didn't realize before because you have to go in so much detail about it. Yeah, just some examples to that. So, thanks for your talk, Cedric and maybe see you at a trunk, at a mart or at your next talk. Awesome, thank you so much.


Now I would be remiss and cheat you guys if I didn't talk about the other Element of what I teach which is the intergame part So let me tell you a story here to introduce it.  He says I Don't know what you're doing with all these other men, but it's not working with me my name so-and-so He lived in 20 minutes does a couple of my patterns 20 minutes later. They're making out like crazy He takes a break looks around and people are like What is this guy doing? He figures you're gonna play it cool mistake number one. He didn't try to get her out of there the same night He didn't get my memo He gets a card with her phone number that has her cell number on it He figures I'm gonna be a cool dude I'm gonna make her wait Waits three days three days come by Sounds like the story of Christ after three days Ransacks his desk can't find the card Goes through his condo can't find the card anywhere.  It's dumpster night.  My old self came back I've been depressed for six weeks. I can't even talk to a woman.  I? Have a question for you guys Where was his old self when he was making out with that woman and getting the envy of every guy in the club Where did his new self go when his old self came back Pismo Beach, Jerusalem, where did it go? Come on. I want some answers anybody Huh subconscious anybody else it's still there The old self was still there when he was making out with a check Okay, I'll take that one more answer Yes, the new self. Oh Interesting I'm going to give you a b-minus on that but not bad Here's what I think say again Who said that? Very good answer. That's an a-plus Very good answer. Here's my idea Here's the really interesting thing for me as a hypnotist. I asked myself the question How is it possible that he hallucinated a way that he gave himself an amnesia for everything? He did right because he did a lot of things right that evening. He prepared himself, right? He got dressed properly. He worked on his attitude. He made the approach Introduced himself right did the language patterns got a great make-out, but in his mind all he saw was failure and my teaching to you is That he looked at his progress through the pain of his past He looked at his progress through the pain of his past in an area of life that has Pain and confusion and trauma and by the way half of what I do is Healing work with people a lot of my seminars. I'm doing a lot of covert healing work Lot of guys who come this community have a lot of trauma that needs to be gotten out of the way It's not about them being beta males.


I think I'm looking for Baraka as well out of Tweetie here. It's a character that I like myself, my slashy boy. Very hype character. I mean, he's very good. Very, very hype character, very strong character. He's a strong character. He's not a particularly wild mix-up character. He's just solid, tons of damage, can pester you. Now he knows. He's good. He has seen what Samish can do to gear us at this tournament already this weekend. Baraka might be a very smart pick to change things up and Baraka's projectile and spacing game. I think he can go toe-to-toe with Katana any day of the week. Blade Spark is very fast. Yeah. Metal Recovery, almost the same as Phantom Katana. It's fast, yeah. So it's going to be kind of the same pace there and it's going to come down to can Tweetie stop the approaches of Samish or is Tweetie going to be the one to be aggressive, which is what he's known to be. Sure, sure. But yeah, I think you're right. The character definitely can play a strong mid-range game in the match-up. So here we go. Tweetie off him for no headphones. Here we go. Second match. Winner's side top bait. It's Tweetie versus Samish. Let's get into it here. Oh, dude. Patient is Tweetie trying to comment with that extended jump in. Katana has one of the best down threes at anti-airing. One of those tools that I talked about that Samish looks for in a character and he's going to enforce that throughout this match. Wow, the knockdown comes in. You can see it's really an intended approach here. The giant button jumped down one in the sky. It's huge. Needed to get out. That would have been so much damage. The Tech turns into town three. That's fine. Speaking of giant normals, Morada Baraka giving up that extended jump one range. Absolutely incredible. Looking for those big reeds. Easy hit the berms and the massive damage with that gutted. Oh, no. It said his game failed. A throw. Either direction is going to be a crushing blow. Now you're in trouble. Looking for the flutter cancel. Trying to bait Tweetie into pressing a button. He was going to have some frames on the ground that were negative for his way out. Not only is the great footsie tool, it's great mobility tool as well. Absolutely right. Yeah, it can even be plus on block. He tried to get something there. It didn't work. Samish is going to get a punish. Oh, he's trying to stay alive in this round. No. And so he came off the two defensive bars for nothing. Very aggressive is Samish right there.  Likes to be in the face. Likes to be making you guess in his pressure. What's something that you're going to look for right now? Are those jump kicks going against the down three of Katana? So far, about 50-50. Because Brock and players aren't necessarily used to getting anti-air. You're right, yeah. And Samish is going to be the one to do it. He has some extremely strong buttons. Wow, neutral jump. Yeah, you can really tell these guys play all the time. Some of the tendencies have been read very well by either side. Yeah, there aren't many that have the stones to do a dash up into someone's face and just neutral jump, empty jump. Looking for a whip. Throw it down. Getting it down three. No chop, chop. Oh, punish. Oh, no. Into the corner. Here we go. And Katana not known for damage, but in the corner. Going above, I believe it is 11 hits. Rain hits, not too sure. We'll get a crushing blow. It's a massive damage. This is going to be chipped out with a chop, chop. He put it in. In his face. Are you kidding me? Not easy to do. Of course, he can continue chop, chop. Wow. Okay, little down three comes. Fast tool. I'm almost shocked that Samish didn't simply hit confirm into a fan lift and go for the fatal blow in that situation. He made maybe a miscalculation. Who knows? Oh, this could be huge. You need to get out of there, Samish. He does. But already good damage taken. Boom. That would have been a bad day in the office if that fan did not show you to the corner. We go on the air escape. Trying to sandwich and jump out. I'm making a read on the throw right there is 3D. Giving up that meter. He's going to be much less dangerous if he finds a hit. Trying to get over. There it is. Boom. And that is among the best jump kicks in the game. But Katana did the trick as the crush comes out. Wow. Wake up. There's no escape. Oh, big damage on deck and you are going to be bleeding right now. Somebody call an ambulance. She's still bleeding. Look at the live up there. It's going away. Tried to flood her on top. No answer. Did not take the bait there. Tweety. Fatal blow locked and loaded for both of these guys. He tried to beat him in the sky. He got out of there in time. And the champion. Absolutely incredible. First game between these two. If that fatal blow came any sooner. Would have sniped him out of the armor. Would have been 1-0 Samish. Not the case coming up into the pressure. Into the trap chop. Last breath does not matter. Tweety striking into game number one. Starting out. Look at that knee. That frustrating knee. That forward four. What we like to call auto neutral from the beta. Back in the day. Easy. Hit your firm. That max distance shot from Tweety. Getting out of the air escape. You're still going to lose some damage.  The pressure is on. Oh in the. Oh this is trouble. Big trouble here. And he can keep it on chop chop. There was a lot of damage up on that combo. We just didn't even need to see it. Tweety looking strong here. And set the tone. Of this match. Aggression paying off. Can he get something going here? These Antares are definitely going to help. Oh my goodness. If that doesn't tell you these two are training partners I don't know what does. Yeah exactly. Surprises the Nazi the breakaway. Instead Tweety now is going to have a little bit more control. Slight life advantage. The Antire comes underneath the jump down one hard to do but he got it. And he knows that timing. You can really see how often they play. Empty jump and the pressure. And that's the mind game they've been playing on. Wake up Simon. Wait a minute. Oh my goodness. That was after breakaway as well. So those two bars are out of here. And now Tweety with a life advantage. With the meter advantage. And he's in the great spot to meet. What's he going to do? What's the call? No defensive bar. He can simply chip out. And the damage buff was there. Yeah. On the flag. Just some added flair for Tweety here going to 2-0. Oh there's the DJ team. There's the chop chop. Bring in the height force here is Tweety. 2-0. We saw someone in the same exact situation in game number one. He's going to turn things around. That's true. It's not over. Although there's no character switch. It's just going to be Katana but that doesn't mean that you can't still do it. Escaping just like that. Tweety did not want to see the rest of that one. I've had such joy watching these guys handle how their opponent wakes up. The distance that they make. Whether they're going to pressure. Whether they're going to make a read on the jump. Read on the wake up. They haven't really woken up. It's either with buttons or jump which is crazy. I think about this high level. Anzai was there. No keep up. Threatened with the flood or nothing. He didn't believe in it. He's had the full splots but not too sure if Katana has the range to actually get the punish on the up to or up three. He gets the grab. It's very close now. There's no defensive bar to avoid. A chip kill right here. And so of course Tweety goes in at its match point. That was so smart by Tweety. Chop chop is if you guys know for a moment multiple hits. If you have any defensive bar on anything that will chip you out.  Will save you from chip death one time. If you have two bars it'll save you twice. But chop chop. It's going to chop you to death baby. Okay. Escape failed. Gets the knock down there on the second plan. The amplified version. Mixed by the amplified version there. It's rare that you see Marauder Brock come up both just like that. He's going to be much less damaging if he finds the hit. No gutted amplified. Coming in from downtown. Beating up to startup on the forward four. Not easy to do. Very startup. Fast startup. We've got 14 frames right there. Wait a minute.  We go but in striking range distance once he walks in right there. Forward four range. Back two range. Chop kick range.  Samisha's still in trouble though. Still match point. Tweety is trying to slow it down a bit for himself here. Just controlling things. You were talking about Bladespark. It really is one of the better projectiles in the game. You don't think of Brock as being his owner per se. But he can't handle the mid-range against a lot of the cast. It really has helped Baraka against these heavy zoners just to set the tone and find his way in slowly but surely. And the amplified Bladespark out damaging most projectiles in this game on trades. Yeah. So it is very good for him. Oh my goodness. He was ready for the tech right there but I like how Samisha slowly trying to approach. He ends it early this time. That is the mix of chop chop which is in this community known to be one of the most unpunishable things in existence. Pretty annoying. Has gotten punched a couple of times so far from Samish but not a big punish character. This Katana is going to take a bite of that arm back to the full screen. We go and there's the trade of the amplified Bladespark. So much more damage than that fan. Oh, the trouble comes. Even that at this point is a big deal. Okay. He didn't press the second button there. Chip is a big deal. Look at the patience. Is everybody's video okay? Throw. He needs to get back in. Oh my goodness. He's not going to get it. It's the boot to the face. That heal comes for the heal up on the stage. Tweety takes it and moves into winners finals. Where he will face Scar coming in later this evening in this top beat. Congratulations to Tweety. That was a dump.


So we are waiting for marriage to do, but in the Bible, sex is considered penetration, so we don't do anything relating to penetration.  Is this a biblical statement? To find out, let's go to Matthew 5, 28. But I say to you that everyone who looks at a woman with lustful intent has already committed adultery with her in his heart. To lust is to think sexually about someone that you're not married to. Believe it or not, there's no exception given to boyfriends or girlfriends. Jesus is not just concerned with your physical activity, he's concerned with your thought life. For this is the will of God, your sanctification, that you abstain from sexual immorality, that each one of you know how to control his own body in holiness and honor, not in the passion of lust like the Gentiles who do not know God. Seek to develop godly boundaries with your significant other. This isn't God just trying to ruin your fun. He truly has your best interest in mind.


Freely the banana girl. I don't know about you guys, but I know plenty of banana girls in New York City And this girl originally got famous for promoting a diet eating 20 bananas per day and eating up to 51 bananas So very cult-like way and also she is a nudist that lives in a South American jungle So some sort of weird following here, but we're gonna look into how contradictory Her diet is not only saying she's off-grid and buying things from the supermarket But just how unrealistic it actually is to be on a vegan diet and the lack of nutrition Specifically in raw vegan diets. So today's video is a what I in a day off-grid vegan video So we're gonna start with orange juice and these oranges do look a little bit green, but they are actually quite sweet inside They could be a bit note quite sweet inside It's sweeter, but we don't have a lot of fruit at the moment. So we've got to make do with what we have So when I have orange juice, I don't muck around I have a lot of orange juice because that is my meal. So this is my morning meal This is my breakfast. So I need to get enough energy enough Sustainance to keep me going to the next meal So I will have anywhere from one and a half to two and a half liters of orange juice in one meal She just said we don't have a lot of fruit and then she went to say I drink a lot of orange juice Am I missing something here? According to historical data oranges actually have been eaten for over four and a half thousand years That is a long long time That is a long long time Not in the context of evolution and foods we have been eating Four and a half thousand years is nothing and I mean that's even more recent than rice. That's that's pretty crazy to think That someone's going to use forty five hundred years to justify consumption of a food very unrealistic They are a great source of antioxidants. They help boost your immune system They help improve your circulation blood pressure reduce inflammation lower cholesterol levels Drinking orange juice is absolutely a great way to start the day. So as you can see here I'm getting up to two liters there and this orange juice isn't like super super sweet So I like to add a little bit of coconut sugar Stir that around stir it in and just brings up the richness of the orange juice and it is healthy in itself Don't let anyone tell you coconut sugar isn't healthy for you Not only is two liters of juice not sweet enough for her She has to add sugar to it and then say oh don't let anyone tell you it's not healthy for you This is completely nuts guys Not only that she says all these benefits of orange juice.  Are you living in an orange grove? Like what's going on here? So then I would take my orange juice and go and edit some videos actually editing this video that you're watching now and Yeah, just drink my orange juice while I'm doing my editing and get the morning started Many of you have asked me what movie editing software I use I've used heaps of different ones over the years I found a cup pro Mavavi screen flow, but now I'm using Adobe Premiere Pro Robin actually put me on to it and I would say it is my favorite editing software so far Okay, so now it's lunch and I'm gonna have a chocolate mint milkshake. That's right. You heard right chocolate milkshake Is this like what a five-year-old would eat in a day if they could choose their diet and The base is going to be dates so I'm pitting the dates and I'm having about 20 dates that's how many I usually have and then add some water at about 1.4 to 1.6 liters of water and Next comes the organic cacao powder and you might want to just start with a teaspoon or so I put a little bit too much in and I don't usually I even have this recipe But I thought I'm gonna go a little bit gourmet today and there's a drop of peppermint oil Lives off grid Has electricity has running water. She's not getting this water from a stream to put in this milkshake And all of these foods the dried dates the cacao powder the peppermint oil Foods that she's obviously not getting locally. I'm sure this girl could do this stuff locally and eat local foods But that's not what she's doing. And I think this is very misleading Just blend it up the choc mint flavor is just incredible It's really really nice But I wouldn't recommend it every single day because cacao can be quite stimulating but here and there You know, it's fine. It depends on how you feel. I feel like I'm gonna shit my pants looking at this You can't be serious you cannot be serious this girl literally Buys all this stuff in a supermarket Blends it up with electricity From her tap water takes the blender walks out into nature on a log Dressed up like she just escaped a cult. What is this? Is this like? Who watches this seriously who supports this? This is completely Nut job stuff like completely batshit crazy How could you go out in nature holding a blender in your hand on a log? That was clearly cut by a sawmill and then vegans have the nerve to say oh You mean teeth and canines and fangs to hunt animals like seriously You think a blender is less advanced than a deadfall trap then a wooden spear then a knife I'm losing my mind Looks really comfortable sitting there Real happy drinking that huh okay one thing I've noticed when I used to be a bodybuilder drinking a lot of shakes is Consuming liquid calories is very stressful on the digestive system because there's so much excess liquid you pretty much have to pee it out So if you need to drink two or three liters or well up to like a half a gallon a gallon of water or liquid form Like even dairy for example It's very stressful on the digestive system to get that many calories in because humans would never have had access to That amount of calories in a liquid form so Realistically speaking you would never consume more than a cup or two of liquid from any source of calories in any meal ever This is excluding water. So it's very difficult to justify Especially people consuming such large amounts and I can't imagine how many times you use the bathroom that during the day It must be insane So let's go on to the dinner recipe and I know some of you are probably wondering why I put in the title goodbye cook food Well, I'm thinking about doing a video about it. I am going all raw again and Notice how she said I am going all raw again raw vegan is Even more nutrient deficient than regular vegan and you'll notice in many vegans and vegan youtubers They try to go raw and then they end up back to cooked foods. They're not getting enough calories They're not getting enough sustenance. They're barely able to do it on a vegan diet Let alone a raw vegan diet and there's people that for some reason think a raw vegan diet is gonna detox me It's gonna cure my ailments. It is literally starving yourself to death. These people don't understand They think a vegan diet is healthy and then oh raw vegan. Obviously. It has to be healthier. No It's it's completely misguided conventional wisdom that fruits and vegetables and plant foods are healthy is really killing us It really is I'm not gonna go through the rest of this video. It's just hurt literally like eight minutes of her going through this recipe you know she adds Like she adds like seasonings and spices like garlic to muric Ghost chili pepper lemongrass obviously things she didn't grow in her backyard and she had the nerve to say Fresh from the garden like she grew this stuff herself and then she puts like She holds a coconut with the machine and a glove on She's showing all these foods like plantains and eggplants and yams And things that she could not have grown on her own. I remember her saying in another video She did that she tried growing eggplants and the enders or maybe yams and then the pigs literally ate them classic example of Not having access to plant foods in nature and then she pretty much adds the whole force to this meal mushrooms a bunch of other things and You know adding artificial seasoning salts coconut various things that to make it more palatable because vegan food is disgusting and The irony is that you know She puts these things on like a wooden board like the chickpeas and lentils like she didn't buy them in a giant bag And then she goes to cook it in like this iron steel pan that was forged by modern engineering. So I Guess the last thing to touch on is just look the crazy amount of food they're eating and the volume of food This girl is eating throughout the day is insane. I think she's at least eating six to eight pounds of liquid and food so another funny thing about her is When she goes to do like Anything in nature like besides washing herself She ends up Let's see if I could find it She's wearing a hat gloves like she's suited up completely suited up to be in nature And then when she like goes to take a shower and stuff then she takes her It's just so out of touch with reality. You're pretending. You're in nature.  He's doing everything and literally the most natural way possible Definitely out of touch in regards to her and if we look at fully raw Christina She just literally has like an unrealist. It's like a farmer's market in her kitchen She has so much produce that would never be available at all parts of the year in all climates This is just completely unrealistic and crazy and you know My detox experience when I first went fully raw vegan another example of people going raw vegan and not being able to stick to it and just this this girl in general is This is completely if you look at her fridge like the amount of produce is it just so Unrealistic and so out of touch with reality I just found a lot of irony in vegans trying to act natural that this is how humans supposed to be and All the methods and things that they have to actually use to get nutrition from foods, especially plant foods Also, you know the high sugar consumption of the diet the lack of just fat soluble vitamins in general And yeah I mean a lot of foods that vegans eat and that these whole foods plant-based vegans eat might have been present in some blue Zone diets, but what these vegans are missing is the very high fat soluble vitamin consumption from super super high quality animal food So thank you guys for watching Where's the bananas what happened I guess, you know, there's plenty of banana girls over here in New York City So you guys can maybe come visit me one day. I got plenty of plenty of stories from bartending in regards to that but If you guys like to support me, please just share the video I got some affiliate links I have to talk about because I like showed my salt in a day of eating video and it was sold out the next day and Like you guys are buying like I have seaweed down there I have any product like my eye masks cod liver Things like that are in the description under Amazon affiliate links. I got to do a video talking about those in general Maybe I'll touch on it briefly in every video in regards to my patreon check that out See what levels of support you guys get it's pretty much dietary analysis at reduced rates And if you do want a one-on-one consultation in regards to diet fitness Sunlight exposure as well as water consumption very important water consumption one of the most important things people overlook Shoot me an email Frank a to phono at gmail.com. Everything is in the description guys


Look at this. This is step up. This is step up. Good job, bruh. She's like, but let me get some for myself. Yeah, that's what step up is. Oh, do you know what shoulder time is? 20 minutes. There's another African green. I'm glad you're here. Jersey's excited. Obviously, Cody's excited because Cody flew over to see you. She's gonna be excited because she has a little anxiety. Oh, since Mandy left? I don't know. Hi, thank you little footsie. How are you doing? What's going on? Tell me all your details. Hi. How was your weekend? Was it good? Did you bite anyone? Did you fall in love with anyone? Huh? Do you have a new boyfriend or is there one added to the list?  So, there's a new development. Yeah. Yeah. Blue comes to see who's after her boyfriend. Blue, Jenna's after your boyfriend. Do they hang out like this video, like Blue? The other day, yesterday, they were just walking on the floor together, but Leo doesn't appreciate the stalker's tendencies. Like, get out of here, man. Get out of here. You don't like sparky stalking. When did this happen? Here comes Blue. Oh, Blue. We're all going to get in on this loving Leo action. Oh, we love Leo so much. Leo doesn't love Blue as much as, uh, yeah. What happened? What happened? What? Why did this happen? Now what's happening? Do you think Blue is just excited that he's chasing him? I don't know. Blue is very like in love. Blue has done a mating dance for Leo. Did you capture it? No, I was just, you're having dinner. Listen, we're having dinner and Blue, Leo was right here and Blue was right here dancing for Leo. They were just sitting on the table with us. Okay, well that's something you need to film. This is my phone. I can touch my phone. Oh my god. What? This is my phone. Okay, wait, actually I need to press this button. He likes my pen from my phone and I use it to steal this. So, look at that face. Look at this face. Did you already did it? What? You said it? Yeah, I know. Is that his little feather colors coming? George goes like this to me.   He's like, I don't think it's happening. I'm like, it's happening. Don't worry. Did you hear he kissed you? Yeah. He also does the long kiss. You know the like kind of kiss. Mom, you let me pet his beak a little bit. I tried to massage his head, but he kept doing that thing and then Nellie came over and Nellie got me pet her head and then this one obviously came right to me and then I said, hi to Leo. Leo was like, are you having a good birthday? Yeah. Nellie, am I cute? We'll be back, Cody. So, he's big. Yeah. He's big.  Oh my God, absolutely. George was like Cody size. George is more filled out than Cody, but like, they were the same. This one. Oh, your head is okay now. Before he was like, I just cracked my neck doing that. Not only does he do that, but Jenna, he spent the whole first hour in this house with his head completely upside down, like looking at the ceiling. Really? His head was freaky. He doesn't step up, but he lets you give head scratches completely, like his face, everything. He does some very specific talking. He says something so specific and he says it all the time that I'm confident if the owner comes forward will know it's their bird. Oh, does he, does he have an accent? Oh, there's the head thing. You think that's bad? He didn't sit all the way upside down with the bottom of his beak to the ceiling. Who did you do to? So guys, Jenna's here now because we're going to take him over to a vet to get a scanner in the area where we found him close to. So maybe the vet will have heard something like that it's, that it's their bird, but also to see if he has a microchip. I did my wrapping. See, I put my finger and hold this part. So a few bites, then it doesn't hurt me at all. So he's not going to come to it. Come on, step up bud. I really just wanted to teach him step up with this, but it's okay baby. We came out. Hi, could I get you to come out? Let's try to hear him with this thing again. See if he'll step into this. Oh, okay. And there's no way that's gonna come. I know. Do you want to show Jenna how nice you are? Show Jenna how nice. Well, this is a new bird. Huh? This is Cody. Isn't this Cody? Yeah. Where's the other one that you rescued? That's the one. Red tail, gray body, similar attributes to Cody. Why do I feel like you're messing with me here? What about what? I want to try your food. Huh? Give me some food. Oh, not Cody. Not Cody. Hi. Scratch his hand. He likes it. He doesn't step up. Oh yeah, he wouldn't be Cody. He's gonna run along. Good, good, good. Detective Tracy is on the case. I'm nervous, buddy.  The whole shebang of petting. Come on, it's okay. It looks like his name should be Harold. Megan said his name should be Mickey. No, his name. You know who he reminds me of? I nailed it, Winston Churchill. He looks like, oh, I'm appalled by all of this. Somebody come get me. These peasants are trying to lift me. He likes to dance. That's what you came up with. You got a friend. You got a friend. You can only skip like 12 verses there. When you're feeling nervous. I just made that up. Let's do it. So shake your feathers. The weather. It's okay because you will stay and be my friend today. Yankee Doodle went to town riding on a pony. We did not take the feather from anybody. He likes it. He wants more. E-I-E-I-O. Wait, what? At a duck. E-I-E-I-O. With a quack quack here and a quack quack there. Here a quack, there a quack, everywhere a quack quack. He's trying to have a farm. E-I-E-I-O. Give it to Tracy. Do you have any? Tracy wants this. She's saying Joanie. Are you saying Joanie? Yeah. Compute all over. Oh yeah, you give it to Tracy. Tracy does want it. She came in asking for it. If you saw my face. I saw it. It's like a little Cody. Yeah, but you see I don't want to trick her.  Like I don't want her to know that like I genuinely care that she's giving me her throw up. Look at this. This is step up. This is step up. Good job, bruh. She's like, but let me get some for myself. Yeah, that's what step up is. Oh, do you know what shoulder time is? Ooh. What shoulder time? What shoulder time?  No, it's cool. How's this sniff? Ah, it's, I mean, it's not, I've smoked better. Support. Support. She loves me. I think we just witnessed like the whole process of gaining trust from a bird. Yeah. Do you see that? I saw it. I was like, okay, she was weirdly walking. I'm like, I'm convinced it's a female now. Yeah, I'm convinced it's a female. So I saw her like, while we were talking and not paying attention to her, she was like having a little bit movement walking towards us. I was like, maybe if we let her be. Do you need help? Here, go on that side. So okay, come on. Good girl. Good girl. Wow. Watch the tail. Hi. Well, that's so good, good for you. Trying to take a ride. She's big. Yeah. She's big. You got it closed? Yeah. Well, let's go. Here's the car. Should we take your car then? Yeah. Do you want me to take something? The bird, like, do you want me to take the bird? The bird. I think just food for dogs.  No, we can't stop. What is it? There's beef and russet that's underneath meat. Is there a vegetarian one? I'm tracing. I'm looking for a camera to see this bird by any chance I'm looking for my picture. Happy. What kind of bird is it? African gray. Hey, how are you? I'm going to hop right over real quick and then if you can, you guys can have him to step up and I can just kind of hover around. He doesn't really step up, but he's definitely into loving on step up. Hey, bud. Here, you want to try? No, he can eat. I promise you. He won't. I tried. But he's really sweet. He's really friendly. Yeah, you're friendly. But I could just, if anything seems so friendly, I could just hover over him. No, I don't care. Yeah, if you can hold him, and I can just go towards the back. Yeah, just like that real quick. Usually you try to go right back here by the neck. No, they can get microchipped. Nope, there's nothing on this guy. Have you heard about anyone losing an African gray? Oh, you guys found him? Yeah. Oh my god. Let me go. Let me go ask him. Wow. Yeah, he says he hasn't heard anyone. Okay. Let me try it real quick on the belly. I mean, I know they don't usually go right there. Oh, I don't know. I don't know how we're going to do the belly. Just look at your feet. Are we straight here? You okay? You okay? Sorry. You're okay. You're okay. Should we turn him around to get the belly? How much does it cost to microchip a bird? Since it's a bird, we have to sedate him. So it's going to be the mask, the feet, and then just the microchip, I think it's like 56. Yeah, try to do this. Perfect. Yeah, just like that. Yeah. There you go. Nope. It's chipless. Do a lot of people call him a microchip a bird? Yeah, well, like we had people it was a celebrity, I forgot her name, but she because she loves her bird. We had her microchip and then we have a couple sanctuary's that do the microchip. It was pretty common. It's yeah. What were you asking? It's you can microchip every animal and then like a giraffe came to mind. You think a giraffe would fit in here? They'd probably like have to go through the giraffe. So 74 times two. So it'd be like $148 to microchip a bird. To microchip. Yeah, because of the sedation. Yeah. But you would just be 74 like dogs and injections. I figured out what happened with this bird. What? Keep saying murder. So like witness or murder and then the owner was like, you're going to like turn me in. So I got a carrot. You're talking to you? Yeah. He's like mumble. What's he saying? I don't know. Calling Jenna a murderer. That's the most famous one. What's your name? What's your name? Can you say your name? Hi. I'm Jenna. What's your name?  What's your name? Can you say your name? What's your phone number? Marlene trying to teach that to George. Yeah. In case he flew away, he'd be like, Do you have an email? Okay. It's time for you to go back. I'll get you guys there. Okay. Okay. Sounds good. It was good seeing you. Thank you guys. Thank you. Fun box is our new subscription box of toys, treats and other surprises for you and your bird. So don't forget to check the link below and get in on that. And guys, don't forget to check out my new line of organic bird food called Marlene Signature Blend made with tops. The link is below.


давайте move on to the next half session so the problem, Miklos described the problem, last year maybe give the mic to you, David also talked about it, how to monitor a very large number of mounts in a mount namespace, when mounts go away, and mounts are created, how to monitor this sufficiently, I don't know, אני אעשה לך לתת לך את הנושא, זה לא ממה, אבל אני רק רוצה להכתב את זה, כשאני מנסה, הוא already has an un-mount event, ויש להכתבים לנסה את זה, ולמנסה את זה, ובכלל, לנסה את הולכת על הולכת, ולכתבים לנסה את הולכת, כשזה נמנסה, זה מאוד רביעי. אני כתב את הולכת על זה, זה מאוד רביעי. והשאלה, שהשאלות הולכת, גם את זה, אתה יכול להסתכל על זה? תלך, זה כבר נפואה של עצמם, ואז אתה יכול להסתכל על זה. איך לדזכת את העצמם? אם נפואה את עצמם, תלך, תלך, תלך, תלך, תלך, תלך, תלך, תלך, תלך, הוא מבחר מסביר. נבחר עוד עד כך, אנחנו יכולים להשתמש... אנחנו יכולים להשתמש מרק בתיכון, בתיכון שזה מונט. הוא שיש להכנות. -כי מונט, כן. לא, אני בתוכנת, אתה כונות מונט, מנכון שיש מונט, אם כך מונט, אתה שיש לו אינטרון. אני חושב, אתה יכולה לשמוע על כל התנועה, וזה פתוח רגע. אני עשיתי את זה, ובאיש כמו שאני עשיתי את זה, אני עשיתי את התנועה, ולתת על פתוח הרבה, זה כאלה איך וינדורס שלהם, כי everything is addressed, there are no hard links, you don't need to consider a hard link type thing, so a path is almost an exact script of a mount, except that paths, you can have mounts, multiple mounts with the same path, so I just said, you put a watch there, you say everything under this path, anything that happens under that path you get a notification for, or a mount, so basically a mount tree is what you need to watch, but you place it on a namespace, on your namespace, you don't actually, the path is just a filter. So each namespace has a separate mount tree, so it's slightly different watching mounts because you need to have information about which mount changed, so you need to have the mount ID, for example, but the problem with the mount ID is that mount IDs are reused whenever a mount is... So I added an extra 64-bit mount ID, well, it will eventually get reused, your computer probably be solved first. When you say added it means RFC patches? Well, yeah, I've posted patches for it, it works. Yeah, but your patches are not for FA notify, but for WashQ, so it's the same concept, but with a different interface. Well, it produces, it produces notification, we could make notifications come out somewhere else. I think the notification stuff and for example the FS info stuff should have been decoupled in that patch set, and in your patch set it was like one big thing, right? The problem is that the notification queue has a limit, and you can overrun the limit. So if you move a whole mount tree, or if a whole load of people do mounts, it will generate a notification of each of them, and if you're not quick enough, the notification queue in the kernel will overrun. I don't think we need a ring type of efficiency for this use case, so FA notify is less limited in that perspective. Well, it still has a limit. Yeah. And if you overrun the limit, you then have to find some way to find all the... So with FA notify, you can have two types of queues, one is in principle unlimited, but obviously there is, and then it means basically the allocation will block until it can succeed. So basically you will block the mount until you can generate a notification message. So then you cannot lose events, that's one way how you can set up FA notify. The other one is that you will simply drop the event if you cannot accommodate it because the queue is already full, but you generate a special type of event like there is lost information. Yes, the reason I added the FS info thing was so that you had a faster way of finding out what had changed, because otherwise you end up parsing proc of the mounts. So we... We're too slow. We also have the eventual user of this API here. I made him come along. How perform and sensitive this is actually for you? I don't know, right now we parsed proc of mount info and that of course is terrible, but for me, if events are dropped, that's fine as long as we have a way to figure out what happened in the meantime later. So we want an event that tells us very clearly the way events dropped and we want an API that gives us this information on our own terms afterwards, which is better than proc of mount info. So I'm actually... I mean, Amir knows this, I guess. I'm actually very much important having an API that tells me for a mount all the children of that specific mount, the immediate children of that specific mount and gives me the events when immediate children come and go. That's all what I probably really would like to have. Because that's what I did in FS Info. Ian Kent, I don't think he's here as he implemented the thing for system D to use FS Info and apparently performed quite a bit better. But yeah, I mean, I'm totally important if we can get a new API. Right now it's really horrible what we have to do because in large Kubernetes installations we get like these millions of events and we can't cope up and we're just busy with dealing with that stuff. Does it matter if it's for notify? I don't really care. I mean, to me, it would be natural if... I mean, FA Notify sounds great to me because it has the queue handling already and things like that. So I'm reinventing it. I don't know. I've never looked at what's your thing called again. The watch queue. Yeah, the watch queue. I don't know. What does it have for... You know what I like about FA Notify? I think it can even call ask events, right? Like, kind of not so that it drops. Which events? Like it can call ask events subtree-wise? Can it not do that? I don't know. I think it was discussed... Coalesce events? Yeah, so basically is that if it drops events somewhere that instead of replacing them by a single bit of information that tells me that some information was lost, it would rather replace a couple of these events with a new event that just tells me below the subtree something got lost. It doesn't have that. The only call is events with exact same information or very, very similar information. But for example, you could do something like that. If you're watching, I mean, it's pretty equivalent to watch a namespace and to recursively watch direct children of a mount.  For FA Notify, if you would have to do like mount events for direct children, it would be pretty simple. Because one, at least limitation of the current API, it needs to get a path as the descriptor of the object. So it's easy to give a path describing a mount or a file system or an Inode. But if you're watching a mount namespace, maybe it's hidden. Maybe it doesn't have a path yet, so there are complications. But if you're watching direct children, you can always set up a watch. Let me know if something is mounted on top of that mount. It's pretty easy. And then what information you get in the event, that could be configurable. So maybe you'd get the Inode where it was mounted or other things. Or maybe you get nothing. You just get an event, something was mounted. And this will be coalesced. So multiple events of something was mounted. What I would like to have is an opath file descriptor simply as part of the event, like of the child. You have an opath file, FD or NFS file handle where you can get it, but of what? Of the mount point? Of the mount point, yeah. You can get it or not get it, then it's easier to coalesce many events. So something has happened to the direct children, and then you only go and probe what happened. I may have a silly question, but what's the actual use case for this? What are you guys intending to do with this info? In system D currently, the boot up process to a large degree is starting services and waiting until things happen, starting the next services. And a good part of that is setting up mounts. So we can do stuff like, yeah. We start that service during late boot once flash var MySQL or something has been mounted. And so we need these notifications. And this is actually, we use it heavily for individual services and people use it heavily. Like they run app so and so, and then immediately before starting app so and so, they mount something and then it gets, because it's a dependency tree, it gets pulled in until that service, at the point the service runs, and then when service dies it gets automatically removed, unless somebody else also has dependency and things like that. So for that we need to track exactly what's going on in the system. And right now we do that always proxf mount and it's just terrible. I don't know how I will describe the event of tacked mount. I'm not allowed to call it that anymore. I was instructed to call it belief by friendly people on social media who pointed out that this has also other meanings that I wasn't aware of. And also for, yeah, the danger, the danger of mistyping people before IOU ring was merged. By the way, just to explain why I think that having this per mount point would be really nice, right? Like so that we can watch the immediate child mount points instead of having something system wide is mostly because we generally in system D want to only watch a subset of the entire tree, right? And because like things that I show up in below slash sys, we generally do not care about their API file systems. We never wait for them. They're just there basically. And that's why, like if we had the system wide stuff like proxf mount info, mostly the ignore events we get from there. So for me, it would be much like the thing that I find nice about the recursive thing is that even though it would be a little bit more work to manage for us, we could very specifically select the sub trees that we care about and ignore the sub trees we don't. Yeah, but that brings us back to the I&Otify recursive watch, which is racy and we need to see that we can implement that without races when a child one is created, you need to automatically add a watch or something like that. Yeah, the problem is that if you are watching just whether something gets mounted under this mount, then so you learn something gets mounted, but then you have to place the watch on the new thing. Yeah, but that's fine with me. I would just consider that I lost the mount event, like what I have to handle anyway if the queue runs over, right? It's not different from the queue runs over event. Yeah, but you would have to note you would have to do a scan of proc mount info. Yeah, I don't want that. I mean, that's a thing. I definitely always want the ability that, yeah, I want the notifications and an API that gives me the immediate children so that I always can catch up nicely and the API for getting the immediate children, I don't know, maybe that's FS, but that is kind of for me the other side of the metal. Having just FANotify without an API for getting the immediate amount of the children won't be useless, but... You get a notification for a mount event happening, or a human event happening, and then you could pass this FD that you, for example, get to some sort of system call or whatever in what form, and that gives you the list of child mounts, identified by what. So that's the thing, I think, and I think you wrote this on the list, I think we need a unique mount ID. It's created, I guess, and the notification needs to contain the idea of the new mount. I'm not sure how it could be solved to make this complicated underneath thing because if some mounts are immediately mounts over the top of it, you can't get the mount underneath unless you can adjust them by some ID rather than path. So for the system to use case, I don't care because we only care about the topmost mounts. Okay, I see what you mean. If you have the mount ID, we need a way to query by a mount ID, but it shouldn't be a problem. No, I think the problem is currently the way it at least works for mounts. If you have an FD, that FD will refer to a dentry mount pair, and it will be fixed. That struct path usually doesn't change, but as soon as you enter into the mount system call world, the first thing that we do is lock mount, and then we walk up from that struct path to the topmost mounts. So your FD technically refers to something else or can refer to something else by the time the actual mount operation is performed if you have mounts on top of it. I mean, what I would ideally want is a file descriptor to the mount FD, so that it's clear what it is. At some point, though Christian may beat me to it, I want to make it so that you can stick something to intercept mount calls that are being made and do something without the network being enabled on the system. Even though it's a deviation, we're giving a presentation tomorrow on how to do unproved mounting right now, but it would be nice if we had something like this. But upcalls for user space are a giant pain, usually not something that we... stuff like user mode helper or also the key ring stuff. Pretty untrustable. Christian is absolutely right. User mode help us need to die. They're terrible. They're past the internet system so they don't get security stuff applied. They don't get resource management applied and they live in their own environment. Are you staying for the security summit because there's some people who disagree with you there? I'm staying around there. But I mean, I know a lot of people who agree with me too on this, but yeah, it's just terrible because they don't have security settings at all. Like they don't inherit anything from anywhere. Fuse? What's with fuse? Upcalls. That upcalls in that sense, yes. Anyway, that's a different discussion. It's one of my personal things that I really want to see them go. So the thing I was saying about up path is, okay, if you were to mount on top of it, then you would get some other behavior, but that's just part and parcel of how mount works with everything. Like if the idea is that you want to get the sort of child mounts or something like that, or you want to do some FS info or whatever, whatever that ends up being, effectively some way of like, oh, I have a thing, please give me the info about this particular mount or for this particular FD, that presumably would not be translated because it wouldn't make sense to translate it in that case. Or would it have to be translated? Is it like a future of? No, it wouldn't have to be, no. So it's just like if you wanted to, I guess, you mount or you mount? If you wanted to, you mount. Or, yeah, you mount and unmount because for example, amount property changing code also doesn't look up the top most mount usually. So it's just really adding or remaining amount. Yeah, then it goes with all mountry and everything else. And it makes sense, it makes sense because of at least the times that I had to be in this code is like if you were to try and you mount in the middle of a mountry, you had all sorts of issue because you would need to reparent mounts and so on. And what mounts survive and it's pretty messy to do cleanly. The way propagation works where your propagation starts from the parent mount that you're trying to unmount. So the reason I'm bringing the old part thing up is because if the issue is, okay, we're talking about having mount IDs, which we hope won't overflow, but may or may not, it's some independent point of the future of a system that's in up for 35 years or something, right? Because even if you have 64-bit whatever mount ID, the thing is that if you use like KCMP or whatever, if you had KCMP, you'd be able to compare two file descriptors. Assistant would have to open like 5,000 files for this to work, but you would be able to check it that way, right? To check if the two things are the same, right? Just a second, nanosecond? You will only overflow it in about 500 years. So, I don't think you're any... Yeah, sorry, yeah. It is 64, yeah, yeah, sorry. No, you're right, yeah, you're right, sorry. My bad. 64-bit all the way. The current mount IDs are a problem because they're not even allocated, like pits are cyclically, but it's if you do mount dash dash bind opt and you get 364 and you U-mount and you mount immediately again, you get the same ID in case no one in another namespace beat you to allocating that ID. Can you... Can you... Can't you just extend them to 64-bit? I mean, they're not exposed in binary to use this space, so you can just... But then you'd suddenly have 64-bit numbers in PROC self-mount info. Possibly. Yeah, that doesn't seem like a regression, but it is exposing a name to handle at, actually. Around this, I propose that we have another at flag to get the MNT ID. I take back my idea. It's not this... What am I supposed to do with that info? The MNT ID is exposed in the system call as a system call argument even, right? An open by handle at... Yeah, but honestly, open by handle at two, whatever. I mean... Huh? Or an at flag, but I mean... I mean, it's not that... I think in terms of adding new system calls, it's much easier than this.  You have to have something different, because otherwise you will break user space even if you exported it as a text file because we actually looked into the kernel to see what it's used internally, and then we used the same type. That's one of the reasons I added a new value that was independent of the old one, because the old one is recycled, too small, people assume it's too small. Honestly, you should just all use UUIDs and the problem goes away entirely. I'm serious. Is that the reason why you put them in TID64 between statics, but then it returns the old one? Static returns 64 value, but it returns the old small value, so it's an at flag. Do you have coffee?


So my favorite part about HDF and this conference here, or actually about the whole HD field basically, is that everyone is working together, everyone is very communicative, and people are like working cooperatively towards a common goal. That's perfect.