I was over here, champing my cuticles, trying to think about what my raps would actually do to you. Yo, they wouldn't do nothing, it's true cousin, you're buzzing.  Yo, yo, yo, I've been on the true buzzing, taking too much truffles. You've been too much trouble. Yo, I'm blue, I do this cousin.  We're here in Philadelphia, USA, please make some noise, let's go! Hey, guys, thanks for being here, for the people that you support, you know the score. This is the first battle of our first day of the video event. I'm sure it's the first of many. Straight into the battle, MC on my right-hand side, Dave, UFC with Don't Flop. We have J.R. Spangler, make some noise, let's go!  Don't know, don't know. MC on my left-hand side, another day of UFC on the day. You've got to sit over yourself, make some noise, let's go! First time of Philly, first time of Don't Flop. Don't Flop. Let's go! I said it's my first time of Philly, my first time of Don't Flop. But that's not it. Because this is the first battle ever where I could never possibly hear an eyebrow joke.   You know what, 20 minutes.   So I don't even want to battle you, J.R.     He can't even get in the dock, he's here in organic bitching. When he told me about it, his hands were twitching, he damn near had an aneurysm. That's how I know it's not because she won't even say it to organic chicken. If you pay her any money, I will beat you up because that will be a wasted dough. Like you ordered her pizza, but you didn't eat the crust. This is word to McDonald's. Word to McDonald's chicken McNuggets.  I know you probably brought a lot of jokes, J.R., but that won't be enough. And I know Uno probably put some good weed in his burrito ducks. He'll be the same day, day and night, gets his 30 Gs from Lux, or he gets a retweet from Lux.   He goes, my best friend's with Olavos.  But it made me sick, it made me think that he turned about their relationship.  So like a fish steak with tips, he won't eat late, so he makes homemade potato chips. He even brings the queso dip for dessert.    He was talking about his drive out here. I figured that's cool when he might as well. He said he got a ticket. How, bro?  Y'all, it's great to meet you. Welcome to the show, Don't Flop. But this is where the promo stops. Left hand, throw a shot. But you can get this one for free like a bogo spot. So get your cameras ready, opponents. What exactly have you done so far? Lex Luthor suggested you, sir, and that's the entire reason you're on the undercard.  Where'd you get the name Celebrity Cell anyway? Brock and Diane have found out. Cause cells are known for having bars. You're more like Celebrity Time Out.    You should have stood there and fought like a man. How are you Italian? And I gotta teach you how to talk with your hands. So I tell him to break a leg, but for you I can't walk for assurance. Should I woke up this morning and got myself a gun to turn his son into an orphan?  I'm Tony Soprano, he's Ralph Cipriano.  See, you was a leg and you ain't got the tools to take to battle me. It's the 29th of March is made to mark the day that you became a casualty. Getting hit with a cyclone.  But I gotta put on the shoulder to prove that I've been a boss in them.  Because the Red Bull never lost.   I hate everybody in this room that understands me clear. Worst case scenario is I go home, lose no sleep, have a beautiful family and make a hundred grand a year. I was ignorant, but it's true. I said, I will find you a bit impressive.  Everybody says this kid's aggressive. But I don't believe a motherfucking word they say for a millisecond.  Detective, do you get the message? He's from Jersey. When was the last time you've been a newer? The last time you took a trip to Trenton.   I was what his face in the hot tub. It was my dick that the chick was sweating so I pulled out. Kids in the water. That's a swimming machine. That is some G-shit. I've said, my eyebrows say I'm Italian, but yours tell me that your dad is from Egypt. He just took a second.  You just look like you went to Hitman Hollis Barber and asked for the reading.  Nothing in the middle. Yo, you couldn't even pat it down.  Even your best friend wants to laugh out loud. See, what we have here is a classic game of cat and mouse. This is not the rookie version of Mayweather vs. Pacquiao. This is a trained pit bull blacking out in a bass attack. You have to be smoking DNA, city tower scream. I even backed out my third through half the round, plus I sell Hyundai's. What more could you have to rap about? That's about it.    But I attributed it to a long work where you can smoke and call you loud.    It's because I thought if I drive five hours, an herb doesn't put this battle out.  And what you need to do is stop bringing up my past accounts.     It's all of his opponents who are rematches now. And I feel that all of his opponents are why rematches now. That's round two, man. Let's go. Round two, okay. Okay, others are suffering because of them. Favoritism is plaguing systems.   And with all that being said, the celebrity cell is not one of them. Now I'm not claiming to be the best. Big battles are something I'm kind of new to. And I ain't saying that I'm the greatest, I'm just letting y'all know who's who. He paid for a battle with DNA. A guy that averages $500,000, right? Hits on YouTube and your battle with him got $30,000. That makes you a negative $470,000 if you do it. You're reckless in selling cars that simultaneously fund terrorism and Kim Jong-Kil's war efforts. Yo, so if you see ARs, you don't even have to pay that much for them. You think ERG's got a bad accent. This dude sells a ton of them. I've been smashing his Hyundai.  Damn! You need to stick the garden stock in the parking lot.  Then I carve his top like a barbershop. And then look for a new spot like an apartment shopping. You're garbage-ocking. Your bars are sloppy. It's gonna be hard to not think. You're hardest-stopping. Who cares if you drive in a laundry? Your image is so nada.   See, I'm better at this. You never have, but then you don't like facing it, do you? This celebrity deathmatch, I built this lane out of clay and I've been aching to school you. Surgery. I'm just taking the time out to start shaking my pupil. That's it! You need to go, man. I've taken the 413 Battle League, and it's true. And I wish it wasn't. I just hate it, y'all. I even called up to do it on YouTube.  But talk about the two-verse battle. There was something he was saying, dawg. He said y'all were supposed to battle before that. And you hit up Logic a day before with that Link Jake from State Farm call. And you had the famous call. Right? That's what he said. And then Uno showed up and choked every round. Wait, I'll leave him hanging, dawg. I think you know where I'm going. I think you know what I'm saying, John. I might have backed out in a third, but at least I showed up to take the loss. Jesus Christ! J.R. Hideo Jean wrote before the battle. He was like, we got to stick together. Because as far as white people in this car, we're the only ones, y'all. Then there was an awkward silence, and I said, what about Rosenberg? We're all... No? No, can you? Something went off. You can't hear me. Right? You can't hear me. Yeah. Because I saw him. That's cool. He's a mad car. But I can put a dealer's play on a Range Rover and drive around like it's NASCAR. What do you pay? $3 a gallon? I have a NASCAR. My brows thicker than mine. At least I get mine trim when it's time.   She said, yours is so short, if you make a porno, it is still considered a fight. Jesus Paul, listen.  Tell us, Joe, is that aggressive and then stares at the crowd's weight because he loves attention. There's not going to be that soon.  When him and Nuno go both ways, they listen to one direction. Let's go, right?   People like it because any great classic is like, hey dad, for the rest of my life can I wear a wig? He orders me lovers. When he knows the dominoes, he orders me lovers.  He's burned so many bridges. I'm talking king of the die. Eye battle. Just because you want to be stubborn when he's homeless in 10 years he'll have no bridge to sleep under. When he showed up looking homeless, that's because he spends all his time selling on days, writing bars and getting focused so he can hang those new Jordans next to his eviction notice. It's your son's birthday party to be here. I happen to think that that's so sad. It was yesterday. What'd you get your kid for his birthday? No dad? These y'all are better off as neighbors. He'll show up at 3 a.m. and then he'll clean. He lost his pager. When he carries the chap the kid is scared to sack like he's the Boston strangler and that's ever since his mother taught him not to talk to strangers. And you're out here buying battles. You should be putting food on the table where you both live. Instead you paid for DNA.  He must have felt like he got pretty low on Santa's list when all he got for Christmas was bad memories and abandonment. In retrospect, Zach in retrospect, that probably wasn't the best present.   There you go. Let me tell you something that you don't know.   Now you want to live your life fine but it's kind of like your battle career because you got to break out your credit card when it's time to do your little white lines. You see if you carry yourself with some dignity and respect your own people would support you when that holds true. I guess there's no iron integrity, right?  There's no you. This whole scene the event was packed with the raptors and the nosebleeds and the rap of a passion.  This carried Quinn after two rounds and said he was finished with rap to his fans in his home team. Now it's ironic that I haven't looked at him this whole scheme and the fact is it's symbolic but what does it actually all mean? I just did it to demonstrate how you left the game and turned your back on your whole league. What was the time, man? So now she's very cell and falling for ballers so it's going to take more than to be on the NBA roster to call I'm talking bachelor pair with a match and a greed with a couple lofty endeavors which happen to be showing the things that differ in aberration from the men she distant more like a non


Okay, it's two o'clock now. I know people are still coming in but this part the introduction part is not important anyway So I'll get over it. So we don't even bother listening My name is Steve Rosted Red Hat pays me to play with Linux My hobbies include working with the real-time kernel This tracing utility called ftrace testing utility called K test the make local mod config things like that That's what I kind of play on Not really much else to say about me Let's say back in 2007. I gave the talk to introduce the real-time patch. It was called inside the RT patch I gave it the Ottawa Linux symposium or less Obviously in Ottawa, and it was extremely popular there. No one knew exactly what this RT patch was they kept hearing about it They heard Linus Travolz. Just call us crazy people So people are very very interested and then last year I came and gave a talk about K test and people were asking me. Why aren't you talking about the real-time patch? It's been a long time since you presented it and I said, okay, you're probably right I probably should reintroduce what the real-time patch is about. It's becoming more popular a lot of it is now in mainline So I yeah, we do have a real-time patch But it's really focusing going on mainline is that it's not about if it's going to go in mainline It's when it's going to go in the mainline. That's the true question of today This is the slide the actual slide that I started with back in Back in 2007 OS So they asked me to change it so I updated it So it's my updated version of the slides and But things have changed within back then for example Well, I should call it understanding preempt RT because now it's It's no longer like I said the real-time patch is the preempt RT is the config option that you enable and since we expect it to go In mainline, I don't want to talk about a RT patch. I want to talk about preempt RT Well, I said things have changed for example Therein works it for Intel now. He doesn't work for IBM. So he was IBM now. It's Intel But that doesn't really matter because he's not part of this presentation anyway, so I Gave this exact same talk in Barcelona a few months ago at the Barcelona E ELCE you and Hello And free electrons posted it And actually it's kind of funny because that's Darren Hart right there and they I love free electrons they posted this and they said hey They not only that they post this a video of it of my whole talk is on video you go online see this whole talk there and On the blog they even said hey, we recommend seeing Steve Rostad's pre-time real-time patch I'm like wait wait. I'm recycling this talk now like people are coming to see me now like really you're going to be giving me Or showing this you don't need to come here. Just watch the video. So What should I talk about? Now I visited and we had the real-time conference in Raleigh, what's my right side out Raleigh, North Carolina where my brother my brother lives with his and my sister-in-law and They're into pumpkin chucking so You know, maybe we talk about pumpkin chunking But we want to be more real-time stuff. So what really interested me is really, you know trebuchets So actually I was like, wow, maybe it'd be cool to make a real-time trebuchet So, you know these how do you design it? There's level different way designs Well, maybe I can make a simulator that simulates it and So I started working on a simulator. I thought this would be cool to add like a real-time simulator and So I developed this thing whoops and said let's try to simulate it One thing I learned was after I started it That's a wacky trebuchet I'm way, this is like my new obsession. I'm gonna make this thing actually work. So, uh, let me kill that. Whoops No, it's not just friction. There's a lot of things the problem. There's a lot of things A trebuchet happens to be a double pendulum algorithm a simple pendulum is very easy when you attach another pendulum on top of a pendulum That's called a double pendulum and it causes a lot of calculus. So I decided to give up a mess and talk right back about real-time So Where do you get the real-time patch right now? I have a stable repository. I'm the maintainer of the stable tree I right now maintain a 3.0 RT patch, which will be I'm gonna stop maintaining it as soon as Greg Crow Hartman stops maintaining the 3.0 mainline Which is this year. I can't wait for that to happen It's becoming more of a burden on me because there's a lot of things that are broken about the 3.0 that I just won't fix That's fixed in 3.2 3.4. I have a 3.2 RT patch and a 3.4 RT patch Thomas Kleichner is still working on the 3.6 RT patch I believe he's going to come out with a 3.8, but he has he says I'll do it if I do it So there's no guarantee that it will come out at 3.8 But when he does I will be the 3.6 maintainer, but there's no 3.6 RT patch or mainline So I might still maintain it just Will just for temporarily Right now 3.4. I believe it's the long-term patch If three I don't know 3.8. It's not gonna be long-term. I know what's the next long-term 3.10 I think it's gonna be that Greg's going to do so we do a 3.10 So basically the idea is I will maintain a real-time stable branch as long as there's a mainline version of that same Brant or that tree is being maintained patches are there the wiki page is there You could get this slides up like when I'm done with it since I just finished writing or finishing tweaking this about 10 minutes ago I will upload these these slides right after this talk So what is the real-time operating system?  It's abused Real-time could mean anything. It means right now. Okay. That's what real-time is I Really wish they called it, you know a deterministic operating system, you know DOS It's exactly actually DOS was actually private best real-time system ever So what does a deterministic operating system do well? Basically, it does what you expect it to do when you expect it to do it. That's really what it's all about It's about no no surprises. No anomalies. That's what real-time systems about it's not about You know real-time because people thinks real-time is fast You know I was at one customer site and they said I can't wait to see you know get your real-time kernel up to see the performance It's going to give I was like, you know Real-time means Determinism determining is some kind of ads overhead, which is kind of the opposite of you know What real fast it's about so yes, we try work very hard. We do lots of tricks to get throughput out We do a lot of things to make this as fast our goal is to be just as fast on the average case as mainline That won't be but we're probably we are we are worst case faster than mainline Mainline has anomalies mainline. You'll see something. I could you know you could run this command in like 25 microseconds And mostly time and when you run on RT. It's 35 microseconds But every so often it may take two milliseconds to run that same command on mainline for us Maybe 40 milliseconds huge difference. That's what's about worst case fast times. That's what real-time gives you You want bill make make your deadlines you want to set up your system understand your system and that requires homework Real-time OS is only as good as what the hardware. It's running on some people forget this if your hardware Has you know cash issues SMIs going off you're going to have issues with your real-time response I mean if the system hardware stops because of the bus is now all congested and you're you have to wait for Things to happen the real software is not going to help you there and people forget that they sometimes blame us all the time We say well, let's look at our hardware. That's why we like to at red hat We have a certification of hardware that we actually look at the hardware We run a bunch of tests on it and make sure the hardware can actually do somewhat what we want We'll actually rate I'll give it say well expect maybe like you know 100 or 150 millisecond latencies Mills microsecond latencies and I'm those like it microsecond latencies Well, of course 100 100 millisecond latencies may be acceptable with some or one millisecond latency may be acceptable for them It's not acceptable for what we would do but it's there and believe it or not The mainline kernel is not will not guarantee one millisecond latencies There's several times where I could trigger three to four millisecond latencies with a real-time kernel even or sorry with the mainline kernel even with a lot of The stuff that came from real-time in there already. It doesn't have everything yet So what is the goal of the real-time kernel? Basically, we want to be a 100% preemptible kernel so What does that mean that means that we could react to anything at any time the interrupt comes in I know it triggers like you get a response from the network some device some sensor sends a Interrupt to the CPU we want that CPU to be able to stop what it's doing to respond to it if it's required to do so So we need to be able to stop at any time. We can't be processing, you know MP3 play play So when playing their MP3 player or something when you know it interrupt comes in We want to be able to stop going and work process anything that may be more important So that's what we want Obviously, you can't have a 100% preemptible kernel, but as Lita says we're crazy. We will try We haven't succeeded yet, but no one has so that kind of makes us feel a little better So how do we do this? Well, one thing is we try to disable or do not disable we're trying to disable disabling interrupts. Does that make sense? We try not to disable interrupts as much as possible and We also try to make sure that any type of any type of instance where preemption is disabled preemption means you can't stop Then you'll switch out this task and work on another task. We try to get rid of that throughout the kernel We want quick quick reaction times So When you install the real-time patch and hopefully when it's a main line, you'll get a menu that looks like this In your you know make menu config you'll see there's the full real-time pram kernel Yeah, let me explain some of these just to review I don't know if people do defaults, but I'll go over for those that you know just never thought about it What do all these things mean in the preemption models? No preemption that's the top one This is basically was the idea was for servers that don't need to do like not for servers that acts for web servers, but basically servers are doing Tasks batch batch processing of tasks things that you don't care about a reaction You just want stuff done just send the stuff out get the job done your compile Maybe compiling something that does just a lot of hard work, but you really don't care, you know the reaction for it You just want throughput Server is like don't disable preemption because whenever you disable preemption There's overhead to do a little sweat or sorry don't table preemption whenever you do a Context switch there's a little overhead there So sometimes just serializing everything is faster as usually some of the best throughput you get is by serializing all your tasks Let it run to completion do the next one run to completion do the next one server model like that is what we do No one actually enables this anymore that I know of Voluntary preemption this is what a lot of the distribution is actually default to Because they're still a little bit nervous about preemptible kernel, which is in mainline. It's been a mainline for a long time Voluntary preemption as you'll see a lot of places that say might sleep might sleep is like a debugging situation where If you have a preemptible kernel You don't there's some places you don't want to schedule So if you call something save you disable interrupts, and then you schedule call the scheduler utility You could break things because what happens is If you disable interrupts and you schedule out You kind of have to state in a kind of weird state because you're not supposed to schedule out Well interrupts are disabled actually worse yet if you grab a spin lock, and then you just schedule out That's even worse because you go away then someone that comes in goes to grab the spin lock He's only spin and and you could block or he could block you from coming back and releasing it deadlock so we put might sleeps in places where if you if you set it to bug option and It hits this might sleep it'll actually give you a warning say hey You know you are in a situation where you shouldn't be sleep scheduling But this what you just called could schedule because not that's the thing about debug or that's the thing about Kernel development you might be calling this function hundreds of times with interrupts disabled or spin locks held But if it never schedules out you don't care like it won't worry about it So this lets you know like this you know this might block on something might grab a mutex and block and you're basically screwed So what does voluntary preemption have to do what does it have to do about voluntary preemption? What's really nice about all these might sleeps? It's saying that this function can schedule we are specifically saying we could schedule at this point So what voluntary preemption means is when it gets to hits one of those might sleeps It will call a might rescheder which calls condition resched which will say hey if you know Something came in and wants us to wake up. Let's schedule at this point. We already know it's safe because we'll warn about it So voluntary preemption is just all these might sleeps turn into preemption preemption points within the kernel So that's how that works The preemptible ground was created by Robert Love His idea of rampable ground way. He's came out about is it was based off of basically S&P the multi processors because multi processors is the way you think it's a spin locks so if you have some critical section that cannot be touched well, it can't be re-entrant and It's great on a uni processor system with no preemption. You just Work with whatever data you have. There is no critical section. Everything is protected Well, if you have a critical section where if you know two Processes we're trying to access the data or modify the data at the same time You're going to corrupt the data So you protect it with a little spin locks everyone shouldn't understand what spin locks are Well, Robert Love realized was this actually this model also works say if we had a preemptible model where you actually could preempt the kernel These spin locks could actually represent places that we can't preempt So what the preemptible kernel does even on a uni processor system is been locked now becomes functional And it says just disable preemption while I grab the spin lock So all spin locks you grab just disables preemption and when you release the spin lock preemptions enable again anything outside that Spin lock can be preempted. There are some cases where that's not always a case And that's why there's a preempt disable function that you'll see throughout the kernel preempt disable preempt enable because we say well We're doing something special here. We're not grabbing any spin locks, but we can't if we schedule out here We could break things on uni on a uni processor system or a not uni processor system But when preempt is turned off these preempt disables are no lumps The preempt kernel basic basic RT now, this is where the RT patch now. This is not a mainline not yet Well, actually this may never be a mainline I'm just popping this up just because you'll see it if you download install the real-time kernel You'll see this big option. I'm going to explain this basically Thomas Kleitzner likes to debug the real-time kernel And since there's a lot of things that the real-time patch still does There's a bunch of things it adds and then it also adds Spin locks turning into mutex's which I'll talk about and The thing about this is he wants to make sure that there's something breaks. He knows what broke Was it the mutex's or was something else I add so what he did we added this config option Just so that he doesn't need to if something breaks and doesn't break under this convection config option He knows kind of like okay. It's the mutex's so this is sort of debug Debug what's called option basically ignore it Ways it too much time on that ready Now this is the thing that everyone here is here for well when you select the fully preemptible kernel This is the real-time patch. This enables everything that you want it gives you The almost preempt everywhere type of mentality What happens is all these spin locks turns into mutex's interrupt interrupts get pushed off into threads So there's really no hard interrupt context We also add a priority inheritance Feature to all the lock all the sleeping blocks in the kernel. We had priority inheritance Right now today priority inheritance is in the kernel, but it only works for few texas So if your system or if you're using like p threads and you have p thread mutex create and you give the attribute I can't remember the exact attribute name, but you can say prior inheritance or whatever That few text will get priority inheritance. So one thread high priority thread blocks on a lower thread lower thread p thread mutex it will Boost the priority of that lower thread Task holding the mutex and run there. So in user space this works But it's not in the kernel not without the real-time patch Let me talk a little bit about the sleeping spin lock. So what happens is when you enable this? Well, let me State that what's first thing I said is the pre-config preempt is a global lock What I'm saying is when you disable preemption You just said I'm the most important task on the CPU No one is more important to me It's like having priority 100 where you know 99s max 100 nothing to preempt you so That's this is bad when you have a low priority not important task disabling preemption It's now the most important task on the system. Well, it thinks it is So whenever you grab a spin lock that are the spin lock it disables preemption excuse me like Robert loves Talked about I talked about with Robert loves work and we don't want that So the way to do that is let spin locks sleep So we actually converts the spin locks into a mutex So instead of grabbing a spin lock or spinning we actually if you block We actually go to sleep now. There's something called adaptive spin locks, which Is what we really do which is if you block on a spin lock or a sleeping spin lock You check to see if the owner is awake because most likely spin locks in the kernel are very short time periods So instead of scheduling out and then scheduling back in that might be too long. You just actually oh the owner is awake Let's wait till the owners while the owners awake and still has a spin lock We're just my spin if the owner goes to sleep then we'll go to sleep too or the owner releases lock We'll get the lock, but a lot that speeds up things tremendously People and what's called Susie actually came up with that that was actually a great idea Now we have like two types of interrupt. Oh, yeah, let's have interrupt friends Okay, spin interups have our inner panelers have a lot of spin locks So if you turn the spin locks into mutexes and you actually run into a hard, you know interrupt context You hit you block on a spin lock You're going to sleep and now if you guys realize but sleeping in a hard real-time or a hard interrupt context is Not a good idea. So we have to push out all the handlers into threads. That's a must So once interrupts are threads, then that works actually makes it things. It's not really reasons to disable interrupts And again, this uses priority inheritance. It's not just for few texas anymore. I Add this is I added this from Barcelona preempt lazy. I want to talk about this another enhancement Remember how I said the real-time kernel number one importance is determinism Number two importance is throughput. Now these two are almost inverse relational to each other The more deterministic you have the less throughput you have How do we handle this one of the trip one of the things that we found out with the preemptible kernel? We're preempting all the time and then we have this timer tick that goes up in the for a sked other tasks Everything's kind of given a certain amount of time to run.  He's only block on it has to wake up this guy just went to sleep Give him the priority inheritance let him run release the lock and come down where if we just let the task finish While he had that lock finish and release the lock We want to have that contention this constant This what's called the CFS scheduler was causing a lot of contention of spin locks much more than a normal Linux kernel And it was killing performance. We still had the latency We still had the small latency. We still had the term determinism, but throughput on everyone seemed to be going down Tremendously so what we do now is with the preempt called preempt lazy So if the tick goes off and says oh this guy's holding a lock It just says hey this guy should schedule when he releases a lock, but don't schedule them out now Only time that's that's Conditional if a real-time task wakes up and or anyone that says I need to schedule now It will stop them then you'll run.  So that actually Throughput went up tremendously once we implemented this priority inheritance okay Priority inversion is bad, but you can't get rid of it What we get rid of is what what we don't want and what we can get rid of is unbounded priority inversion Anytime you have a task running that when another task wakes up of higher priority and it wants to run But it has to wait for another task to finish doing whatever it's doing whether it's doing it process I interrupt or just holding a lock. That's a priority inversion That means that this lower priority task is running while a higher priority task wants to run But can't due to whatever reason You can't get rid of complete priority version But unbounded priority version means that there's a chance that this higher priority task wakes up blocks on a lower priority task There's lower priority tasks may continue forever. You don't there's no bound to how long that task will run and That makes your system Undeterministic you can't claim to be a deterministic Operating system if you allow unbounded priority inversion That's why the real the mainline kernel right now can't have that one two or three millisecond latency or jitter is due to priority inversion It's complex a lot of people have criticized priority inversion because it's a complex algorithm and we've actually tried to make Multi-owner for rewrite locks to have priority inversion with rewrite locks that just made it more just like exponentially more complex That's why we want another reason why we hate our W locks is because all our W locks now It's a single mutex in the real-time kernel, which kills our throughput. Let me explain what unbounded priority version is We have this task, which is a lower priority task. We have a b and c three tasks a is the highest priority b's second priority sees the third priority he's Sees running along and gets preempted by a but in the meantime c grabbed a lock a runs highest priority task It's that same block. Okay Go back and continue, but before c could finish and release the lock It gets preempted by b because b is of a higher priority and b runs Let's say b require it will run until a tells it to stop See what happens. There's no bound now b will just continually run and a is blocked forever Our highest priority task in the system can't go continue What does prior inversion give us her priority inheritance give us? See runs blocks as soon as we blocked here what that means is c will now inherit the priority of a so c is now running at a temporary priority boost So when b woke up it won't preempt because now c is about a's priority Which is higher than b's and b will have to wait so when it releases a lock You know this guy goes up a continues to run finish when a finishes now b can run That's priority inheritance and this is where everything's bounded You can actually measure this and see how things what your max latency will be Now there's some places in the kernel that we can't have sleeping locks. The scheduler is the main one Okay, you can't really have a sleeping lock for the scheduler And there's a few other places the timing code does this so we have this thing called raw spin lock Back a couple years ago at the real-time one of our real-time workshops We try to brainstorm, you know, how did what we should call this and you know Because we can't one thing at least our balls told us is The original way of doing spin locks was we had this magical macro that would magically turn some locks into sleeping Socks and some locks would not stay into sleeping locks and just by looking at the code You couldn't tell if this spin lock was a sleeping lock or not sleeping lock because we had this magic macro That was determined on the type of that lock and Lena said that's not acceptable because it either has to be a Lock that could always say I could sleep when you turn this feature on or it could be a lock that doesn't sleep We had this whole thing about what name what should we called it? I like to the name live lock and dead lock or goldly locks But we came up with raw spin lock So what we did is we converted everything that it's going to always be because before we're going to label what there's sleeping locks Maybe but we that was too many locks. There's only a few locks. It doesn't should never be converted So we said let's call those raw spin lock So if you look in the kernel today, you'll see a bunch of raw spin locks that really in the kernel There's no mean for having this raw spin lock mainline doesn't need it again This is another reason why I'm saying real time will be in mainline because we already have code in mainline That's only for the real-time tree So we are setting this up so when you see raw spin lock, that's telling you this lock must always Spin it can never be a sleeping lock so Thread and errors Now this side, okay interrupt when interpandler comes in a lot of times you'll have like say if your disk drive goes off It call calls interpandler and the interpandler runs well again interrupts a hard interrupt will disable interrupt Well when interrupt is triggered most likely to disable interrupts and then it will run on a not in a it can't be preempted A non-preemptible mode which means that this interpandler is not most important task in the system again And you know if you're doing interpandling for tasks that you don't care about it's going to block your real-time tasks So we don't want that we want to be able to you know push that off And let our real-time task if our real-time tasks more important than these interrupts We want to be able to let our real-time task user space task run over an interrupt handler, so Another some of the things that are kind of nice about making interrupts as threads You could prioritize each interpandler is sometimes some hardware supports that you've actually say this interrupts Handler is more important than this interpandler But some are a lot of architectures don't handle that give you that or it only gives you a very limited version now you actually go through every single interpandler and Prioritize what you want how he's put that let that's that came from the original path back in 2007 because update DB every time I was working on my working at home I was working to like 3 a.m. And update DB would kick off at 2 a.m. and I'd always know it because my system would just slow down to a halt and Actually, I installed the real-time kernel purposely so I could actually continue working and let update DB not bother me anymore today It really my machines are fast enough that I don't even notice anymore like kept it in there Here's what interrupt happens task goes off interrupt happens this device handler handles goes back to the task This is the interplay. It's what I call an interrupt latency This is the latency or actually the Yeah, I guess latency is good word, but You know the into the disturbance of your task is going to be this huge big that this device handlers really long Handler is you'll only see that in your task with threads We still need the hard interrupt handler. Okay, that's still a hardware feature We can't change the design of hardware. So we have to have software. So we're coming up with our priority high priority task interrupts Going come it's going to go up regardless and we go we wake up the device thread and come right back So this is the latency for us. It's really really tiny and it's consistent. It's almost Every time we have an interrupt handler go off. They're all the same length Then later on this got our high priority test size ago. Okay, I'm done Go to sleep now the device handler could go and work and finish up what it's doing There's some non threaded IRQs Basically the timer interrupt Which is mostly in charge of scheduling the real-time tasks that are going on So we don't want the you know the time if the timer up was a thread and you made it Low priority, you know, basically, how do you stop in schedule? Is there a question? Well, actually we want to get rid of that. I may talk about that later They say whether we want to get rid of taskless and soft IRQs Yes, they sometimes the soft IRQs run an interrupt context sometimes most most of time But they can't be running in a threaded context as well Actually it would help more Yes, in fact, actually I'm working on there's nappy, which means new API I'm actually working on something called a nappy, which I'm going to call it even newer API Which actually is converts everything to threads because the whole idea is I'll talk about right now. This is good. What happens with the nappy code It was basically interrupt will go off process it interrupt go off process it interrupt go off process it and actually It could the CPU was actually the bottleneck here Well, we found out was let the buffer handle it interrupt goes off kick off the soft IRQ Let it wait and then the buffer will handle all these Processes and then when the software you would wake up then it would say okay Now we just do them all at once or we almost and we go into a polling mode And we just like let's keep on just going in a soft IRQ context We just go into a polling mode now imagine if we made it into a thread where we get our first thing We just kick off the thread it goes into a like a permanent polling mode later Q fills up start going to polling mode and as a thread you're still handling the delay actually is Effective to have like the inner goes off post it off let it go Handle the into a polling mode because you can't do it in a hard IRQ soft IRQ You could go into a polling mode, but even then it's the polling mode of that could affect the rest of the system as well So yes, it might help your network, but it's not going to be maybe it's not going to help the tasks that need to start processing These tests so now you can actually control let my task Oh, you know let the interrupt happen, but the tasks that needs to process this start having it work as well so Yeah, basically we're adding the latency to the to the networking stack, right? Oh, yeah Well, the whole thing is you didn't the thing was you didn't want to do the polling in the hard interrupt context Because when you do polling in hard interrupt context that that stops all interrupts so you know on that CPU or you know whatever's maybe you just have like one CPU that's Committed or had affinity for that IRQ when that's running it's going to stop all interrupts So you've got to actually push off to a soft IRQ which is kind of an interrupt context But the hardware doesn't consider it interrupt context. It's only a software interrupt context That's why it's called soft IRQ So I'm actually be working right now. I'm doing some basic tests to see it There's some tests that show that this actually improves networking performance. So I'm we hopefully something will come out with that. Yes Well, the threads will have affinity the threads have affinity to it and actually in our t-task We actually when you set the IRQ affinity it tries to keep the threads with the same affinity We switch back and forth to that Also, there are some things that if you say it's called no, if you put in Some when you declare your interrupt request your interrupt You can say this interrupt must always be a real interrupt and it won't thread it. That's Don't do it. Don't think that. Oh, this is this will help me. It's mostly for like timer interrupts By the way thread interrupts is today in mainline You could actually right now We actually have two ways of doing thread interrupts one is per device where the device could say I want my task actually this one will be working on is making the driver itself say I want my Handler to be a thread so you don't have to create your own you have to do anything about creating work queues or whatever you want You could just say hey give me a thread dedicated for my interrupt and How do you do that request threaded IRQ instead of doing request IRQ? so that will give you a threaded IRQ and What's also nice about this you can actually have other you can have some that one interrupt line if you share an interrupt line You don't like some of the devices could actually have different Priorities for that one same interrupt line. So you can actually have one interrupt line that Handles three different devices and each device have you a separate priority on how that interrupt line works as they're all threaded We also have the big switch One big switch hand way where you could just say give me all interrupt handlers You can do is a mainline say give me all interrupts to be a Threaded interrupt, but if you have three devices that don't have their own interrupt Like threaded interrupt didn't say give me a threaded interrupt. They just said request IRQ Those devices will actually run Those devices will actually share the thread of the other devices that's on it. Yes Well, yeah, but today's systems can handle it Back then they really that was an issue and stuff so today threat and we also we we fixed Contact switching is much much faster today The overhead of threads is much lower. There's a whole bunch of things that the kernel has improved So sometimes looking back the way you did things is beneficial because the whole you know Paradigm of what you're working in has changed where the old some of the old ideas now are applicable today. I Mean who look at your cell phones How many people would use like look back at installing a full kernel with tracing and everything on it on a embedded device? You know look at how embedded device devices have changed See I said the request started I already talked about this. I said too fine. I've this I already talked about this When you'd say like when you flip the big switch We actually create our own little function that will just is what's called for all the IRQs that didn't say I want to thread it will be given a threaded function to use To enable the big switch you not only have to configure it in your kernel But you actually have to go into the kernel command line and say are the current the kernel command line parameter When you boot up the kernel you actually have to say thread IRQs To have it implemented so you can actually have it compile like on a production kernel if you want you actually have threaded IRQs The big switch enabled, but it won't default go that way unless the person put on the command line Hey, I want threaded IRQs Okay Things did not to do in the kernel local IRQ disable. I hate seeing this Really you should never have local IRQ disable in any driver. I Really get ticked off when I look in the driver code and see local IRQ disable because you shouldn't be doing that You're saying that my driver is more important than everyone else's driver. That's what you're doing and Well everyone thinks it is It's that old like it's like my patch is more important than your patch review it now You know, it's the same mentality Don't do IRQ disable in the mainline core kernel. We use it, but we try to we're trying to get rid of them We hate local IRQ disable. Sometimes I see local IRQ disable and I'm looking I'm like why what is it protecting? There's no comment No, nothing to see a local IRQ disable here and bunch of functions are called and then local IRQ disable I'm like, okay. What is it doing? Maybe they just said oh, I just don't know why this broke But if I do look like you disable it will it works now. I really believe that has some of the reasons And the thing is like I said, that's also probably means that there's SMP bug in there Yeah, it works because you just made it non-re-entrant for this one CPU But another CPU could come in and hit that same side. So a lot of times I see local IRQ disable And yes, a lot of times it's buggy.  This is the angel of it. We want we love to see spin lock IRQ save Custom preempt RT spin lock IRQ save doesn't disable preemption because the reason why you have spin lock IRQ Usually is because you have that spin lock will be used or somehow be associated with a interrupt That's using spin locks that's going nest within this guy And that's the case that if you don't use spin lock IRQ spin lock IRQ save or spin lock IRQ If you grab that lock the inner goes off grabs the same lock you just deadlocked But since interrupts are now threads, you don't have that problem anymore All those deadlocks just disappeared because you could sleep the inner go up grab your same lock But you could sleep too because they interrupt as a thread so it could sleep and then go back So all the in real time spin lock IRQ spin lock IRQ save. It's just a spin lock. There's no difference The interrupts are not disabled. Nothing else is done differently But what's nice about mainline doing this is you're actually kind of labeling giving the name the spin lock You're labeling what you're trying to protect. So it's actually good to do this Okay, the less obnoxious but still obnoxious Collar from instead of local IRQ disable we have preempt disable which is the same as exact same problems as Local IRQ save although you allow interrupts to come in you still don't let any task continue So you're like I just don't want any tasks to happen So I'm like preempt disable do a bunch of stuff preempt enable and a lot of times again I don't know what you're trying to protect. Why why do you have this preempt disable? A lot of these things are not commented so What's worse is like preempt enable no resched when you enable it because when you preempt disable if the interrupt comes in It says hey, you need a schedule when you call preempt disable it will actually call schedule But then you had this preempt enable no resched because for some reason you know that you don't need to do that check Usually this is called within a preempt disable So if you're doing preempt disable do something else preempt disable you already know preempt disable is already happening I don't know why you do it twice. Anyway Or preempt disable and you do preempt disable again. They do a preempt enable and you know you're in a preemptable situation It's sometimes okay to use the no resched Because you're not going to schedule anyway because you already have it's a nested preempt disable You could throw this in but this will warn if you do it if you don't if you use this when preempt enable no resched and you use that Outside of a nesting preempt disable it'll give you a nasty warning.   So forget about the slide No, it has the same problems as low-caricule disabled. Yes Yes, yes preempt disable disable preemption on that CPU only so same same thing per CPU Now this is the one time where people will use preempt disabled They'll say preempt stable because I or IRQ disable and they'll do a bunch of stuff with these per CPU variables because hey I'm not preempted disabling. I don't want if I don't preempt disabled I might migrate to another CPU and now the variables I'm working with is not my CPU variables I started with so people use like low-caricule save this but this hurts real time So what we've been time for what we kind of added we added some Functions and that will be coming they're not in mainline yet, but they will be coming soon One thing you always do is you pin your CPU or pin like if you use a pin CPU thread So this task only runs on this CPU you can always use you know per CPU variables without worrying it because that task will never migrate But we added some stuff They said get CPU var actually disables preemption. So we don't like that. So we added get CPU light on non preempt RT It does the exact same thing as get CPU nothing different But when you enable preempt RT what it does is disables migration now. There's some cases where Now you have to wear it out. You can still schedule out So if you are going to modify some per CPU variables and you scheduled out something else may come in and modify those Same per CPUs variables. Well, I'll talk about that later Get local var is the replacement for get CPU var, but there's the same thing it just disables migration They don't migrate disable handler. That's kind of nice So say if you have per CPU variables that you have to protect from other If you have some per CPU variables that if you get migrated out Or if you get sorry if you get scheduled out and someone else could schedule then and goes to the same code path It might use those same CPU variables and mess up your whole situation Where in mainline preempt disable and preempt enable does all the protection you need There's no bug with SMP because it's a per CPU variable And you just want to make sure that no one else gets in there Well, we've added was this local lock which also local lock RQ local RQ safe Which is basically the same as local RQ save or local RQ disable or preempt disable depending on which version you use But in mainline, it's just like that. It either does preempts disabled or disabled IRQs But when you turn real-time on it creates this it creates a sleeping spin lock for you That's associated with the variable you're getting or with the name So you have to this thing actually pass in a name to it and you pass a name in it and it will associate It'll create a lock for you So when it goes into the section you'll grab that lock and then do your work and then but still you could be preempted And if someone else comes along another task comes along it goes to that same section It will have to grab that lock too, but you already have it so it will stop So it acts just like mainline doing a preempt disable, but using locks and allows More reactions happening.  It actually helps you label the code for you. So it basically labels the The critical section so it's more documentation, which is a good for mainline RW locks don't use them. I'm not gonna go talk about it much Is that definite determinism there's problems with because as recursive locking you could there's two types There's unfair RW locks and there's fair RW locks also RW stems I think are fair with her so if you have a read writer lock, which means multiple readers one writer and Say if you have a bunch of readers, let's say it's unfair Which means that first come first serve or you know if you're waiting on it It's basically it's Everyone tries to grab the lock whoever happens to be first to get that gets the lock There's no queue you try again if you miss it you try again miss it you try again go into a loop Constantly trying so say if you have a large CPU system and you have a bunch of readers and they all read it A writer comes along tries to grab a lock, but it can't hold the lock while a reader has it Well, if one reader let's go of it Well, actually if one reader grabs a lock you go to grab the writer goes to grab the lock Can't because there is a reader on it another reader comes along it grabs a lock the first reader Let's go but because the first second reader came in the writer still can't have it You just basically what's called a live lock of the writer lock the writer lock writer will just sit there trying to go And it goes on forever very undeterministic I mean it could actually halt the system per se Then there's a fair system fair lock where you have a bunch of writer reader locks and What happens is when a writer comes in any new reader comes in they'll block they'll have to they can't grab So they block behind the writer now the problem with that is a lot of people are actually a lot of people I wrote code with the reader a locks into me go like oh, it's just a reader lock It never blocks so they'll put it before and after another lock Well, you just created the a bba deadlock associated with that because now a reader lock once a reader lock and block It's now untrue lock and you can actually have it where you do a reader lock Or whatever you have you grab your lock do a reader lock and block where someone else has the reader lock goes to grab The block that you have it blocks the writer never continues deadlock of the system No hurts great for power I just wanted that I throw this up just because if you really if your CPU goes into a real deep sleep when you Enable or no hurts and then you expect quick reaction times when we've had people complain Well, I got this huge latency Well, I looked at the eye like you have like no hurts enabled and you got real deep sleep going on It takes like two milliseconds to come out of the deep sleep sometime You know some systems are gonna take a long time to come out of your system Like I said real-time OS is only as good as the hardware if your hardware can't react to something because it's going into deep sleep You know really the best real-time Idol is idle equals pole.    Don't implement your own spin locks in user space You know if you have a real-time task that depends off try not to use don't have it depend on like I owe that might take a Long time like be smart about it you can you know use memory messaging basically and M-lock all or M-lock whatever get your system. So you don't have to worry about page faulting in and Question actually before questions. I just want to see something. Oh SMI you can't see this. Let me get up it Darn it. Ah What's the control plays plus it doesn't want to give me a there it goes I was running at the time. I just killed it But if you can see I know probably can't see this but I was running cyclic tests During this whole presentation had one this is a laptop on my laptop. I'm running a real-time kernel So and I ran everything I ran my trebuchet program everything had a 231 Micro say max microsecond latency I guarantee you there's SMIs going on in this laptop almost every laptop has SMI's it usually handles thermal If you go in and turn off your SMIs you're most likely fry your laptop So don't do that But that was just the idea this I'm running the real-time kernel on this laptop So any questions?


Well, it's that time of the week again. It's time for Chit Chat Across the Pond. This is episode number 772 for June 26th, 2023, and I'm your host, Allison Sheridan. One of the joys of traveling with the University Alumni Program is that your trips include lectures by professors and a variety of interesting subjects. One of my favorite people to learn from on our trip to Antarctica was Professor Jason Briner. He's a professor of geology who studies Arctic environmental change at the University of Buffalo. I am very pleased to introduce Dr. Briner to you today. Welcome to the show, Jason. Hi, Allison, thanks for having me. Hey, we had fun hanging out with penguins, right? Yeah, that was a blast. That was enough penguins, what did they say? 40,000 on one beach? I wasn't gonna stand there and count them, that's for sure. Well, while we were in Antarctica, you gave a fascinating talk about your research work in Greenland, and when we were working on when to schedule this conversation, you said, you know what, I'm about to go again. So I think it'd be even more fun if you talked to me right when I get back. So I'm just gonna open up the doors and let you roll.  Yeah, well, the Greenland ice sheets are pretty important component of the Earth's climate system. Of course, everybody these days is well aware of climate change and global warming and sea level rise. And globally, one of the biggest contributors to sea level change currently that sea level rise is melting of the Greenland ice sheet. So I'm part of a giant fleet of scientists that are studying the Greenland ice sheet and how it's responding to climate change and how it's contributing to sea level. So like there's people who study the ice from space, there's people who study the ice from airplanes, there's people who study the ice from a boat looking at it, cabbing and making glaciers, and there's people who study how it's changing today and there's people who study how it's changed in the past and trying to link them all together. So I'm like, I bring like one kind of science to the big problem of Greenland melt. So explain what a nice sheet is separate from other kinds of ice and you talked about that some on the trip. Yeah, a nice sheet is a type of glacier. So you've heard of a glacier and glaciers can exist in a variety of sizes. They can be small glaciers like the ones you might envision in Alaska. And then our planet has two ice sheets, basically the biggest category of glacier, something that we consider sort of like continent and scale or nearing continent and size. And the two ice sheets we have in the planet are the Antarctic ice sheet which sits on the Antarctic continent. And then in the Northern Hemisphere have the Greenland ice sheet which sits on the island of Greenland which is part of the North American continent in this case. And these glaciers are significant because they form on land in cold areas that receive a lot of snowfall. And then that snowfall doesn't melt off the following summer. And so in polar areas of the planet you have these conditions where the snowfall builds and builds and builds and builds and forms glaciers. And then they like flow downhill or cav off ice birds into the ocean and then they melt. But in any case it's the poles where these things exist and Greenland is the one in the Northern Hemisphere sitting on the island of Greenland. It's three kilometers thick. And if it were all the melt it would raise global sea levels by like 24 feet. Geez, okay. Okay, that gives me scale. I can't picture it in width or length but when you say 24 feet if the entire thing melted. Okay, so we don't want that to happen. We do not. Okay, I'm definitely not 20. I don't think we're 24 feet above sea level. We might be, but that's tough. So yeah, and the thing about that is it's like you don't want it to go away but even like minor oscillations of that ice sheet have a big impact on global sea level change. Like even a foot of course is really significant. And that's just Greenland. So if the planet is doing something crazy enough to get a foot of sea level rise out of Greenland and it's also doing something crazy on the other glaciers, on the Antarctic ice sheet and these other components that also lead to sea level rise. Right, right, okay. So you go to, I've seen photos and I've got a link in the show notes to Jason's galleries about showing his trips to Greenland and to Alaska. And this is not that you go check into a Marriott when you get there. Describe what it's like going there. Right, usually the style of research I do where we want our samples from and the parts of the Arctic that we wanna study the parts of Greenland that we wanna study aren't in like the towns there, there in a remote part of the landscape or on the ice sheet where there's no infrastructure and people don't typically go. So these are areas we've identified that have scientific importance. So I might go with a colleague from another university. I might bring along a couple of graduate students, maybe an undergraduate student. So it's basically like a couple of professors and a few students getting flown in to some remote location in a float plane or a helicopter, maybe a plane on skis and they kick us out into the tundra or on the ice and all our duffel bags. And we basically create our camp, a remote camp on the spot. So we have all the food we need with us for the next whatever weeks. We have tents and we have our scientific equipment that's gonna allow us to conduct our studies or collect our samples. In your talk in Antarctica, you said that you love that first feeling of being completely disconnected. Why is that? Yeah, I think that this noise that we're surrounded by in our daily lives these days with social media and just plugged in digitally, email on my phone, it's really quite amazing to be dropped off in this location. And the last thing you sort of hear sense of civilization is the plane engines cranking up and then like flying away until you can't hear it anymore. And that's like literally the plug unplugging from the outlet is this thing going away.  I also have young kids at home. And so there's all these elements of how life goes from complicated or there's just the number of things you think about in a day is vast to maybe there's three or four things you need to think of. Your life boils down to some few things and few simple things. Wow, when you describe it as being remote, I could kind of picture it, but then when I looked at your photos, oh my gosh, it's remote.  So let's get into the research of what is it you're doing when you're there? On this most recent trip, what were you studying? How do you go about it? Okay, so I'm just back from Greenland a couple of weeks ago and I was there for about a month and I was part of a project that we call Green Drill. So this is drilling through the Greenland Ice Sheet and in particular, this kind of drilling or why we wanna drill to the Greenland Ice Sheet relates to trying to get samples of the crust, the Earth's crust, the surface of the Earth, the rocks and stuff that are buried by the ice sheet. And so basically some recent headway in scientific progress has resulted in some really interesting new insights about ice sheet sensitivity to climate change in terms of ice sheet history, like how long have these ice sheets been on the planet? How often do they grow and shrink? We know that if we look to the Earth's past and we look at temperature, temperature changes a lot up and down, up and down, up and down, up and down. And this is the Ice Age cycles that you might have heard about. And what we realize is that the Greenland Ice Sheet actually grew and shrank a lot in these Ice Age cycles under natural climate variability in Earth history. That's before we were here.  But we didn't know really how much. And it turns out when we started to analyze, we got this really special sample from the bottom of the Greenland Ice Sheet. It was almost collected by accident. And I was part of a team that made some analysis on these rock samples below three kilometers of ice. And the analysis of those rock samples came way down there. And this is like, this is almost as rare as getting a moon rock. Like there's no other, no one gets a rock from below the middle of an ice sheet. Like it just hardly ever happens. And so this sample was so precious. Like I said, almost collected by accident. So we finally measured it and it told us that that ice sheet, that huge ice sheet that locks away 24 feet of sea level change was gone. Not that long ago, like sometime within the last million years, maybe even more recent than that.  What do you mean it was gone? Well, below it's its thickest part. The rock sample told us it had no ice over it. As recently as sometime in the last million years. So you're saying there's a pocket of something above the earth when you got down there? The ice sheet is covering the sample now. But the chemistry of the sample told us it was exposed to basically daylight. Oh, as in no ice sheet at all? And in order to be exposed to the sky, you can't have the ice sheet there. And that wasn't that long ago? And it wasn't that long ago and the earth was in the middle of this ice age period. And so we're like, why did the whole thing disappear? You know, this recently in the geologic past, the scientific community at that point was like, it probably grew several million years ago when the ice age started and oscillated, it pulsed, but it never went to zero. We wonder now if it actually goes to zero more frequently than we previously thought. So back to the screen drill project. So now this research is now intentionally putting holes in the Greenland ice sheet to stick a needle down and we're taking the biopsy of the bed of the ice sheet. We're trying to drill through the ice and pluck up a piece of rock from below the ice so if we can ask it the same question, when's the last time you saw the sky? And this will give us more information about these oscillations of the ice sheet. Does it really go away periodically? Was it really gone recently? And that's what we're trying to address. So is that to verify the results of the first accidental test? Yes, verify those results. Because your first thing is you've got to assume you either made a mistake or... This is something Andrea just said. You either made a mistake or you've discovered something and you need to know which one it is. Yes. Geology is tough that way. It's not what we call experimental science. It's not necessarily like lab science. You can't just set up another experiment and redo it. Right, right. There's no control variable or control system you set up, yeah. It just doesn't really work like that. So geologists are like Sherlock Holmes' scouring of the earth, trying to sniff up clues and make interpretations about earth history. So in this case, we need to go sniff up more clues. We need more data.  It's not gonna be an exact replication of this first accidental finding. But we're trying to build on that to see if we can basically, yeah, confirm, add, grow, build on that study to learn more about the problem.  Could it have just been a hole in it there but you take a sample someplace else in this giant ice sheet and you don't find that result? Could they sort of both be true? That there was a hole in it but it wasn't gone? Yeah, yeah, yeah.  In this, therein lies the problem. This first accidental sample I'm referring to was some rock collected at the bottom of one of the long ice cores, really famous ice cores that tell us a lot about the climate history of the planet. The diameter of the ice core and of the rock that was accidentally collected below the ice core is like as big as a tennis ball. It's like the diameter of a tennis ball that you cut in half. So really is that middle biopsy? It is and you have this entire Greenland ice sheet. Oh, and we have one sample that's only this big. So your question, what we really, how we try to tackle that is with like models, computer models that try to mathematically simulate the Greenland ice sheet. Wow. And those models say that if there was no ice at our little tennis ball location, then probably that means that the ice sheet had to have really, if not completely gone away, like 95% gone away. Okay, okay. Wow, it would be a whole discussion separate from this, but it makes me wanna go, okay, how do you write these models? How did you figure out how to do that? But that's probably a whole another discussion. It's a huge discipline within climate science. It's these same models that predict future sea level rise. Okay. You can't do that without having a whole research lab devoted to running fancy supercomputer models of ice sheets oscillating through time and how they get impacted by climate because in those models, you need to have the physics of how glaciers flow need to understand that when the atmosphere is this temperature, that means this much melt. So this is a discipline in earth science, this kind of like glacier ice sheet modeling. I like learning this stuff from you because my entire view of what a geologist is is the geologist on Big Bang Theory. And I know you said you hadn't seen him, but I mean, he's just this big, slow-moving, boring guy. Like you can't breathe, he's so boring. And it's like, that's what a geologist is. And it's obviously nothing like that, but learning about all these different fields that you could go into as a geologist doing the computer science of it, that's cool too. But I was really struck by some of the photos that it appears to be your part mechanic when you're working on the ice sheet because the drills and stuff, there's some really great photos up close. I'm a mechanical engineer. I can't help myself. I was really excited by looking at the drills and the equipment and described like the process. How do you drill holes in an ice sheet? Yeah, so I'll start with the answer to that question by relating it to what you were talking about the iconic geologist and what my definition of geology is or the way I would characterize it are, it's as a field, geology or earth science is maybe a more modern lingo buzzword to describe it, is by definition interdisciplinary. It's you're integrating physics and math and chemistry, archeology, paleontology, volcano studies. In order to understand a process on earth, you might need to rely on some computer science. You need to go out in the field and do field work. You need to collect data. You need to do data analysis and all these processes, understanding of how it works requires fundamental knowledge in physics and math and chemistry. And now going forward to talking about this drilling. So I might bring a particular expertise to this project but I'm not a mechanical engineer. I don't know how a drill operates. So we partner with, we collaborate with mechanical engineers who are professional drillers and they're the ones who will collect ice cores for climate studies. And the drillers that we hooked up with have expertise in drilling through the ice but they're not interested in the ice. They're interested in then going into the rock below the ice. So it's this program of drillers called the Ice Drilling Program, US IDP, Ice Drilling Program. And they're the group of people that gets the contract from the US National Science Foundation to support scientists like me in their pursuits to get a sample to study. So we partner with this engineering group. Oh, sorry, go ahead. So we partner with this engineering group, the Ice Drilling Program, and they send drillers with us, the scientists. We all go to the remote corner of Greenland and we work together to get our samples. Okay. Now it's intriguing to me that you talk about drilling just down to the earth. When we were on the trip, you talked about doing a core which is you're pulling a cylinder out. Are you not pulling a cylinder out now? You're just going, I'm just going all the way down. So this first sample that I talked about earlier was that was accidental. To get to the bottom of the ice sheet took like three or four drilling seasons. Oh, wow. Because they were collecting ice cores. You can imagine, here we are, society on earth kind of freaking out about some sea level rise. And we want to go build on this first study to try to learn a lot more information about the Greenland ice sheet. We don't have time to sit on the ice sheet and drill for three years before we get our sample. We need to speedily get down to the bottom and get our sample out. And in order to do that, you need to not collect ice cores as you go. You need to just basically auger through, make ice chips and flush those ice chips out to the surface and then keep drilling and try to get to the bed as fast as possible. So this is like two miles though, right? In our case, we use drills that are good at like, let's see, so the site, our main site that we drilled this spring that I just came back from was 500 meters. I'm an ugly American, I'm typing 500 meters. Like, just go for it. 1600 feet. Okay, type 509 meters. 509? Yeah. 1669. Oh, that's only 12 inches. Oh, that's about 1700 feet. Quarter mile or something, 1700 feet. Yeah. So that was the thickness of ice at this site. So the drill technology we're using, we need to fit it into an airplane, actually a lot of different trips on that airplane. And we need to get it out into some remote corner of Greenland and we need to drill through the ice and then we need to get it all back out all in one season. So that limits how thick you can go through. So the kilometers, the ice sheet is three kilometers and it's thickest. We went to a site that's 500 meters or maybe about 1700 feet thick. And so that was our drill site. We went through 1700 feet of ice. Geez, that's still. And we, it took a long time and we drilled, these drillers are so amazing. It's just incredible, the skill set of these drillers. It turns out, my view is that mechanical engineers, you included, you guys are like professional problem solvers. This is everyone, you're all MacGyver's. It's not so much that you know how to use a piece of equipment, that you know how to fix it or you know how to tweak it to do what you want. It goes way beyond just getting a tool and using it. It's every day, almost every hour, problem solving along the way and always making constant improvements or debugging something. So these drillers to get this thing to work the way they did was just absolutely astonishing. I'm picturing Ben Affleck and his team in the movie Armageddon that can drill anything and they can solve all the problems. That's what these guys were. By the way, that's a favorite movie of these rock drillers. Is it really? Quotes from it the whole time, I like it. So you mentioned when we were arranging when we were gonna talk, you said this trip was challenging. So what went wrong? There were quite a few headaches. I would put those in two categories. One is weather. We had really crummy weather. We had a lot of blowing snow, a lot of high winds, a lot of ground blizzard conditions. I was at, we actually did two drill sites. One at 509 meters and one at 97 meters or like 300 feet. And I spent my time at this 300 foot ice sickness site where we were using a drill. And at our camp where I was, we were getting like 60 mile an hour winds. And mind you, we're intense on the surface of a glacier on a downward sloping glacier. And these winds are blasting off the glacier. There's nothing blocking the wind from you at all. There's no trees, there's nothing. Yeah, exactly. So it was, I mean, that's as extreme as it got, but just even like standard 20, 30 mile an hour winds with all that blowing snow means it's not only uncomfortable to work, but the power supply for the drills are generators, gasoline driven generators, and they don't like blowing snow. So if there's blowing snow, you basically clog up the generator and it dies or you can't use it. So at both drill sites, the reliance on generators meant that any time the wind was really blowing, we had to not do operation. So the first of two major challenges this season was the weather. We had a lot of blowing snow. So we could only work maybe like half the days we were there. So that really cut down at the time to get the job done in the time window that we had. The second challenge was more technological. So this drill, that got us through 1700 feet of ice. The previous, it had only been successfully used once before to go through a glacier, that was Antarctica, and it went through about 500 meters of ice. Okay. Not 500 meters, sorry, 500 feet. Okay. So we're basically more than tripled the depth of ice that this drill went through. So in this new territory of these great depths, there's just new problems that arise that the engineers need to like problem solve solutions for. One of the things that you need to do, remember how I said that the drill goes down and it makes ice chips? If you didn't get rid of those ice chips, the hole would just clog with ice chips. Right, right. So you need to remove all the ice that you're drilling, you need to flush it out of the hole and that's done with pressurized drilling fluid. Okay. So you basically have a constant circulation of fluid. You have a pump on top, you're pushing a fluid down the hole, maybe all these rods that go down that they're drill bit is on the bottom of, they're hollow. Oh, okay. So the simple way to think of it is that the fluid goes down the middle of all these rods and then flushes out the bottom of the drill bit and then runs back up the outside of the drill rod pipe up to the surface where the fluid is re-collected. Okay. So there's a simple pressure, pressure at the top drilling fluid circulation. Simple. Okay. Yeah, yeah, yeah. Simple in concept, but not in practice. Okay. Now the problem with pressurized fluid down the hole, at some point that pressure could theoretically get high enough that you can hydro fracture the ice. Oh.  Oh. And you can no longer drive those chips up to the surface and then you can get clogging. So what makes the pressure go up? Just because of the head? How far down you've gone? Yes, correct. And the amount of pressure you need to flush those ice chips all the way up to the surface of a very, very long hole. Right. So it turned out that we drilled to 370 meters and then the hole got a fracture in it. 370 feet or meter? This was meters. So whatever that is in feet. So remember we're going to fight, we eventually got to 509 meters. Okay. And at 370 meters on our way down before we got there, the hole fracked. And we lost all our drilling fluid or up to a certain level. Can you even get the drill back out at that point? So the drill itself was still bathed in this drilling fluid. Okay. And then water because that would freeze. So it's like a purify kerosene kind of thing. So that's collected in the hole in the glacier and the drilling bit is still bathed in this drilling fluid and it's up higher than it leaked out. Okay. So when that happened, we basically thought the show was over. Yeah. That we would have had all these resources and all this effort out on the glacier and we weren't going to get our sample. Mind you, this is like to put all, this is 60,000 pounds of equipment we had to fly under the ice sheet. Right. We had about 12 people out there. We had this like mini tent city. You can see in this, some of these photographs on my gallery. So it's a pretty sizable encampment. This is a pretty big operation especially relative to what I'm used to which is like a couple of professors and their grad students. This is like a pretty big operation for what a lot this kind of work is. And you're only able to work every other half of the days because of the weather. Half of the days because of the weather and then this thing happens and then this thing happens and it's like, well, we only have like a week and a half left before the planes coming back to start shuttling us off the glacier. So there was one thing that we could try which is to pull everything out of the hole and, okay, maybe I need to back up. The surface of a glacier is snow. And eventually at some depth that snow gets compacted into ice. Okay. That's how like most glaciers are. In the snow part of the glacier, the top part, you can't make a hole and circulate fluid through it. The fluid would just leak into the snow path. Oh, right, right.  The top of the hole is lined with this metal piping that we call casing. And the casing comes down that goes both through the snow part of the glacier. And once we get into the ice density range then we can just drill a hole into the glacier ice itself. Makes sense. The junction between the coast part of the hole that where the casing is and the ice part of the hole is the specialized coupler that couples the case part of the hole with the uncased part of the hole called a packer. Okay. And the packer is basically like it's in a long dated intertube.  Okay. And it sticks it into place and the casing screws into the top of that. And then below that it opens up to just the ice hole. Yeah. So, okay. So it turned out this packer was about where this fracture took place.  Put the packer back in. Then the casing gets to go lower or the packer gets to go lower. Yes, exactly. Then you could put the casing lower and cover up that hole. Seal that fracture. So that's what they did and it worked. Oh my God. But it took them like a week to do it because they had to do all this incredible manual labor to pull everything out put everything back in, re-drill it, reset the packer, re-put the casing in. So like a week or something later meanwhile there you can't work a couple of days because there's bad weather. You're looking at your watch.  So they put all that infrastructure back in and then after a few more problem solving steps they hit the bed at 509 meters. Wow. Planes come in the next day. Oh geez. Then a storm comes in. No. Planes not coming the next day. Yay. Storm and yay. Storm ends at like 5 p.m.  So the other thing is once they drill through the ice column and they hit the bed, they need to pull everything up and put a new bit on the bottom. Because you gotta go pick the stuff up. Yes, not only, yes. Because now you're going from not collecting any core to collecting core and you need a new bit that's, the first bit is just remember it's, all that needs to do is chip out ice. Now we need to drill in rock. So we need to change the bits from an ice bit to a rock bit. And we need to put in this particular tube that collects, that is designed to collect the rock as we drill into it. 5 p.m. Storm abates. They all run out to the tent. They work all night long. They collect seven and a half meters. Okay, so like 25 feet. Oh, a lot. Of core material. Like a 25 foot tall core. In increments, they pull it up. And then they sleep like 24 hours and then the plane starts coming and shuttling people off the ice. Oh my gosh. I'm sorry, but I can't stop having the analogies to Armageddon. You know, I'm thinking, no, we've only got 15 more seconds before the nuke blows. You know, here he comes. Here comes the plane. We gotta get it. Wow. It was stressful. Wow. Yeah, yeah. It was tense and these drillers, you know, not only are they professional problem solvers and like they just made this happen, but they have a tremendous amount of pressure on their shoulders, right? Like, all the resources, yeah, yeah, yeah. And it's never a given, right?  You know, it just hasn't been done that often. It's not like some like surgery that's so routine that it's just a no-brainer that it's gonna happen. Like this is really at the cutting edge. Like I said, this is the first time this drill's been used, first time it's ever been used on Greenland. It was used through an ice thickness three times more than the previous time it was used. And so it's really at the leading edge and these drillers need to acknowledge that its success isn't a gimme. Yeah, definitely. Can I ask a really obnoxious question? What are you and your team doing during this? So you guys playing poker while these guys are drilling? What were you doing? This is a great question actually. So like I said, we had two drill camps and I myself was down at the drill camp closer to the edge of the ice sheet where we were getting these windstorms coming down the glacier. And our drill was a smaller drill and it was designed to go through while we went through 300 feet of ice. That actually required all hands on deck because every time we were going into the bed below our 300 feet of ice, it turns out we would get a piece of the core like 10 centimeters in time and pull it up. We eventually from the camp I was at with this 300 foot drill, we got six feet of a rock core. But it came out in like, it came out in like six inch increments. Every time it came out, we had to put rod 300 feet down a hole. So these are like five foot rod sections. They screw together. So you're screwing together a five foot rod to another five foot rod, dropping it down the hole, screwing another one on top, dropping it down the hole, screwing another one on top. Repeat, repeat, repeat until you get to 300 feet.  Can't get any more. Okay, pull it up.  So the drill camp I was at, not only were the drillers busy, I was busy, my graduate student was busy. We were all really busy. It was a lot of manual labor. Yeah. Up at the 1700 foot camp, the drill was a bit more mechanized and the drillers there were a little bit more separate from the professor who was there, the graduate student who was there and so on. But it turns out that that's a huge camp that needs to be maintained. And here's one example. I talked about these wind storms. It turns out that if you're on the ice sheet and there's nothing else on the ice sheet and the wind blows really hard, the snow just keeps blowing and the surface of the ice sheet doesn't change from before the storm to after the storm.  And on the downwind side, you get a huge snow drift, snow drift. Now picture not one obstacle on the surface of a glacier with a lot of wind, but what, 15 different tents and like equipment and barrels of fuel and barrels of gas and generators, the whole camp had these ginormous snow drifts everywhere. Then the next storm, the wind might change direction and the wind comes from another direction. So now he has huge snow drifts in this other direction. So basically it was a full-time job to be constantly digging out the tents after each one of these snow storms. You don't know grad students this stuff beforehand, right? You know, I've purple tunnel from making these connections. And being- And how strong is your back? Exactly. Wow. So there's a lot to do to maintain a camp that even if you're not in the drill tent helping the engineers as the science team, you're maintaining camp. Wow, that is, I'm just baffled at how you get anybody to agree to go do this, but I bet it's an honor to get to go do it. Yeah, you know, the kinds of people that are attracted to this discipline or this kind of science are those folks who are interested in this kind of adventure. Yeah, yeah. That's interesting. So the total length of time that you were there this time, you said it was a month? I was there for a month. The team, the whole team, was there for basically two months. There was a week or two, two-ish weeks in the beginning of the field season when we were sorting through all our equipment, staging what's gonna go with what plane load and getting everybody and all the equipment shuttled onto the glacier. That was a couple of week period and there was about a week-ish or more period at the end where things were like shuttling off. So time spent on the ice by anybody was probably like seven or eight weeks. And then various people kind of came and went during that overall period. And I was one of the people that kind of came and went. I came in after everything was put on the ice and then my objective was to work with the subset of the drill team and we got deployed to the second drill site. And then when that job ended and we went back up to the top, that was about when they were really struggling to know if they were gonna get it or not. I shuttled off and came back to not rate that point. And then they did a bunch of problem solving and that's when they later then got that 25 foot long core from the bottom. That's crazy. So as a scientist, what kind of science are you doing while you're there? Or is it really mostly the manual labor of making things go? It's mostly a sample collection mission. So we're collecting, the goal was basically to collect samples. Those samples we analyze back home. Most of the science we do is after the field season when we have these samples back in our labs. Okay, so how important is it to you to be there and experience it as part of what you're doing? Yeah, I mean, there's maybe two reasons why we would go. One is so we can appreciate what was all involved. But the other reason is that there are decisions to be made about how the sampling is going and should we go further to get the sample or not? And these are decisions that are best made in the field when you're part of a team. When the sample's coming out, do you need somebody to catalog it? You need somebody to maybe sort of like log it as it's coming out? It would be important to keep those in order, probably. Keep it all in order, yeah. So when we have this 25 foot long core doesn't all come up in one big, long 25 foot straw. It comes up in chunks. And so we gotta be really systematic how we catalog all that. This is amazing. It's just stuff that I don't know anything about. And I love learning new things that I don't know anything about. And the importance of the work you're doing, obviously there isn't anything more important to the world than what you guys are working on that's really astonishing. Yeah, that's nice of you to say. I think the climate change issue is we know a lot more about it than we did. I think a lot of the challenges we face as a society is on the action side. But certainly there's still some questions on the science side as well. One of the things that really struck me on the trip, everybody on this trip that we were on was from some university somewhere. And one of the things I love about that is anybody you sit down next to no matter where you have anything in common in life with they're interesting and interested. Everybody was talking and telling stories and listening to each other. And it was really great. You did a talk about the core sampling done in Antarctica into the ice sheet there. And you explained how through the core sample they were able to find pockets of air going back. How many, I wanna say 800,000 years? Nice. There you go. Yeah, exactly. I'm really bad at remembering numbers like that especially the scale, I get it wrong all the time. But so they were able to go back 800,000 years and look at the air samples over time and they could measure the CO2 and methane in those pockets of air over time. And that's how we get those nice cyclical graphs you can see where people say, look it's not humans because look it's doing this nice little cycle. You see it happens all the time. But the graph as it goes to when industrialized society starts suddenly the scale of CO2 and methane is off the charts in height. It's nothing like the natural characteristics you saw before that. And I had seen that graph before but I hadn't heard it as well described as the way you described it. The fact that I can repeat it since you explained it really well. But I sat down with somebody at lunch right after hearing you talk. And I said, wow, that was really amazing.  And the person I was talking to was an engineer and his answer back to me was, yeah, but I'm gonna need to see the data. And I couldn't speak for a minute because I couldn't think of any answer other than that was the data. You just saw the data. The data is what you just saw. That what is wrong with you? And how do you, how can you stand it? Cause I'm sure you get this more than I've ever heard it. How do you even respond to something like that? Yeah, it can be tough. Yeah, the data, I'd like to see the data. Just showed it to you. Do you want to say it again? It's a real shame that something that is 100% scientific, grounded in science, has been hijacked as a political tool to bend people into their political parties. This is what politicians do is they choose a topic, polarize it to try to garner the votes into their camp. And it's a real shame that that happened to climate. And I start by saying that because in some ways, there's some people that you could have a conversation with and you'll get somewhere and there's others that you won't. And I think it's a little bit sad, but in my experience so far, maybe it sounds like a little bad to just say, I've given up, but like there's certain people, no matter what you say, they'll come back with the yabba or the sound bites that they heard somewhere. And it's just, you can tell you're not getting in and you might never get in.  You don't get anywhere with it. The thing I always think about with a subject like exactly what we're talking about is what possible motivation would anybody have to get everybody all excited about this if it wasn't real? You know, there's no motivation. There's motivation on the other side to say it's not real because we can make a bunch of money drilling oil and burning up dead dinosaurs and ruining the atmosphere. There is motivation on the other side. There's no motivation to creating panic or fear or worry other than we need to fix this. There is no other reason to do it. It's not like a hobby. Yeah, my favorite one is that the professors are in it for the money. Grants and billions of dollars that goes straight into our personal pockets, you know, pass me that. Obviously, yeah. That's what I've always heard professors make just huge amounts of money. Yeah, yeah, yeah. That's probably the biggest problem in society. Well, I think this has been really, really interesting and I love talking to you about this. Maybe when you learn more about what you found, you could come back and tell us about it. I would love to hear what you found in seven and a half feet of core of the earth from underneath the Iceland ice sheet. That is just fantastic. Yeah, it's definitely something we're really excited that we have these samples. And yeah, I mean, we're just like jumping into our labs right now to try to analyze it. Yeah, go science everything now. Yeah, we got to science the blank out of it. All right, again, you are from the University of Buffalo and we've got a link to your website where everybody can look at these pictures and really appreciate the work that you're doing here. Thank you so much for coming on. This was just such a joy. Yeah, it was a real pleasure. Thank you so much. I hope you enjoyed this episode of Chinchat Across the Pon Light. Did you notice there weren't any ads in the show? That's because this show is not ad supported. It's supported by you. If you learned something or maybe you were just entertained, consider contributing to the PodFeed podcast. You can do that by going over to podfeed.com and look for the big red button that says support the show. When you click that button, you're gonna find different ways to contribute. If you'd like to do a one-time donation, you can click the PayPal button. If you wanna make a recurring contribution, click the weekly Patreon button. You're only charged when I publish an episode of the NoCillicast, which let's face it, it's every single week, so I don't charge Patreon for Chinchat Across the Pon Light or programming by Stealth episodes. Another way to contribute is to record a listener contribution. It's a great way to help the NoCillicast ways learn from you and takes a little bit of the load off of me doing all the work. If you wanna contact me for any reason, you can email me at alison at podfeed.com and I really encourage you to follow me on mastodon at podfeed at chaos.social. Maybe you wanna talk to the other NoCillicast ways. You can do that in our Slack group at podfeed.com slash Slack. Thanks for listening and stay subscribed.