Team keep it clean. You know, I love y'all right like and I gotta let y'all know that I love y'all and the way that I can show My love for y'all is by letting y'all know like look sit down Relax, we're probably gonna be here for a little while cuz I feel like we got a lot to talk about So I don't know if you're gonna have to break this video up in parts Watch it at different hours of the day watch it on different days Whatever you got to do, but I appreciate you're watching Whenever you do watch make sure you subscribe to the channel so you don't miss nothing because it gets It's always some going on with the ravens man. It's always something. But anyway, um last night Lamar Jackson, I guess he said he did a full body workout So he had just woken up from a nap So he was up. He was up and he let the twitter world know that he had been up um, and Steven the Steven Ruiz on twitter um, he had put up Uh, he said two minutes of throw from Lamar Jackson's last full game and it was a game against the Jacksonville Jaguars and That was some bangers in that game. But anyway, all right, we all know how that game ended. That was a heartbreaking loss Oh, man, that was a that was a tough one. But anyway Lamar Jackson, he had quote tweeted the clips of that game and he said this game And he put a sad face. He said replay replay brah And that was a game that Lamar Jackson. He took that loss very very hard Very very hard as we all know. I mean we took the loss very very hard because that was just like Ah It was right there the ravens were up big the jaguars came back Ray and y'all know the end and then they'll super long field goal. Justin tucker if he would have made that we would have Oh my goodness. But anyway, um So Steven Ruiz the Steven Ruiz also tweeted he said this is Lamar Jackson's career stats versus the coaches of teams Who have publicly said they are not interested in him So he put up all the numbers and stuff and Lamar Jackson also quote tweeted that And he said I can't wait to see you all and put a winky face So basically saying like all right all these coaches who've been publicly saying oh, y'all not interested in me All right, I got something for you um now Somebody replied to that and they said uh, this is coach kush zero zero He said had the chance to see them in the playoffs last year, but you sat out And oh, we know that that was big last year because there were a lot of people that believed that Lamar Jackson He chose to sit out instead of being out there with his teammates and that had been circulating for a while The rumors were flying and especially What added fuel to those rumors are really in my opinion what started those rumors were John Harbaugh, we know john harbaugh. I was just talking to my guy agent bond zero zero about this on twitter a couple minutes ago And he brought up how john harbaugh knows john harbaugh. We all know john harbaugh is very vague when it comes to injuries And that was no different with the whole thing with Lamar Jackson My point of view. I said I wish that harbaugh would have just said if he knew that Lamar Jackson was done I wish he would have just said hey Lamar Jackson is done for the year. That's a wrap. It's over His season is done Um because that would have let us know that would have let the media know that would have let everybody know Okay, Lamar Jackson. He can't play this year now on the flip side on the business side of things I get why you be like hey, well, hey Lamar's week to week they did the same thing last year And he was done. But they said Lamar's week to week Because that keeps fans Interested that keeps hope alive for Lamar Jackson returning So that keeps all eyes on your team watching and waiting to see a is Lamar Jackson going to come back this week Could this be the week? And like that helps so I get that too But if they knew that he was done I wish they would have said he was done now Another possibility could have been like hey Harbaugh didn't know if Lamar Jackson was truly done or not because there were rumors that Lamar He was going to his own doctors and not the ravens doctors and if those rumors are true I mean especially after everything that's come out over the past couple of months Especially when it come to Ravens team doctors I could see why he would do that but either way. I just um Something that we talked about before too I wish Ravens would have put everything to bed sooner Because they didn't and it really Was a bad look They made Lamar look bad And because I know a lot of people they were like oh man It's immature when when Lamar came out and addressed the rumor on twitter when he came out and addressed the injury on twitter He said hey, look, I want to be out there with my guys, but my pc is it's not good I can't I want to be 100 and I can't do it. I can't do it I just wish the ravens would have addressed that before the way before Lamar did There was one point in time when um There was the rumors going around oh Lamar is not showing up for the for the training He's not showing up for the treatments and whatnot Um, and that was going around for a little while then ravens on Baltimore ravens.com. They had um Was it Mike Preston put out an order? No, they had a cliff cliff I can't think of his last name right now, but they had him put out an article Oh, yeah, it was Mike Preston that put out of the article Saying that Lamar Jackson had been missing the treatments. Um, but then they had clifton brown That's his name They had clifton brown put out the article on Baltimore ravens.com That Lamar Jackson had been making his treatments and then uh, they I guess Somebody from ravens. They tapped Mike Preston on the shoulder and said hey fix that Fix that article and then Mike Preston did a did a 180. I was like, oh, no, no, sorry Lamar Jackson has actually been showing up for treatment Okay, what is it gonna be? um But anyway, I had wished that the ravens would have addressed stuff sooner Instead of like, oh, well, maybe he's weak to weak But again It's just it's so much that we don't know about that whole situation But we we saw so many people turn On Lamar Jackson calling him a quitter saying that he sat out Saying that he quit on his team But Lamar Jackson addressed that anyway, um, coach cush zero zero again He said had the chance to see them in the playoffs last year But you sat out he was directing that to Lamar Jackson and Lamar Jackson said if I wasn't hurt Lord knows we would have been hunting. You know that stop trying to be a comedian So Lamar Jackson again letting it be known that he was hurt So then I guess Lamar Jackson was like, you know what look Let let let me clear this up because I've heard this about me heard a lot of people been saying this about me You know what let me clear this up right now and he said Uh, I don't remember me sitting out on my guys week one versus Jets to week 12 versus Broncos How come all of a sudden I sit out because of money in which I could have got hurt at any time within that time frame When we know the Super Bowl been on my mind since April of 2018 So There were people that were not only saying that Lamar Jackson quit on the team But he quit on the team because he did not have a contract And again It was circulating around like crazy So this is Lamar saying like hold up That doesn't even make sense because if it was really about the contract Why not just sit out week one through 12? Why not just sit out from the beginning of the season because he didn't have a contract then Why would he choose to sit out? All right. Well, why don't you sit out then instead of well now instead of then It just wouldn't make sense and then he followed that up and said let's get real I'd rather have a 100 pcl then go out there and play horrible forcing myself to put guys in a bad situation Now that's selfish to me So Again, and this is what he said before too that he'd rather be 100 He'd rather be 100 instead of going out there halfway Half-stepping and just it wouldn't be a good look for because they feel like that would have been putting himself before the team Because it's almost like this whole situation was like a loose loose because he's hurt obviously so he can't play But if he would have played and then especially if he would have been out there hurt and playing bad and not being able to Be himself and whatnot people would have been like, oh man, man, this dude should have sat out. He should have stayed He should have stayed at home So again, it was like a loose loose form regardless um, then somebody asks, uh Boats and balls. I don't know what that word is. But anyway, somebody asked him they say real talk Lamar How come you did not travel with the team for the for the playoff game? Because there was some game where we saw Lamar Jackson travel with the team and then there were other Other times where he didn't travel with the team and that was a thing that a lot of people took They were like, oh man, he's not traveling with the team. Oh man. He quit on his team. He's a bad leader. Oh my goodness Not realizing that Ronnie Stanley, there were times where Ronnie Stanley didn't travel with the team when he was hurt JK dobbers didn't travel with the team when he was hurt. That was like this happened but with Lamar obviously Lamar is The player on the ball tomorrow Raven. So he's gonna garner the most attention. So I get it But anyway, um, that was another thing too. Why isn't Lamar Jackson traveling with the team? He said after I traveled to the Pittsburgh game my pcl got inflated So a few of us discussed it and I got the okay to stay so I could try and recover faster So there you go There you have it Lamar Jackson just addressing everything that people had been saying everything Because that that that was so much of what was going on and what was going on through a lot of people's minds All Lamar Jackson quit on the team all Lamar Jackson He ain't playing cuz he ain't get paid from the Ravens All Lamar Jackson not even traveling with the team. What a bad leader And now this is him saying oh no, no, no, no, let let me clear all of that up for you Just so you don't have anything twisted any longer I said, hey, I love it and and this was late last night And I was like, look, I was thinking Lamar like man It's too late. I ain't about to make no video on this last night. No, I was in my bed relaxing I looked at it. Oh, okay. Cool. But yeah, we talk about it tomorrow come on for like getting up but anyway, um continued uh Hector Scott Hector. He said Um, there's no denying the Super Bowl is on your mind considering how much you've improved But money is clearly on your mind as well. We don't know what's going on behind the scenes But you stated yourself you wanted to be a billionaire. Why does that matter to you? I will never know but at the end of the day I hope it works out for you and you don't forget who had who had To your back from day one. Oh, I guess he meant to say who had your back from day one The Ravens is at the top of that list despite what is being speculated So, um He's saying the Ravens had Lamar Jackson's back from day one And this this this part right here Um, not saying that Hector's saying this it seems like it but not saying that he's saying this but Even if he is or is not Outside of what Hector just said Um, I see a lot of players. I've been excuse me a lot of people So, oh The Ravens were the only team that drafted Lamar Jackson. He should be appreciative of that And I'm sure he is like they gave him an opportunity he played and whatnot did this thing out there but People make it seem like he wouldn't have been drafted by somebody else That he wouldn't have got picked up by somebody else Like the Ravens were the only team that would have given Lamar Jackson a shot and whenever I hear people say that it just I just hear so much entitlement Like oh He should be proud that our organization the Baltimore Ravens gave him a shot And it's like they making it seem like he doesn't appreciate the shot that he's been given And also like he would have never got a shot from anybody else But anyway, just my two cents on that but back to Hector um Lamar Jackson replied to me he said who doesn't want to be a millionaire what And Hector says someone who is humble and knows that money ain't everything money is just a tool man Get what you're worth. But like biggie said more money more problems And Hector took the hip hop route. I see And Lamar Jackson said he said Hector Zaroni now. I'm not humble I understand that biggie quote But you might as well tell everyone in the world with a little more than someone else the same thing If this is the case, but Be blessed and love you broski So Lamar ended that on a positive note. Um, because it could have gotten really ugly, but uh, Lamar ended He ended it on a positive note. So it didn't get ugly It seemed like he was heading in that direction, but um, he didn't let it And he had also said all in all, I love you all don't ever think I have an attitude speaking on the net Only if we're losing I don't know anyone who smiles after losing or play around See and that's him saying like look like when he loses he be heated as we know as we've seen Uh, but anyway, he continued. He said you should not play sports, bro Other than that, I'm in great spirits at all times.  I just got to make sure you having the lj fund again this year And to that he did not respond. So Okay, but I did see it on the on his uh fan page that it's going to be I think it said on july 8th of this year So I got my question answered, but anyway, um That was that so Kalaeus Campbell Kalaeus Campbell and we're going to talk about this a little bit later probably too But Kalaeus Campbell was like hey look if Lamar if Lamar's out then I'm out too because the Falcons they end up signing Kalaeus Campbell But one of my guys my guy jt He said he feels like the Falcons are actually, um doing negotiating with Lamar With everything that they're doing he said they're negotiating with Lamar in plain sight And initially I was like, hmm Because the Falcons were one of the first teams I think they were the first team on the their official twitter account There was a report that came out that said the Falcons. Oh, they're not going to be pursuing Lamar Jackson and the official twitter account Look in your twitter accounts like team twitter accounts. They never do this ever But there was a report. I forgot what reporter said it but the reporter put a falcons are not going to be pursuing Lamar Jackson And the Atlanta Falcons they retweeted it But Team team accounts they never do anything like that ever ever So I was like, oh that's that's weird That's really weird and obviously it's been extremely weird how so many teams and reports have come out But teams saying oh, we're not going to be pursuing Lamar Jackson and it's like it's like People have talked to different gms and head coaches and whatnot and it's like Teams and and coaches and whatnot front office people have been almost going out of their way Going out of their way to let it be known how much they will not be pursuing Lamar Jackson And but my guy JT went when he said that initially I was like, hmm But then I thought about it. I was like, wait a minute It actually kind of does make some sense Because if we remember if we go back little before a free agency officially started It was a report that the falcons and the Ravens were very close With trade talks when it came to Lamar Jackson That was a report and I know some people believe that some people didn't want to believe it But that was a report The falcons and the Ravens were very close when it came to trade talks They're very close, but obviously nothing happened um But uh Arthur Blank I think it was he came out I think it was yesterday And he talked about Lamar Jackson's injury history that that is injury concerns with his style of play um And he said it's a it's a different scenario than it was with the Sean Watson um So with him saying that it was it was interesting it was I'm just like, okay And then uh, of course they recently got or they today they just got Kalaeus Campbell and Kalaeus Campbell continues to talk about how he wants to win a Super Bowl or whatnot He wanted to win the Super Bowl. He's going to Atlanta That's what he had it too Nobody all that my friend. I think hey, they must have cut that check right But then at the same time my guy brought out he said that uh Atlanta said that they're comfortable with their QB's That is a different feeling from feeling good about your QB's so It's something to think about It is something to really really think about when it comes to Lamar Jackson um That would be something I know Jeff Zribick in an article that he had posted Recently, um, he talked about how the Falcons their offensive coordinator is one of the best in the league and he talked About some of the best fits for Lamar Jackson And he talked about how their offense Offensive coordinator is one of the best in the league and what he would be able to do over there It would just be a really really good fit um So I don't know Let's see again with a lot of this stuff. We just we really won't know until we know Now, um, jocina Anderson um, she addressed The Odell Beckham jr. All the talk surrounding him Because uh, obviously yesterday and we of course talked about it Um, jocina Anderson reported that Odell Beckham jr. He was going to be talking to some different owners Over in arizona. That's where he trains at And that's where the owners meeting is at. Um, but he was going to be talking to some people over there Just talking having some conversations and she mentioned the Baltimore Ravens That were going to be one of the teams that he was talking to And we know that the Ravens this whole Odell Beckham jr. Ravens thing has been going back and forth for a long time A long time I already told y'all like i'm not getting my hopes on it I think it'll be nice be nice addition, especially if he is 100 percent healthy and everything is cleaned up um, the right way But i'm not i'm not getting my hopes up. I'm not I would like for it to happen along with another receiver too But we'll see. Uh, but anyway, she said, um That your team like somebody asked her Uh, they said 90 percent of us Ravens fans thought that jocina Anderson misspelled baltimore included myself because In the report yesterday, she said built more. She said that uh, odell beckham jr. Was going to be meeting, uh, and built more With the ravens and yeah, a lot of us did think that Um, and he said sorry jocina Anderson, uh, Ravens and built more arizona are way too close to baltimore If you're talking about the ravens without the arizona behind it and built more And she replied to that she said yes as in the built more hotel in arizona And she said no worries your team is putting in a noteworthy and competitive effort To add a top flight receiver for lemar jackson and their new offense under todd munkin period so that That was significant too With this whole thing It's funny. A lot of people um, I saw some people say that last night with lemar jackson They were like, oh lemar jackson answering uh ravens fans questions. Oh, yeah. Yeah, he's staying for sure You are definitely gonna be a baltimore raven this year And we'll see we'll see. I still got my doubts um But we'll see um, but with jocina anderson saying this That the ravens are putting in a noteworthy and competitive effort to add a top flight receiver for lemar jackson. She didn't just say um, oh dale beckham jr. She said they're putting in a noteworthy and competitive effort to add a top flight receiver All right ravens, let's see Let's see Because We know the ravens are the kings of trying at the yrc position We know the ravens are the kings of almost We know they are the kings of hey, we tried but It just we just didn't get it done let's see Let's see what happens Again now getting my hopes up for anybody now get my hopes up from deandre hopkins Now get my hopes up all day back on jr. Now get my hopes up for definitely not from mike ever Like that's my wish list, but I know that ain't happening But let's see Let's see Another offseason hoping that they would do something like that. But hey, we'll see But that stuck out to me, but also when she said for lemar jackson and their new offense under tide monkeen period So um by her saying that the way that she said it um She obviously hardball and they the ravens. I'm sure they don't want lemar jackson to go Of course the report came out yesterday that if they got the right offer And that was actually from jeff's rebix article uh in the athletic if the ravens got the right offer for lemar jackson then They could trade them Yeah, they'd be interested in trading them if they got the right offer and I mean Yeah, that's that could be said about a lot of players and a lot of quarterbacks even but um That would like make it seem like the ravens are really really confident In lemar jackson being that guy It would make it sound like they're they're really really planning on lemar jackson being that guy Will he be that guy? We don't know They don't know I hope he can be but again, like I said, I I got my doubts But Hey, my doubts could end up being wrong Martin does ends up staying Or whatever happens Whatever's gonna happen. It's gonna happen. I know that that's what my boy kev told me the other day He said hey, what whatever whatever whatever gonna happen gonna happen That's exactly that's true. That's right. That's real Whatever's gonna be is gonna be and we'll see when we get there Um, but anyway, I love y'all team keeper clean. I appreciate y'all Thank you for watching. I don't know if you had to break the video up Go too long a little long, but yeah, man I just had to make sure we got we talked about everything that we wanted to talk about I love y'all so much. I appreciate y'all. I know some of y'all have sent questions from subscribers. We got them We got them. So just be patient, please Even though I know y'all already are So thank you. I hope y'all have a really really great day And like maybe lemar jackson will be maybe he won't be we don't know but we out


Good afternoon everyone. This is Melissa with the stockswish.com and welcome. Welcome, I thought I'd do a market update here. It's one o'clock, one o'clock on Monday, June 16th. To start off the week, what is the market doing? Holding, holding very strong. Despite the fact that it came down, didn't go over the high, came down and looked like it was gonna break the low. I saw that happen after I closed out the trading room this morning and I saw when I was doing this right bar and I said, don't think it's gonna break the low. It's not gonna break the low. The market's gonna hold. Even if it comes down, it's gonna fake like it's lower but it's still gonna hold. And it's holding. Market triggering the buy set up today. I didn't think the market would even do that today. I thought the market would hold bullish slash neutral today. Call for today for me when the market was bullish slash neutral, meaning sideways. And however, we actually triggered the buy set up very early today, happened by 10, 10, 15-ish. And then I said, well, I guess it's possible we can go over the high today but I really didn't think that we would go over the high today. But we did trigger the buy set up. And I think we closed green today and we are gonna close green today and it's one o'clock and there's three hours left in the day. Market isn't even over the high but it's not gonna break the low. It's markets along. And people still keep trying to short this market. I think I'm gonna write a song. It's gonna be called Don't Short This Market. Don't short the market. Don't short the market. Don't short this market no matter what you do. That's gonna be the song. Don't short this. It's really actually hard to believe that people are still shorting this market. Eventually they're gonna give up on it. Target for the spy is 200. It's gonna get there. I don't know when but it's going to. The sooner it does, the cleaner it'll be here. And if we don't, okay, get over the high and make a new high and get up to that number in the next two weeks, really this week basically, before the holiday, before the July 4th holiday, then it's gonna be a choppy month. It's gonna be choppy, choppy, choppy, choppy for probably about a month's time. And then the market will finally do it. So what I'm looking at here for this market is a possible pull in all the way into here but low odds. I don't think that happens so because this is so close and not that far away, I don't think the market is gonna take as long as it needs you to get up over the high. I think the most likely thing we do is continue up and we may base and base and base and then do it. We could come in a little bit here more if we want to before we do it but I have 100% conviction we're not breaking this. So I don't know why people think that we're gonna come in because we're not because even if we come in, this isn't that far away. It's only $3. And this will take us into July if we do come into here even and retest it but low odds that it happens. Better that we come into here, follow through higher or just follow through higher or slosh around in here until we follow through higher. So what's happening here in many, many stocks, okay. Bullish stocks bearer stocks. Things aren't seeing the continuation follow through that they wouldn't normally because there's an uneasiness going in here where people are not making decisions in reference to the market positioning of their trades. People are not wanting to go long even though that's really the right direction to be playing core trades right now unless you're in something that's very weak. People aren't wanting to take new positions or go long and they're not sure if they should hold the long because they have because people think the market's extended but it's not. It's not and people will know that once the spy gets to some crazy number like over 200. And then eventually everyone will stop shorting this market but until then I'll keep singing the song don't short the market because people are gonna keep trying to short it and in reference to bearish stocks to get the follow through all we need is the market to make a decision either one way or the other. And the decision really is to go right now break over the high sooner rather than later or slosh around and then get over the high in the month but then this market isn't gonna come in. I'm not even looking at that as a possible option because it's just not gonna happen. So here we are again here now we're continuing continuing up look at this one o'clock. So and people shorted this but we had already gone over the high of the day when people shorted this. Knew we weren't gonna break this low. 193.68, 193.60 could have actually bought the market here in the retest of the low today because it was like so perfect. So perfect. Anyway, it's very strong market. Nice pulling in here that's happened. So, you know, we'll have to see how this week shapes up. It's gonna have to do how we really gap tomorrow and then follow through whatever we do or where we gap tomorrow whether it's up or down or neutral. I we're probably gonna have a neutral gap in the market tomorrow morning. I have to look and see whether he comes out but we're gonna close green today. I think we closed in the buy setup today. I don't think it's gonna be a monster green bar. I think it's gonna be medium but either way we're not gonna break the low of the day today and the spy here is just not gonna happen. Markets very nice and strong. Any retest of the high that does not break over it will be very sloppy for this market. People will make an attempt to short it then again. In other words, if we follow through here this week and we go up to 196 and don't get over it and don't trade over it bullishly and retest it, market will be sloppy for another month, you know. So like the second 30 week in July. So the best cleanest option for the market would be to follow through in the current trend. What's happening to people now is the indecision. Not sure what to do. Feeling like it's too late to buy but it's not. And people not knowing whether or not to dump stuff that's weak or shorted because I don't know what's gonna happen with this market because they may think it may come in and they're not sure what to do. But the fact is that the market really isn't gonna have any significant pull-in and even the maximum level of support pulling that this market could do, it's just, I don't see it happening. So nice, nice beautiful bullish market here today. Spy should close looking very similar to the way it looks right now. Maybe a little bit bigger maybe a little bit smaller the green bar but still closing bullishly today. And tomorrow I have to look at the econ app but I bet we're gonna have a neutral gap tomorrow. Probably neutral or teeny weeny down. Neutral or teeny weeny down or teeny weeny up. I don't think we gap any significant way tomorrow but look for follow through. Look for the market to cleanly get over the high and if it doesn't retest it, it's gonna be sloppy. If we don't break over the high the next week it's gonna be sloppy till July. So we'll see what this market wants to do. Do what it wants to do naturally or wait and wait and wait. Because if we don't do it it's just a waiting period until we do. The next golden gap class is this weekend, the last class, the last class to do before the price of the golden gap class increases. Price of the class is increasing to 2,999 as of July 1st. And if you would like to do this class you can find your watch in here if you're new. I wanna learn the system. The price of the class this weekend is 24.99. There will be no exceptions made once the price goes up. You can pre-register for a class in the future or do the class this weekend on June 21st and 22nd. The trends classes next week, June 24th and 25th if you'd like to do that one as well. And I'm running the Combo Special for June for a half off that class with golden gap class participants. If you wanna do both classes and you get half off the trends class. If you wanna do the trends class it's 9.99. If you wanna do the gap class it's 24.99. If you'd like more information email me at melissa at thestockswish.com. I have a webinar coming up this evening at 4.30 Eastern time. Should be interesting and we'll see everyone there. Have a wonderful day and don't short the market. That's the name of the tune. Name that tune. Have a good day everybody.


What's up everybody and welcome back to the channel I'm EvilRabbit here staring at my R35 GTR that I have tuned up for Eustacepec. Today we're gonna be testing the car out and trying to get smoother on our steering. A lot of people have been blasting me in the comments recently about being too sloppy being you know inconsistent and being just very twitchy on the steering. And I have been and I've been watching the videos to try and figure that out. So I've adjusted my wheel settings tuned up a different car and I feel I'm getting a lot smoother on my steering here on fours motorsport seven. So we are gonna go rip this car around here today in a track that I was shown by a couple of the I tandem guys and it's a definitely a fun a fun way to basically practice precision in a sense and have some you know elevation changes some jumps and stuff like that they call it the roller coaster track. So we're gonna go hit that today in this GTR do that a couple times have a little fun with it get sideways and well try and get smoother on our steering. So we are in the open airfield like I said we're gonna head to two of the hangars and they like I said they call this of course the roller coaster basically has some precision moments where you gotta get close to walls through doors as well as some jump trips and sections we've been using these last four airplane hangars. We're just trying to be a lot smoother on steering we're kind of gonna try and let the steering wheel slide through our hand a little bit more and be a little more precise. So we're gonna rip this real quick so I can show you guys pretty much what this track is pick it in the fifth hard but hardly needed to but no handbrake there we go around the outside cut our way out the door get us out out to the wall bring ourselves back in bring ourselves back in the hangar try to be as smooth as I can on the wheel a little better throttle control right there on the outside of that wall super close to that wall so we you know had a very close almost taken that wall out and then come on out up against this wall and pretty much bring it back up top so that is the rolling coaster track it's a lot of fun to do I tell you guys do it yourself I tell you guys definitely show me this and it's been a lot of fun messing with it this GTR is handling so much better right now than my other cars that I was previously using for ESDA get close to that wall here we go and get as close as I can to these doorways pop and flames and when I go there trying to let the wheel do a little bit more by itself through my hands and not force it as much trying to just be smoother the first feedback in fours and seven on xbox is definitely uh can be very twitchy if you're on pc you have emulators to help with it an xbox you're pretty much just basically set with what we got oh I thought I could hold that but fortunately could not hold that so we're just going to pretty much finish this out and then go throw it again shouldn't have gone all the way out there but the jumps the bumps around and everything make it so much more fun so I think we're gonna attempt to do this first person for all the haters out there who say I drifted in third person we're gonna be throwing this in you know first but like I said recently I've been blasted in the comments a lot about being too twitchy and super sloppy and that was very sloppy and super sloppy with my driving and stuff like that so trying to fine tune my settings on the car as well as on the wheel and this is much more difficult than I thought I was gonna be on the wheel and get smoother it's a bit of a different perspective doing it in first because I'm throwing more angle than I should be we're gonna hit that oh failed failed in first person unfortunately this car doesn't look where I want it to on a set of when I was running the Oculus I was able to look where I wanted to look so a bit of a more of a technical style track to kind of try and smooth this car out I was testing this car in tandem with a couple of the teammates and everybody trying to see if this would hold up to how I needed it to and well it did so I'm gonna throw some tandem clips in before we end this episode off but the r35 definitely a very nice chassis I don't think we're gonna make that oh I didn't think we'd make that we're just kind of throwing it all around so I'm gonna throw some tandem clips in here for you guys as always make sure you install my Instagram and Twitter I appreciate all the support and everything throughout the competitive style of Forza so we're gonna be doing the PC build next month so we'll be doing more set of course of content on the PC and doing the set of course of competitions and stuff like that but right now I think we have pretty much decided which car we're gonna run for USDA Drive Research now it's just a fight of getting smoother kind of letting the car do its own thing which is this r35 is kind of allowing me to do it the long ebra grab but so as always make sure you follow me on Instagram and Twitter if you're not subscribed we do appreciate the support if you tap that subscribe button so I hope you guys enjoy the tandem clips with some of the guys in the GTR and as always I'll see you guys on the track second drive a lot better than that oh LZ was part of the Boxing Boy gang he gets banned from wearing a tree every time and he has a little bit what kind of dog do you have I have a mini australian shepherd there's one that's like a couple doors down from me in my neighborhood so I'm really just crashed into the wall I think it was LZ uh no I didn't crash though probably Ranga Ranga was an FD with the guy what the fuck Ranga in our exhibit wait is Ranga playing forza yep oh this is lost for our old jojo I didn't see Tom by the steering car anyone oh how did we survive well somehow I didn't manage to murder yeah and somehow I didn't manage to murder anthem oh Ranga that was almost a perfect lead holy crap that was nice but I fucking walked down at the last part fuck I went wall to wall to wall I thought I was gonna biff it into a wall but I did not no you were insane you were freaking nice and perfect probably let me follow you I love this car I think it's a combination of the car that I refined tuned my wheel settings and my fantech feels so nice right now give them a bit of a go all right I already know this stuff is in the wall oh man oh my LZ oh my god oh shit there's a lot of people I think it's just everybody in the party what the fuck is going on Jesus Christ he's joining up CSW they didn't say I think you guys had pretty similar yeah he he has a modified one that is pretty much equivalent to mine you guys made a mistake that dream set up I want one day I didn't think I was gonna pull out of that Ranga sorry I had to shoot wide yeah no worries this thing is fucking nutty this thing is like actually dirty the thing yeah you know my tires are completely cold and I was in the wrong gear I wonder whose wing that I just punted my wrong gear I mean I have mine had some good tandem clips and there's a lot of TUS here so why not you know throw up a you know some tandem videos oh that's a wall and that's an anthem a new meaning to fill in my entry what about blue I can't hit my car because how many people is it well you are you or yeah you know I mean it's a standing guard everyone in your videos that was three wall taps I run out of gear I ran out of walls to hit wall tap wall tap wall tap well that was long oh you're good my car's already destroyed anyways here we go oh I just died oh sorry whoever that is pardon me oh yo thanks that's a slow down it's just not you throw you reckon froze froze oops wrong gear who's panicking he hugged it in half a race I do that on a brief move I just don't I just forget I'm a first I mean oh shit no


Hello shipmates. I'm here at the Ford Center for Education and Leadership reading a book, Good Book, the last stand of the Tin Can Sailors, and you can see here quite a monument here, a sculpture of 6,000 books on President Lincoln. I've chosen this location to talk to you about the new reading list. Next month, when we celebrate the Navy's birthday, at about the same time, we're going to be rolling out my new reading list. Now we've worked with the Naval War College to put together a list that I think you're going to like. They're kind of set right along my sailing directions, and they're really designed to give you an insight to your history, your tradition. They teach you a little bit about management and about leadership. Now look shipmates, I'm not trying to make historians out of all of us, and I don't want you to get a history degree out of this. I want you to be interested in your Navy and see how your predecessors, sailors just like you, made your Navy great. They had a lot of the same challenges, a lot of the same anxieties, and they got the job done just like you're getting the job out there in the Navy. Now we're going to bring this to the NKO where you can download these, and they'll be available at your libraries on your ships, on your shore sites, in your units, and of course available at NKO where you can download them for e-books. So look, take a hold of this thing, read about your Navy, learn about the heritage and tradition, learn about those that came before you. I'll see you out there in the fleet.


Thanks, Larry, for the introduction. Hi, everyone. I'm Sam Gross. And I'm here to talk about my project that I've been working on for the past few years to let Python run multi-threaded programs without the global interpreter lock. I'd like to start with a bit of background on multi-threading and Python's global interpreter lock. Multi-threading, using multiple threads, allows an individual program to perform multiple tasks at the same time. Threads are useful for two related reasons. First, threads can exploit the parallelism available in multi-processor and multi-core computers. Threads can run on separate cores, so a program that can divide its work into multiple tasks executed by separate threads can run faster on a multi-core computer than if it used a single thread. The second reason is that threads can be useful for structuring concurrency, even on a single processor system. Threads might be used to handle multiple requests in the network server, to overlap IO with computation, or to perform long-running tasks in a graphical application while keeping the user interface responsive. Threads aren't the only way to structure concurrency. For example, modern Python includes the async IO package to structure concurrent asynchronous code. But this project is mostly focused on the first reason. We want Python to be able to take advantage of the parallelism available in multi-core computers by making multi-threaded code run more efficiently. One of the big obstacles to efficiency is that the global interpreter lock, or GIL, ensures that only one thread can run Python code at a time. When Python was first publicly released in 1991, it didn't support threads or have a global interpreter lock. Support for threads was added about a year later, in 1992, together with the GIL. For context, the 1990s saw a number of operating systems had better support for threading and support for computers with multiple processors. I think there's an interesting analogy between the introduction of threads in Python and multiprocessor support in some of these operating systems. At least in the cases of Linux and FreeBSD, support for multiple processors was accompanied by the introduction of global locks, the big kernel lock in Linux, and a lock called Giant in FreeBSD. In the original implementation, the big kernel lock ensured that only a single thread could enter the kernel at a time, somewhat similar to the way that Python's global interpreter lock ensures that only a single thread can run Python code at a time. Over time, the uses of the big kernel lock were gradually replaced by fine-grained locking and other techniques to improve scalability. The big kernel lock was completely removed from Linux in 2011. From a technical perspective, the global interpreter lock makes sense for a few reasons. First, it's simple to implement. It doesn't require a lot of code. Second, the global interpreter lock is really efficient for single-threaded code. The lock is only rarely touched, so it doesn't substantially affect performance if you only use one thread. Third, the global interpreter lock avoids a lot of the potential dead locking hazards of using multiple fine-grained locks. In particular, the locking discipline for the GIL is pretty straightforward. You need to acquire the GIL once before calling into Python code, and you should release the GIL before calling an operation that might block the thread. Finally, the global interpreter lock still allows for some parallelism. C extensions can release the lock around long-running pieces of code that don't access Python objects. We're now about 30 years after the introduction of threads to Python, and as Larry mentioned, the computer hardware landscape has changed. Multicore processors are on desktops, laptops, servers, smartphones, and today I learned they're in glasses as well. In server environments, it's come to see computers with dozens of CPU cores or more. Especially as the core counts increase, it's difficult to take advantage of multicore parallelism from Python due to the GIL. Even with the GIL, there are a number of ways to parallelize Python programs today, but they all have substantial limitations. First, in C API extensions, you can parallelize operations internally in a way that's invisible to Python. For example, Psykit-learn uses OpenMP to parallelize its K-means operation. When you call the Psykit-learn's K-means function from Python, Psykit-learn distributes the work to a number of threads, waits for those threads to finish, and aggregates the results and returns back to Python. This works well when individual operations are large enough to be efficiently paralyzed, but not when there are a lot of small steps. A lot of scientific computing libraries, like Psykit-learn, PyTorch, and some versions of NumPy and Psypy use this technique. Second, C API extensions can release the global interpreter lock around long-running operations so that multiple operations can overlap. The previously mentioned scientific computing libraries do this, but it usually doesn't scale well beyond a few cores because contention on the GIL quickly becomes an issue. You can also use multiple processes, including through Python's multiprocessing library. Processes are like a heavyweight version of threads. The main challenge with processes is that communication between processes is usually more expensive and more difficult than communicating between threads. I'm going to expand on this point. With processes, objects need to be serialized to be sent from one process to another. You can't share Python objects directly like you can between threads. This is a performance issue because serialization is expensive and requires making extra copies. It can also lead to usability issues because not all objects support serialization. Additionally, many C and C++ libraries you might want to use from Python don't work well across processes, even if the libraries support multiple threads. Some libraries have opaque data structures that might be difficult or impossible to share between processes. Removing the global interpreter lock is a challenge because besides the actual removal of the lock, you have to make Python safe to run with multiple threads. You have to address things like reference counting. Python primarily manages memory by keeping track of the number of references to each object, and the current implementation isn't thread safe without the GIL. You also have to figure out how to handle concurrent accesses to Python's collections, things like dictionaries, lists, and sets. Once you remove the GIL, unless you've made other changes, concurrent modifications to things like dictionaries would crash the interpreter. There are a lot of other internal components in the Python interpreter that need to be made thread safe. Many of these are easy to fix and only require small changes, but other require more substantial changes. Some of the specific internals I want to talk about today are Python's built-in memory allocator and the cyclic garbage collector. Finally, all these changes need to be made while keeping good single-threaded performance and while allowing for efficient multi-threaded scaling. This project isn't the first attempt to remove Python's global interpreter lock. I'm aware of three previous attempts in CPython. The first was in 1996 when Greg Stein released his free-threading patch. The patch used atomic reference counting on Windows and a global reference count lock on Linux. List and dictionary accesses were protected by mutexes. Parts of the patch were actually adopted upstream. In particular, it introduced a PyThreadStateStructure and correct per-thread exception handling, both of which live on in Python today. The second attempt I know of was Python SafeThread by Adam Olson in 2007. Adam pursued an interesting design. Most objects were owned by a single thread which could use normal reference counting operations. If another thread needed to modify the object's reference count, it would have to pause and wait for the owning thread to give up ownership. The implementation also had restrictions on sharing objects between threads. You could share immutable objects like numbers and strings and some special objects like a new shared dictionary type. Trying to share other mutable objects like a regular dictionary or list would raise an exception. The most recent attempt to remove the gil from CPython is Larry Hastings' Galectomy, which started around 2016. The galectomy used atomic reference counting for objects and later versions experimented with buffered reference counting to improve performance. The implementation used locks to protect accesses to lists and dictionaries. The galectomy was the effort that had the most influence on this project. I'd seen Larry's PyCon talks on the galectomy where he described parts of the implementation and the performance challenges. I think the biggest takeaway from these previous efforts is the recurring challenges due to reference counting, both in how it affects single thread performance and efficient scaling. Reference counting is by no means the only issue you have to address when removing the gil, but it's essentially the first problem you have to solve because the performance impact is so large. A number of people have come to the conclusion that reference counting is the biggest obstacle to removing the gil and that we should ditch reference counting and replace it with a trace and garbage collector. So why not ditch reference counting? The primary reason is that to getting rid of reference counting would make it much harder to get a reasonable degree of compatibility with existing CAPI extensions. I think the effort to get a reasonable degree of compatibility would probably be bigger than just directly dealing with the issues that reference counting poses. The most straightforward way to make reference counting thread safe is to use atomic operations. The problem with this strategy is that atomic constructions are slower to execute than their non-atomic counterparts. For example, if you modify Python to use atomic reference counting and run the PyPerformance benchmark suite, the benchmarks take an average about 60% longer to run. This chart shows the slowdown of using atomic reference counting in each benchmark in the PyPerformance suite. There's big variance in individual benchmark performance, so don't treat the 60% number as particularly precise. A few benchmarks aren't affected by the reference counting change while others take more than twice as long to run. Biased reference counting is an algorithm created to address the overhead of atomic reference counting. It was created by Jiho Toy, Thomas Schwul, and Joseph Torres at the University of Illinois at Urbana-Champaign and published in 2018. It's based on the observation that most objects are only accessed by a single thread, even in a multi-threaded environment. Their version was implemented in Swift, but the same technique can be applied to Python. Biased reference counting works by splitting the reference count field into two parts, as well as storing the idea of the thread that created the object. The first part of the reference count field stores the local reference count. The thread that created the object uses plain non-atomic operations to modify that local reference count field. The second field stores the shared reference count. Other threads modify the shared field using atomic instructions. The true reference count is the sum of these two fields. Because most objects are only accessed by one thread, the technique avoids most atomic operations while still maintaining thread safety. Here's a basic implementation of incrementing the reference count using biased reference counting. The algorithm checks if the object's thread ID matches the current thread ID. If they match, the algorithm increments the local reference count field using normal non-atomic increments. Otherwise, the algorithm increments the shared field using an atomic operation. This is faster than just using atomic reference counting because, at least on current hardware, the extra comparison to check the thread ID is cheaper than the cost of an atomic increment. This technique has some similarity to the strategy used by Adam Olson in the Python safe thread project. Both associate a Python object with a thread that creates it, and that thread uses fast non-atomic operations. However, biased reference counting avoids the blocking operation present in Python safe thread, where a different thread accesses the object and has to wait for the original thread to give up ownership. Here's an updated version of the previous chart. Atomic reference counting overhead is still shown in blue. Those are the same results as before. Bias reference counting is in green. The overhead of biased reference counting is much less than the overhead of atomic reference counting, although it still has some extra overhead compared to the regular non-thread safe reference counting that's currently implemented in Python. Using Python 3.9 as a base, biased reference counting is about 10% slower on average across the PyPerformance benchmarks compared to that 60% slowdown we saw with atomic reference counting. While biased reference counting does a pretty good job of reducing the execution overhead of atomic reference counting, it doesn't address the scalability issues when multiple threads frequently modify the reference count of the same object. To expand on this, I'm gonna make some generalizations about how memory operations scale on modern computer systems. This table is divided in columns between operations that are read-only and operations that modify memory. The rows are divided into private accesses, that's memory that's accessed only by a single core, and shared access, memory that's accessed by multiple cores. The checkmarks indicate if the operations scale well as the number of cores increase. The first thing to notice is that read-only accesses scale well with lots of cores. Memory reads from multiple cores both to the same location scales well as well as reads to different locations. The same data can exist in the cache of multiple CPU cores as long as the data isn't modified. The second thing I'd like to point out is that write operations scale well if the data is private. In other words, multiple CPU cores performing write operations scale well as long as they're to different memory addresses or more accurately to different cache lines. This includes atomic operations. Things like atomic increment and decrement are slower than their non-atomic counterparts, but both atomic and non-atomic operations scale well if the data is private. Finally, concurrent modifications to shared data doesn't scale well. So for example, lots of threads concurrently modifying the reference count of the same object won't scale well on multicore machines. This is relevant for Python because there are some objects that are necessarily shared between threads and they're frequently accessed. For example, the constants none, true, false, some small integers and some other objects are shared by all threads in a Python program. Like other Python objects, they're also reference counted even though they can never be collected during the lifetime of the program. Absent other changes, reference counting on these objects would be a bottleneck for multi-threaded programs once you remove the gil. The fix is to skip reference counting on these objects. The objects are marked as immortal using a bit in the reference count field. The reference counting code is changed to be a no-op if the immortal bit is set, and the objects are kept alive for the duration of the program. This works well because things like none, true and false already stay alive for the lifetime of a Python program. Immortal objects aren't a new idea. They're used in the Swift as well as some alternate Python implementations like Cinder and Piston. To go back to the scaling table, immortal objects work because we've transformed writes to shared data to read only accesses. We've gone from modifying the reference count fields of objects like none, true and false to only needing to read that field to check if the object is immortal. So why doesn't Python already use immortal objects? Well, for single-threaded code, immortal objects aren't a clear win. The cost of the extra branch to check if the object is immortal is typically higher than the work saved by skipping the reference count operation for single-threaded code. At the language summit, however, back in April, Eddie Elizondo and Eric Snow presented an implementation of immortal objects with smaller costs than previous efforts. So we might see immortal objects in C-Python independent of any guild changes. Early on in this project, I decided to test out the implementation on Larry Hastings' Fibonacci benchmark from the Gelectomy project. The benchmark's pretty simple. There's a function fib that calculates the n-th Fibonacci number, and it's called concurrently for multiple threads. The Fibonacci calculation is deliberately slow. It's mostly there to do some work. I was running the program with an early version of Nogil Python with eight threads, and I was pretty happy to see the top command show all eight cores being fully utilized.  And indeed, this would have been great if the chart showed throughput versus the number of threads, but the y-axis here is actually time taken. Even though the program was using eight CPU cores, those cores were being used so inefficiently that there wasn't a speed up. In fact, adding more threads made things worse. Here's the same data but inverted to show throughput instead of time taken. If things were going well, then eight threads would be eight times faster than a single thread, but here eight threads are actually only running at 25% of the speed of a single thread. So what's going on? It's the same issue I mentioned previously. Reference count contention here is so bad the program didn't scale at all, and in this case reference count contention wasn't solved by immortal objects. Let's look at the Python bytecode. I've highlighted the two load global bytecode operations. These correspond to the two recursive calls to fib. They load the fib function from the global's dictionary and push it to the interpreter stack. As part of that operation, they increment the reference count of the fib function object. Later, when the function call is finished, the object is popped from the stack and the reference count is decremented. Since all the threads are calling the same function, they're also all concurrently modifying that same reference count field on the fib function object. We could try making all functions immortal, but that has a number of downsides. Functions don't necessarily live forever. In addition, they hold a reference to the global's dictionary. So making functions immortal would prevent the global's dictionary from being freed, which would in turn hold onto global variables unless those variables were explicitly deleted. Some programs and libraries depend on running finalizers for global variables at exit, and making function objects immortal would prevent that from happening. Do the reference chain from function objects to the global's dictionary to global variables. This project uses a variant of deferred reference counting to address the reference count contention on function objects. Deferred reference counting skips the reference counting operation for objects as they're pushed to and popped from the interpreter stack. Other operations like assignment to a dictionary list use the normal reference counting behavior. Because we've skipped some reference counting operations, we can't free these objects when the reference count reaches zero. Instead, they're only free during the cyclic garbage collection. The garbage collector is modified to inspect the interpreter stack. At the start of a garbage collection cycle, it looks through all the objects on the interpreter stack, adds back any missing references. That's the deferred part of deferred reference counting. It then does a normal garbage collection cycle and finally removes the reference count that it added in the first step. So deferred reference counting avoids some reference counting operations, but it also delays the allocation. Normally, that could cause problems, but we only enable deferred reference counting for some types of objects, module level functions, methods, and some type objects. It turns out these objects would typically only be deallocated during a cyclic garbage collection cycle anyways. I mentioned before that function objects have a reference to the globals dictionary. Well, the globals dictionary also has a reference to functions defined at the module level. Functions in Python naturally form reference cycles, and there are similar reference cycles for methods and type objects. So deferred reference counting on these objects doesn't typically delay deallocation compared to the normal behavior in CPython. Here's the Fibonacci benchmark again, but with deferred reference counting enabled. The results are much better now, and we get a nearly eight times speedup with eight threads. The Nogil project uses a combination of these reference counting techniques to achieve good single-threaded performance and avoid a lot of the obstacles to multicore scaling. Most objects in Nogil Python use biased reference counting. They are deallocated immediately when their reference count goes to zero with a few exceptions. A few types of objects, such as function objects, methods, and some type objects use deferred reference counting within the interpreter loop. Outside the interpreter loop, they're treated like other objects and use biased reference counting. They're only deallocated during garbage collection cycles. Constants, such as true, false, and none are immortal. Reference counting on these objects is skipped everywhere, both in the interpreter and in library code. They're never deallocated, at least not until the interpreter shuts down. Now that I've talked about reference counting techniques, I'd like to move on to collection thread safety. This is how things like dictionaries, lists, and sets handle concurrent accesses. Dictionaries are particularly important because they're widely used in Python, including for storing object attributes and global variables. Like many previous attempts at removing the GIL, the Nogil project adds per-instance locks to collections like dictionary and list. I'm going to talk a bit about the locks used. The implementation is based on the locks used in WebKit, the browser engine for Safari. The design and trade-offs are described in detail by Phil Piesle in a blog post called Locking in WebKit. If you're interested in how locks work, I highly recommend it. It's one of my favorite technical blog posts. And if you search Locking in WebKit, it'll be the first hit. I'm not going to describe the implementation too much, but I'll describe some of the more important attributes of the input of it. First, the locks are small. They can be as small as two bits, but due to alignment requirements in C, in Nogil Python, they typically require the same space as a pointer, so eight bytes in a 64-bit system. That's still a lot smaller than P thread mutex or C++11 standard mutex, which take 40 or more bytes. The locks also only need to be initialized to zero, just like an integer. It doesn't require any memory allocation. It has fast, uncontented access. That's really important because by far, the most common case is that only one thread, access is a given collection at a time. Finally, it avoids thread starvation in heavily contended locks by eventually handing off the ownership of the lock to the threads in the same order that they attempt to acquire it. All right, that's it about the locks themselves. One of the challenges with introducing locks to collection types is that you have to be careful to avoid introducing lock ordering deadlocks. For example, storing a value, excuse me, storing a key in some dictionary might involve comparing that key to an existing key already in the dictionary, comparing it for equality in particular. For string keys, that comparison's really simple, but user-defined types can override the quality operator. And if that user's code accesses either this dictionary or some other dictionary while implementing quality, then we risk deadlock by holding multiple locks while another thread might attempt to acquire those same locks in a different order. For dictionaries, a reasonable solution is to unlock the dictionary before calling the quality operator and relock it afterwards. Essentially, we make sure that a thread can hold the lock for only a single collection at any time, and that's what was implemented in this Python, or excuse me, in this project for a while. It turns out there's a lot of places you might wanna call back that might call back into Python and incidentally acquire locks. For example, the PyDeckref macro can call arbitrary code through an object's finalizer. Identifying all these places is doable, but it's tedious and error-prone. It'd be really convenient if we didn't have to be so careful about unlocking these locks before calling back into Python or calling things like PyDeckref. Notably, the global interpreter lock doesn't have this issue. You just need to acquire the global interpreter lock once, and it's only released in a few places, usually implicitly. For the most part, you don't have to worry about lock-ordering deadlocks with the global interpreter lock. Fortunately, we can make these locks behave more like the GIL. The basic idea is to allow threads to implicitly release locked collections in places where the threads would have normally released the GIL. We don't do this for all types of locks, but instead introduce a new internal API, PyBeginCriticalSection, and PyNCriticalSection. PyBeginCriticalSection locks the past in lock, and then pushes it to a stack of active critical sections for the current thread. Subsequent calls to PyBeginCriticalSection also push onto that same stack. When the threads would have normally released the GIL, the threads instead release all the locks for the active critical sections and then mark those sections as inactive. This includes times when the thread is blocked trying to acquire a lock held by a different thread. In places where the thread would have normally reacquired the GIL, it reactivates the top most critical section and locks the corresponding lock. However, it doesn't lock any previous critical sections. The PyNCriticalSection releases the lock for the top most section and pops it from the stack. If there was a previous critical section and it was inactivated, it's now reactivated and its lock is relocked. The final PyNCriticalSection unlocks and removes the last critical section from the stack. The scheme avoids the risk of any lock ordering deadlocks. To understand why, consider that a lock ordering deadlock requires a thread to hold one lock, let's call it A, and be blocked while trying to acquire a second lock, say B. There also has to be another thread trying to acquire those same locks, but in a different order. With this API, the moment a thread is trying to acquire that second lock, B, it will release its previously held locks, allowing other threads to proceed and avoiding the deadlock. So, since this avoids lock ordering deadlocks, why isn't something like this more widely used? Well, with this sort of scheme, there's a lot of places where the held locks might be implicitly released. And that's not something you really want in a general locking API. But it's fine here for replacing the global interpreter lock because the places where these locks might have been implicitly released are the exact same places where the gil could have been implicitly released. There's one additional bit to the critical section API. Sometimes it's useful to lock two collections at once. For example, when extending a list with the items of a dictionary, it's useful to lock both the list and the dictionary. The list, because it's being modified in the dictionary so that the extend operation sees a consistent view of the items in the dictionary. So there's one additional API that locks two mutexes at once. Locking collections works well for most accesses, but there's a few operations where acquiring a lock would inhibit scaling. The most important is when accessing a single item in a dictionary, because things like globals and object attributes are implemented using dictionaries, and a single dictionary may be frequently accessed by many threads, just like we saw in the Fibonacci example. The single element accesses for list and dictionary are implemented without acquiring the collections lock. This is made more complicated by reference counting. The acquiring the accessing thread needs to increment the reference count of the returned object, but another thread might have simultaneously modified the dictionary and freed that object. To handle this, the fast lockless path only handles the case where the reference count isn't zero and where the value is still in the collection. If it fails, the operation falls back to a slower path that retries the operation after acquiring the collections lock. Normally the scheme wouldn't be safe because when an object is freed, its memory, including the reference count field is invalid, the scheme requires cooperation with the memory allocator to ensure that the reference count field remains valid for the duration of this access, even if the object's freed. That leads us to the memory allocator. Python has a memory allocator called PyMalloc. It's optimized for small allocations and it's used for allocating Python objects. Unfortunately, it's not thread safe. This project replaces PyMalloc with MeMalloc, a memory allocator written by Don Leyen at Microsoft Research. MeMalloc's performance is slightly better than PyMalloc and it's thread safe, so it's like that MeMalloc would end up in Python independent of any GIL changes and there's an outstanding PR from Christian Himes that implements MeMalloc into CPython. MeMalloc with some modifications is also important for the lockless dictionary and list accesses we just talked about. Finally, MeMalloc makes it easier to make Python's cyclic garbage collector thread safe.  The first challenge with cyclic garbage collection is that the mechanism used to keep track of objects isn't thread safe. The current garbage collector relies on a global linked list to find all Python objects that support cyclic garbage collection. The linked list uses two fields in the object's headers, GCnext and GCprevious. Objects are inserted into the linked list shortly after they're allocated and removed from the linked list before they're deallocated. This poses a thread safety issue because threads allocate in free objects concurrently. The list also can't be made thread local because objects might be allocated in one thread and deallocated on a different thread. This project gets rid of the doubly linked list, although the GCnext and GCprevious fields are still in use for now. Instead of the linked list, the garbage collector uses MeMalloc to find all the objects that support garbage collection. These objects are allocated from a separate MeMalloc heap than other data. The collector scans each block in the heap to find objects, unallocated blocks, and objects that aren't tracked by the collector have their least significant bit in the first word set to zero. Tracked objects have the least significant bit set to one. The cyclic garbage collector can't run safely when other threads are accessing or modifying Python objects. The global interpreter lock previously ensured this, but now the collector needs to pause all of their Python threads before running. Python already had a mechanism for requesting threads to handle asynchronous signals and exceptions, which we've extended to handle requests to pause for the garbage collector.  The other threads are resumed after the collector finishes identifying unreachable objects. I wanna talk a bit about compatibility because a frequent concern with removing the GIL is that it will break existing Python code in extensions. For pure Python code, that is Python code that doesn't make use of the CAPI, I don't expect the code to require any changes. Now, that probably won't be 100% true, but I think it's pretty close to it. Importantly, things that looked atomic with the GIL, like this dot append, still look like they're performed atomically due to per-collection locking. For CAPI extensions, many don't require any changes, and those that do the changes are pretty small. I'll talk about a few examples later. The most common issue is global or shared data in C that's implicitly protected by the GIL. A less common issue is due to the use of borrowed references. For example, pi-dict-get-item and pi-list-get-item return objects without increasing their reference counts. This can cause problems if the dictionary or list is concurrently modified. Most uses of borrowed references aren't an issue. For example, pi-dict-get-item is frequently used to parse the keyword arguments dictionary, and that dictionary is effectively private to the called function. It's not shared with other threads. I'd like to give some examples from real Python packages that use the CAPI. Scikit-learn is a Python library for machine learning that builds on top of NumPy and SciPy. Under their hood, it's a mix of Python, SciPy, and some C++. Thanks in large part to Olivier Grissel, Scikit-learn now works without the GIL, and there's even a continuous build that runs with no GIL Python. I want to go over the bugs we fixed while making Scikit-learn work without the GIL. Getting all the Scikit-learn tests to pass required fixing 10 issues. Of these, five were in the no-GIL project, but unrelated to removing the GIL. The other five issues were related to removing the GIL, essentially thread safety issues in these projects if you ran them without the global interpreter lock. One of these issues was in the Scikit-learn code base directly. Three of them were in dependencies, two in NumPy and one in Scython, and the last one was in Vis Tracer, which isn't really a dependency, but a useful profiling tool. Here's the bug in Scikit-learn caused by removing the GIL. This is an excerpt of some Scython code used to compute k-means. When run without the GIL and with OpenMP parallelism enabled, the operation would give an incorrect result triggering test failures. The problem is that Scythons with GIL construct won't prevent multiple threads from running at the same time if you've already disabled the GIL. The fix was pretty simple. We just replaced with GIL with a different lock. Since the code is already using OpenMP, we use an OpenMP-based lock. The actual PR that landed in Scikit-learn is a bit larger than this because the same pattern is repeated a few times in the file, but this is the essence of the fix. I'll only briefly go over the other issues that cropped up when removing the GIL. For NumPy, there were four total issues, but two of those were fixed before the Scikit-learn effort. Of the four issues, three were issues due to caches that were no longer thread-safe without the GIL. Some of those caches could just be disabled when running with no GIL Python, but one of the caches had to be protected by a lock. One issue was due to the use of borrowed references in NumPy that was no longer safe without the GIL. That was fixed by switching to an API that returned a new reference instead. Scithon had an issue with locks allocated from a pool that wasn't thread-safe without the GIL, but it turns out those locks weren't needed if atomic operations were available. Finally, VizTracer, the profiling tool, had some internal state-and-see that needed to be protected by a lock. The critical section API came in handy there. The fixes for Scikit-learn and VizTracer have landed upstream in those projects, but the changes to NumPy and SciPy as Scithon are still in forks maintained by me. Overall, the changes were relatively small. The Scikit-learn test depended on a total of 25 other Python packages. Other than the changes I mentioned here, those dependencies didn't require any other changes. The other project I'd like to talk about is work done by my colleagues at MetaAI on Hanabi. Hanabi is a card game that's been proposed as a benchmark problem in AI. Hanabi is an interesting AI challenge for a few reasons. First, it's a cooperative game. Two to five players work together to place down cards in numerical order. It's also an imperfect information game. You can see other players' cards, but you can't see your own cards. And to play well, you need to think about your partner's behaviors and how they'll respond to your actions and the limited information that you're allowed to share. My colleagues have developed agents, bots, that play Hanabi, both with other bots and with people. The bots use artificial neural networks to decide what actions to take, and they're trained using reinforcement learning techniques. The code is partly in Python, using PyTorch and NumPy, and partly in C++, using PyBind 11. Here's a diagram of the system used to train the bot. The system simulates thousands of games, that's the section marked environments, at once using multiple threads. Each thread is responsible for simulating more than one game. The inference model and the training model are the bot's neural networks. The inference model determines what action the bot will take in the simulated games. The results of those simulated games are fed into the training model to improve the bot's behaviors. Every so often the training model is copied back to the inference model. The Nabi system was built with vanilla Python and with a global interpreter lock in mind. The worker threads and inference loops have to avoid running Python code. Only the training loop and the training model run the Python interpreter. This means that many pieces of code had to be written in C++, or like the inference model, written in a subset of Python called TorchScript, that serialized your representation that can be executed from C++ without invoking the Python interpreter. Having so many components in C++ made it harder to explore new research ideas. My colleague Hanyan implemented a version of this system designed for running without the gill. Most of the components were moved to Python, including the worker threads, the inference model, and the logic for each player. The environment, that's the component that implemented the rules of the game, stayed in C++ because there isn't a compelling reason to move it to Python. The research ideas don't involve changing the rules of Hanyan. As I mentioned before, the project uses NumPy, PyTorch, and PyBind 11. Those projects were already patched to work without the gill, although we found and fixed another issue with the PyBind 11 patch in the course of this work. I'd like to show the performance of this system and the steps we took to optimize it. First, here's a chart showing the performance of the new, mostly Python system with the global interpreter lock. The x-axis shows the number of worker threads, and the y-axis shows the number of games simulated per second. The gill is pretty clearly a bottleneck here. Using more than one worker thread doesn't improve performance. Here's the performance once the gill is removed. Now, adding more worker threads improves performance at least up to a bit. This only shows up to eight threads, but the computer it's running on has 40 hardware cores. This is the same system, no gill, but with even more worker threads, up to 80 instead of only up to eight. We hit a bottleneck pretty quickly. Increasing the number of worker threads beyond 10 didn't improve performance. Running with more worker threads actually hurt performance. To investigate this, I used Linux Perf Tools and GDB. Perf Top showed that the majority of CPU time was spent trying to acquire a lock, but didn't give much information beyond that. GDB narrowed the issue further. At any given point, most of the threads were stuck trying to acquire the same lock. This is a lock I added to our patched version of PiBind 11 to make it thread safe. PiBind maintains an internal hash table called registered instances that maps C++ pointers to their corresponding Python wrappers. This is done so that if you return the same C++ object multiple times, you always get back the same Python wrapper object. The lock I added to protect the hash table was the source of the bottleneck. There are a lot of techniques for implementing concurrent hash tables, but one of the simplest is to split the table into multiple pieces. Each a hash table with its own lock. The key, in this case the C++ pointer, is used to determine which sub hash table to use and to index into it. This worked surprisingly well and was easy to implement. The turquoise bars show the performance after splitting up the registered instances table. This new system can simulate nearly 1,200 games per second with 60 workers, which is about a 20-time speed-up when you ran this program with a global interpreter lock enabled. I suspect there's still opportunity to further improve scaling, but we think this system has performance comparable to the original C++ heavy system, and having more of the components in Python made it much easier to explore new research ideas. You can try out the no-goal version of Python yourself. Both URLs here go to the same GitHub page. The easiest way to install it is with pyenv, a Python version management tool for Linux and Mac. There's also a Docker image, or you can build no-goal Python from source, if you prefer. You can pip install most packages, including many scientific computing libraries. Please let me know if something doesn't work by filing an issue on the project's GitHub page. In the next month or so, I'll be working on a Python enhancement proposal, or PEP, proposing to remove the global interpreter lock from a future version of Python using the ideas from this project. To help address compatibility issues, I'll propose placing the gil changes behind a compiler flag so that the default build will still have the gil enabled. I expect this to take some time, but in the meantime, I'd appreciate your feedback on the project, either on GitHub or elsewhere. And if you're interested in efficient multi-threaded Python programs, please try out no-gil Python. Thank you. All right. We have time for a couple of questions. I've been asked to ask a couple of questions first. So, Sam, you didn't know how to do this. My experience in working on the galectomy, I had a miserable time debugging because debugging across multiple threads makes it so hard to go back and reason about who changed what and how it happened. I relied very heavily on what's called reversible debuggers. What has been your experience with debugging C-Python, figuring out what happened? Yeah, so that was actually one of the things I took from your project. So I didn't use the same reversible debugger. I used RR, which is a project for Mozilla, which is another reversible debugger. And it made debugging both thread safety issues and reference counting bugs much easier because you could go to the source of the failure and then just go back in time to figure out what operation was responsible for corrupting the data. Okay. If people have questions, by the way, you can line up at that mic and I might call on you. Okay. So you talked about how you're... I can't remember. There's five or six different, like X reference counting and Y reference counting. The one where you're... the bias reference counting that has the per-thread local count and then the global count. So one of my first experiments with the Galactomy was to just switch to atomic reference counting for everywhere. And performance was tragic. And in particular, the really tragic thing was that the per-threads you added, the slower it got. And I'm not good at computer architecture, but my understanding is that essentially the way that atomic Inker and Decker works is that there is an internal bus that's shared by the cores on your CPU that they use to talk to each other, to tell each other things. And in particular, when you atomic Inker and Decker, it's like there's one CPU telling the other CPUs, hey, forget about this cache line because I've blown it away because the more Inkers and Deckers that you do atomic, the more contention there is for that bus. And so that's a place where you're going to have scaling issues where atomic Inker and Decker is going to fail you. So how much overhead did you find? You talked about a certain amount of overhead. Did you see problems with this contention where adding threads made the atomic Inker and Deckerers get worse because you were doing enough of them? My understanding of the way caches work now is that there isn't a global bus locked. I think that was the early implementation on multiple... Not a bus lock, but a bus where they have to communicate with each other and they have to... Well, I guess a lock, yeah, okay. So caches are confusing. I'm pretty sure if I say something, it's going to be at least a little bit wrong, maybe a lot wrong. So atomic Inker and Decker don't do a global lock anymore. They effectively lock the cache line. What that actually means in how it's implemented in hardware is a little bit complicated and I don't think I can explain it. What it means is that atomic increments and decrements to different cache lines still scale well. So that's only kind of relevant to the reference counting here. It means that if you have a bunch of threads and they do some atomic reference counting, it still works fine as long as they're not to the same object. But biased reference counting doesn't really address the scalability issue. It only addresses the single threaded overhead. And the way scalability is addressed is by avoiding reference counting contention on shared objects. So mostly skipping the reference counting either by deferred reference counting for functions or immortal objects, or immortal, like skipping the reference counting entirely for immortal objects. There are some objects that are going to be shared across threads and they're going to have multiple threads modify the reference count. And that's often okay. It's sort of a matter of scale. Reference counting is just incredibly frequent, so doing it everywhere doesn't scale at all, but doing a little bit is okay. Having some objects that are shared is still fine. And that's what we saw in the Hanabi system, is that there's a bunch of Python objects that are shared between the threads, and it didn't affect performance too much. It was only the few things that were accessed so frequently, like the dictionary lock or the function objects that really affected scaling. Okay. That's heartwarming. We do have a question for the audience. Thank you for the amazing talk. I just wanted to understand, do you think in the future we will be able to get some semantics behind the whole work of paralyzing Python and removing the GIL? Because of course introducing, removing the GIL is going to introduce some edge cases in a lot of stuff. Do you think we'll be able to have guarantees on what we have in the Python code more or less or a model or a proxy? Especially because you've shown a lot of research ideas like bias, reverse counting, so it looks like it's possible. So I think one thing that would be nice is to have a formal model of what operations are performed atomically, but I don't think that's practical because I think it would be hard to come up with a formal model of how things behave currently in Python keeping existing behavior I think is more important than formalizing the model. One of the challenges with both the way the current implementation works and no GIL works is that there are a bunch of operations that are seemingly atomic if they're performed on simple types but might not be atomic if they're performed on types that like override equality or some other function. And it's really hard to reason about that in general. So I think what will be necessary is sufficient documentation but I think a formal model will be pretty difficult. Also if I can ask another question do you have, have you considered lock free data structures for some structures in Python? So I think lock free data structures might make sense in some places. I rejected lock free data structures for things like the built-in dictionary type because in general the lock free algorithms are slower to execute for at least for like single thread cases than algorithms that use locking. And the most common case here is lots of threads or threads accessing individual objects without any contention. Thank you very much. Yep, next question please. Hi, thank you for the talk. You mentioned the critical section technique and that it works because of the current design with the GIL. Once it's removed, is there anything you need to, we need to do like for further maintenance to make sure that nothing breaks? Yeah, that's a great question. So one thing I didn't mention in this talk is that the APIs that currently exist to acquire or release the GIL, like PyBegin allow threads I think and PyEnd allow threads are still used and are still important in this project. The good thing is that means existing code continues to work but it can be a little bit confusing and it was one of the things that came up in the Hanabi project is that my colleague wrote code not using this and it ended up causing issues. Essentially, you still have to use those macros at places before the thread would block. And it's important for two reasons. One is like the critical section API and the second is to allow the threads to pause at the correct place for garbage collection. If you don't use that, if you skip over that and block while I guess still attached the Python interpreter, you can risk introducing deadlock in the same way that if you block a thread while holding the GIL you can risk causing deadlock. So they're still necessary but they're used in exactly the same places. Thank you. Okay, one more from the audience. Thank you. I love this work, so thank you so much. So I have a question more about the community and the process. So you're in the process of submitting this as an upstream patch to the Core CPython interpreter and I know that you have been in conversations with people from the Core Python development community and the same way that in previous terms they say, no, 60% slowdown is too much. So what do you need to achieve before saying, okay, this is ready? Do you have some information about these are the goals that have been reached and the ones that I still have to do more work to reach before the community accepts this? Yeah, that's a great question. In terms of what's acceptable performance I don't know. I think it's something we'll sort of have to figure out. But to answer the question is, no, I don't know. Okay. And other things beyond performance like impact on other libraries or stuff like that? So I mean, having low impact in other libraries is clearly important. But again, it's one of those things where it's much harder, performance itself is kind of hard to quantify, but impact is even harder. So I think it's going to be hard to know what the requirements there are. They'll know when they know it. All right, I'm going to take the session chair privilege. I'm going to ask one more question before I let you go. Sam, you get to borrow somebody's magic wand for 10 minutes. If you could, would you give up on reference counting in Python and actually go to tracing GC? Because as we saw from your slides you've had a very complicated and difficult trail to get to where you are with performance. So reference counting is now very complicated and it's still causing performance issues. My understanding is that tracing garbage collection in a multi-threaded system is fundamentally more multi-threaded friendly.  So the short answer is no, I wouldn't. And well, this is a tough question. So I think the typical answer would be yes. It's much easier to get better scaling. There's a lot of issues you can avoid. There's a lot of issues that have gone to cyclic garbage collection. But one thing I really like about reference counting in Python is that it means that destructors are called as soon as the reference count goes to zero. Essentially as soon as the object is no longer referenced, as long as there's no cycle, which is the common case. And I found that this makes it much easier to integrate some C++ APIs, especially ones that handle external resources, either lots of like memory on the CPU . Before this, working on this, I worked on PyTorch and before that Torch 7, which was written in Lua. And Lua has a cyclic garbage collector. It doesn't use reference counting. And we ran into issues pretty frequently due to the garbage collector. So switching to Python and reference counting was actually really nice in the process as we were going from Torch to PyTorch. So I like reference counting. I feel like that might be a little controversial. But, yeah. I'd keep it. It's good news and bad news for PyPy. Thank you, ladies and gentlemen.