So, last time we wrote out this theorem which is let G hat be obtained from G by the elimination a weakly dominated. So, let G hat be obtained from G by the elimination of a weakly dominated strategy S hat J for a player J every equilibrium of G hat is an equilibrium of G. So, recall what the setting was we had G which was a game comprising of n players these were the strategy sets and these were their utilities. This is when I put these curly brackets around ESI means that I am looking at S1 to SN similarly curly brackets around UI is U1 to UN. So, this was your game G and then we considered another game G hat which is formed from the same set of players and these strategy sets S hat I and the same utilities UI and what was the connection between S hat and S hat I. So, S hat I was actually equal to SI for all I except for a specific player J and for player J it was equal to X SJ minus X hat J that chain my notations this was I use the notation X hat J. So, for this is for I equal to J right. So, in other words we all other player strategy sets were held the same it was player J strategy set that we change and how did we change it we remove one particular strategy X hat J from this and what was special about X hat J? X hat J was so X hat J is a weakly dominated strategy and what does the theorem say the theorem basically says that every equilibrium of G is G hat is an equilibrium of G which means that you look at the smaller game the reduced game that is formed from after elimination then all the equilibria that are there of this game are also equilibria of the original game and the we summarize this by saying that no that essentially if you eliminate weakly dominated strategies no new equilibria get created ok. So, if you eliminate weakly dominated strategies then no new equilibria get created ok. So, now let us see so this only says that no new no new equilibria get created all the equilibrium of your of G hat were already present in G, but you could have potentially lost equilibrium right. So, which now when you are losing equilibria it means that there is some possible strategic outcome of the game which has now which has been eliminated because you decided to ignore the particular strategy for a particular player right or you do not ignore I would say overlook a particular strategy for a particular player right. So, now question is when can we say that even that no equilibria are lost as well the non-equilibria are created, but can we ensure that no new no equilibria are lost in this process. So, that is essentially the main thing that we are building towards theorem. So, if x hat j if the above x hat j is strictly dominated then the set of equilibria of G is equal to the set of equilibria of G hat. So, this is the this is what you what we wanted to this is the main result of this particular exercise that if you eliminate a strictly dominated strategy for any player then the smaller game will have the same set of equilibria as the original game. So, neither are the no new equilibria are formed and no equilibria of the original game are lost in the process. So, let us prove this. So, as far as proving this is concerned we already know that no new equilibria are created. So, it suffices to show that no new no equilibria are lost. So, it suffices to show that if x star is an equilibrium of which one of G the original one then x star. Now, I want to show that it is an equilibrium of G hat, but remember we also showed another small lemma which said that if you take if you have a big game and I consider a smaller game and if the equilibrium of the origin of the bigger game is present in the smaller game then it is necessarily an equilibrium of the smaller game. So, it suffices to show that that if x star is an equilibrium of G then x star just has to be present in G hat then that is enough then it follows from there that x star is an equilibrium of G hat then x star is a strategy profile in G hat. So, in other words what do we need to argue that we need to argue that if x star is an equilibrium and x hat j is the strategy that you eliminated and x the eliminated strategy could not have been part of any equilibrium. If that is that can be argued then you are done because then you are sure that the equilibrium is present after even after elimination. So, which implies that it suffices to show that is not an eliminated strategy. So, what strategy profiles get eliminated as a result of elimination of x hat j? Now x hat j is a strategy of player j. So, x hat j gets eliminated for that player, but profiles strategy profiles that means for n players what gets eliminated? Yeah x hat j comma x minus j for any x minus j all other combinations the others can play whatever they want all of that gets eliminated. So, when you are removing that particular row for that player every combination with every other every column or whatever you however you want to think of it all of that gets eliminated. Now, let t j be a strategy dominates x hat j. What does this mean? That means regardless of what the others play t j is better for player j than x hat j. So, that means if I look at u j of t j comma x minus j this is strictly better than u j of x hat j comma x minus j and this is true for all x minus j in s minus j. So, now what do I do here? I just put in now this has to be true for every x minus j. So, I can put in my favorite x minus j and in this case the choice is natural I just put x minus j as x minus j star the equilibrium strategy the part of the profile from the equilibrium.  Let t j equal to no we cannot take t j equal to x j star because x star is an equilibrium and t j is just a strategy that we know it dominates x hat j. So, we have taken that x star is an equilibrium of g. So, and t j where is t j again remember t j is a strategy for in the original game because it was dominated x hat j from that original.  And this in turn is we know that this in turn is from here we know that this is actually less than equal to u j of x hat j comma x minus j star. What does this mean? So, we now have that the left hand side here is strictly less than the right hand what is the meaning of that? This means that basically you are this x see what is common here x star minus j is here x minus j star is here as well. So, the inequality is because of this one here we are still looking at u j the inequality is because of this x star j and x hat j means that these two cannot have could not have been equal because if they were equal then you would have got you could not have gotten a strict inequality here which means that x star j is not equal to x hat j. So, which means x star j is not equal to x hat j which means what? Which means actually x star is present in the as a strategy as a profile in g hat because x star j is not x hat j it is not the eliminated one. So, it is definitely present in the in the game that you get after elimination. So, consequently you get we have that the set of equilibria remain of g is equal to the set of equilibria of g hat. So, what this is what this is told us is that now that the elimination of strictly dominated strategies is with you know there is no punishment for that you there is no nothing you lose you not you do not create fictitious new equilibria nor do you lose any equilibria as a result of this. Is this clear? So, this basically justifies for us the whole process of iteratively eliminated eliminating strictly dominated strategies the exactly the thing that we use to solve for solve for the solve the game the in our previous class this justifies it that essentially after doing this elimination if when you are left with only one only one particular strategy that necessarily has to be the Nash equilibrium of that of the original game right. So, actually let us just write this down formally. So, if the so corollary if the process of elimination of strictly dominated strategies results in single strategy profile profile x star then x star is a is a Nash equilibrium of the original game. So, if you I should I should have made use the word iterated elimination if that is implicit here that if you keep doing this elimination again and again and you are left with a single strategy at the end then that has to be a Nash equilibrium of the original game. Why is that the case? Because we just saw that every time you eliminate you are neither losing equilibria nor creating equilibria and this happens at every step right and so finally if you are left with just one single strategy profile right one strategy for each player then it has to be that that itself that particular one that was left is necessarily the Nash equilibrium of the previous the immediate previous game and also of subsequent all the previous predecessors of that right you in short the original game that you started yes. So, I will tell you so if this happens then it is a Nash equilibrium of the original game. So, this automatically guarantees also that if you have if you end up with one then this is also a proof of existence that is definitely a Nash equilibrium of the original game right because when you are left with just one strategy that one strategy profile you have a trivial game and that that one point then is the Nash equilibrium of that G hat of the reduced game that you have found and that is by the theorem is equal to the Nash equilibrium of the yes. So, that is what I just so I can show you examples where I will come to that. So, stick less is required if you eliminate weakly dominated strategies this result is not true anymore okay you will lose equilibrium in the process yeah. So, if you end up with one it is it is it is necessarily an equilibrium then it is not then the statement is not true anymore then it is not true that no it is a Nash equilibrium of the original game okay, but it is what all equilibria are not retained. So, I will there is in fact a stronger version of this since Shashank has bought this up let me mention this. So, can I stand in this by saying something more is a unique Nash equilibrium right is the unique is rather than a is the unique Nash equilibrium of the original game. So, if you if after weekly the elimination of weakly dominated strategies if you are left with a single one then that is necessarily a Nash equilibrium of the original game correct that much holds, but that it is the only Nash equilibrium is not true anymore okay. So, it is the so when you are left with a single profile you have effectively then by this process discovered the Nash equilibrium of the game okay. Now the another corollary of this is actually which is implicit in the way I have stated this theorem see if it so happens that you are getting eventually to one strategy profile and that is the unique that then turns out to be the unique Nash equilibrium of the original game. Now suppose you eliminated strategies in a different order okay and then ended up again at this x star right what would that that would again tell you that it is the it is the Nash equilibrium it is actually the unique Nash equilibrium of the original game what this means is essentially the if you are ending up at a unique strategy profile then the order of elimination does not matter okay because even if you in fact forget ending up with a unique strategy forget about ending up with a unique strategy profile the very fact that you neither lose equilibrium or gain equilibrium means that you can eliminate in any order the smaller game will always have the same set of equilibria as the original game okay. Now the order this invariance of with the order of elimination is again not true for weakly dominated strategies once you have if you are eliminating weakly dominated strategies the equilibria that you end up with right because you lose you potentially lose some equilibria the equilibria that you end up with is sensitive to the order in which you have in which you have eliminated okay so let us actually do one example I yes but they would they would all have the same set of equilibria as the original one yeah so I have another example okay so layer 2 layer 1 top middle bottom and so okay so player 1 has so this is the game player 1 has 3 strategies top middle and bottom player 2 has 3 strategies left center and right okay and the payoffs are this 1 comma 2 2 comma 3 0 comma 3 2 comma 2 2 comma 1 3 comma 2 2 comma 1 0 comma 0 1 comma 0 and again both are both players are maximizing yeah so there are there are several options so let us let us there are strategies here in this game that are weakly dominating other strategies okay so and what I the point I want to make is that actually the order in which you eliminate will matter so let us let us go through one or two different orders okay so first let us first first observe let us take this so t is actually dom weakly dominated by m right so I will just write it like this you know t is weakly dominated by m for player 1 is that clear why because you are comparing 2 strictly greater than 1 2 equal to 2 3 greater than strictly greater than 0 okay so t is weakly dominated by m so but there is weak dominance so because in this case you get the t and m give you the same thing okay okay so t weakly dominates m now so suppose I then suppose I did eliminate t okay suppose I did eliminate t then in that case what would I what would I be left with I would be left with this smaller box here this one okay so now in this anything dominates on anything else yeah r actually dominates l again weakly 2 equals 2 sorry l dominates r sorry minus 2 equals 2 and 1 is greater than 0 so l dominates r okay so l dominates r I can remove I can suppose I remove r here now I am left with this smaller box where now m weakly dominates b so you can again remove this okay and then c is dominated by l and so you are left with an equilibrium which is followed by r dominated by l followed by b dominated by m followed by c which is dominated by l and that is that leaves us with the equilibrium ml okay so ml is a nice equilibrium of the original game this that give that came about from this order now let us see if we can go through another order so now let us suppose we take the let us take another order let us take the order b is is actually dominated by by m right so I can eliminate b so b is dominated by m so get rid of b first okay I am not going to keep making marks now so let us let us remove yeah b is removed so you are left with just the upper box okay now in this anything dominates anything else r dominates c let us see r dominates c okay let us let us see what happens r dominates c so c less than equal to r okay r dominates c so I have removed you can remove c then and you are left with then p m and and l l l and r okay r actually then dominates l okay so then l is gone so actually r dominates both in this case r dominates c as well as l right after having removed b r dominates both c and l right so you can get rid of both so l okay so l and c are both been removed and now you can come you have to just compare t and m in the in column r so then obviously t is worse than m you are left with so you can remove that gives you then m r is a Nash equilibrium okay so you can see here what happened was when we eliminated l actually the previous Nash equilibrium which was m l previous means the one from the earlier earlier order of elimination that Nash equilibrium is now got eliminated when you eliminated l whereas and and and you are left with m r as a Nash equilibrium is clear so so what does this mean essentially it means that the reason this has happened is because you are eliminating weakly dominated strategies there are these ties here right so for for if you take m m l and m r for example for the for the for player 2 they both they both give payoff 2 right but you said that r dominates l and you removed removed l but actually but it was it was strictly dominating only when player 1 was playing t then player 1 was playing m they were equal and he eventually ended up playing m in the equilibrium and that is why you lost out on one equilibrium right okay so in other words the set of equilibria the set of equilibria of the reduced game obtained by elimination of weakly dominated strategies is dependent on the order of elimination does the remainder set of profile yes it could it could change based on the order because you may yeah you could that could happen but of course if by by elimination of strictly dominated strategies I I mean you follow the same process you means you eliminate strictly dominated strategies of course you if you eliminate the exact same strategies regardless of the order you will get the same profile but if you are so in the in other words you eliminate strictly dominated strategies but you all eliminate them in any particular order and therefore you are not eliminating the same set okay then you will you you could get yeah could be could be I am not sure but I think one can construct an example where you eliminate something prematurely that could happen I think yeah yeah so I mean what a player we are not analyzing this as players right so we have as observers of the game we have eliminated this on behalf of the players okay yeah so all of these are Nash equilibria there are actually more you can play around with this game a little bit more you will see there are more Nash equilibria also you can come up with other orders of elimination and so all of them are Nash equilibria it is our our effort at trying to arrive at one of them by elimination which is the problem so you you basically we had no we did not have as much justification to eliminate weakly dominated strategies and as a result we have not properly analyzed the game that is what the conclusion is yeah no not at all not at all so if if you have a unique Nash equation is if there is a unique Nash equilibrium can you necessarily find it by elimination of you know dominated strategies no so you will you will in general end up with a game you will often have a game where there is in fact generically you will have a game that has no dominated strategies but does have a Nash equilibrium some dominating strategies but an Nash equilibrium does not exist yes yeah that is possible okay so what this is basically done is now it has helped us in some cases in certain types of problems wherever this process actually converges to a eventually one unique strategy profile in that case we have basically through a logical elimination you know solved for the solution of the game and it turns out that actually coincides with the Nash equilibrium itself okay so this this way of solving a game you know like a puzzle trying to say okay this would happen then okay then that would happen and so on and so forth almost you know trying to like a logical logical puzzle this way you can you can do in up to this up to this point when the game has the property that it ends up eventually this process of strictly eliminating strictly dominant strategies ends up end you with one strategy profile okay now in general what you would be left with is some is a set of strategy is basically a reduced game from which you have removed all these dominated strategies and but that is it you cannot analyze that any further with this logic okay and that is where you need you need a leap of logic which is which is what the Nash equilibrium does that it tries to learn reason about okay what are the communication requirements in the game and so on and so forth okay so you go into that you need to get into that whole line of thinking once you get once you have done this kind of basic elimination okay all right okay


Everybody forced here rip in 2022 select WWE hobby one box number six random pack style With random a second I'm gonna pull the packs out just as they lay here and set them face down Just as I'm pulling them out like that top left side is gonna be packed one bottom right is gonna be packed Well, I'll do my best to keep them on the side. They should be on so It is a bit tough with selects sometimes. Let's do the random now 12 numbers 12 names will put them side by side to see who's got which pack before we get started Eight times on our random couple of fours one two three four five six seven eight Jordan W down to Joshua W and Eight times on the pack numbers one two three four five six seven and eight Four down to three. I've got Jordan W four Ray G 10 Jordan W one Valentin two Jordan W eight Chen Kai 11 Jordan with six and seven Chen Kai 12 Jordan has nine and five and Joshua W has three We'll get these results posted. We'll get ripping. Can we hear just a little bit behind guys? It's tough to do these in catch up a little later All right One box number six this one is laying here. There's also two in the back So this this is tough this one right here on the bottom is going to go to the right side because it's laying more that way and that one's Going to go left side. It's the only way I see to do this This one goes here These stay together. This one is packed one. That's kind of there you go That's where I can see it. It makes more sense to me I Stand by your decision. We only uh Can only run with what the select box gives me pack one right here got Brunner San Bernardino Becky Lynch looks like we got a E.O. Shirai insert NXT insert Got a blue premier level Alondra blaze 125 of 199 and a Jordan Devlin ringside. I don't come back back to Do drop happy Corbin Godfather a silver premiere level Seth freaking Rollins Not numbered in a comma number three Stone cold AJ Styles Wesley Blank there a Seth freaking Rollins Tri-color concourse and a Natalia Yeah, there was a ton of hype when it first came out. Let's get that as settling down though Make it affordable pack number four now triple H JC Jane an insert here is Jerry Lawler Hall of Fame selections scope concourse Demian priest and a Jerry Lawler Pack five Ricky Schell looks like there's an auto in pack five here Bobby Lashley It is 46 of 99 legendary signatures red Nikki Bella Nice hit Let's go with the red.  It's just that hype of like first time it's hit prism, you know, yep McFauley a Tri-color concourse Champa non-numbered and a J. Uso Got pack 10 now Rob Van Dam Shamus King Woods on the up a nomin on Maroon Oscar premiere level 39 of 149 nice color there a rookie premier level Valkyrie pack 11 Vaughn Wagner Nicky ash make us out of more a silver concourse rookie Grayson Waller and a Carmella and pack 12 Thing should be an auto here Harlan's the auto is Bushwacker Luke Signature selections that is awesome big-time throwback there Bushwacker Luke Who's the other one? Premier level tri-color Vader and Shinsuke Nakamura that'll do it for one box number six one box number seven bill bill Thank you for this number seven is up next


There's that saying that you never know what you have until it's gone. You never appreciate what you have until it's gone. But with Ravens and Marlon Humphrey, that definitely is not true at all. We know what we have in Marlon Humphrey, we love what we have in Marlon Humphrey and we appreciate what and who we have in Marlon Humphrey on the field. One of the best cornerbacks in the league, one of the top pay cornerbacks in the league now, a playmaking cornerback and he is a grinded out type of cornerback. He's not the pretty guy that's not going to get all the sexy interceptions and what not. But he does make the play and continues to put in the work. Fruit punch, that's all I got to say. And then of course, off the field, we all know Marlon Humphrey especially in the post game Instagram live videos when they on their way games. Once he get that camera in his hand, that's a wrap. That's a wrap. Once he start, once he goes live, oh yeah, that's it.   This was a move that we all knew was coming because once he got on the not so team keep it clean list last week, then we knew he was going to miss the Colts game. Then they officially announced that he was going to miss the Colts game. And we expected him to be back for this game coming up against the Patriots and now it's official that he is back and he'll be practicing today in the first team practice this week of the week for the Patriots. So that is very good news, especially comes just after the Ravens signed Tramon Williams and with Marlon Humphrey coming back and Tramon Williams being inserted into the lineup. We'll see how often he plays. We'll see how much he plays. We'll see if he ends up taking over for Terrell Bonds and Terrell Bonds gets pushed more to a special teams role. And I know a lot of people have been getting on Terrell Bonds and they say, oh, he's bad.  He's this, he's that. And I'm like, again, same thing I said in the past couple of videos about Terrell Bonds, like it's not like this dude is giving up a bunch of touchdowns or anything like that. And I know for Ravens fans, we do have high standards when it comes to the cornerback position. I know that. We definitely have high standards when it comes to the cornerback position, but at the same time, like, who do you expect the quarterbacks to go up against? Who do you expect the quarterbacks to go after? Marcus Peters or Jimmy Smith or Marlon Humphrey or Anthony Avent when he was in a situation. Marcus Peters, Marlon Humphrey, Jimmy Smith or Terrell Bonds when he was in a situation. So with him giving up the plays that he gave up, like it's I just I think sometimes we as Ravens fans just got to tone it down a bit and be more realistic because we're not we're not going to shut down every single receiver every single week. It will be nice. Be great. But that's just it's not going to happen. Every cornerback can't cover for all the time in the world and Terrell Bonds is obviously the most inexperienced one, so they were going to go after him. But we'll see what Tramon Williams role is going to be. Now a big point about Tramon Williams role and that what we hope to see is that he ends up making Marlon Humphrey kicking Marlon Humphrey back to the outside because again Marlon Humphrey is primarily outside corner again. We appreciate his playmaking ability from the slot now to but that's just that's not him. That's not his position. He's filled in well there and Ravens could do a lot worse at the slot. So it's been like a good problem to have for the Ravens kind of because it's like man we have all these corners here and there even though we lost a lot and we were depleted for a little bit but they added Tramon Williams and now we back in business but it was like with Marlon Humphrey we will lose our slot corner and then they would have to put him into that slot role. So he wasn't at his natural position on the outside but now I'm just wondering if they're going to actually put him back there once Tramon Williams gets on the field. So it's just going to be something that we just got to wait and see as as time goes as the the season goes and whatnot and maybe even we may get that question answered starting even maybe this even maybe this week when the Ravens go up against other Patriots on Sunday Night Football for the world to see. Oh that should be a fun game. That should be a really fun game. So it but it's nice to be getting some depth back. It's nice to be getting the boys back all together. It's nice to be just getting everybody healthy and everybody team keep it clean because it was it hurt like last week when we found out that Marlon was on the not team keep it clean list and he was going to miss the game because we know Marlon's impact on the game. We know that this dude can take away he can take away a receiver out the game and we know that he's good for we know that he's good for making some plays Marlon Humphrey ever since his rookie year been a very physical cornerback. That is one thing that I don't get. I don't I don't hear that get talked about enough when it comes to Marlon Humphrey is how physical he is and he showed that from jump. Soon as he stepped on a scene with the Ravens he showed that I remember in preseason. I was like oh that's the rookie who he drafted. Oh wow that's number 29 who throwback Marlon Humphrey right there. So him when he went out it was like oh man even though we still have Marcus Peterson Jimmy Smith so we know we're gonna be all right but still we are of course our much better team with Marlon Humphrey with our best defensive player then without. And with Marlon Humphrey I remember tweeting it during the game I said man I said I know them boys gonna hold it down I know they gonna hold it down during the coach game I said I know the defense and all of them gonna hold it down but we certainly miss Marlon Humphrey because he does so many things he makes so many different types of plays. He'll get an interception here and there of course forced and fumbles we all know about them for his fumbles but Marlon Humphrey he helps him run support too. He helps him run support a lot actually again very physical corner and I think I think the game on Sunday Marlon Humphrey being out forced Marcus Peters to step up because I remember on I think it was on the open and drive there was a run play and Marcus Peters was back there. He ain't made the tackle but he was back there and he just knew that okay I'm I'm gonna have to step it up a little bit and I mean he been doing this thing pretty well for the most for the majority of the season besides that Chiefs game but the Chiefs game was really rough for everybody so I can't even single Marcus Peters out for that one but that Chiefs game was rough everybody but besides that he's been doing this thing but in the coach game I think he really took it upon himself like hey Marlon Humphrey's out I'm the guy I'm the guy and I have been the guy before so it's nothing new but at the same time hey I'm the guy so he was like shout out to Marlon Humphrey I am going to try to do both his and my job this game and he did both because he came and he tried to help him run support it didn't work out so well but then on that force fumble what cornerback we know who usually forcing the fumbles on the Ravens it's Marlon Humphrey but what cornerback ended up forcing the fumble that ended up getting scooped up by Chuck Clark and going for six it was Marcus Peters so he did Marlon Humphrey's job and then he turned around to in the second half of the game Lamar and Ravens driving down the field oh man Lamar is hot they up the tempo and everything again that whole room about that it was hardball caught in the plays I still haven't seen anything that acknowledges that so let's see but anyway they got the up tempo offense going Lamar connecting and whatnot hitting people running everything's working clicking they get to the goal line Gus go ahead finish it off boom fumble Marcus Peters said wait wait wait wait wait I'm gonna still do my job I got it don't worry Phillip Rivers takes a shot and that's what you do that's what you do if you get a big turnover like that you take a shot take a shot because the defense is coming out and they're like oh man we back out here we didn't get the touchdown really upset they whine and they're so you take a shot he took a shot on Marcus Peters and Marcus Peters say hold up I did Marlon Humphrey's job now let me do mine's too interception questionable interception but interception so shout out to Marcus Peters and shout out to Marlon Humphrey and just shout out to the Raven secondary man because as hurt as they have been as depleted as they have gotten they are still a very good secondary because our top guys are like some top guys in the league not just on the team not just in the AFC not just in the AFC North but our cornerbacks are some of the top in the league and that's something that we definitely appreciate especially as Ravens fans and then you add Tramon Williams to that the ageless wonder a guy who actually when I was watching him yesterday he didn't play bad last year now I read something to that said that he was like ranked in like the top 10 of cornerbacks now I'm not sure if that was the top 10 in slot corners of just a top 10 period but regardless man you added a good one and you added a small one too so that makes this team just that much more dangerous and just that much more lethal in the secondary because I know it's a lot of fans is a question about the age I question it myself like 37 years old oh my goodness but again age doesn't tell the whole story doesn't tell the whole story so we'll see how it works out with Tramon Williams and just this entire second area as a whole as a unit but moving forward they they are in some pretty good shape and I'm looking forward to a team keep it clean I love y'all so much I appreciate y'all thank you for always listening thank you for rocking with us I'll see y'all later on y'all have a good day we out


No promises. OK. Who's ready? Already? Good? OK. I'm not big on intros, so Eugene offered, but I politely declined. Thank you anyway, Eugene. So thanks everybody for coming. Who knew that this was a technical meetup? Hopefully everybody. OK. So it's not going to be super, super, super technical, but it may be more technical than ones that you've been to in the past. Who would like to see more technical topics and meetups from Data Science Singapore? That's everyone here. Not surprising. How many of you have given or have offered to give a technical talk here? Oh. So, hand. Thank you. Now we see the problem. Who's going to give these technical talks if it's not us? Because it's quite hard to find technical speakers, right? Super hard. So I'm sure you know some stuff that we would all be interested to learn. I'm positive. I'm really, really sure. So if you want to see more technical talks, it's up to us to give these talks. So please, if you have some topic that you think is interesting that's technical, talk to Eugene. Where's Kaishen? Talk to Kaishen. Talk to whoever, somebody, anybody. Just shout around and say, I have this technical thing that I want to talk about. When can you give me a slot? And I'm sure they'll probably be very happy to do it. So just a bit of preliminary. If you want to see more tech talks, please volunteer to give tech talks. Next steps. Today, I'm going to talk a little bit about data processing on laptops, because that's what I like to do. I don't like clusters. I'll go on the record saying that. I'm on video. So I'm on video saying I don't like clusters. I'm not a big fan of Spark. I'm not a big fan of Fadoop. They're really good tools when you need them, but most people don't, most of the time. Some companies do, but not always. And my claim is that you definitely don't need them for machine learning. Do not. You might need them for pre-processing of data, but for actual machine learning tasks, it's very unlikely that you need them. So that's me. That's my... Can you read that? Is it small? It's kind of small. You can find me on Twitter, Aadrake, and my website as well. So if you're on Twitter, and you think what I'm saying is really awesome, you're like, this guy, he's cool. Aadrake, hashtag hacker. If you are on Twitter and you think this guy is nuts, he's crazy, and I don't even know how he got the position to talk to us. Aadrake thought leader. No one else will know, but we will know.  And this is not mine, by the way. I stole this from Josh Wills, who was formerly with Cladera now at Slack. So hacker, good, thought leader. So these are my claims. These are important. The RAM that you can get in a machine is growing faster than the data that you need to process. This is an important one. There are lots of techniques that you can deal with so-called big data that don't involve these big systems like Spark and Hadoop and stuff. And yes, I know you can run Spark in a local cluster on your laptop. I don't mean that. I mean the big stuff. So you can do everything on one machine. I do everything pretty much on my laptop all the time. This you may not be able to see. We can't cancel it. This is a slide from Kray. Who knows what Kray is? Did anyone know this? You're laughing, so I know you know. Okay, how many? One. They build really, really, really big fast computers. And they've been doing it forever. I don't know, like 40 years or something. One of the recent ones that they released could accommodate 500,000 CPUs, which is really a lot. So my favorite part about this slide is the little part in the bottom that says Kray confidential, do not distribute. That's like maybe my second favorite. But what I thought was really cool about this was if you look at some of these claims and you do data science work in industry, you've seen some of these. So you have all these different processing systems and the data is siloed and it's everywhere and it's hard to get to and oh, this team has this data and that team has that data. Thank you for being on time. You're awesome. We're not shaming you publicly. I'm supporting you. But this is like part of the problem is that companies and teams set up these big, big, big systems. And I'm not saying Kray is doing the wrong thing here. They're actually marketing some pretty cool stuff, but their slide is really indicative of some of the problems. So you get this big, huge cluster thing for your data prep ETL. Then you get your stream processing system maybe, which is like Storm or SAMHSA or whatever. I don't even know how many there are now. And I think Twitter just, they're releasing some like every few months. Then you have data mining stuff that you have some interactive query thing, which is like your Teradata or your data warehouse or whatever. And all these things are separate and usually the data doesn't agree and things like that. So the idea is that instead of having bigger machines will be a little more intelligent about how we do this. So what if you have too much data? And too much is subjective. So there was a Kaggle competition recently. I think it just ended like a day or two ago or maybe today for Expedia hotel recommendations. Was anyone looking at this? Am I the only one here? Who here has ever done a Kaggle competition? Just so I'm curious. Okay, so quite a lot, quite a lot. Okay, cool, cool. So there was one today that just ended for Expedia. I think the data set was about four gigabytes plus minus. Was it? And when I looked on the forums like a couple of weeks ago and some people were asking questions, how can I process this data so much? How do you start up a Spark cluster on Amazon? And credit to them. If they're doing it for fun, for learning, that's awesome. But I mean, this laptop, by the way, is not new. It's like three or four years old and it's got more than four gigabytes of RAM in it. So I'm confused about why they were asking, but it's a very common tendency that I've seen on Spark or on Kaggle. That people want to use Spark, they want to use Amazon, they want to have clusters and do all this stuff. And well, one thing you could do is just add more RAM. Is that an option? And you can kind of see that. This is the frequency of people who have worked with certain size data sets. And if you look at it, I mean, 10 megabytes is like Excel size. So that's no problem. There are some people that have worked with like 10 terabytes or one petabyte, but it's not common. So if you chop out like the top 10% and the bottom 10% that are kind of skewing this, and then you kind of run a regression line through it, you get basically that there's a 20% year on year increase in size of data set that people are working with. So it's not like massive, right? It's not like the data sets we're dealing with for machine learning stuff are doubling every few years. And this source is from a Katie Nuggets poll. So in fair interest of full disclosure, there could be some sampling bias here. The people who are actually responding to these polls and surveys, something to keep in mind, but maybe it's representative. So let's say that data sets are going up 20% year on year. This is the progression and availability of RAM on Amazon EC2 instances, which you can now get a two terabyte EC2 instance. I would be super surprised if you have two terabytes of data that you need as your working set for your machine learning. Does anyone here have two terabytes? I don't, I don't. I'm so sick. They don't have two terabytes of data they need for machine learning stuff, not just two terabytes ever because maybe you have like movies and stuff, but. Anybody? I don't, I don't. And I know some companies that have, but it's not that common, right? Maybe you have like two terabytes of logs that you need to process into something that's usable to do machine learning. But usually it's not a huge need where you have two terabytes of data. And I put this note on here because I think it's cool. Tyen actually has a new system available that you can put six terabytes of memory in, six terabytes of RAM. Like that's a lot. When I looked at the case and all the RAM slots, like the case is mostly RAM slots. It's insane, it's insane. So even if you use EC2, you're getting like 50% year on year increases in RAM size, which is more than 20%. So clearly one option is just add more RAM. But that's kind of lame, right? That's the argument about vertical scalability versus horizontal scalability. So who here has heard the word scalability at work or school or something in the last three days? Yeah, it's crazy, right? Everyone's saying that word. It's kind of odd. So usually when people are talking about that, they're talking about horizontal scalability. We wanna add systems and we wanna be scalable and oftentimes they're using it in the generic sense that they want something to be fast or they want it to be fast even when we get more data or fast even when we get more users or something like this. But that's not always needed. So one of my favorite steps is just ignore some of your data or as statisticians call it, sampling. So just ignore some stuff. The classic case of this, there are many, many, many cases. So if you have terabytes and terabytes of log data, like for example, clicks and views for online advertising, is anyone here working online advertising or something like it? Nobody here works in online advertising, are you just embarrassed? I worked in online advertising before. Okay, one. I feel you, I feel you. Sorry? Sorry? There was. Okay, okay, okay. Was it Vito? Okay, last year, right? So just as an aside, if you don't do Kaggle competitions, it's a good idea to check it out. It's a good idea to check it out. The problems are, it's kind of a bit contrived, right? Because the data's already been processed, so a lot of the legwork's already been done for you. But you can experiment with a lot of different machine learning techniques and feature engineering and all kinds of other things that might be really useful. So if you haven't tried out Kaggle competition, I suggest you check it out because it might be. So yes, there was one last year for Vito, which was about click-through rate prediction. And most ads don't get clicked on.  I have ad blockers. I don't want to see them. They're not relevant or interesting for me and they take up my bandwidth and they make my browser slow and it's just not great. So no surprise that few people click on ads, but I mean, really few. 20 people in 10,000? It's not that many. So if you have a data set of clicks and views, you can get rid of like 90% of the views and still train a classifier on that data set and it'll be fine. You're not gonna have to worry about it. You can literally throw away like 90% of your data and use the remainder as your train set and it will be okay. Now, that's a very coarse way to do it. There are a lot of theorems and stuff in mathematics that you can dig into if you wanna have guarantees about your sub-sampling and how you do it and if you wanna do under-sampling or if you wanna over-sample, under-represented cases and all this kind of stuff. But for this particular example, you can pretty much get rid of your views and it'd be fine. You look skeptical. Are you skeptical? Okay, I thought you were looking at me like, no. That's the way I look at people, sorry. That's good. It's always good to be skeptical. That's good. Oh, by the way, if anyone has questions or something that you wanna bring up along the way, please do. This is not really a lecture. This is more like a knowledge sharing thing. So if you have questions, oh, say that was fast. How do you know it's a representative of the sum? When you say you would throw 90% of the sum. Because if the difference is so big, like in this case, 20 clicks per 10,000, I'm not gonna worry about whether or not it's representative. Because the probability of having click is so low that you can get rid of the views. If you just take away a random sample of the views, it's not really gonna matter that much. Because your classes are so unbalanced. It's not like 60, 40 or something. It's like really, really skewed. So you mean are we sample only the click data? No, I mean take 90% of the views or a lot. I mean, I don't know what makes sense in a particular case. So it's case by case, right? But take a big chunk of the views and just ignore them. Just pretend like they never were in your train set to begin with. Seems weird, right? Like who would ignore data? Why would you do that? Statistician been doing it for 300 years. They're experts in how to ignore data effectively. Actually they do it backwards. They start with the important thing, but same idea. They use sampling techniques so that they make sure that it's the representative of the whole population. They just don't throw the data. Well, your population of people that click is so small that you can get rid of 90% of the views and the chances that you're gonna have any different. Unless you have like, okay, maybe you have something weird like you only have two features in your whole data set. And I don't know, there could be some contrived like pathological example where it wouldn't make sense. But in general, you can do this and it's not a problem. If you just take some uniform sample of the highly, the much bigger class, it's not gonna have any effect on the accuracy of your classifier. Assuming that you're doing some kind of classification problem which the ad click problem is a classification problem. Yeah. If you wanna learn more about sampling, though there's a lot of cool stuff, especially in the literature, if you wanna look on it. There are many, many papers on how to do effective sampling if you wanna get mathematical guarantees on it. So you can do that, but I didn't go into that today. Sorry. Sorry, I'm skeptical just because I think you would skew the distribution point severely the other way. This is a rare advanced search. So you're taking a data set with rare defense. No, not the whole data set. I'm not saying down, okay. Two things. One, if you down sample the whole data set uniformly, the chances that you're gonna down sample a click is basically zero. Do you agree? Yes.  That was only my first claim. So claim number one, even if you took the whole data set and just randomly threw away like X percent, the chances are extremely high that you would only be throwing away views. So that was, that's claim one. Claim two. I'm not actually talking about doing that. What I'm talking about is just take your views and then get rid of those. What's the difference? Not really much, but you get some guarantees that you're not throwing away clicks. Non clicks, views, yes, sorry, sorry. Yeah. Yeah. So in this kind of problem for anyone who's worked on it, usually you have tons and tons and tons of views. So think of them as training examples with label zero and then you have very, very few clicks which are training examples with label one. And you can get rid of most of those training examples with label zero and you'll have no negative impact on the accuracy of your classifier. Sounds weird, right? Super true. Super true. Sorry? Not in this talk, but we can talk about it afterwards if you like. Yeah. Okay, so one option, get more RAM, but that's kind of like cheating, right? Another option, sampling, which can be done in a lot of ways as we just talked, you can do uniform sampling, but basically any time you have a classification problem with really unbalanced classes, there are ways that you can cut down on the data volume and still, in fact, it can improve the accuracy of your classifier because depending on what kind of metric you're using for success for your classifier, if you have this problem, you can make a really good classifier that just always says no click. It will be right, whatever, like 99.99, like all the, it'll be right all the time, basically. So you have to be a bit careful about how you're structuring these problems, but in general, you can get rid of data. So sampling, that's one option. But what if we don't wanna do those two things? Let's say we wanna deal with data sets of arbitrary size, we don't know how big they are and we have constraints on hardware. So let's say we only wanna do data sets of arbitrary size on this four-year-old MacBook Air. So what do you do? One, you have to stop thinking in terms of data set and start thinking in terms of records because you're not gonna fit your data set, whatever it is, in your RAM. Assume it's size infinity, gigabytes or something. So you have to start thinking about records. And you may not know exactly which features are coming in on this data set. So you need a way to do feature extraction on a per record basis. So this is the part where I'm gonna ask, does everyone understand what I mean by feature extraction on a per record basis? Okay, so if you have, so you get a CSV file. It's got five columns, right? You know you have, let's say, five features. But that's kind of a nice example and you don't always have that. Sometimes people push data into the system and maybe it has five features or maybe it has seven features or maybe it has 10 features or maybe it has 5,000 features. So what do you do in this case? Because you don't know in advance what's in the data necessarily, right? Because it's of infinite size. So there could be other things in there that you haven't seen before. So you need a way to deal with features that you haven't seen. And the way that you deal with it can't depend on any other records. So if I get a training example, the way I deal with the features should only depend on that particular training example. So I'm, basically that's the case where you can only fit one record in memory, which is not true usually, but it's good for our purposes. Does that make sense? Because if you need to see the next five million training examples to figure out what the features are, maybe you can't fit that in your memory. So you need a way to extract features that only depend on one record at a time. That's what I mean by stateless. So you're not keeping the state of future extract. Make sense? So where do we find some kind of data? What do you mean? I mean what kind of data fits this description? What kind of data? Generally we see like some, like five features or number of features are possible, but you are saying that every record might have different number of features. So what kind of data has this kind of data? The favorite example I've seen is also the one that I liked the least, which is anything that's in JSON. Sorry, I just don't like JSON. It's really nasty. It's like, you can take, yeah, I saw there was a funny picture one time that somebody said, oh, I had big data and then I converted it to XML and now I have even bigger data. It's like, it's true. Has anyone ever converted JSON data just to CSV? You save like a lot of space, right? Just because you don't have the keys and stuff. Anyway, if you're dealing with JSON data, you don't always get all keys in every record. So sometimes, depending on the system, people will send keys with some kind of null value, but it's not really required from JSON, right? So you don't actually know in those cases. So you need a data source, which you can think about as a stream instead of a big file. That's one thing. You need a way to process your features without knowing what came before or what came after. And you need some kind of model which can support incremental learning. And what I mean by incremental learning is you have a model, you give it a record for some records, some examples, some training stuff, and it learns those, and then you give it more, and then you give it more, and then you give it more. And once you have these three things, you're actually not limited anymore by what you can do with your machine because you can process a stream. You don't care what features come in, and your model actually doesn't care how many records you give it at one time. So you can give it one record at a time over and over and over again. So if you have those three things, you have a lot of flexibility on the size of the training set that you can manage on this machine. Okay, data source. Let's talk a little bit about functions in Python. So somebody asked what the language was gonna be in the talk.  Ah, Python, right? Yes. Python. Yes, thanks. It's gonna change. It was a trick. Okay, so functions and generators. Is anyone comfortable with the difference between a function and a generator in Python? Let me try this a different way.  And raise your hand if you've ever written a Python generator. See, that worked a lot better. That worked a lot better. Okay, so I'm gonna take that to, I'm gonna assume then that not everyone's comfortable with generators, which is cool. It's like a function, except when you call it, it doesn't go away at the end. It sort of hangs out and it keeps the state. Keeps the state. So if you have a generator that's like incrementing a number, every time you call it, it will increment a number. And it will return the next one. It will return two, within three, and then four. Because it's keeping the state instead of just returning a value internally. Which is kind of convenient. Because then you can make a generator that reads lines from a CSV file. And every time you call it, you will just get the next line from your file. So now we have a data stream. It's magic. It's really cool. So I use this all the time. This is actually, I think this might be, this is code maybe from a Criteo competition that Kaggle did a while ago. I don't know if they've done more than one. But I use this stuff all the time. So if you have a CSV file, you just give it the path and how many columns are in the file. And magically you now have a stream of data. So you already fixed problem one, right? You transformed your problem into I need to load this big file into RAM to I'm going to process this line by line. Or for people like Pandas, there's a chunk size argument to read CSV. Who knew that? Not so many. Why are you laughing? Yeah. Okay, so a couple of people knew, but not that many. So did anyone here use Pandas? I do sometimes, okay, some people. So for anyone, so let me rephrase, who hasn't used Pandas before? Okay, cool. So Pandas is a very convenient library for Python. It has all kinds of nice stuff in it. If you've never used it before, I highly suggest you check it out because it will make your life much easier if you're working with data. And one of the things you can do with it is you can read CSV files, which is very nice. And if you can't read the whole file, you can supply this chunk size argument and it will just read in that many lines. So it's kind of the same thing that we did except instead of reading one line at a time, it's reading many lines at a time. And then you can do something with whatever that chunk is. Okay, so we got our streaming part. Do we start right at seven? This is about to get intense. Okay, feature extraction. Has anyone heard of the hashing trick? I love things in machine learning that are called the whatever trick. Makes it seem so fancy. Okay, so the idea is you take your features, you hash them, and you use those values as an index into an array containing your weights for that feature. Did I get it right? Did I get it wrong? Who's heard of the hashing trick? Did I get it right? Okay, cool. That's the other thing. If I say something you think it's just totally crazy, I'm depending on you to tell me that it's totally crazy. So now you have a way where you can get in features. You don't know what they are. You get an index into an array, and then you can have weights for that feature in your array weights. Now you can also do a nice trick with this that we'll talk about later, but this is what gives you the ability to deal with arbitrary features in your dataset. And if you think about it, it makes sense, right? Because if you have three columns, like first name, last name, location, the information contained in first name, Adam, if you just concatenate that together, this is the same amount of information. And if you hash it, assuming that that hash is unique, it's the same amount of information. So there's no information loss that's happening here. You will get information loss if you have hash collisions, but we won't talk about that. So this is a very nice thing to do. I use this all the time, all the time. So hashing trick. If you haven't checked it out, definitely check it out. And then the third step was we needed models that support incremental learning. And actually, there are quite a few of them. So who here has used scikit-learn? Maybe about the same people, same pandas and scikit-learn seem to go well together. So all these are in scikit-learn and they support incremental learning. So what that means is they have a partial fit method. And if you use the chunk size argument from pandas, then you can supply that chunk to the partial fit method of any of these and be just fine. So now we have all three things that we needed. Caveat, not all the models can deal with unknown, with arbitrary features. Some models need the set of features to be declared in advance. So just be careful if you're using some of those models. However, if you're using SGT classifier in regards to those don't care. So you can just throw whatever you want into those models and they'll be totally happy to learn for you. Or you can just write your own, which is kind of nice. It's not that complicated. And it looks chopped off, which is awesome for this projector. But still you can see the code. I think it's like 13 actual lines of code. And this is not my code. This came from, I think it was the Criteo competition on Kaggle. There was some very cool person who posted this on the forums. It was like beat the benchmark in 200 megabytes of memory, something like that. People who are laughing probably remember reading it on the internet and face-palming.  I was like, ah, okay. So this actually will get you in a top X% place in Kaggle. And it's like 13 lines of actual code. So this is stochastic gradient descent with logistic regression, no fancy stuff there. And you can't really see the log loss, but it's like 0.463, which was pretty good. But the problem is it doesn't use all the cores. It was Python. So it's gonna use one core on your machine. And if your machine has four or eight, or if you have one of those big servers with six terabytes of RAM, maybe you have 100 cores or something. Or if you have one of those cool CPUs, those sort of power eight CPUs or something from IBM that have like 96 different threads, you won't be able to use those if you're using Python in a default, in a normal way. The problem with this is it performs well. So the log loss is four, six, three, but you can't really see it, I think it's chopped off. The runtime on my laptop is like an hour and seven minutes. Which is, I mean it's okay, but for people who do a lot of data analytics work, one of the things that really slows you down is the lag time between running your models. So if it's like an hour plus every time, that's like the old school version of I can't work, my code is compiling. It's just like the modern version of well, I don't know, my model is training. So that kind of thing. So the faster you can train models, the better. And there are other ways you can do that. You don't have to train on the whole data set. You can do some sampling stuff like we were talking about earlier, but if you want to run it on the whole set, on my computer, this takes like over an hour. Kind of long. So ideally I'd like to be able to use all my CPUs, which I was not using in the Python version. And if I share any memory, I want to use locks to make sure that I'm not writing to the same area of memory from different cores. At the same time, because that can be bad. But this was a question from Qashen. What about the GIL, the GIL, the global interpreter lock? Has anyone ever battled with this in Python? Couple people. Now we're getting into more esoteric territory. This is good, this is good. So in Python, when you're executing all these instructions, the Python interpreter has a nice little lock that will only let one instruction execute at a time per thread. So if you run Python, whatever, your script, one instruction at a time. So if you have, whatever, eight cores on your machine, it's not going to use those other cores normally. Even if you spin up more threads. But that's the important part is that this is per thread. If you have a totally separate operating system process, like if you have eight interpreters running, then you're actually OK. Because each interpreter will have its own interpreter lock. And all you have to do is share the memory. All you have to do. So there's this nice library in Python, multiprocessing, which you use to do this, to spawn these multiple processes. So you can get around this global interpreter lock problem. And there's a nice structure in there called raw array, which gives you a raw C array in memory that can be shared between these processes, which is quite nice, because now it seems like we're starting to get around all of our problems, right? Like, we bypass this lock. We're going to use all of our cores. We're going to make it faster. We have access to shared memory, which is really fantastic. And we can run multiple processes. So if you run it, you'll get this. This is like 10 processes. Each one gets access to the same chunk of memory and just increments some integer of whatever process number they are. Basically, just to show that all these processes are accessing the same memory. So it's not that hard. And once you do that, you can have a queue. If you remember before, we had those records that we were generating from the file. You can put those records in a queue. And then all of these processes can pull from the queue and do the work in parallel. So it sounds pretty sweet. It can look just like this. It's not very hard. But that's actually slower. That's slower. So even though you're using all the cores now, it's slower. So when something is slower, more experienced tech people, what's the first thing you do when something's slower? Somebody's got to answer. What are you doing something slower? Give the process in the file. Yeah, that's step zero. Good point.  And don't say go have a beer or something like that. Like what do you do technically to try to fix it? Check if you're mixing all those CPUs. Yeah? Yeah? The i will pull. I don't know. I don't know. That's the point. We don't know. So what most people do is they just start changing stuff. Bad idea. First thing you do is always profile. Always profile. So I profiled that code. Why is it slow? Let's run profiler and see why. I don't know if you can see, but this is the code. And it's calling out to raw array, which takes, you can't see it because it's fuzzy. It takes like 0% of the time and some other stuff. And then there's put, to put the item into the queue. So we created that queue, right? And then we were putting stuff in the queue. So all the workers could do it, could work on it. And this put method is taking up, you can't see it, but it's taking up 74% plus minus of all the runtime. So my cores, my CPUs, cores were 100% maxed out, fighting to put stuff in the queue. Why? If you follow over to that yellow box, that says method acquire of blah, blah, blah, semlock. What that means is that that queue has a lock on it. And all these processes are fighting over who gets to use the queue. So the main process is trying to put something in. The other ones are trying to take something out. And there's all this stuff going on. And they're all just sitting around fighting over this lock on the queue. It's often called lock contention. It's a big problem. So awesome. I made my multi-core Python code that can process this stuff really quick. But it can't process it really quick. Because even though I bypassed the interpreter lock, there's still a lock on the queue. If I want to get around this, I need to do some really complicated magic, probably with some C module in Python, so that I can have a queue without a lock like this. And now we're getting to the point where it's starting to be silly to try to do this with Python. Now we're going to change. Sorry. But there was Python, right? So it's OK. So this happens to me. It's not unusual. This happens when I'm helping companies. So for those of you that don't know, most of the time I don't get to write code and work on technical stuff. Most of the time, I help companies with leadership, operations, organizational structure, and all that kind of thing that you would expect somebody from McKinsey or whatever to do. That's what I normally do. I advise growing, mid-size, and large companies on how to do this. But sometimes I get to play with tech stuff. And when I do, I run into things like this a lot. I run into things like this a lot.  I'm actually not a sticker person. I just did that because I wanted to get rid of the stickers. I use Python a lot, but there's a point. And this is an example of a point where it doesn't make any more sense to use Python. You're at the end. Now, you've got to move on. Yes. So Jython, so that's a nice question. Jython and IronPython do not have a global interpreter lock, because they run on the JVM and the .NET, whatever it is, what's it called? .NET thing? Yeah, but it's the something runtime, ACR. I don't remember the acronym. Anyway, so IronPython and Jython do not have the same global interpreter lock problem, because they don't suffer from that. However, I bet you if you try this code, you'll still see they have a lock on the queue. Still see they have a lock on the queue. And there's no easy way to get rid of that. There's no easy way to get rid of that. But I didn't try it, to be fair. To be fair. How about Scython? Scython. So if you created your own queue data structure that was lockless somehow, and you did this in Scython or in C or whatever, and you use this as a module, to your other Python code, you could do that. But in Scython, I don't know if it worked. If you write it in C, and then use it as a C module in Python code, then maybe it'll work. But in Scython, I'm not sure. In Scython, there's a with statement that allows you to bypass the global interpreter lock. But I don't know, you probably still have to write your own data structure from scratch. And at that point, I would argue, you're still, just give up on Python, man. It's not going to like, no one's going to come to you. Travis Oliphant and Guido are not going to come to you and be like, why did you do this to us? You betrayed us. I go to C, but then you go to Python extension. OK. So if you start doing stuff like that, then you have a lot more options. But most people, I'm guessing all the people here, basically, are not going to write C modules for the Python code just so they can have access to a lock-free queue. That's kind of a lot of work, kind of a lot of work. Yeah, what tool did you use? So this is the profiling tool from PyCharm. PyCharm. So if you don't know it, PyCharm is a development environment for Python from JetBrains. I used the Community Edition for a long, long time. And then they changed the licensing. And I got really angry. And then they made a modification. And I got less angry. And I paid them for it to show them how less angry I was. But they make good stuff in general. I like PyCharm a lot. It has a built-in profiler. So if you have code, you can just say Profile. And I mean, you can do this without PyCharm. You can do Profile. You don't need PyCharm to do it. But it's integrated. So that's the tool that I use. So I switched to Go because I like Go. Does anyone know Go? Go Language from Google, a couple of people. OK, I like it a lot. It gives me a lot of things that I can't get from Python. The multi-core support is better. Concurrency, la, la, la. We can talk about it later. The Go people, I'm sure, will be happy to evangelize, right? OK, cool. So what I did was I basically rewrote the code in Go, which wasn't hard. It took like 20 minutes. And you can't really see because it's chopped off on the screen. But actually, it's not really any longer. In fact, if I can do this. Yeah, I don't know why I got cut off. OK, real managed. So I rewrote in Go. And I started with the base case. No concurrency, just one core. So same as the Python version, basically. I read one record from the file. Do the machine learning stuff, la, la, la, go, la. So this was the Python version. This was the Go version. They're not that different. But man, was it faster. Man, was it faster. So just switching from Python to Go dropped the runtime delay eight minutes, which is already not bad. I mean, you can, with eight minutes, like an eight minute cycle time to train a model is pretty OK. I think this is about 12 gigabytes of data I don't remember exactly, plus, minus. So it's not bad. So with 20 minutes of effort in doing this in Go instead of Python, I've shaved out quite a bit of time. So we're on a good path. We're on a good path. So same data set, still my small four-year-old laptop, eight minutes. But actually, we wanted to use all the cores. That was the original idea. Like, there are three other cores on my machine that could be used. So that's pretty easy to do in Go. Because it's kind of built in. You can just, if you want to have one worker, that's fine. If you want to have four workers, that's also fine. And if you want to have four workers, that's what it looks like. You just make a for loop that goes up to four, which is actually kind of hard to, you make a for loop that increments to the number four. And then you can have four workers on your machine at the same time, all using the same data. It's all very nice. So. So the goal is, is it still should be, or is it not should be? It is a compiled language. But even if you ran that code with PyPy, it's still not that fast. Because I'm also a big fan of PyPy. That's my sneak tool that I use when my Python is too slow. I try PyPy. So if you guys don't know PyPy, it's a jit for Python. So as long as you're using standard Python code, it will just in time compile your Python code while it's running. And it can give you really, really good speedups. So if you haven't checked out PyPy, I can highly recommend it. As long as you're not using a lot of libraries, because the library support isn't always 100%. That answer question? Cool. So this is the idea. We want to use all the cores. Just now, it was also on a single core. Sorry? Just now, it was also on a single core. Yeah, it's exactly the same thing. And so which part of the code has got speedup to PyPy jumping, and when you profile that, that should be a hard way to text the code. That's a really awesome question. I was so happy that it was faster, I didn't check. I mean, I'm just being honest. But that's actually really good. I didn't think about it. I really didn't think about it. That's my fault. I should have said, OK, well, why is this faster? But I was like, oh, it's so much faster. Yeah. But OK, so if I had to guess, I would say it probably is string parsing and arithmetic operations. Because when it pulls the records in from the file, it has to split on the comma. And that's actually kind of a nasty thing for code to do. String parsing is always nasty. I think I just complained about that on Twitter a while ago. So I don't actually know why it was faster. But that's an awesome question. I think tonight I will find out. Come to me afterwards so that we can talk. And then if I find out, I can tell you. OK, cool. Because now I'm curious. I can't believe I didn't think of that. Oh, well, OK. Sorry, was that another question? I was just getting you to repeat your question. Oh, so the question, which was awesome, was why was it faster? Which, if you think about it, is a very good question. And the answer that I gave was, I was so happy that it was faster, I didn't check why it was faster. That's my fault. But the follow-up was, I'm going to check why it's faster. Because I also want to know now why it was faster. But yes, it went from an hour and six minutes to about eight minutes. Because my fault. I don't know. I'll find out. OK, so this was the idea. We want to use all the cores that we have, which is very easy to do and go. It's not so easy to do in Python for the reasons we already discussed. You can use the multiprocessing module like we talked about. But if you have certain needs for shared memory or cues or something, and you're not spending a lot of time in your CPU, you may not get benefit in the Python world due to lock contention. So that's where we are. So then, our go version is not really that much faster. So even if we're using all the cores, it's not that much faster. Why? It should be quite a bit faster, right? Going from like one core to four cores. But that's like what, I don't know, 9% difference or something like that? Not even, I don't know. Not a lot. And now the math. So I think I said last week there was going to be math in the talk, and I actually took a lot of it out. But there's still something there. And I'm actually starting to hit the time limit. So I think I've got about three minutes left. So I'm not going to go too deeply into this. But stochastic gradient descent is ubiquitous for scalable machine learning. I just used that word, scalable. But it's kind of true, because it's quite fast. It's easy to work with. And in our particular case, we only are operating on one record at a time, which is a perfect fit for what we're doing with stochastic gradient descent. And the code that mirrors this math is that. It's when we're updating the weights for our model. The nice part about that is if you think about these updates to this big vector or array or whatever of weights, it's pretty unlikely that two cores are going to try to update the same weight at the same time, with some loose assumptions on how sparse the data set is. It's pretty unlikely, because if you have a bajillion features and lots of records, the chances that two processes, especially if you only have four, are going to try to update exactly the same weight at exactly the same time are really, really small. So how can we make this faster and use all of our course? Well, one option would be that we take the locks off of memory and just let the cores update in a round robin fashion. And for this, you can leave the cores on. You can leave the locks on, the memory. But just synchronize or kind of structure the way they try to update it, so that they're not really fighting over it, but they're sort of updating the memory in an organized way. You can do this. It's a bit faster, but not much. Or you could just ditch the locks. If the chances that you're going to update the same memory location at the same time from two different processes are really, really small, then why not just ditch the locks in the first place? Because that's what's slowing you down. Because we basically traded one problem in Python for the same problem in Go. Except instead of the Q, it's locks on the memory. It turns out you can do this, and it's not that big of a deal. And in fact, it can actually be helpful for you. There's a nice paper on it that I'm not going to go into, because I'm almost out of time. But I will try to put the slides online, and you can read it. The basic idea is if you have a problem that's sufficiently sparse, who's comfortable with the word sparse in this context? That's the second question. That's the second question. Sufficient sparse enough that you don't have a risk. OK, so for people who haven't run into this before, a good example of sparsity is something like a recommender system in the e-commerce website. You have tons and tons and tons of products and lots of users, but most users have not purchased most products. So if you put this in a big matrix of users and products with ones in the places where users have bought the product, it's going to be mostly zero. Mostly zero. That's a really good example of a sparse problem. So when you're doing stuff like that with machine learning and you have sufficient sparsity for some nice definition of sufficient, which they talk about in the paper, you actually don't really need the locks, which is cool. So let's take off the locks. Now we go from eight minutes to four, which is better. I mean, we didn't need to go down from eight minutes, right? Because it was still pretty good. But we go from eight to four. And the original Python was an hour and six minutes. The multi-core go with no locks is 355. And one thing that I do is I usually operate in requests per second, or records per second. So when I build systems, I usually don't care about how much data, like how many quintillion zillion bytes or something, or how many terabytes or something. I care about how many requests per second, how many records per second can I push in there? So the Python version is about 11,000. The Go version is about 200,000. Nice. 20 times speed improvement. Plus minus. Plus minus. So on my laptop, this is perfectly doable now. You can give me twice as much data. I'm still happy. It's no problem. It's no problem. And to be fair, there's still locks in here. The Go version, do I have it in here? The code's not in there. So in the Go version, there's something called a channel in Go, which is same as a queue. And a channel in Go actually has a lock on it. So even though the Go version is still using some locks in some places, it's still way, way faster. So if you have a laptop, even if it's a four-year-old laptop that doesn't have too much memory and maybe only four cores like mine, you can still do quite a lot in terms of data analysis. You don't need a cluster. You don't need to worry about Spark. You don't need to try to figure out how you're going to pay for AWS stuff. I see people doing this, too, on Kaggle. Like, I spent $2,000 last month on Amazon. What? Wow. Wow. Why? $200,000 per second on my old MacBook. How much? Wow, OK. So you don't need all that. You can get it by just fine on a small machine, processing as much data as you want. So I'm happy to take any other questions you have. What about the library support of Google? Do you have a specific concern? Like, is there a library that you need that's not there? By the beginning of the process, there was a library that I needed that I needed. What about Google? You mean for exploratory data analysis specifically? So there are tons and tons of libraries for Go. I guess that's the first point. The other point that I would make is, for me, the way I typically work, I do a lot of exploratory analysis in Python. And when I'm actually ready to build something, then I do it in Go. So Python, for me, is an exploratory tool most of the time. Sometimes I actually use it for things that are in production or bigger projects. But most of the time, for me, it's a tool that I just use for exploratory analysis. Like you said, for the library support. It's got pandas, got numpies, all these things. There are some things like that for Go, but it's not a tool that's been focused on data analysis. Like Python has been recently. I mean, it's kind of the same as if you, so numpy has been around for a while. The panda is not that old. I don't know. Does anyone know when pandas was released? It can't be more than like six, seven years, right? I think it's about six years, plus, minus. So until six years ago, there was no pandas for Python. And I haven't found it to be a hindrance. But if you're doing exploratory analysis, you're still understanding your data and what you want to do with it and what kind of models might work, then it's really hard to be pandas and scikit-learn. It's really hard to beat it. There was another question. Why do values of normals are different? Because the hashing functions are different. So when I talked about the feature extraction, sharp. I was wondering if anyone was going to catch that. In the Python version, do I have the hash in there? Yeah. So code line 123, index equals end value, la, la, la, weakest hash ever. That's not my comment, by the way. But I think it's a good one. So that hashing function is different from the go one. Did I? Where is it? Yeah. In the go one, I wrote my own hashing function. So since the hashing functions are different, the weight increments might be slightly different. So that's one reason why the log loss might be different. However, I will call attention to the fact that it's better. So it's basically the same, right? It's 4546. It's not that much different. Is there anything like how many places you can rank? Ranking-wise? In Kaggle. This will, I mean, in Kaggle, 0.01 will take you from 10,000th place to first place. Roughly. Roughly. People are laughing to get it, the Kaggle joke. So the thing with Kaggle is I never play to win with Kaggle ever, because you will lose large chunks of your life trying to do that.  You may develop a larger social community online that way. You might learn more doing the Kaggle one. But anyway, the point is Kaggle, as I mentioned earlier, is really contrived. You're getting a data set that's already been prepared. It's already nice. The loss function is given to you. You know how you're going to be evaluated. Everything is very straightforward. And then there's a leaderboard. So you can rank against other people. But what happens is, once you get above, so I usually go to the top, I don't know, once I get to about the 80th percentile, 70th percentile, something like that, after that, no point. Because in the real world, you're not going to do that. In the real world, if you're actually solving an actual data problem in a company, or a university, or wherever you are, they're not going to let you spend four months getting a 0.001% improvement in your model. Nobody cares. Like, what they're going to ask you is, is it better than what we have? OK, deploy it. And that's going to be it. Maybe you do another version of something like that, but you're not going to spend a long, long time. So what would be the ranking change on Kaggle? A lot. So what would be the ranking change in the real world? Nobody would care. Usually, unless you're like UPS, they have some place where they would care for routing problems. So can we say that you have a few ensemble mentors in our work? I'm sorry? You have a few ensemble mentors in our work? Have I built what? I'm sorry. Do you ensemble your models in our work? Do I ensemble them? Because you mentioned that nobody really cares if you spend 0.001% on your work. And ensemble, it takes a huge amount of time. So are you asking, do I personally do it, or have I seen it done? No. But I have seen it done. There was a company that was using a cluster, Spark cluster, of not a huge one or anything. It was like six machines or something. And they were spending about 25 hours of compute time with Spark to process all these models and do cross-validation and all these kind of stuff and then ensemble them together at the end. And it was quite cool what they did. But it was replaced with XGBoost on a small machine that ran in 25 minutes. So that's pretty much what I do in a lot of companies when I help them with technical stuff. I like to make the joke that I've shut down more Hadoop clusters than I've ever set up. And it's true. It's true. And that's an example of it. Sometimes people start with a really big approach to this problem, but you don't need it. I mean, just like we saw tonight, you can do a lot with a laptop. This is not a powerful machine. It's just a regular four-year-old laptop. And I typically don't ensemble models in the real world, for me, for the problems I've seen. Does anyone else? I mean, is there anyone here that's seen significant improvements from ensembling models? Does anyone know? For Kaggle, yeah. Yeah, but that's like pretend world. That's like pretend world. Yeah, for Kaggle, yeah, because you're fighting over that tiny, tiny improvement. But for real world, I've never seen it. I've never seen it. Maybe some people do. What about data size with hundreds of features? Sorry. Do you make a difference? Sorry? Data size with hundreds of features. Hundreds of features? Does it make a difference? Does what make the difference? I guess I have to try with data size with hundreds of features. Like this? This method? Yeah. Oh yeah, that hashing trick is perfectly fine for that. And the other thing I like about it is when you're using that hashing trick, you're pre-allocating how much memory you're willing to use for this problem. So it's quite sneaky. Maybe it didn't show up in there. You want to look at the Go version or the Python version? Probably the Python version. OK, so on that third line, there's like value plus key, la, la, la, la, ma, d. And what I'm actually doing is I'm saying I want to use two to the d slots of memory for this. So I can say in advance, I want to use 1 gigabyte, 5 gigabyte, 8 gigabyte, whatever gigabyte memory. And then when the features come in, it's going to hash them, mod whatever. So if they don't fit in there, it's going to put them somewhere else. So you may have collisions with your features, depending on how much space you're willing to use. Like hash collisions or memory location collisions. But that's OK. It usually doesn't cause a huge problem. Now, if you say I'm only going to use two bytes of memory or something, then yeah. But in general, no, generally it's no problem. You can put many thousands of features. That may be more of a model question. Which models are able to handle many, many, many features and automatically deal with ones that are less important, more important, so like regularization or these kind of things. But in terms of the hashing capability, no. No problem. I mean the overall acting. Yeah, it won't matter for this one. It will run at the same speed. Yeah, because if you have many, many, many features, it's still going to hash them into the same memory OK, so it won't matter. There was, you had the question. You know, I was a comment on software. So I do a lot of software at work. Oh, OK. You have to work well if a lot of features you have start coming from images and start coming from graphics, and later from images. So if you just train the base of all the features, train one out of one of your first thousand, fix one out of the fixed thousand, notify 100 of them on software all different times, because you didn't know that the different features come from different data sources, they are the components. OK. So it's one example that I really have. You have a deep view of that and produce a lot of features. It doesn't really make sense if you just don't text the specific data. So you're basically partitioning your training set. And then you're training different models on each partition, and then ensembling them later. OK, OK. And you do that at work. OK. Anyone else do that at work? Oh, oh, be proud, man. Come on. OK, so a couple of people doing ensembling stuff at work. But you have to know what you're doing. It doesn't mean that you have to control like you don't ensemble. Yeah, it's not the Kaggle approach where you do an ensemble of 1,000 different models, each one with a tiny different version of some parameter. But it's actually, I mean, it's ensembled, but you're really kind of doing some kind of model selection by hand for each of those partitions. Sorry? It's more like stacking. Stacking? You're building a bunch of models on different things, and then building a model over it that uses those as features. It's not so much ensembling as creating it. But you're not using those features, right? You're actually taking the outputs, and you're just Yeah, the pseudo features. Sorry? So it's stacking because the pseudo features. Yeah, you're going to use all those specific model features. So for example, there's a really high level idea sense on certain data, it goes to the driving So then, sorry guys, we're getting a little specific. You too? Let's talk afterwards, OK? Cool. Anybody else? We thought we would be the loss of memory, assuming that under a very rare scenario, that the schools try to access the exact same memory space with the process die, or is it just get corrupted data out there? So the answer is it depends. Because I mean, it's like a more deep technical question. But if you have a CPU that supports certain atomic operations on your memory, common ones are like increments, like an atomic add, or add two numbers atomically, something like this, add something to a memory location atomically, then in some sense, you still have a lock, but it's enforced by the CPU. So one CPU operation actually does that add for you. So it's impossible for two processes to do that at the same time. So you're basically moving this lock out of your code, and maybe down to the architecture level of your machine. But I was actually arguing, debating, whatever you want to call it, with somebody a couple days ago about whether or not that's an actual lock. Because it's like one CPU instruction, right? So is it still a lock, maybe? I don't know. But in general, it's not an issue. As long as your CPU supports those atomic instructions. If it doesn't, then if I built a system that doesn't use locks like this on an architecture which doesn't have the atomic operations, shame on me, because I made a bad choice as a developer, as an engineer, as a tech person, whatever you are. You have to be aware of the architecture issues. But in general, it's not a problem. So wondering whether you've tried a higher order system optimization method instead of just casting gradient descent. Because in theory, things like this method would be a lot faster in terms of the project. So that actually is actually for single computer. Would that actually be the same method? So I'm jogging my memory now. So Newton's method converges faster. Yes. I think it's a quadratic method versus linear method. It does converge in fewer steps. But the processing requirement for each step is quite heavy. Not sure. Yeah. You're right that Newton's method will converge faster in terms of steps, like how many steps? So the linear method, or if you use SGD, it will converge more slowly. But the computational requirements to execute one step in Newton's method can be quite heavy if I remember correctly. So we can check afterwards. But I haven't looked into that in a long time. So honestly, I can't remember the details. But I seem to remember that being the reason. So if you have less data, then yes, Newton's method can work quite well. But if you have a lot and not a lot of data, it's maybe not the best option. Which in this framework, where we don't know how much data we have, could be 10 gigs. It could be a 1 terabyte. We don't know. So if you have like 1 megabyte of data, maybe Newton's method is ideal. But we're not making the assumption that we know in advance how much data we have. So that's the use case you have seen in that loop, or just like plus-go in that sense. Who laughs? You think I'm going to say something funny? Never makes sense, right? That's not true. So it does make sense where you have a really large amount of log data or something like that that you need to preprocess so that you can then do machine learning with it. So if you have, I don't know, petabytes or something, I don't have petabytes of data. If you have a lot of data, it can make sense. Yeah. Oh, so excluding ETL stuff, when does it make sense? I've never seen it make sense. There may be cases where it does. I've never worked at Facebook. I've never worked at Google. I've never worked at Twitter or Microsoft. So maybe those companies have cases where it makes sense for them to do their actual machine learning stuff on Hadoop or with Spark or something like that. But I'd be kind of surprised because I have acquaintances that work at Amazon and they say the same thing. It's like, there's no way you need more than an EC2 instance to do machine learning. Just no. And they were doing image processing, image recognition stuff, image-related machine learning for Amazon. So I've never worked there. So I can't say with 100% certainty. But my guess is that with extremely rare exceptions, there's just no need for the machine learning case. For the ETL case, I think it can make sense. And the machine learning also feature extraction is the next one. In this case, we do streaming feature extraction, actually. So our feature extraction, we don't actually have to worry about because we do the feature extraction when we see the record. So we're basically doing the hashing once we get the record. So it's on a per record basis. So in that case, it's not an issue. But I'm not saying there isn't the case. I'm just saying I haven't seen one. Yep. We will be nice. But what I think is that if I have a lot of data, I'll just select the right constant. That just looks like it takes really long to do it. So if I'm doing some kind of exploratory stuff, I would just have a sample of the data. I won't try to do it. Because I'm just trying to understand it, right? What kind of features are in there? What does it look like? Probably look at some histograms because I love histograms. Those kind of things. I'm not really worried too much about doing a full scale model training at that point. If I needed to do that, then I probably would use a different tool. But for the exploratory stuff, I use Python, pandas, and scikit-learn pretty much. Sometimes I'll use PyPy. There have been times where I wrote some scython stuff for some problems because there were some computational things that were a bit heavy. And writing five lines of scython fixed that problem. So that's a good example where extending and using Python more made sense. But in general, just the same stuff, the normal PyData stuff. Python, pandas, scikit-learn. Other questions? Yeah? You said that the bottleneck is usually the computation. I'm sorry? The bottleneck is usually the computation of Python. You're not a Python. The bottleneck of what? In this case? I mean, it definitely wasn't IO for me because I'm pulling from SSD. So I'm still fine on IO. If you have an IO bottleneck, this is actually a good question, if you have an IO bottleneck and it's not computational, then the global interpreter lock is not an issue. Because global interpreter lock is not in effect for IO and for certain computations that are done out in the sea land. So for some NumPy stuff, you don't have to worry about pulling the interpreter lock. If you're doing a web-based API or something, then you may have more of an IO bottleneck. But it's kind of a different problem, right? What we're describing is how much data can you reasonably process on a small machine? And for that, you pretty much have to discount the possibility that that small machine is well, discount's not the right word. But on my laptop, I'm not worried about running models, running an HTTP endpoint for my model. I'm just pulling the data from disk. But in this case, no, there was no IO bottleneck. It was all CPU. Other questions? So I'm curious, what's the limit about creation? Oh, this will work on anything. You can do this on a Raspberry Pi. It's not specific to the machine. What is specific is you have to decide. I won't go back to it. But you have to decide in advance how much memory you want to allocate for the weights. And that's machine dependent. But other than that, this approach, in general, will work on any machine. Are you mean in terms of CPU and stuff like that? Yeah, it's not specific. Let's say what you can run on a Raspberry Pi, you want to? Sorry? Something like a lot of work is done on a Raspberry Pi.  You won't have as much RAM. But that's OK. You just won't have as much space for the weights. Any other questions? Yeah. So what kind of machine learning algorithms does this approach conflict itself to? And you're out of supply, but I think you're going to find yourself in a kind of supply. Right, so in, you mean for the feature extraction or for the piecewise fitting? Piecewise. Does it not, when it's up to? What kind of problems or which algorithms? Which algorithms? That's a really excellent question. I can't think of any off the top of my head. I mean, there are some. I'm not saying I can't think of them because they don't exist. I'm saying I can't think of them because I don't know. Does anyone else know? What's the question? Oh, so the question was, are there any algorithms where you can't do this? You can't partially fit them. I mean, if you want to do, like, regular gradient descent, then you can't, right? Because you have to calculate the whole gradient for the whole data set it was. So that wouldn't work. Sorry? I don't, I don't, I really, I genuinely don't know. But you can kind of get that from some regularization, right? And you can add regularization terms to estrogen and logistic compression. And then you get the same result. Maybe we talk after. It's not parametric, that's why. OK, maybe we talk after. So the answer is almost certainly, but I don't know. OK, other questions? Cool. Thank you very much. This is my information. You can find me around here. Thank you.