Hey everyone and welcome back to the Broogly Channel! I'm your host, Broogly, and in this video I think I found a level that rivals the pool rooms for a chill liminal vibe. So if you like chill liminal vibes in the back rooms then you've clicked on the right video because today I'm going to be talking about back rooms level 6999 or the neon pool rooms. This level is awesome and it's pretty chill and it's actually kind of a sublevel from the pool rooms level itself. So yeah, let's get into it. Level 6999 is classified as a class habitable and is literally 100% safe, like there's nothing here that can hurt you, I mean unless you can't swim. This level has a kinda similar mythos and story as the promise land does where it's a level that everyone seeks out and tries to get to but no one knows how to get here or where the entrance is or anything like that. So it's safe to say that the level is pretty much a hot commodity. Think of it like an oasis in a desert. Everyone wants to be here but it's virtually impossible to find and it's mainly talked about through whispers and stories throughout the back rooms. The methods used to actually get here are very strange and so far it's only thought that two people ever have made it here but even then no one knows if it's even real or if it's just a figment of people's imaginations. The level first started to be rumored to exist back in 2019 and that is when apparently the first person got here. The physical description of this area looks like a never ending complex with roofs and walls made out of rooms with pools in the middle. On the outside of these pools there are chairs and tables and lanterns and lights in the corners of the floor and in the ceiling and these lanterns and lights are responsible for most of the lighting here. The level almost feels futuristic in a way like it's almost too clean and precise to be from the modern time but it's the back rooms so no one knows what time it takes place in. Most of the time this level is colored in purple but apparently sometimes it can be red, pink, yellow, blue and even blacked out so it just depends on what time you go there I guess. As you can expect it is foretold that this level has abundant tools and supplies laying around like almond water and such on top of there being chill pools that you can relax in here. It almost seems too good to be true in a way. The pools are said to go very deep, deeper than the pool rooms pools, just said pools like 15 times and they aren't actually made out of water but instead it's a water like substance. Getting into these pools is like an instant sanity cleanse because it calms you down and helps you relax and chill out and forget about your anxieties kind of like the fountain of youth that rejuvenates you in a way. The pools are also emitting a relaxing water flowing noise that can help you just chill out fully and fall asleep even. Apparently there's also an entity or two that just float around in the pools but they're assumed to be passive and they won't attack you and they're actually facelings and they kind of just float around. To enter as I said it's not known all the tasks and all the stuff you have to do to get here or what you have to do in order to do anything because this level might not exist no one knows. To exit you have to ask one of the facelings that you saw the pools to send you away and they will but yeah the reason I actually like this level is because of how mysterious it is you know I feel like the whispers floating around the back rooms of this perfect level where you can finally relax and finally chill I feel like that's pretty cool and it's kind of like a heaven area for the back rooms you know something for people to look forward to and have a purpose. It would give pretty much any wanderer something to look forward to finding and it would give them something to hope for and maybe even give them purpose or all of it could be fake and there could be no such thing as the neon pool rooms but it would be cool if there was. Maybe one day we'll know for sure but as I mentioned and hinted at earlier in the video it's actually thought that you can get to this neon pool rooms level from the main pool rooms level but no one knows the exact way to do that but those two people I said that have been here supposedly came from the poor rooms and got there so if you're lucky enough you might jump into those crazy pool rooms and be sent here which would probably be more relaxing because it doesn't get dark or scary anything like that it's just a bunch of infinite liminal pools that have purple lighting I mean literally how much better can a level get how much more relaxing can it get. Alrighty that is it for this back rooms explain video I hope you enjoyed and thank you for watching until the end I genuinely appreciate you thank you for all the support on the channel it's about to hit 400k that's literally utterly insane to me and I just cannot believe it thank you all so much if you want to help me get to 500k be with me on the slides 100k push to 500k just drop a sub it's free and when I do get to 500k I am buying the full back rooms outfits the pants the shirt the hat the socks everything and I'll be making a bunch of videos in it and it's just going to be awesome and it's going to be a cool thing to do 500k is going to be amazing and the push itself to 500k is going to happen over this fall which is also the best time of year so from you know fall to winter we're going to be aiming for 500k that's awesome it's my favorite time of the year about to hit a huge milestone I'm so hype hope you like for it too make sure you check down in the description below for any links and stuff you might want and also make sure you check out Spoogey and Tugley because on Spoogey I just uploaded a really cool Trevor Henderson Creatures Explained video and on Tugley I'm always uploading bangers over there real life stuff tier list horror games gaming videos pretty much everything that's cool and fun I'm playing so if you like cool and fun and me you're going to like Tugley and Spoogey just hit 15k so I'm really appreciative to you of that thank you for helping me get there I appreciate it now that Brugley summer spectacular is over let me know what levels you want me to go I have a ton of them planned out of course but I do love here in your opinions and if any of you suggest a cool level I might stick it in my schedule so let me know in the comments thank you for watching and supporting me I will see you later I hope you're hyper fall I hope you're hyper Brugmas it's going to be awesome I genuinely cannot wait seriously I love fall and winter my favorite time of the year my favorite time to upload and hope you guys are ready for all the content coming on all my channels peace


And so, you have women who will meet a guy and maybe this guy has had a couple of good things going for him, his confidence is up and she's like, I want him. And a guy who isn't used to controlling his energy, this is his death in a certain sense because she will take the energy out of him. We talk about the term succubus. Succubus is an example of, it's an extreme example of feminine energy but all women to a certain extent are succubi. The flip side is that men want that and so women will seek out guys who have energy. If you ever go for guys, the whole thing about hypergamy, you can distill it down to women will never choose a lower energy guy than a higher energy guy. But the problem is that if a guy doesn't know how to control his energy, the woman can take all of it out of him because the guy surrenders into that death. He does not want to continue to produce anything anymore. And then what happens? You have a couple of months of the honeymoon phase, having sex all the time and then eventually she's just starts to feel I'm not really that interested anymore.


fastball was good tonight location with it in and out and Change up keeping them off speed a curveball could have been a little better But it was pretty good when I needed it and just really feel our strikes. Good. Did you feel like the change was it was good? leaned on it a lot They're good hitters, so I'm just trying to keep them off balance best I can you mentioned the strike throwing a lot of first pitch Strikes for you attacking the zone early. How much did that play into your success? Yeah, pitching ahead is make it makes life a lot easier, so I'm trying to Get them out as fast as possible get them on the ground and let the defense play You've asked you about it so many times and you said the runs will come the runs will come But is there any part of you that's wondering where are the runs for me? No, maybe we just need to stop talking about it and it'll happen I trust the guys and I'm just happy to be out there and keeping us in the game Happy to go that deep right? Yeah. Yeah, I mean my pitch limit's been Good, I mean I've been keeping my pitch count pretty low So just trying to build some confidence in late in the game He felt the wind was affecting some of his pitches. Did you feel any effect from the wind the wind? I Don't really throw anything straight, so I wouldn't know I Think he's been so good this year, so I guess it is sooner or later. He's gonna give up some runs with He gets it. He's very forgiving when he has one bad day, right? King's awesome. He's been Incredible this whole year and I'm sure he'll keep being incredible.


Matthew, it's been a long year here, but how does tonight rate for you? It was an awesome win tonight. Great way to finish off the season, especially for the supporters. I think we really deserved that win tonight to take the next step in this tournament. It's been a fantastic year, very successful year and we capped it off in a beautiful way tonight. Obviously, congratulations on your selection to the Soccerers' preliminary squad for the World Cup. Obviously, comparing yourself from the start of the year to now, what's changed to help you come along in such a short time? For me personally, it's been great to be involved in this club, playing in front of these fans. To contribute to this year's success from the back of last year, it's been very enjoyable. Obviously now it's huge honour to be involved with the national team. I'm looking forward to the journey ahead. Hopefully we can go over there to Brazil and make Australia proud. Obviously, we're saying goodbye to a few players who have played in this club since its beginning. As someone who's come into the squad this year, did you feel an obligation to do them proud and to give them a proper send-off? That just shows what this team's made of. Credit to all the boys, even the ones that are leaving. The way they finished it off in style tonight, it's credit to everyone. Obviously, those boys will be very much missed. It's a new chapter now. I think we're confident that we can take off where we left it and it's going to be a great journey ahead. Thank you.


What's up guys, it's Harry. Just a few things to note before this podcast starts So unfortunately, it is a bit laggy this week due to the fact that I was on shitty Wi-Fi At a cottage. We basically rented it like my family and I and it was kind of like a little family vacation We recorded it everything seemed fine and then kind of after that I went back I reviewed it and it definitely does lag in some spots We decided to post it anyway because like we thought the info was really good You're going to definitely learn a lot about the economy inflation We talked about partying in Miami a bunch of stuff like that like how that type of vibe has changed And just a bunch of stuff like that how things have changed from Kobe till now So it's definitely a good one. Just mind the leg Please we'll make sure that does not happen again But we just are going to post this anyway because we feel that the information is is super super good So apologies from us. I'm sorry about the lag and I hope you enjoy this And thanks a lot What's going on guys? We're back with another episode of the after-hours podcast Today we got Alex who has been a He's been on a few times So lately we've kind of taken the podcast in a different direction, you know We've talked a lot about like the overall market And just trying to touch on all different subjects as far as trading and stocks on that goes So we wanted to bring Alex on and kind of get his insight and everything. So yeah, man Thank you for coming on. Yeah, man So I want to try to come on more often too I hope to come on at least once a month because I think there's new direction of the podcast is gonna be good because a Lot of people want to hear about the overall markets They want to hear about maybe alternative ways to invest money and what you could do and you know Future episodes I'll explain that what I do with my excess cash and you know, how I try to grow my money because there's no Linear path to do it. It's just based on, you know, your personality what you want to do how to get whatever So I think first it's August 5th 2022 and we should talk about what's been going on the market.  It's being caused by supply supply chain issues because of the Russia Ukraine war by China partnering up with Russia not giving us materials Whatever it is. So that's number one. Now number two is with this elevated inflation There's not much demand anymore because people don't have money People don't have money because all they spent all their money on stimulus checks Whatever money they had they already bought a Tesla. They already bought a Peloton.  So the market now as of today is started to rebound a little bit It's probably rebounded because people think that peak inflation is over and in a way that you could judge how inflation is over is the three well the main things are energy prices commodity prices and housing prices So these are the top three things that you know judge The inflation so energy went sky-high because the Russia Ukraine war is starting to come down Commodities with sky-high because the war in China started to come back down and now with hiked up interest rates Real estate is the next thing to come down So the market is starting to rebound in anticipation of us hitting peak inflation now the thing to note is the market is What you do in the stock market is you pay today's value for future earnings or future whatever future whatever So even if inflation peaks in six months the market is already pricing that in right now So although we might have another month or two of inflation that's lagging The market is telling us now based on commodity prices based on energy prices based on housing prices that inflation has peaked And you know, that's kind of where we're at right now. Who knows what's gonna happen. Yeah A lot of people are saying that there's gonna be a recession But I mean, what do you guys see on your end in terms of all this stuff? I think I think the scary thing to me is like what you said like the market's kind of telling us the future direction of where we're headed But outside of the market, I'm seeing more of like recession activity being like my friends have no money to Spend when it comes to like buying a house people have no money anymore. They just have afford an average mortgage They can't do what happened is this interest for you with the let's say you're making X amount of money You're able to buy a $1 million house now with the interest rates going up You're able to buy a $500,000 house. Well, so your buying power got And and and I'm noticing what people are doing again.      Yeah, we've been in one and It might be that You know people yelling recession recession it might be that Inadvertently, yeah fear of a recession right the fear of a recession Is enough to panic people enough to say alright recessions coming recessions here everyone's selling everyone's worried everyone's whatever Yeah, and we're almost by causing a mild recession like you're saying. Yeah It fixes it as if people are expecting a large one does that make sense? It's like yeah, that's how I feel to My yes, it's a little bit weird, bro And to be honest, I'm gonna tell you from like a product perspective So I know people that move products where there'd be like vacuums speakers Anything any type of product a microphone whatever a type of product So what ended up happening is there was so much demand built up so much demand built up because so people so many people have so much money Now what happened is because of the supply changes I don't know if you guys know but China was not letting you ship containers and the prices of containers quadruple So you are not able to get your products from China.   So that's what's been happening guys and that's that's just That's literally just in like the product industry.   Yeah, exactly I still think the scary thing is though We even though we see it coming and I agree Maybe it's gonna be much more mild than everybody anticipates, but it doesn't change the fact that people just don't have money there's a lot of people that are just the wealth gap is getting bigger and bigger and Unfortunately, it's also stupidity bro. It's also stupidity. I mean like look bro. They don't get me wrong like Like bro, I got I got a good amount of money.  Okay, no problem I buy I bought a samurai sword.   They're putting on their goddamn credit card Yeah, it's an illusion man. It's all illusion.   Okay Recession is fixed by improving spending by improving demand people buy when people buy shit recession is fixed Inflation is caused by the print Supply chain issues whatever They got to pick which problem to solve how do you fix each problem? Okay number one you want to fix the inflation problem What you gotta do is you got to create a recession You want to fix the recession problem the way to fix it is to print money give people stimulus and cause more inflation So these the Fed is stuck in the middle man There's no directly in the middle What do I do do I fix a recession or do I fix inflation?  Yeah, because what's gonna end up happening is this One year one year inflation is gonna be fixed. It's gonna be gone. It's gonna be back to normal. Okay, sure So inflation issues is gonna be fixed. There's still a recession spending issue in all of history in All The way Is to have money like we printed it vietnam war. We did not have money. We printed it 2008 housing We did not have money. We printed it. We didn't have money.   Really.  So the system is rigged to be honest the whole system is rigged Money is fake.  It's like housing is there one Eggs milk whoever if the federal reserve really weight Of inflation for example, if milk was 10 percent of the weight And milk was the highest inflation They would say, you know what? Let's bring milk from 10 percent of the weight to 2 percent To raise the weight of eggs from 2 percent to 10 percent It's not in the inflation number changes bro. Yeah, and they could do that. Shit.  So big money Is saying all right interest rates are up.  So i'm gonna give you an example Let's say a million dollar house. Okay million dollar house in two months is 800 thousand 800 thousand. Let's say the interest rate is five percent i'm gonna buy that house at 800 thousand Next year when the federal reserve starts dropping interest rates to stimulate the economy I'm gonna go and refinance that house at two and a half percent. Okay. Okay That actually works.  Yeah, and bro If you buy that 400 thousand dollar house, maybe in two three years, it's gonna be worth 600 You know, yeah, so there's plenty of opportunities to make money, but you just gotta be able to have cash sitting Waiting to be able to deploy and bro.  I'm up three percent It's not a lot of money, but the naziq is down 33 percent. Yeah No, I mean it's the same. I was the same way. I I was very lucky I'll call it and I I talked about in the last episode I woke up one morning and I saw these prices of like apple amazon like you're saying And I felt like a moron not buying. I'm thinking these companies are great. They're gonna last forever and that's it I was the same thing I said, this is it so I don't know I want to think everything is bottomed and I personally do And I think from here, we're just going to continue to rally and I don't know what 2023 is going to bring But for now, I think I think we're going to keep pushing here's the thing bro There's only been so many times in the stock market history where shit has been out 30 percent Facebook meta is down 50 percent bro 50 percent Yeah, but anywhere, bro. I'm walking all around people pictures are still using instagram, bro They sure they might have a bad year.  Yeah amazon might be over stock. What's god damn amazon, bro Their cloud business is gonna be worth a trillion and ten years like these are good companies.    I just can't be very brave for Oh my god Someone clip that I'll clip that But yeah, that's kind of that's kind of where we're at right now in the overall market guys the overall market is starting to slowly rebound Um, you gotta remember that we've been in a downward cycle for eight months So even if we go up for one month, it doesn't really mean anything It just depends at same thing as support. If the stocks hold support, they're gonna go higher If the stocks don't hold support, they're gonna go lower So it depends every single dip probably mark is gonna crash today We'll figure because we're talking about it. But every single dip has been holding and You gotta account for that and if I remember correctly, I remember seeing a statistic out there that There is the most amount of people Shorting the market in history in these last six months Yeah, I what do you guys think about, uh, if we kind of bounce from here and we go to like 420 Uh, like 430 and then what do you guys think about like, uh, we fade until kind of the winter and then we like, uh We like, you know, set a good bottom in like october november kind of december and then we rally up No, the thing is bro, for stocks to go higher, they need to go down and digest For stocks to go higher, they need to go down and trap shorts.  There's gonna be bear markets. There's gonna be recessions.  Let's come back. Let's do the same goddamn podcast Let's compare prices Yep 100 and I also think that like the thing is is that I feel like a lot of people who are Like before like we had a lot of people who were bearish and everyone was saying This shit's going to zero. This shit's going to zero. This shit's going to zero Which is fine. It's okay.  This is gonna rally. This is gonna rally.  Yeah I got 100 grand. I'm gonna make it easy. I got 100 grand in facebook Meta Got 100 grand leverage it up to 400 grand. Yeah, okay If facebook goes down 25 percent 400 grand 25 percent You get liquidated God A lot of these stocks dude are down 50% So even if you're using two to one leverage two to one liquidated So I think what ended up happening was a lot of traders A lot of investors They got liquidated No more market participants for you, bro So what's gonna end up happening is a lot of these sellers, bro.  You better believe it, bro. You better believe it They got all liquidated at 600 700 now we're at 900 And the same thing with coin Coin absolutely and kathy wood or whatever like she sold at 60 now. It's back at 100 Yeah, bro Exactly. Yeah, so that's the type of market we're in guys. I mean look I think If the market is going down based on interest rates and inflation I think inflation peaked because of oil commodity and housing prices going down and I think that um Over the next year the federal reserve is going to lower interest rates And if they got some fucking goddamn balls, they're going to print some money again and start the cycle all over again They're because it's the same cycle has been happening, bro in 2008 when they printed money.    We're here to save you.  We love war.   Don't listen to the white house.    You got a great topic about the miami club scene So you want to talk about that? Yeah. Yeah.   They're still hanging out, but they're not balling, dude. Yeah Yeah, they're not bringing out the fucking, uh, champagne or anything No more champagne, bro.   I went to the bathroom the bathroom attendant was a yo, what should I do with amc the bottle girls amc Bro like at that point Every single motherfucker owned amc stock bro. It was crazy Yeah, no my girlfriend's sister bought a couple shares too Bro, my boy dan.  I mean, that's not crypto rich people, bro. These are rich people. Yeah crypto Dude, do you know how many people I know took up like refinancing their house? They took out money invested all of it in bitcoin at like 60 k How are they doing? Yeah, that's that's what's been happening, bro. That's what's been happening. So like This is the world.  They're just laying off the fat I'm sure a lot of these companies had a lot of workers that were getting paid to do nothing Yeah, so it's smart Layoffs are smart. It's just not good. It's smart for the business.   It don't matter, dude. Yeah, it don't matter It's true. Yeah. Yeah, that was solid Alex you kind of killed it on the end there That was good So what else bro, let's talk about other stuff. How's trading been? It's been really good for me to be honest. Like I mean a lot of people have been like kind of I know it hasn't been great for like a lot of other people but like last like three weeks have been super solid for me Um, just because the thing is is that my process is a little bit different. So like Like it's like the inverse of Alex where like We always have one like hot stock to focus on that I can like either scalp around or just like do something like that Whereas like I feel like a lot of people Like it's easy to get squeezed in this market I feel like And then like by the time you get blown out like a lot of people don't want to like short the back size Or like they're already squeezed out or like whatever and I just don't want to What Harry does is very smart. So like how do I how do I explain it without giving too much waste what Harry does is like We short at resistance so we wait for a stock that's already broken To bounce and short of resistance What Harry does is he waits for it to go to support And he's just like all right. I'm gonna buy his support worst-case scenario. It's already at the lows I'll lose 10 cents. Yeah, that's case scenario. It's gonna ramp up bro. So like Logan you're wrong.  It's very simple. Well, like people just don't do it, dude No, it's it's like I find like for me like For example, like yesterday, what was it nviv or something like that?  It's down from 460 to 380 I'm sure if it goes down to 380 harry's gonna buy it.    Like 370 470 something like that.    It's very low risk, bro. There's low risk.  So by the time we get to three bucks It's like it's not really like a good risk reward opportunity for me So I'll just leave that alone and then I was like, okay.  So I'll just come back tomorrow And then yesterday was like my process to a tee where it was like, okay Sopa had tanked money was going into another one It was a great situation to fucking, you know, just take some and we ended up just It was like two halts or something like that two halts up.  Yeah, maybe we should cut this. All right guys I think that's probably gonna be it Yeah, we're recording this at like 8 a.m. So it's been like 8 30 right now. So So guys what you should do is leave a comment in the video and let us know what you guys want to talk about on next week's uh Or not next week next next podcast Yeah, uh, so that we have a list of topics and we could just kind of keep going through stuff that you guys want here And if you thought that You know, this conversation was helpful and you want to hear more about this stuff Just let us know and we'll kind of turn the podcast into whatever you guys want to see so good stuff guys For sure. All right. Thanks everyone


Done. Okay, is that better? Yes. Nice. That's good. Nailed it. Thank you for hanging in there. Okay, so I want to ask what isn't machine learning. So as math underpins everything we do, they're kind of, I think you get to a point where everything is machine learning. So, you know, if you think about when you're a child and you're trying to learn to walk, it doesn't look that unfamiliar to, you know, when someone has reinforcement learning algorithms that are teaching something to walk. So from an offensive standpoint, I think it's an important distinction because, you know, where offensive folks try to live in the realm of possibility rather than in a sort of a box that we try to define. So, you know, maybe if you start thinking of everything as machine learning, you're going to find more opportunities to attack machine learning models, use them in your ops, whatever that might be. But for this attack, in particular, you know, we're going to, we're going to do a copycat model. And I think this, this kind of concept will come out as we go through it. So I'm kind of tired of the main copycat models. So I'm just going to call in pink panther attacks from now on. One, it's more fun to it's still kind of catlike. And three, I know pink panther involves a diamond that is always trying to be stolen and preconditions for a successful attack. And these are pretty loose are a representative data set. So for example, in the case of an AMSI provider, we're looking for like PowerShell, VBA code, a big representative data set that we want to model. And we need the ability to get feedback from our target model. And this doesn't have to be direct output.  So even if the model doesn't give you back output, there are still, you know, just do the telemetry nature of our networks. It's more likely than not that somewhere outputs being recorded. So rather than, you know, maybe going directly at the model, try and think of some binary test that you can, you can perform. That is potentially outside of a realm of a machine learning model. So it's pretty simple attack once you kind of get into it.  So we is like 400,000 scripts and the workshop, the lab piece of it, we'll talk about that. Effectively, you know, the scripts get fed into an AMSI integration. And this just came off of the Windows sample.  And we feed it every script in that corpus. And that gets passed to Defender or whatever AMSI provider that might be. And it comes back with a score. And then we turn on, we just collect those scores offline. And once we have, you know, we have known inputs and we know what the output the founder thinks of the output, we can, you know, model our model. And the hardest part about the hardest part about, I think, adversarial machine learning isn't the machine learning piece. You know, most of us don't have to invent math. People, you know, much smarter than myself have already done that for me. But what is difficult is the engineering piece that's getting access to the right data. It's getting it's getting data in a way that isn't going to get you caught or you're going to distribute your traffic accordingly. And you can kind of fly under the radar in the future. You know, currently attackers kind of have free reign and, you know, get away with quite a lot of querying and noise in the network. That's not always going to be the case. And I think it, you know, is helpful for us to start thinking about limiting our queries, thinking about information density of a command, you know, a PEN versus a process list, for example. You know, PEN, you're going to get, well, if a host is up or not, or you're going to get TTL, which is going to tell the OS. But a process list, you're going to get, you know, that and more. But it probably just spends on what you're after.  So if defenders are using machine learning to model, you know, whatever logs they're looking for, there's nothing to say that we as offensive folks can't also use machine learning to say, model C2 traffic, or let's say you have semantic, you have some product that is blocking you, and you can't quite pinpoint it. Machine learning will be able to do that for you if you can set up the right experiment.  So I guess I just come back to the question like, what isn't machine learning? And in my mind, everything is machine learning if you kind of go back to the math, which is kind of fun. Okay, so in the lab for the workshop, all the code is in there for you. So the first half you'll step through, and it'll be, you know, very explanatory. And then as we go through, you know, it'll be less and less explanation, so you'll be required to kind of look more asked, ask more questions, etc., etc. Break stuff like you're not going to break it, you can always reset it, Google it, ask us for help. AI Village has easily, probably the highest concentration of some of the smartest people in the industry. And there's so many PhDs and super experienced people. So, you know, we love talking, I like talking offensive ops, but, you know, a lot of people like talking stats and topology and homology and all this other stuff. You know, the other day I learned that there is more than one algebra. But yeah, so, you know, this is the place to ask, so go ask for it. As you go through, I would challenge those of you who are more experienced to produce a better and more efficient attack. And in there, there is also a massive data set of VBA code that I found. It's like nine gigs of VBA code. So if you want to go through this code would work the same and you could get your name on that leaderboard for CVs for machine learning systems. So if you want to do that, you can go for it. So feel free to ask questions discord, there is there are no dumb questions. And, you know, we have three TAs that are helping me out here. So if you have, you know, specific questions, we can ask for it. So hopefully you have the the link. You have everything. So effectively, this is the you call this is the work workshop. And there are two, two files to Python notebooks, one's the solutions and one is the workbook. And you should start with the workbook. If you do get stuck or you need code help, or you just want to grab it and go, the solutions ones gonna work for you. This amsy.h file is if you do want to compile the amsy stream.exe so you can recreate the full attack path. You're gonna need this header file. The collect.py is what we use to gather all the information. And then this insights.xlsx, this Excel file. This is the version one of the insights that we pulled out of Defender. So this would be like that that proof point style insight.  This is it. So obviously, you can see, you know, there's a lot of binary blobs in here. But these would be the most malicious tokens. And this makes sense to me. Code command. And at the bottom, you have the least you have a giant binary blob. And at the bottom, you have the least, you know, malicious commands, or I shouldn't say commands, most least malicious tokens. Yeah. So this that's free. And yeah, so you guys can get started. Do you have any questions right away? I haven't seen any questions so far. Perfect. That's good. That's good. Is anyone is getting 403? Is anyone else getting 403? You're not seeing any chat on Twitch or Discord. It's just reopened the link. And it's building the container for me. Okay. Yeah, me too. Okay, that's good. I am in the the Jupyter Notebook. Nice. Yeah, so the first little bit, just I'm gonna give you guys like 10, 15 minutes to rip through the first little bit. And if you guys have questions, remote is always a bit weird. You know, it always gets me in the mood. Can you guys hear that? What always gets you in the mood? Tron. Yeah, I don't think it's a good idea to stream copyrighted music on here. Oh, right. Twitch doesn't really have a problem with that. But the YouTube part at the end will be removed. That's fair.  So So, well, I can send you the notebook link again, or myself and other people are using on the chat. That's good. I have. Excellent. Okay, so yes, I'm just going to talk. Everything's pretty self explanatory. In terms of the instructions. So we need to zoom in a little bit. It's kind of hard to see on the street. It's a better There's a one year old. I can see it very clearly. Excellent. So I'm just going to talk to you guys for free to run through. So sort of the basis of this attack is really the Annie scan and Mallor scan interface. Amzi is something they introduced in I want to say PowerShell five little sooner than that. But effectively, it's just a DLL that gets called into PowerShell, the IC dialogues, any J script, Phoebe script, and now dot net 4.8 and beyond. And inside, it has a number of blacklisted functions that whenever one of those functions is called, Amzi will get called to scan content. And Amzi is not a it's not a security boundary. It's not a security product. It's it's an interface. And so all it does is collect output, or so I collect whatever content is being put into the buffer or the string, and it passes it back to an Amzi provider. And the Amzi provider then has an opportunity to scan the content and make some determination. So obviously, this workshop is about Windows Defender. But there are several other there are several other providers, Amzi providers that this attack would work against. And that's, I suppose, kind of the nice part about machine learning currently is there's a lot of attack surface. So once an Amzi provider, you know, scans the content, and it decides what it is, there's, you know, a range of scores it can give currently, it only gives back zero or 132768. But in Microsoft documentation, they talk about a range of scores. So if you think, rather than a hard label as a zero or one, eventually you'd get some sort of regression type continuous variable continuous. Yeah, continuous variable on the out. And so you'd get, you know, some heuristic, I suppose. There is a great talk by actually Microsoft, badly behaving scripts. It's probably like an hour, an hour long. But I think they gave it two years ago at Blue Hat, I L. And it's two engineers from the machine learning team. And they're just discussing kind of how Amzi works, the things that they're looking for, the things that they are kind of the kind of models that they're building and so on. And so this is kind of our first indication, at least officially that, you know, machine learning was being pushed onto the client. And I think it would seem anyway that overtly malicious things get stopped on the client. But anything that's kind of in between seems to take, you know, seems to go to the cloud. One way to test this would be to have some sort of timing attack, you could get the response times for submission to the reception of a score. And you could maybe see if there was a significant difference between, you know, the worst scripts and then the medium bad scripts. So we've none. You guys can obviously read. So the first thing I've seen you do to create a copycat is we needed a data set. And there's a talk by Lee Holmes and Daniel Bohannon in 2017, called revoke obfuscation, power show obfuscation detection using science. And in the day, basically so Daniel Bohannon wrote invoke obfuscation, which basically takes a script or, you know, any number of different languages. And it puts it through obfuscation. So you know, it breaks it up into million different pieces. So it'll break those rejects is that, you know, those brittle detections. It's awesome, awesome tool. These guys wrote it, wrote the talk and part of that research they collected like two gigs worth of PowerShell scripts. And they labeled them both benign and malicious for us. And so they're already, you know, you can actually the links here. So you can go pull it down and rip through it. Once you have that, you know, the rest of it is just getting target outputs from the target models. So, you know, if we want to create a copycat, we want to know what defender thinks of each script that we give it. And then, you know, that is we model the model. So we have the inputs and we have the outputs. We don't know what's in the middle. We don't know that black box. But, you know, we can infer it, you know, through with our own model. And you'll never get 100% of the model.  But the nice part about machine learning and the struggle that machine learning has, I think is, is it does introduce a probability into what used to be a static decision. Yeah, the larger positive thing interesting is VBA. So that's just nine gigs of Excel macros that you could do the same with. Which would be pretty awesome. We have, let's see, what is this? This is the, this is the game, you're not on Windows. If this were like a lab lab, we'd have like Windows VMs, and we do this like for real. The this is just the output from the AMSI stream. So invoke WMI backdoor looks malicious, but Defender doesn't think it's malicious. So if you look down, you see scan result is one is malware zero is malware is the is the official feedback loop. And, you know, even in that in that talk that is above, they discuss the fact that you can't trust file names. And that makes sense. You can't trust headers anymore. There's some research that did a number of like three years ago now, against mail filters where you would null out the first two bytes of a duck and file. And depending on how the mail filter decided what kind of file it was, it would either, you know, block it or it let it through because it can't read the magic bytes at the top. And then obviously the documents corrupt at that point. But what Windows does is when you open the duck and file as it would ask you to repair it.  You know, your macro would live. And you could get code execution. Right. But the next bit is the provider display name. So we have, you know, Microsoft Defender antivirus. If this is if defenders turned off for whatever reason, or there is no AMSI provider, it'll give you an error. And you will not get a score back. The next piece is just PowerView. So PowerView is pretty well in the script. And it is definitely, well, I it's not explicitly malicious. But it is used by malicious people. That's not fair to say. It's used by attackers. I'm sure there are some nice attackers. So in this lab, there are uploading 380,000 scripts to GitHub was not popular with GitHub. So in this little bit, you only have 3000. But the collect up UI has everything you need. And when we were parsing, you know, we just did in your data, you have data clean and dirty.  And clean scripts are see clean. There's like 3000. There are about 1200 malicious scripts. They can look at. Yeah, if you guys want to start ripping through this code. As we're parsing, you look through collect dot py. But I personally like to keep. So there's a lot of moving parts, machine learning, you just you kind of can't get away from it. So whenever possible, I kind of like to keep at each step, you know, build a data structure and I'll keep the previous output. And I think it works. I see most of my data sets are pretty small anyway. So it's manageable. But you can kind of see the get screenshot. So this is the file name, the original file name. This is the hash, the empty five hash of it. This many of the result being its malware. And you have the basic C4 encoded text that you can go through. I like this because if there's any weirdness, you have everything you need right in front of you, you don't have to go, you don't have to start again at a particular point, you don't have to go back to the beginning of the process.  But at least for me, it's fine. I'm not dealing with billions of things. So yeah, we just got discussed lists. So if you're not familiar with a list, to think probably most of you are raise your hand if you're familiar with lists. Nice. Okay, it's about 20% of you. Can't hear any giggling. All right, so you have lists, lists are like really nice. So you can just split them on any dilemma you want. But you guys, I'm interested in the lists. So quickly, quick question, Will. This is from the Twitch chat. Should he be able to view the PowerShell code? When they click on a file with clean order to get a mess of characters, but not a script? Yes. So do you? So if you Yeah, so in this in this little code block here, you can open it and it is it is just a so we've already parsed it for you. It was just gonna take forever otherwise. So what you're gonna see is the script name. So this is the original script name, the MD5 hash of the script content. The AMSI result or should I should say the defender result. So what defender thinks of the script? And then the base 64 encoded version of the script. So and can you turn lines? So this bit, yeah, that's what we're doing here. So we're splitting, you know, a file. So what we're doing is we're setting up eventually, we're going to rip through all these, these files, and we're going to build a big vocab. So this is a, this is I just pulled one out as an example. So let's see. So for example, if we want to reference, you know, this is just less stuff. So obviously, index so we can reference it however we want. You can then reference with semi colon, you know, from index one to the end or from index one to the third and you know, third. Or you can even do minus so you can go to the end of the list. And just get the script content. But if we're going to look at the script content, you know, we kind of want to know it's interesting that get screenshot is as counted as malicious. So if we decode it, we can have a look and maybe maybe there's some knowledge that we have that we can think about as to why it might be malicious. So we're just using split to make it a little nicer. And there's a typo right there. It's a double typo now. So to make it nice, actually, this isn't the solutions one. So you guys would know this. So if you want to make it like a little nicer, we're just going to split it again. And we're going to get like a list of the things. So this list, in my mind, the some of the more malicious things would be the add type assembly. It would be convert to in 32 new objects just to my own memory stream. So everything like in memory attacks, any sort of memory streams. And then at the bottom, you also have to base 64 string. You know, I think base 64 is typically something that gets picked up quite easily. But if you wanted to figure out, you know, how this was, why this was malicious or why defender thought this was malicious, does anybody have any ideas that they want to type out to anybody typing can't see. Yeah, it doesn't seem like there is anybody typing on discord right now. Nice. Yeah, so we're just going to go through the dictionary. So and you know, this is just what I came up with. It would be to try and determine the commonality but, you know, between all the malicious scripts and see if see what tokens and when I say tokens, I mean words, words came to the top. And to do this, we're going to use we're just going to do a dictionary. And it's kept, you know, key, key value pair. So I'll let you guys run through that. And if you need playground dictionaries, they're pretty awesome. Like dictionaries. But there is one other data structure that I like even more than dictionaries. Excellent. So we know what dictionaries are now. And we're just going to decode the content. And we're going to add it all to a dictionary. We're only going to do one script. And does anybody know why we might just only want to do one for now? Because it's a good practice. Good practice. But when you are dealing with, I don't know, thousands of scripts, like half a million scripts, if you have something that's going to mess up halfway through, or you start down that path, that you're just going to waste a lot of time fixing errors. So if you can get to work with one and then 10, and then 50, and then 1000, you know, I think that's a much better way to go about processing large data, large datasets. So for this one, we're just going to do one. But we're going to split it twice. And then we're going to just add everything to the word index. So there's a fill in your blank, a fill in the blank bit here. So if you guys want to go ahead and run that code, and then tell me what the issue is. And I'll give you clues. It's on the line with all the question marks. I didn't know that I needed to be as close to the microphone. Or are you guys already way, way, way past this. So to get a proper count of the words as we go through them, we need to just add one. And you would have been able to tell after you went through it, because there's only one token listed for each. But the script has at least one so if we just fix that. Now we get a better representation of what's out there. Going one at a time. I guess I found, you know, I used to have a real issue with loops where I wouldn't break them out properly. And so doing it a little more slowly, at least one time has helped me. And then we're just going to sort the dictionary by token. So now, you know, this equals sign. There's nine of them. There's six curly brackets. There's four new objects. But nothing. So there is a lot of punctuation in there. And but we can deal with that later. So now, because we're impatient, and we just want to get to the machine, but we're going to run through the entire all of the malicious scripts. And it might take a little bit on the binder might even take a little bit here. The yeah, I wish I tried to ship all of the scripts to you. Okay, so now we're just going to sort the words. This is all of the tokens for all of the malicious scripts. And it isn't ideal. So there's a lot of numbers. And this is fairly common when you're tokenizing text. And I would generally just say Google slow but nice bit is we're seeing get proc address. Does anybody know, you know, in what operation get proc address might be used? Does anybody that works in an AV vendor know what get proc address might be used for? You can Google it if you want. Nobody GT, any penetration testers, malware authors, anybody? I'm a data scientist. That's fair. Well, I'll let you look that up. But it's it's used typically when you're looking up front functions in remote in other DLLs. Most notably, I think used in process injection. Anyway, alright, so there's a lot of numbers. It's fairly classic. But you know, we're seeing write bytes to memory, memory address, a lot of malware type things, remote proc handle, obviously, this is all PowerShell. But the numbers are actually kind of annoying. So let's scroll through. I mean, I'm, are there any tokens in there that are particularly interesting to anybody? Okay, Rob, logistic aggression managed to answer a question. Thanks, Rob. I was the man. Switch chat. Nice. What was it? He said you get the address of a DLL function and memory also heavily used in packers. Yeah, exactly. No, I can count on Rob. Okay, so the next bit. So we're just going to continue to filter down. And I would say that's, that's generally the case with most of your data, I think most of your time spent when you're doing machine learning isn't actually machine isn't the bit isn't the math piece that you typically think of it's processing data, it's, yeah, it's processing data, it's filtering data, it's making sure that your data sets are balanced or have a distribution that you want or whatever it might be. You know, I said earlier that we're not inventing new math, like I'm not going to, I'm not going to be the guy to invent new math. So, you know, I, the place that I can be most effective is using applying my domain knowledge to, you know, what I know of machine learning and being extra careful with my data. There's a Abraham Lincoln has a saying about sharpening an axe. Can't remember exactly what it is. Yeah, it's going to cut down tree, I've had 10 hours to cut down a tree, I would spend the first eight sharpening my axe. That sounds right, I could have made that up. But the same is true for, you know, machine learning, and I would say data science. And I don't have comath, do you want to chime in there as, as a data scientist, if it's true or not? Yeah, I tend to get the fly something that works, and then to go back and perfect it to make sure I don't get little mistakes. Because when you scale things up, it really screws you when there's a bug one point point one set of time. Now you have 100,000 things, the bug was his guaranteed to happen. It's better to clean this up before you really Yeah. And if I mean, if you're, if you're a data scientist, like, if in the Twitch chat, you don't try it, what do you think the percentage of cleaning versus actual machine learning, like data processing versus machine learning, I think that would be interesting. And if you're new to machine learning, I would say it's okay. I mean, yeah, no, it's okay to be impatient and like go really fast because you want to get the end result. But if you want to get quality results, it is better to go a little slower. Okay, so we're just going to create a new list. And the difference here is we're creating just a find all. So we're going to start removing some punctuation and numbers. And then we're just going to add it to a new list. And then we're going to print the top 100 tokens. So all the numbers are gone, a lot of the punctuation is gone. I tell that looks quite a bit better than the other one. We can still see we still have get proc address, we still have you know, so we're still seeing the malicious tokens that we would expect shell code.  But we haven't ruined our what we want to model effectively, right by its memory. So that's also there. So this bit I am, I just had picked some tokens and go Google them and make sure that you know, there. See what comes up. That's kind of fun. But I suppose we are live, I can do that. Let's see. You guys are using edge that that's embarrassing. Do you guys have you guys heard mini cats? Do you know if that one's malicious? Yeah, I would say so. Power shell type.  Yeah, yeah, definitely. And I would say as an operator, I think a lot of the industry uses a lot of the same scripts. So if like the deal injection, the deal injection at graver. Yeah, so these these are these are malicious text scripts. Okay, so once once we have that bit, you know, you could we could stop here, right? So we have some malicious tokens that we know that, you know, these are all tokens or words that were labeled as defend by defender as malicious. So we could stop here, theoretically, and have malicious scripts that never, you know, don't use any of these words. Alternatively, you could do the same with the clean scripts and only use, you know, words from those clean scripts, or, you know, you could do both. But I think that would probably work. But I think you would have a hard time making it repeatable. And you wouldn't necessarily, you know, something if it all of a sudden stopped working, I don't think you would have much recourse and being able to figure out how or why. And so I think this is kind of where machine learning shines for us. And it's just the ability to iterate through massive amounts of data extremely quickly. So you know, while I'm going to use the the GPT to fishing analogy, and it's like, you can spend an hour crafting, you know, one fish for five targets, or you could spend an hour correcting five unique fish generated GP to GPT to for five targets. GP to G language models aren't perfect. But they do help scale. So it's like you generate five of them, and you can correct them, make them seem, you know, realistic and not sound terrible. In the same time, you could to do one. So, you know, in terms of pulling out insights from Windows Defender, I think this is also true. All right. So we're getting to that's the machine learning bit. And in this manual, you know, we reference 380,000 scripts. There are I think 410 in the whole piece. So in that PowerShell link at the top, you know, has that many. We just use I just chose the biggest folder that had them in there. And then introduce them to insight. So we're going to get into some of the data representation. I think data representation is probably my favorite part of machine learning. It's really where you get to shape kind of shape the output and shape not the model, because the model has some architecture, but you get to encode and embed your domain knowledge.  And that piece is is extremely important. So, you know, if you want to the outputs of your model, or the accuracy of your model is a direct representation correlation with the quality of your data representation and your feature engineering. And it's, yeah, it's my favorite part. Okay, so GT went through tokenization earlier this earlier today. But tote we've already kind of played with tokenization, but tokenization is effectively the process of splitting, you know, the the words into separate words.  So once we have a tokenizer, rather than we could write it ourselves, but like we did up here, but I would imagine there's probably better developers out there. And I trust I trust them probably more than just no code for some stuff.  So each line is a PowerShell script. You know, just a list of it. And we're going to fit on text. And this is just to create an index. So we're just creating those data structures. And you can see, you know, each each word gets its own sequence integer. And, you know, in terms of sequential numbers, like in a text prediction scenario, like sequences are really nice. Because you can say, you know, what number comes after what, and you can go back to your index and look it up.  These are I don't have a cat, but I assume cats like mittens, or most cats are cold mittens. But we're just taking the first index and we're tokenizing it. Six more cats. It's, it's yeah, it's a word index. It's fairly straightforward. And there are a number of ways to represent text. You have machine learning models, that'll do it. You have frequency models, you have TF IDF, which is term frequency inverse document frequency, which is like a weighted frequency. What else do you have? What am I missing? Obviously one hot encoding, that's where we're gonna go next. So one hot encoding is basically it's a vector, which is the length of your vocabulary. And the presence of a token or a word is denoted by a one. And if it doesn't exist in a document, then it's a zero. And so we end up getting is, let's see how long is there? Well, it's only 17. So each vector is going to be 17 integers long. And the array is going to look like this. So this is just the first sentence.  It's not indications, activations, probably incorrect. You get different, yeah, indications. Anyway, hopefully that makes a little bit of sense.  So if you have a lot of like a really big corp, you know, really big corpus or vocabulary, you get some sparsity in there. But it's really nice if you only care about the presence of a word, it does not. It doesn't necessarily keep the semantics or the syntax or, you know, it's not very good for text prediction. It is good for classification. Just kind of what we want to do. Alright, so now we kind of have an idea what tokenization does and the tokenization scheme that we're going to use, we're going to get into my favorite data structure, which is named tuples, named tuples are immutable, well, they're, they're tuples, but you can, they're like immutable classes. So you can kind of create them, you can set your, your variables to whatever you want. And it's really nice because then you can turn around and reference a tuple or reference a variable inside of a tuple by name, which makes your code read a lot cleaner than if you were just going to use a tuple or a list or even a dictionary dictionaries can be. Actually, I don't know the speed difference between dictionaries and the tuples. Does anybody know there's any? Maybe not. But this is an example of named tuple. I, you know, it's like, you know, when you find something you like, and you use them absolutely everywhere, that this is where I'm at with named tuples. Just because they're super nice. So this code, we are going to kind of create our training structure. So this is you should be familiar with everything in this code block by now. So we are, we have our filter, you know, there, the tokenizer has a filter attached to it. If you want to use that one, we're going to create a primalist to hold all of our training data. And then we're going to hold a we're going to create a list that will that we're going to use for tokenizing. And this is kind of eventually, you know, when when you start building these small pieces, eventually, there's a point where they kind of all come together. And you can transition from processing into learning. And this script here is where we do that. So, you know, all of a sudden, we can, I shouldn't say script, but so we've built all these tiny pieces out. And you can kind of see how that they're coming together. And there's, there's two really important things happening in this little code block. Obviously, the tokenization is happening. And we are collecting all of the all of the name tuples. So I know name tuples, we're going to go through every malicious script. We're going to decode it as we did before, we're going to remove all the punctuation, we're going to split it, we're going to unique it. I did this because I assume it makes tokenization faster. But I don't, as I said, I don't have any evidence for that just seemed to make sense. Maybe you guys can try and let me know. We are going to then, you know, rejoin the unique tokens, unique words and sentence, and we're just going to add them to a name tuple, and then add that name tuple to a list. So the important piece here is the label. And, you know, from malware, the label is a one. And then for clean labels are zero. So this is where we're kind of creating our training structure. The other piece you might notice is originally we had 380,000 scripts that we ran this on for this demonstration, this workshop that get helped and really like it. So we only gave you 3000. But now that's still like a two to one. But when you're having 380,000 scripts versus almost 1200 scripts, you know, your data is very unbalanced. So if you, for example, were to do a similar scheme as we did earlier with just the malicious scripts, we did it for both. What might end up happening is your clean script words will start to the word isn't drawn out. What's the word Eric spin? What's the technical term? I'm gonna say drown out or average out whatever it might be. It's like if your GPA is really bad. Eventually, you just can't get it out. Alright, so then at the end of this, so the outcome of this is we're gonna get a list that has all of our name tuples that are ready for to be processed to be fed into a model. And then we're gonna have all of our documents that are going to be tokens. So at the end of this, we're just going to build a vocab and we're never everything. This is a little blurb. So it takes a little bit to go through all of them. Because I open this, I don't want to tokenize that. Also, you guys probably know this, but if you guys use them, it creates a swap file. So I helped some help divide some issues with where they were, they were opening their data files inside of a folder and running Python out of the same folder. And so their Python was like trying to tokenize their text and they couldn't figure out the error and it was just because they were trying to read a swap file. Anyway, all right, so now we have our our vocab. Yep. And we can kind of see the tokens that came out. So these are the after our filtering, you know, the closer you get to training, you want to be, you know, you want to be increasingly happy with what you're seeing in terms of the words or the tokens. And I think that I'm more happy with this than when we started being definitely play by ear, but it is useful to take a moment and see. Just take a look. You know, remote DLL handle, just take a look. That's gonna be useful. Let's try them. So this is something I haven't dealt with. And you can see in the insights.xls, but if you were going to deal with it, this is probably how you would do it. So you just want to do a regex for this TVQQAAMA. Does anybody know what that is? I bet Rob does. I see you're binging it. Yeah. Yeah, this is the this is embarrassing. The best way to Google. Yeah, so that's, you'd want to remove those obviously. I didn't. It indicates that it is a PE file. Yeah. Yeah, compressed or embedded in some other medium. Okay, so now we're at the point where we are, I already described this, but if you're not familiar with machine learning at all whatsoever, there's a little video series by, what's his name? I don't know. But his YouTube channel is 3Blue1Brown. It's just like four videos an hour. It's really good. It's gonna be way better than I will ever be able to explain machine learning to you. So I recommend having a look through that. You know, watch it a couple times. And if you're really into math, he has a lot of cool stuff like, yeah, his animations are really good. But neural networks, we've kind of all seen this picture. But our input layers represent our input data or our one-hot encoded text. Hidden layers represent an activation function. And the output layer or notice the result of the network or classification, prediction, whatever it might be. There are a lot of, machine learning is more than this little picture that we see everywhere. And I would say this is like the tiniest little bit of machine learning.  And this is just like the most vanilla machine learning implementation. So I think the nice part, you know, the nice part about Keras and whatever the frameworks is they bring, you know, that kind of ability and power to non-mathematicians or non... What do you call non-mathematicians, spin? I don't. Is there an industry word for them?  Students. Students. Yeah, something like that. Normies. Normies. Yeah. Honestly, we just don't think about non... Yeah. I don't want to dig myself into a whole year. You've got too much math to think about. As a non-mathematician, I take credit for this. Yeah. Yeah. So I think people was right. Yeah, I just got it right. Late people. Yeah, late people. And if you are, I think the first book I read was Make Your Own Neural Network by Tariq Rashid. And that was, it was just the most, it was like 120 pages. It was just the most basic, most straightforward explanation of a neural network.  And that is definitely a hindrance. Like, you don't have to be a mathematician to use machine learning, but you should at least spend time learning the basics. Okay. So, let's see. Do you have a little explanation of machine learning? We have our train set. We're happy with the tokenization to a point. I'm happy enough for now anyway. You know, eventually, you know, I think you always want to be moving forward so you always kind of want to be thinking in pipelines. So I wouldn't, you know, wouldn't spend too much time necessarily in one area, but when you build it, make it such that it is modular and can be like put into it into a, into pipeline. Let's see. So now we need to tokenize all the documents. And this will take a list, which I think docs is. We'll find out. Nice. And then we can, let's just double check. And now we're going to create our, our score array. So here we have, so this is a named tuple that we're going to go through. So we're referencing e, which is a terrible variable name, sorry. But all text is the, where the list is holding all of our named tuples. And if this is a, this is called list comprehension. Another, another favorite of mine, it's like creates a list. Put a function inside of square brackets and it'll create you a list. But you can see we're referencing the label. So we're just going to create a score matrix and actually you can see what that looks like. But there's giant arrays. And the reason they're giant arrays is because mechanics of machine learning is kind of, would you say rooted in matrix multiplication? No, because that will ignore all of the decision trees and a bunch of other things. Yeah. Yeah. And canyons, they bruise in a bunch. Yeah. Yeah. Yeah, this, so we, when I was talking about like the, this picture being kind of done, it's the only picture we ever see. And it's like the smallest, it's just one of the smallest pieces. There's so much more out there that you should look at and I probably could have introduced you to. Anyway, so now we have our score matrix. Actually, I'm looking at this. Okay, so these are all our labels. So when the, when the network's learning, you know, it's going to calculate a loss. So it's going to have an input. And then at the end of that, it's going to take whatever the label was and see how close it was to the label. So a large loss is bad and a small loss is good.  And we'll get into that. So I like this. This is part of my playlist. It's a tether of spaces, nano versus VS code, messy versus that other guy. Machine learning frameworks are no different. So do you guys have preferences for machine learning frameworks? Like what, like, why do you, why do you choose the machine learning framework you do? I use PyDorch because it tends to look more like real code than TensorFlow. But Keras is like super easy to get up and running. So it's a good choice. Yeah. And then when you do the real world stuff, I reach into a compiled language like Rust or C. Oh, see, that's brave. Has anybody used ml.net? Did anybody know ml.net existed? No. Depends on what you want to do. Actually, I have a win32. So directX. There's a win32 machine learning implementation. Actually, it's not in it. It's just an interface for TensorFlow on X models. Yeah, I prefer Keras. It's simple. I think if all the math is the same, then you're really just looking at what you prefer. And yeah, this is what I, yeah, Keras is good to get off the ground, but it may limit your customization. And I think that's exactly what an API, it doesn't limit you, it abstracts complexity. So I think the reason Keras is so easy to use, because it's a really good job of abstracting complexity. But if you want that complexity you want to dig in, it's still there for you.  Versus something like PyTorch, which is pretty raw, I would say. That's probably not true. I mean, some guy in the back's yelling about Matlab right now. Oh, Matlab? That's a whole other factor where we all have a little bit of PTSD about it. Yeah, don't ever say Matlab. Yeah, Rich is saying PyTorch is, yeah, Keras is good for 99% of your projects, and then PyTorch, where when you're getting freaky, and I say PyTorch is good for 90% of your projects, and Rust for when you're getting freaky. There's also Jax, all sorts of stuff. Oh yeah, Jax. Does anybody use Jax? Do they like it? I know Jason is seeing here. Yeah, it's kind of the Wild West, I would say. Like, yeah, there's a bajillion different frameworks. Everybody uses something different. I like TensorFlow. It has TensorFlow serving. It seems to have a good ecosystem around it. But you obviously, you make trade-offs, right? But if your verbiage and your fundamentals are good, then I think you could probably use just about any language with enough practice. Okay, so we're going to create our model now. And we're just going to create, you know, it's the same as the picture effectively, but we're just going to create it in code. One thing we didn't talk about are activation functions. Or actually, I didn't actually talk about a lot, but as inputs or as your data traverses from weights into hidden layers and hidden nodes, weights, as they change, their job is to modulate inputs based on, sorry, modulate, yeah, modulate inputs based on their inputs, such that the output is relative. Sorry, I saw some in the chat. So we're just going to use Sigmoid. I think Sigmoid is super simple. Just go place to start. The other, I don't know, what are some other favorites? I know Relu has replaced Sigmoid. Is there a good reason for this? Or is there, does everybody kind of go, Oh, this is better. So are you going to use it all the time now? So the Relu paper was originally basically said, Relu gets better accuracy scores than Sigmoid.  And for images, it tends to, Relu tends to have a higher accuracy rate of score than Sigmoid does. And that's basically why at one is because on images, you can easily show that it's better. I don't know whether it's concretely better in all situations because of that. So you're showing your age with Sigmoid here. Relu also handles the disappearing differential, or disappearing gradient, yeah, vanishing gradient, where if you go too far into the negatives or too far into the positives, you're never going to crawl yourself back out. Yeah. And then there's leaky Relu that helps, that tries to help that even more. There's L who tries to say, Hey guys, you could be Relu, you could be leaky Relu, let's just combine the two. And so I think it never goes below negative one as a way. Yeah. Yeah. And I think this is actually like, I like the discussion. I don't know how many of you are listening or machine learning people to begin with. But as, and this is why the tutorials are kind of, they're nice to get off the ground, but I think they're limiting in the fact that they're always using kind of the same architectures. So they're never, you know, to say, Oh, this is good for this and this is good for this. And that's true. And you might end up there anyway, like they are, maybe there's just lessons they've learned that you haven't. But I think it is important to explore different architectures, different losses, just play with play with the numbers. I think machine learning is ultimately about iteration and experimentation at scale. And that includes activation functions that includes output nodes, includes any, any lever you can pull, you should pull it and see what happens. Also, the, as an engineer, sort of as a data scientist, you should learn pull the levers and see what happens. Because you might find that on like security data, sigmoid works better than value in this case. And you never tried sigmoid because you just go with standard value. You never learned that. So read everything, try everything. Hang out, hang out at AI Village. Super smart people. Just hang out at Journal Club and be a little flounder. That's what I do. Okay, so we created our model now. One thing we need to do is we need to put our documents, our, what do you call it, our vectors into our model. And when I was first starting, this is actually one of the harder pieces that I had to fit my brain. I think that's true is just the shape, the shape of arrays and how they get introduced into models. Does anyone in the chat want to take a stab at what these three question marks should be? Live. I'm, I'm hung up on the missing current disease. Yeah, but Rich has got to write the feature size. Yeah, exactly. So what is that? That would be Pretty sure it's your matrix.shape variable. What are the things that you guys do constantly? You guys must be probably missing some imports. I'll push the new version. All right, so if you're missing those imports, you're gonna want to, so we created our model. It's super simple. The text matrix, the input DIN that we're doing is an array of let's see, probably like 86,000 tokens. So, you know, across this, you're going to see that's going to look obviously like this. So we just have, I think 1200 samples times, you know, 80,000 tokens long and just gonna feed it then. Nice. And then we're going to do a test train split. When you build a model, you obviously have test data and you want to keep some test data out. But the idea is eventually that your model will be deployed into the real world where it's not going to be trained on. It's not going to be seeing test data necessarily. So ideally you want to keep some data away or out of your training set. So when it sees real data, or when it sees data it hasn't seen before, it can make a, you know, hopefully an accurate guess. So splitting them out for that reason. And I've heard you guys talk about like training leaking, leaking into your training data during training. How does that happen? So if you don't, one of the ways it could happen is you are, you could have duplicate samples that are like normally you just say, I pick a random set of indices or pick the last 20% of my indices. And that's my test set. Or you do a cross validation fold where you like divide it up somehow. And if you had duplicate data, say, like the last 20% of your data was actually duplicated with the first 20% of your data, you trained on the first four bits and then you tested on the last one. Well, it's since it's duplicated, it's included in your training set and you do really well. So you have to make sure that you don't have that sort of issue. And then there's other little things that you could have. It's basically a data cleaning issue a lot of the times. And then sometimes it's a weird bug in your code. Sometimes you can just reverse the model itself. And it'll sit, you'll say, Oh, show me what this looks like. And then it'll just print out an example from the training set. Nice. Yeah. Okay. So maybe early stopping twice early stopping. It's super nice. So if you have a really long running task and your, your model is unimproving early stopping will just stop your model. Part of me doesn't like it. And I just feel like it could some get better. But this is a tiny model. So we're not that interested. We don't really mind. Um, Crescent nice give callbacks. So if you want to use tensorboard, whatever batch size, this would be the frequency at which updates to the weights will happen. E pox, this is just time we're going to do. So the number of times that we're going to, you know, run through, we can train. It's obviously seems pretty accurate. I don't know I haven't actually tried to optimize this at all. But what do you guys do when, you know, model trains really accurately at first? Do you think it's beneficial that a you would train, you would try and overfit a model at first because that would at least indicate it could learn something. Yeah.  But something's wrong. Something must be wrong. Yes, something is wrong. And it says probably there could be any number of things. This was this version one. And, you know, I think I gave you the I'm giving you the code so you can go and recreate the model, do whatever you need to. And I would love to see love to see some blog post or some write up about everything I missed. So this is a visualization of training loss. Obviously with the gradient going down, you would like to see this going down if this were inverted. That would indicate that. Well, I could indicate a number of things, but either it's not learning or it's overfitting and it's stopped learning completely. And there are some evaluation metrics. And if it obviously it's going to be like, yeah, I did really well on this as we saw up here. So it's a little suspect.  So, you know, we're just taking the first malicious document in our, in our, because we, we, what do we do? We put them in a list such that they were, you know, malicious and then not malicious. So we're taking the first, we're taking the second malicious document and then we're going to take the last non malicious and we're just going to try and predict and see what they are. And yeah, they're a little too accurate for my liking. But, you know, that's, that's kind of it. Okay. So this is kind of where we leave you. So the first, this Excel file that you're looking at here, when I did it on all 380,000 scripts, it took me, well, and this would be dependent on whatever kind of potato you're running, but it took me 17 hours to do. So we're probably going to run it here, or you can, you might already be running it, but having to reset a few times because you thought your theme froze. But effectively what we're doing is we're just toggling absolutely every possible combination and then making a base prediction and then a new prediction. And then we're keeping a cumulative sum of those. And what you end up getting is a spread of scores across, you know, a number of predictions that will sort your tokens into malicious and non malicious. This is the same code from the proof point research, the proof point research was easier in this regard because they had a wider, you know, they had a range of one to 999, where these are hard labels and it's a zero or one. You know, as a first run through, you know, looking at the most malicious and the least malicious tokens, you know, without any optimization, if it were ever going to work, I would expect, I would expect to see that, you know, at least they're being sorted to some degree, because now you can go back and you can start, you know, tweaking the model or whatever it might be. You know, to really pull out, really hone in on what's accurate, I think a lot of times for attackers like this first version would be just fine. But, you know, if you, as you go into the future, you know, you could even do this, you could always be collecting data. And actually, I remember after my first B sites talk about machine learning last year, Rob came up to me and was like, Hey, you should, you should have a separate data gathering campaign.  And it doesn't necessarily burn, burn you because collecting data can be noisy. So it is a bit over time. But do you guys have any questions, comments? What I would love to see is someone to take this defender model and absolutely crush it. I didn't, I mean, it's a tiny model. I'm not sure Microsoft would care that much. The there's a VBA, this VBA data set I have not touched. So if you guys want to race to whatever, I'd love to see that comes out of it. I appreciate everybody. Thanks for coming. Hitting up in that Slack or not Slack Discord, if you have any questions, or you just want to rag on my terrible code, let's find two. But I will end it there. All right. So I think this is this is the last stream of the night. So we're closing out the Twitch stream now tomorrow. Hopefully we'll go smoother. We learned some things today. But I'll see you guys all in the morning. Hopefully.


the income and layer trade off, the competing substitution and income effects. Dear students, last class we have discussed with the help of examples, we discussed a pure substitution effect and a pure income effect, but today we will discuss the combination of these two, how these two effects will work together. So, in this context, we will take the example that how they work together. So, it depends on the real wage increase, this increase is whether permanent or temporary. A long term increase in the real wage will actually cause both these effects income as well as substitution effect. Due to substitution effect, if real wage increases permanently for a long longer period of time, due to substitution effect worker will supply more labor, because the reward of work is now increased, so worker will work more. Layer will be expensive, it will work more and work more. And because of the income effect, we have understood that because of the income effect, what will happen is that now the layer can be afforded more, layer is a normal good, so the layer will be more and what will happen in the end, the work will be less. So, due to substitution effect, labor supply is increasing and income effect is decreasing the labor supply. So, these two will move in the opposite direction. So, what will be the net effect? Overall, what will happen? With the increase of wage, the labor supply will increase or will be less. It means that when the income effect will be stronger, then what will happen? With the increase of wage, overall work will be less and when substitution effect will be stronger, then due to the increase of wage, labor supply will increase. And if both will be equal, they will offset each other, then it means that the wage is big but there is no change in the labor supply. So, what will be the net effect? Whether income effect will dominate or the substitution effect will dominate. So, in this case, the wage increase is expected length of time. If the wage is increasing for a long time, then it means that the income effect will be stronger and the income effect will be stronger, then the labor supply will be less. And if this temporary period is increased, then the substitution effect will be stronger and its implication is that the labor supply will increase. So, this theory cannot decide which effect will dominate. We will study this empirically, what data shows and what happens. So, the results of data come to us that when the temporary wage increases, then the labor supply increases because the substitution effect dominates and the income effect is stronger. And when the increase is of permanent nature, then the income effect will be stronger and the labor supply will be less. So, theoretically, all possibilities are there. What are the possibilities? That when the wage increases, then the labor supply increases, the labor supply decreases, and labor supply can be constant. When the income effect will be stronger and when the substitution effect will be stronger, then the income effect will be stronger. When the increase is of permanent nature and when the substitution effect will be stronger, then the increase will be of relatively temporary. Thank you.