Hey there, njroute22.com here with another cheap wine reviews vlog Little technical note here the last Few cameras we've been using here The last camera I was using for the last five videos was a Sony a5100 you could see the link in my in the description below It was a nice compact point-and-shoot. It was modern 2017 2016 version It autofocus quick everything about it was great 100 megabit like video transmissions. This is all geek talk by the way I Had to return and I'm part of this beta program. This this there's a website out there called xp's expese Com and you pay like 30 bucks a month and you get to test one piece of technology for three weeks a month And you have this whole list you could get test drones You can test hairdryers and kids toys cameras computers laptops etc. Etc And you pay 30 bucks a month and they'll send you something for nothing well for 30 part of the $30 fee and You test it for three months inside if you want to buy if you don't like you send it back You get a new item the next month That's why I had the Sony a5100 so a good camera you get it for about 400 bucks Here's the thing though. Let's say you like the camera you could keep it and they they charge me if you wanted to keep it You could pay it off over time. It's like having a credit card, but not needing one 40 bucks a month for 12 months. I would keep the camera It's it's actually a pretty cool program you pay as you go if I can find an affiliate link there You could sign up for this expese. It's really kind of cool. It's in beta right now But they're accepting new customers and it I have a feeling this is gonna be the new trend Where you don't you could try stuff out without committing I mean I have to admit You know on go on Amazon buy it use it for 28 days and return it say it's broken and they'll give you your money back They don't care said there's a way to take advantage of Amazon by the way so what I'm when I'm using here is my old 10 year old digital SLR and It doesn't have I can't autofocus like for instance today I'm reviewing a dark horse double down red blend. It's a 15% 799 a bottle Wine this is in the pretty much the mid-range for our cheap wines $10 per 750 being the high $5 or less being a lot. Well, we're gonna talk about that at some point in the future I think there is a too cheap wine like I said before about the With a crane lake that is deadly and I talked to my buddies at one of the local liquor stores around here and We talked about some of the really really hobo low-end wine and they were like 12 and a half percent alcohol I'm really a little nervous about that I'm gonna do it once for the viewers the like a handful of viewers I have for this channel one time I'm gonna and I'm gonna make sure I don't drink like five days beforehand or me one Whatever. Anyway, let's see if this focuses. This is the dark horse double down. Does it focus a little bit I can't tell I'll off seeing post but this is This is a cheap wine and let me let me preface this real quick I've had bad experiences with dark horse in the past and I'm not saying that's going to affect this review at all This damn thing is crooked. I'm gonna straighten it out here. Let's straighten it out. Yeah, that's a little bit straighter now A little bit straighter a little bit straighter good. I'll move it back to so you can see a little bit more of the I'm just such a I'm not a professional a youtuber, but I'm an honest one. That's that's important So let's see how it focusing it doesn't focus too good We'll see if this is dark horse. I choose this one because you have the high alcohol percentage. I typically Have better experience With with the higher alcohol in terms of how I feel the next day. Remember this wine vlog is not About the taste the notes the blend the grapes did whatever that is BS And I'm not gonna curse on this on this channel just in case you're wondering Why should I mean like we all curse enough during the day? Like what what I mean, maybe perhaps if my curses have like comedic timing I might add a fruity little zinger here and there But anyway, did you see did I do my my signature drink? Man, it's so refreshing. That's I really enjoy a nice cold watered-down red wine Six ounces. This is a 30 ounce cup from Starbucks 30 ounces six ounces of wine maybe seven or eight depending sometimes I'm feeling a little saucy and I'll add a couple extra and you get a whoa It really does affect the buzz and then you know five to eight ice cubes, you know standard 1970s ice cube trade nothing from a bag and the rest water filtered filtered water no fluoride And that's how I drink the one. It's just it's the best way. It's not ghetto. It's not it's not White trash. It's none of that It's how I preserve my life. Anyway, the other day I did a video about the 19 crimes and I and The wine is good. It's nine bucks a bottle past the two bottle test But I went on their website and the marketing was in another league. It was it was all about Something that had nothing to do with the product or the wine. It was just some sort of sort of fruity just Like reality show fake kind of I don't It bothered me that I couldn't find out about the wine, but I realized and as I've told you guys That this little write-up on the back of the wine is to get you mentally tricked Into thinking something before you've been drinking I never read those fricking things never never read this never read this Never read this just buy it. Here's what you look at You know find the varietal that you like is a cab or a merlot or Pinot noir, whatever Pay attention to the alcohol percentage and they're different. Sometimes they put them in these weird spots like over here That's 15% and dark horse makes a 13 and a half and all I Bought this for some friends of mine and it always turned out kind of bad like everybody got angry It was like an angry wine This was a few years ago, and I've kind of stayed away from it But I picked some up for the for the local Local nut nut house here, and I think it still has an angry feeling so if anybody's a little bit Don't don't buy this one. I'm okay. I'm stable almost always unless I drink like hard alcohol like vodka or whiskey. I Don't trust myself drinking. That's why I don't drink it anymore. Anyway This for me I'm on bottle number one I finished one whole bottle and there's an unopened bottle here proof evidence never hasn't been tapped There's no trickery going on here I'm gonna crack that open, but I I don't know if this is gonna pass the two-bottle test I mean I had a weird night last night. I'll explain it some other day But I looked on the dark horse wine Dot-com dark horse wine dot-com and I looked up their wine and all they talk about is is the emotional triggers Okay, bold luxurious dark food hints of caramel and everybody likes to hear like that the sugar sugar sweets And there's a harvest grapes. We're gonna harvest them at the optimum ripeness. Isn't that the objective always me? Would you ever want a frickin sour grape? There's a reason there's a that's a phrase. Hey, you got sour grapes when you or how about Harvesting when they're past their optimum ripeness. Well, these are rotten crud.  I mean, I'm sorry I'm a loss of words for the lack of creativity. I mean if that Like thrills anybody. Hey, man They this one is good because they pick them when they're at their Optimum ripeness.  That's when you're supposed to pick all grapes always. Ah Anyway, and then they use Most sorts of other trigger words. What you'll taste luxurious tannins with rich color I know how can you taste rich color? It says luxurious tannins with rich color. I can't taste color I can only taste taste. Anyway, I'm really just a little bit irritated with the how how Marketing has has completely circumvented just common sense quality price. How do I feel? Etc. So That's basically it. So I'm on bottle number one I'm gonna take a break here and and I'm gonna tell you that their marketing and and and website is is Lacking of any real honest information. It's just BS marketing By the way, they this they say for this particular grape this Bob double down Red blend on their website says 14 and a half percent, but when you look on our bottle It's actually 15% so maybe this is either an old here and again This is another one of those wines without a vintage. There's no year. So you can't really figure it out So just like the Rex Goliath I mentioned again that I talked about a few videos ago I said, don't worry about the packaging or any of the marketing. Look at the percentage Like if this this went up to 15% if I feel good tomorrow I'm gonna give it a thumbs up regardless of how much I dislike their marketing and it doesn't do anything to me If I does something for 99% of everybody else, but I'm not I'm the 1% Out of don't take that out of context Um If this works, I'll I'll keep it on the list 799 a bottle is is a good Middle ground. I mean it's $16 plus tax for for a 1500 milliliter and it makes me cringe when I used to get Uh the Rex Goliath for 1009 at 13 and a half percent and it was a perfect wine I mean $5 a day or it's eight seven seven bucks a day Times, you know, 365 is over $2,000 close to $3,000 a year. Wait a minute Yeah, seven times 400s 20 hundreds between two and $3,000 a year I would say if I drank two bottles a day every day of the year, but I don't drink every day of the year just most of them um Anyway, so so their websites useless like like nobody's I mean, it's just all about marketing as wine Wine will always be all about marketing and tricking your brain into into liking something and every there's all this Fancy crap going on. I mean if I could find a $3 bottle of wine that makes me feel fine every night and it has So so taste I will buy because I'd rather use the money for something else I will not invest my even if I had A million dollars a month salary or 10 million dollars a month. So I still wouldn't buy a $500 bottle of wine I don't why I mean like common sense should be Just because you have more money doesn't mean you should spend everything you got right? I mean, I like to win. I mean, sure. Maybe maybe if I had a million dollars a month I might go up into the $20 category. Perhaps the the two bottle rule is more adequately passed in that and It's struggle to find a cheap wine that doesn't make me feel like I need to call the garbage dump and take myself to the trash and I don't know. Um But anyway, I'll come back in a little bit To talk about maybe how I feel halfway through and I'll try again to do My morning after how do I feel? But again, I've been really bad and a little bit like I haven't followed my own rules where I if I ever review a wine I should probably do it When I'm I have a clear day at least before me so I can reset and then truly like level the playing field But I think I'm old enough and experience enough to be able to figure it out Even with an unlevel playing field. I I think I know So we'll come back in a little bit and see how we're doing with this. This is one bottle in and one Okay We're back with uh, this is the last glass Of the whatever that was. I don't remember the name of it anymore. Uh, some sort of a double double Double blend That's just the most satisfying feeling to to take a after a nice cold slug Of uh, cold wine.  I'm maybe I'm getting tired. I don't know what time it is here. I can't even see One or two This uh, this this wine we're drinking right now is is perfectly fine On the expense event 17 bucks or higher for two two bottles Again, I'm gonna say it again Your best bet is to find something that's around five dollars a bottle and that's why box wine such as Shania Is is the best because you save a man of money Save a lot of money to save a man.  I don't get mad. I get silly more silly than than anything else but I hope uh I hope my my wine reviews here are helping somebody because If I could help one person Not get worried about the the marketing behind a wine and just figure out what the the money The the experience and the after fact the after effects Like how you feel the next day is a big deal. It really is a big deal. That's why I think those 3.99 I'm tempted so tempted to buy a three dollar 99 cent bottle of wine At one of the liquor stores here and do two bottles of that and I looked at the label There's no I mean the fact that there's no right up about it. I mean As much as I bash the write ups. Oh, it's a it's a ripe plum blah blah blah blah The fact that there's no write up Almost makes me also nervous about it. Um I think it's kind of funny.  I have nothing to say about it.  I'm gonna do it. I'm gonna do it.   It's either Um Because the 599 wine that I like per bottle 750 bottle that I like the best is explore door available at bourbon street It's good. I've drank two bottles and I'm fine.  Okay I haven't played a video game in like four years and I just got hooked on it And that's what they do. That's how they do it I drank four cans of spiked seltzer afterward and I was I'm pretty much okay today. I slept a few hours extra But the three that was a 599 wine on sale from 899 or whatever it was whatever they the perceived prices are God knows you don't know what they they really sold for what the wholesale cost was um But a 399 wine I'm like I'm very curious. That's eight dollars for for 1.5 Do I want to put myself through The like almost almost guaranteed pain the next day and maybe two days afterward I don't know so so the cheap wines has like Like like a limit a high limit of 10 for me in terms of financial reasons and a low limit of five or six for physical reasons if I go below that to like three Like I may just be like volunteering for for pain and I don't want to volunteer for pain ever so Hey, I'm back. Uh, this is the next day after drinking two bottles of the dark horse Double down whatever the wine I drank in the video Proceeding this it passed the two bottle test because I drank even a little bit more of something else And I feel fine today. So I think we're all in the clear with this So I pat I don't know what bottle doesn't pass the two bottle test. We have to maybe find out. So that's it


There's a little bit of we're old men who are not good at RTSs, baked in here. MOBAs, well it depends, there's a lot of things wrong with MOBAs. MOBAs have the go problem. They don't give good feedback because- At least the ones that are out now. Yeah, most of the MOBAs that are out today, it's very difficult to learn how to play them without reading guides that fans wrote explaining the math. Or having someone who knows what they're doing, sitting down and literally teaching you, walk here, do this, stand exactly there for five minutes, then back up and stand exactly here for five minutes. If you f**k up- There's magic in you from this story. It reappears every- exactly this many seconds, so every that many seconds you have to go over and take a look and see which one it is and make sure there's no one else trying to get it. Yup. So they're great games. If you learn them, they don't teach you how to play them. They hide the information you need to get good at them. And as- and because they're competitive, they're ranked, there's leaderboards. If you f**k up at all, you get kicked. You're gone. No, but if you- I mean, you play- you- those of you who play MOBAs, if someone on your team is just feeding, how bad do you feel? How do you feel about them as a person? And often- and the games that try to punish you for leaving, like in Dota, it's like, well, I can't kick them, because I'll get in trouble for that, for, you know, but what am I supposed to do? Just stay here and lose until we're done? It's like- I'll just rage quit. Yeah. That's fine. You know, it- it punishes you for that too. So it basically forces you to sit there and suffer with the rest of a bad game. MOBAs are bad at getting new people to play them who don't have a tool. Yeah, somehow they have the most- they somehow have the most players of, like, any game. Somehow. It's some kind of miracle. Quite a reaction from the crowd here on this one. MOBAs are bad at something else. This might sound kind of- kind of off-key here. MOBAs are bad at being spectator sports, despite being the biggest spectator sport in all of gaming right now. Oh, well, I guess football was gaming. Sure. Yeah, hockey is gaming. Right. MOBAs are bad at being spectator sports unless you know how they work. If I watch a MOBA that I've never played and the crowd starts cheering and usually, like, nothing happened on the screen, I can't tell what's going on, they're only great spectator sports if someone next to you is like, Yes!  And then they explain what happened. So it's the same thing. We were at the big League of Legends area at PAX Prime and there was, you know, big game going on, whatever, everyone's watching it. And you saw some crazy stuff happen on the screen and some people cheered like, Yeah, and this is a delay.  All right. You go to football game, you ever see a touchdown? Like, three people are like, Yes! And then everyone else is like, Oh, it was like, Oh, yeah! Now people know it's a touchdown. They can tell. But football has that too. Like, say, I don't know about football and Scott does. We're sitting here and like, they're setting up for an outside kick. Like, something weird is about to happen in football. And I don't know until they kicked it. Yeah, like, you suspect. Like, you suspect, like, you suspect, like, This is not the situation. They might do the crazy thing. Something cool is going to happen. So the expert football person is going to be like, Hey, something cool might happen. They must start explaining like, Why this thing's exciting. It's the same thing with MOBAs. MOBAs are only exciting to people who don't play them when they're watching them. If someone else is pumping them off and telling them what's going on and getting them going. The breakout for eSports is going to be the accessible spectator game. Come on, heroes in the storm.  Doesn't have to be accessible to players. Please don't suck. It only has to be accessible to spectators.


I'm honored to be here at DEF CON and grateful to be able to share with you some research where you do three things. First, we're releasing a tool to attack next-gen AV, and you can find that GitHub address. I'm going to describe it today and demonstrate it and show you how to use it. So first, let's set the stage. Today we're going to be talking about evading next-gen AV that uses static analysis for detecting Windows PE malware. And to motivate this, let's first talk about rules and how one might write a rule to detect malware. So on this little chart here, I've plotted a bunch of totally fictitious red dots and blue dots, which are meant to represent files as described by first, file size, and second, by the number of registry keys contained, you know, strings contained in the file. A feature of this presentation. And then I'm going to just by hand create a YARA rule, which I have in that black box there, that sort of defines this region of this feature space, file size, number of strings, that cordons off all of the malware in my dataset. So this is nice, but of course it's really easy to break. And if I just, you know, take my malware sample and maybe add some bytes to the end of the file that has no, does not break either the format, the PE file format, and doesn't break the function of the malware, then I can break this rule. So what makes machine learning maybe harder to break, harder to attack? And I guess there are a couple reasons. And one is that you can kind of think of the machine learning as a much more sophisticated and graceful rule. And it learns these complex relationships from the data automatically. And furthermore, it kind of instead of presenting this sort of brittle cliff from malicious to benign, they're sort of the smooth territory where the machine learning model can tell you about its confidence that a sample is either malicious to benign. And this is important because this allows sort of a graceful degradation. If one modifies a malware sample, they're sort of this graceful falloff from malicious to benign. So that can make it hard. So can we break machine learning? Well, the short answer is yes, we can. But the idea for Windows PE is that we like to input a file that our model knows is malicious with high confidence and make a few subtle changes to the bytes on disk, modifying elements that don't break the PE file format or don't break the behavior of the malware, and then trick our model into thinking that's benign. So it's actually become quite fashionable to break machine learning in recent years. And this will be a constant reminder to keep me on my toes. Look at how fast I've done. If you haven't seen this image, this is sort of famous in the image domain that one can take an image of a bus, for example, that image recognition computer vision model knows is a bus with high confidence and change the pixels ever so slightly. And now, even though there's no difference to your eye, now the model thinks it's an ostrich with high confidence.  But there's kind of three takeaways, whether images are not, from this kind of adversarial machine learning research. And the first is that all machine learning models have blind spots. So Next Genevieve, Buzzword, WeDoIt, at my company Endgame, they have blind spots. Number two is depending on how much knowledge an attacker has about your model, they can actually be really convenient to exploit. And the talk we're doing today, the research I'm presenting to you today is actually in the category of the least convenient, the most inconvenient spot for the attacker to attack. And sort of the third takeaway is a little scary. And that's that if I find for my model this sort of bus ostrich confusion example, there's a decent chance that it will also work against your model. So that an attacker doesn't really need, often need to attack your model in order to find some success rate of evasion against your model. And that keeps people up at night. All right, so that's for images. That's for images for bus ostrich. The thing about this, really how this works is that in those cases, there are two things. The attacker knows everything about your model. He sort of has the source code to the model. He knows the weights, he knows the parameters. And in fact, it has to be a special kind of model like deep learning and neural nets that are fully differentiable. And given that, for my image of a bus, I can actually ask the model, what would confuse you the most? Tell me which pixels I should change to confuse you the most? And we'll happily give you an answer. And by changing those few pixels, the good news is by changing pixels, I have not broken what it means to be an image. But let's think about applying this now to PE malware. If I were to present some model with bytes from a malware sample and ask it what bytes or what feature should I change, and I, you know, I change those bytes on disk, well, at worst, I've totally broken the PE file format. And at best, I've totally broken what the malware was intended to do. So two things, requires full knowledge. You have to know everything about a deep learning model. And the samples it generates are not necessarily malware, in fact, are not necessarily PE files. So a kind of cooler attack that's based on a black box, so it doesn't need to be deeply learning, can be any sort of machine learning model that reports a score to you, has been investigated by my co-researchers at the University of California. And essentially it's based on genetic algorithms. And just in a nutshell, you know, these are based on the evolutionary principles of survival of the fittest. And I start with a big batch of malware and sort of breed it with benignware. So elements of a malicious sample will take structures or elements, in this case, for PDF malware. And it will insert elements randomly, mutate sort of the DNA of the malware, and pass it back to the model. And if I see that it's decreasing its score, well, then I'm going to keep that malware sample around for the next round, the next generation of breeding. And after doing this, you know, for two weeks you can evade these kind of classifiers. Now the difficulty of this, however, two things, I have to have a model that reports a score. It has to give me a number between zero and one. Not just a malicious benign, it has to say 90 percent malicious or 20 percent malicious, right? And secondly, there, in this process, it's very possible, quite possible that some mutated variant of malware actually doesn't do the malicious behavior. So my colleagues at the University of Virginia have used a sandbox, an oracle, to make sure that before mutation and after mutation, that behavior has not changed. And that can be quite expensive and is why this kind of attack can take so long. So I'm setting the stage here. I hope you realize, I'm trying to paint a picture by why this is hard for P.E. malware to attack machine learning. We want to avoid requiring full knowledge about, you know, a deep learning model or any other kind of model. In fact, we don't want to care what kind of model we're attacking or even that it is a machine learning model. Secondly, we want to make sure that whatever malware we produce by attacking this model maintains file format and maintains functionality. And thirdly, we don't want to, we want to avoid the expense of running things through a sandbox to check to see if they are where possible. So our goal is to design an AI buzzword in the title, but true, design an artificially intelligent agent that will learn to play a game against your machine learning model. It will choose mutations that are known to preserve file format and function. And for this, we're going to turn to reinforcement learning. And to do that, I'm going to hopefully not insult your retro childhood or current retro lifestyle and explain to you the game of Atari Breakout in two sentences. So this is a game where you move a paddle left to right and you hope to bounce the ball off your paddle and make it launch towards a brick and every time you knock down a brick, you get a reward for knocking down that brick, right? So how would I build an AI agent for this based on machine learning? Well, one way to do that that has been done by the folks at OpenAI is to wrap it in so called a reinforcement learning framework. And it's actually really simple. So I've got a screen shot for my environment that includes the display of the Atari output. It includes an ability to manipulate the paddle left to right or do nothing.  It gives me a reward every time I knock down a brick. And then I train an agent on the bottom side and here the agent learns through some sort of delayed feedback. So given a state of the environment, which is literally like a screen shot of Atari gameplay where it supposedly can learn the position of the ball and of the bricks and of the paddle, it needs to choose the best action. Choose to go left or choose to go right. And based on that, eventually it may receive some sort of award for doing an action that resulted in a reward. So the basic idea here is after playing thousands of thousands of games, then the player can learn an answer to the question, you know, what action is most useful given a screen shot from Atari gameplay? So this is a fun problem. You can actually go and download an AI for Atari from that website at OpenAI that will be better than you at Atari breakout. We're going to change this to play a new game. Let me first describe to you why we wrap this in reinforcement learning. So in the Atari example, when I move my pedal right, there is no reward for that. I get no points, right? But I'll move it again right by chance, move it left, you know, by chance, move it right and by some stroke of luck the ball bounces off my paddle. Again, no points. I move right again but eventually that ball goes and breaks a brick and I get some point. Now in isolation, none of these moves were actually useful and resulted in an award but because I got this sort of eventual reward for all of my moves, I'm going to distribute and I'm going to sort of reward that sequence of actions as having provided some kind of useful benefit. And so this is the same thing, this very same concept we're going to use to break next gen AV. So here's the new game. Instead of a screen shot of Atari pixels, we'll have a malware sample. The scoring mechanism will be my next gen AV. It will give me a score, sorry, instead of a score it's going to say yes, I believe you're malware or no, I believe you're benign where. And the agent now will learn to select from a buffet of options that are known to preserve the file format and the function of the malware by manipulating statically the binary and disk. And by playing thousands and thousands of games, the hope is that the agent can sort of learn basic ideas that given this kind of malware sample I should add an import or I should append to the overlay or I should create a new entry point and use a trampoline to get to the old entry point. Things like that that can sort of hide the presence of malicious activity by creating camouflage in the binary. So we are releasing a tool to do this and you can go to Github in-game ink gem malware and download some very rudimentary codes to do just this. We have provided the following sort of the following elements of this gameplay. This literally is a gem that can be used in the open AI framework for creating your own reinforcement learning agent. And we've provided some very basic ones in there to begin with. But it works like this. So in the case of Atari, the state was a screenshot. In our case, the state will be a feature vector that sort of summarizes poorly but, you know, coarsely the state of the malware. So what is the malware look like that I'm using to attack the next gen AV? That feature vector is based on general file information, header info, section characteristics, strings, file byte and file entropy. Things that are often used in static malware classification by next gen AV. And we're going to feed that into a neural network that will learn this state. So given this state, what's my best action? The actions that I can choose. Right now we've concluded just, you know, our buffet has just a few options. I can create an entry point, create sections, I can add bytes and places that don't break the file format or functionality or modify things that are not known. So, you know, these idempotent operations of packing and unpacking that don't change the behavior of the malware but change how it is presented to a malware classifier. And we're using the very cool tool called LEAF, the library to instrument extable formats by Quark's lab, a shout out to them. And finally, we are also, you know, included in this repo is sort of a toy next gen AV. It's a decent toy. It's worthwhile to attack and see how it performs. The key here is that this game doesn't care what you put in that black box. It could be our toy model. You could rip it out, put in your own next gen AV model. It could be a traditional antivirus engine or whatever. At the end of the day, you just have to retrofit it so that it will report a zero or a one. One for malicious, zero for benign. All right. So, let me just demonstrate how this worked on some samples here. But first, just to drive home how hard this is, you know, I have, the agent has a very incomplete state of the world. The malware he inspects through a feature vector that is noisy and not at all perfect. His actions are stochastic in nature. So, for like Atari gameplay, I can say move right, but I don't know how far the paddle is going to move right. There's a similar thing here. I'll say add an import, but it's going to choose randomly from a list of known benign imports. So, there's this random nature. And furthermore, I know nothing about the model I'm attacking. So, this might be a little bit like you trying to solve, you know, maze or traverse a maze without a map and wearing kaleidoscope glasses and well intoxicated, which I probably describe a lot of those, I don't know. But this is a really, really difficult problem. Nevertheless, we hope that we can learn. So, you probably can't see this. I mean, I just, whoa, I can hardly see that. And so, what I'm showing here is just the output and two examples. At first, the model is just totally guessing at random and getting nowhere. And after I wait for several minutes, the model through his exploration process catches a lucky break and he creates a new entry point which evades the machine learning model on that malware sample and he updates how he learns to evade that model. And by getting lucky enough, many, many times over tens of thousands of games, then, you know, with this sort of rudimentary model that we put in place, we can begin to learn to break NextGen AV. So, here are the results. In one minute, given a batch of malware samples that neither the agent nor the model have ever before seen, we could modify those samples with our agent that has learned to play the game against the NextGen AV and 16% of those snuck past. Furthermore, do you remember how I don't necessarily have to attack your model to bypass a different model? We uploaded those samples to Virus Total, both the pre-modification and the post-modification. Pre-modification, 35 out of 62 caught those samples. After our agent got a hold of them, there were 10 additional antivirus engines that whiffed on those malware samples. So that's pretty cool. We also ran sort of random mutations. We want to make sure our agent was learning something and not just getting lucky all the time. So we did the lucky experiment and, you know, it turns out lucky is pretty good too. But the agent's about 50% better than lucky. All right. We're done. The summary is this. You can go to GitHub, in-gameink, Jim Malware and try this game for yourself. No knowledge of the target models needed will manipulate raw binaries and produce new binaries this world has never seen, some fraction of which may evade your machine learning model. I hope that people will contribute and make it better. We use these things at in-game to help harden our models. Stepping back a moment, it turns out that machine learning is actually fairly robust. Even under direct attack, the machine learning models warded off most of these attacks. Nevertheless, all models have blind spots, so don't buy into the hype. And with that, thank you.