Eric Cappuano, everybody. Okay, I appreciate it guys. Hey, so thanks a lot for coming to my talk, especially being, you know, noon on Sunday, final day of con. I know that a lot of people are already on planes, heading home, so the fact that you guys are here with me is very humbling, so thank you. I'll just go straight into it. So obviously you guys know you're here because we all know that tabletop scenarios are not exactly the key way to train for incidents, right? But what do you do about that? If that's all you know, then what other option do you have? So what I'm gonna talk to you guys about is something that I've actually done personally with my Incident Response Team because I recognized there was a gap that needed to be filled. Tabletop is not it. And so I built an Incident Response Simulation Platform. Now just so you guys know, you probably saw on the summary of the talk, this isn't gonna be like just, you know, waves and waves and waves of data and things that you've never heard of before. A lot of you are probably familiar with virtual environments and things of that nature, but I am gonna go a little bit deeper and go over some of the tips and tricks that I learned in my painstaking process to build this environment to achieve high fidelity training, right? There's a difference between just building a bunch of VMs and hacking them. No, in order to really get the most value out of it, there's some things that you wanna do to as closely simulate your enterprise environment as possible. So quick introduction should be next. So a little bit about me. I am the sock lead for the Texas Department of Public Safety. We have 12,000 endpoints, over 467 remote locations across the entire state of Texas. So it's a very large network. We are the state police, the Texas Rangers, Department of Emergency Management, driver's license. I mean, we've got a lot of stuff, so that's a busy job. I'm also a cyber warfare operator for the Texas Air National Guard. Cyber, cyber. In my spare time that doesn't exist, I'm also a private consultant, and I do information security for small and medium businesses in central Texas and surrounding areas. I'm also a very proud member of the packet hacking village where I run the emerging threats, honey pots and deception systems. So if anybody here got to play in the honey pot challenges, that's mine. I built that with the help of some really awesome people. I'm a cyber patriot instructor because I believe that anyone in this field, IT security doesn't matter, pay it forward because those of us were once in junior high and high school, didn't have somebody to come and teach us this awesome stuff. We had to figure it out. Well now you can be the one teaching the next generation of professionals. I'm a part time blog author, I don't write nearly as much, but security against obscurity is my blog. You see the link there. And lastly and most importantly, I'm just a hobbyist teacher student just like the rest of you. And as you can see, everything revolves around caffeine. So ask yourself this and be honest. Don't tell me, but tell yourself the truth. Is your IR team ready for a worst case scenario? How would you know? How often do you encounter worst case scenario? We're not talking about macros and viruses and emails. I'm talking about APT on the network, X filling, sensitive data. What do you do? Does your team have the skill sets that are required to operate in an environment in a situation like that?  Or do they all run off into separate rabbit holes and stop talking? Which is probably more likely. That kind of rolls into the next one. Do they communicate, right? Because I know how my team is, well the multiple teams I'm on, when they get to chasing something, they go quiet, they go silent, verbally, and if you're in Slack or some kind of messaging platform, quiet. We're in the middle of a breach. Why are we quiet? We should be talking. We should be sharing information. I know you're seeing things. I'm seeing things. He's seeing things. Why aren't we talking about it? So how well are they communicating? And then lastly, organization, right? And I'm talking, yeah, the boring stuff, the administrative, the hierarchy of the incident response team. Is there a leader? Does that leader know his role? Do the followers know their roles? And are they properly executing? Or is everybody just kind of free for all? And I hope you find something. Hope I find something. So those are the things you should be asking yourself. Because look, even if your daily job is working in a sock, you're like, oh, I'm not new to this. I respond to incidents all day long. Yeah, you're responding to those routine mundane incidents that you can do with your eyes closed, right? But it's a totally different situation when you find out that there is a persistent thread on the network. Maybe it's been here for three months and you just found it. That's not mundane. That's not routine. And neither should your response to such an incident.  Obviously the point that I'm making here is that a tabletop scenario cannot prepare you or your team for this. Now it's better than doing nothing. I will give you that. To have a sit down and talk to your team and say, okay guys, here's what's happened. There's a hacker on the network right now. And he's attacked our SQL database. And we're pretty sure he's exfilling records right now over DNS. What would you do? We'd block him, we'd stop him. We'd put in a firewall rule and we'd stop him. Okay, cool, awesome. Prove it. Like, how would you one, even have detected this to begin with? Two, how would you identify what is even happening? You know, okay great, we see a huge influx of DNS activity, but we don't know what it is. Hackers aren't using clear text. Wall of sheep victims use clear text. But how are you gonna figure out what's really happening and respond accurately? Because in a business environment, you can't walk in the bull in a china shop mentality and just start throwing firewall rules in. You stop business, you're in trouble, right? Now you got two incidents on your hands. So tabletop is great, it's a good start. It can help you identify weaknesses and gaps in your capabilities or your technologies, but it's not enough to really validate, can you function with a sense of urgency when it has hit the fan? Are you validating the procedures that you rely on so closely?  You're getting DDoS'd, what would you do? Well, we'd follow the procedure. When's the last time that procedure's ever been put to the test in real DDoS? No, some CISSP wrote it, hoping it was gonna be effective. It's never been tested. So you're waiting for the day that you actually get DDoS'd to find out if your procedures are gonna work. Skills, training, right?  I mean, I'm so lucky, I was really, really lucky to have the team that I have. But let me tell you something, even with the experience that all the people in my team have, years and years of experience, the first time we went through our scenario, the first time we sat in the simulation, no kidding, this network's being hacked. You'd think they were all beginners, because all those years of experience and knowledge and all that stuff, for some reason it almost goes out the window and all of a sudden you're panicking. Oh my gosh, like stop this attack. Like yes, they're X filling records. Even though it's a simulation, it's funny because it has the same effect. You put a time constraint on this, be like you have six hours, stop the breach. It really does something to the heart rate, you know? And so my team that's not new to this responds to incidents all day every day, all of a sudden couldn't remember that Palo Alto has many different tabs to click on to see traffic logs, threat logs, URL filter logs, they forgot all of that and they did not find the X fill.  And then lastly, right? Tabletop, since you're not validating or actually doing anything, it's not gonna highlight where your technology might fail you. It's not gonna highlight where your personnel might not have enough coverage. You got one guy on your incident response team? Sure, he's gonna do great at tabletop. He just has to answer questions and I'll throw him into the mix with a real incident and see how many hats he's gonna have to wear and how quickly he's gonna quit. Doesn't work. And you know, one quick point. Think of this, the tabletop, why it's inadequate. Let's take something like paramedics for instance, first responders. Imagine if first responders trained to respond to life-threatening situations simply by sitting down with colleagues and talking about it, you know?  They're bleeding profusely, what would you do?  Okay, cool, awesome, that's what we'll do. No, that's not, that's not how it goes. They go and they train. Is anyone here CPR qualified? Awesome, what did you have to do in your training to become CPR qualified? Physical testing, on a dummy, right? Why couldn't you just watch some videos, taking a CBT, muscle memory, absolutely. Because there is no CBT, there's no video that's gonna give you that sense of feel. What does it feel like to do chest compressions, right? What does it feel like to breathe air into an unresponsive person?   But you know what you won't forget is that muscle memory of, that's gonna come in handy, that's gonna come in handy. So it's the same concept, it's the exact same concept. Firefighters don't learn to fight fire by watching videos on YouTube. Because when they run into a burning building, fear sets in, panic, you know? So that training has to be muscle memory, it has to be instinctive. Incident response is the same way. No, I'm not saying that incident responders, you know, are paramedics or are firefighters, no, lots of respect for what those people do. But incident response is very similar in the nature that if you don't know how to do it in the calm, you're definitely not gonna know how to do it in the chaos. So let's get to the good part. And just so you know, this is not a very slide heavy presentation, I didn't want it to be. So I'm gonna go, I'm gonna pull up some diagrams and I'm gonna talk about them for a while, so I hope y'all are good with that. So my advice for how to effectively train for digital forensics and incident response is attack your own network, sort of. Obviously we don't want to do this in production. So we build it, right? We build an environment, and let me tell you something. I was able to build a pretty sizable environment on two bare metal ES6I servers. We're not talking about a cluster. I'm not talking about a full rack in a data center. I'm talking about two 1U servers. I was able to build a very, very high fidelity and elaborate environment. So this is something you could do on a budget. This is something you can do with yesteryear's hardware that you've replaced with your latest refresh and it's just sitting in a closet somewhere, grab that stuff, bring it into an area and start building out an environment. And I'm gonna go over some of the things that you're gonna wanna think about when you're doing this because yes, there's a right and wrong way to go about it. So I'm gonna kinda start from the left and work over. So my decision was to actually have the ability to connect this range up to the no kidding internet. And the reason that I wanted to do that is because that makes it way easier for me to have organic traffic generation. Because think about it, right? If you're responding to an incident and you're coming through firewall logs and all kinds of good stuff, it'd be really easy to find a bad guy if there was no web traffic, no activity at all. You just sit there and just wait for the first log to roll by and oh, there it is, we got it. No, for a high fidelity environment, you need noise just like in the real world. Who here has ever tailed the logs of their perimeter firewall? Maybe once, right?  So you need to reproduce that or else you're not doing good training. So I decided to have, okay, I wanted the ability to connect this range up to the internet because when I get to the part later about how we generate traffic, you'll see that internet access is kinda required. However, I'll give you a trick here in a minute just in case you don't have that internet access. It can still be done. And also, I wanted to share a quick resource. So while building this, I had to solve some problems like how do I generate traffic? I wrote a Python script. It's in my GitHub. If you can just remember my first initial E and my last name, which is almost impossible to remember, you can find me on GitHub, a web traffic generator. It's just a little simple Python script that you can feed a seed of URLs to and it will literally headlessly browse the internet just and it'll go a specific depth. So say start at these five websites and go to a depth of 10. So it'll start at yahoo.com and it's gonna click 10 links in. And so it's a really cool way to generate organic web traffic. Because guys, you cannot leverage canned TCP dump like playbacks. And let me tell you why. Because what we're using here, just like in our real environment, are stateful layer seven firewalls, right? Layer seven firewalls, well actually any stateful firewall is not gonna play nicely with your canned TCP dump traffic. You're not gonna just be able to replay PCAPs and have these firewalls pass them. It's not gonna work. So unless you've got a breaking point, which I usually don't drop brand name, sorry. But unless you've got a very expensive system that was built to do that, you're not gonna be able to. So what you need is you need real traffic.  So anyway, so traffic generation and I have another tip for you guys on that one in a minute that's even easier than my script. The next component in this mix would be what would be your external router, but this dotted line right here indicates kind of where this range starts and stops from the perspective of your C-Cert responders. They are not aware of anything to the left of this dotted line. So this router here serves a very unique purpose. You're using that and that could be anything. That could be a Vios router previously or Viota. That could also be a PF since firewall. You don't need any special gadgetry here, but you do need something that can route because what you're doing there is route manipulation. So again, the effectiveness of this training is gonna be how well can we emulate the things that you're gonna see in the real world? Well, it's really hard to emulate C2 traffic to Russia unless you have a VPS in Russia. So what we can do is in this router here that your participants aren't even aware of, in this router we're setting up some destination NATs, right? You can NAT an entire slash 16 if you wanted to of a known Russian IP space.  And so what I can do then is I can have C2 from any of these systems here going to a no-kitting Russian IP, but it's not even leaving my range. It's just once it hits this router, it's getting destination NAT it to a server that I've got listening up here. Why is that good to do? Why is that useful? Well, because we want to generate the same or similar types of artifacts that an incident response team should be looking for. So if you can't simulate traffic to Russia, well, I mean, that's definitely something that would add a lot of value to the training. So, but again, I mean Russia, that was kind of an obvious point, but maybe something less obvious, but the point is if your organization doesn't do business outside of the country and you throw something in there, even something simple, Great Britain, UK, France, whatever, it should still be anomalous. If your incident response team is looking at it, hey, why are we talking to this country that we don't even do business with? So anyway, so the other thing that we're leveraging this router here for is you can hijack DNS. So if for whatever reason, you want to stand up a site like a paste bin clone, which actually we did in our scenario, we created PoneBin.ru, which our red team infiltrators use to Xfill data. Well, to create that record outside of this environment, because you don't want your incident response team to log into the firewall and be like, oh look, there's a static DNS entry for PoneBin.ru. I bet that's going to be used for something.  I could hijack Google, I could hijack Yahoo, I could hijack anything I want. So now I can further manipulate what's going on in this environment to make it do what I want it to to achieve my training objectives. The next thing, oh and also I wanted to highlight some of the resources that I'm using in my environment. So obviously, this is also where your attackers live, so you're going to need someone to red team this engagement. And so your red team guy literally could have one Metasploit box, one Cali system sitting up here in the fake internet and he's got a handful of interpreter handlers on it. And all the C2 and all the bad stuff that's going on in here is literally, it could appear to be going to 100 different IP addresses, but it's all getting added to this one guy up here and one dude's red teaming this entire engagement, which is also a ton of fun. But you know, Apache web server running a bunch of virtual servers for your special sites and services. You want to fake Twitter? Set up a fake Twitter up here. You want to fake Facebook? You know, you can do all of that up here using very simple things. Another thing that I'm doing in my fake internet is, so to really add additional realism to this environment, email, right? Email is the number one vector.  So you need the ability to send email into this environment. So we've got a Python script, literally, that's just sending more Ipsum every minute to every single user in our fake environment. That way, when we send that one malicious email through, it just blends in. It just blends right in with all the other noise. So, you know, we're leveraging some pretty simple concepts here. Okay, so the next piece in this puzzle is your perimeter firewall. Now here's the thing. I meant to touch on this a little earlier, but so here's the thing. When you're approaching this and you're going to build an environment, there's a component you need to figure out first. If you're building this on the cheap, obviously, you're going to lean more towards open source solutions, which if you've been to my previous talk, I'm all about open source is awesome, right? A lot of really powerful commercial tools run on open source engines, so what's the difference? However, know that, you know, the effectiveness of the training is going to be tied to how closely does this mimic your production environment. So there's a lot of value in trying to reproduce. So for instance, if you use, you know, Fortigate firewalls, it would probably be a good idea to try to get Fortigate virtual machines in here and use those because that is another way to one, train your incident response team, to two, validate how good are my Fortigates anyway? Have you ever really put them to the test? Do it in here where the stakes are not high. Well, know that you don't necessarily need to go out and buy licenses for all this stuff. Tell your vendor what you're doing. Hey, John, Fortigate, listen, man, I'm building a training environment to make my guys better at what they do. Can I get a complementary license? I'm gonna tell you right now, guys, I've probably been given $200, $300,000 worth of free licenses for my training environment. Vendors don't care. You tell them it's a lab, it's offline, it's not production, you're just training with it, they will throw product at you. So don't be afraid to ask, okay? So when you're approaching this and you're deciding do I wanna use, you know, Palo Alto, Fortigate, ESAs, whatever, talk to your vendors, you're more than likely gonna be given some leeway there. But if you do go the open source route, I'm gonna tell you right now, you can achieve everything that I'm talking about here with open source solutions. Short of maybe your Windows workstations that you're gonna wanna break into over here eventually. So moving on, so perimeter firewall, you can totally roll like a PF sense if you want to, or you can throw, like I said, a virtual appliance of, or even a physical appliance. If you have a little lab unit sitting around, fork that into this environment and use that. But what that perimeter firewall's gonna do is just like in your production environment, it's gonna have a DMZ probably with some web applications, some web servers, things like that. Maybe hang a mail security appliance. Whatever you're doing in production, you need to do here, it's really the extent of it. And hopefully just like in production, you're running IDS IPS here, right? To simplify, the next piece of it is really just like one giant core router to rule them all. And it could be a copy of the same device you've got here, or it could be a different device, it doesn't really matter, but that's where you're gonna create all your different virtual networks to mimic your different branch offices, or divisions, or whatever subsections you have in your organization. And the reason you wanna do this is because, remember, during our incident response training, we wanna create anomalies. We wanna create things that are out of the ordinary, like someone in HR hammering a SQL server. Well, in order to mimic that, we need to have an HR subnet, right? And then the C-Cert needs to understand that that is, you know, what that is. On your core router, it also would be ideal, hopefully just like in production, to have IDS IPS here, so that you've got visibility on not only your North-South traffic coming into and leaving the environment, but East-West visibility on HR talking to the server farm, you know, or IT talking to marketing and sales, right? Now, again, this is cool because when you're architecting this, this very simple network, it starts to prompt questions in your mind of, well, do we really have that visibility in our environment? I don't think we do, you know? We've got a giant, you know, Cisco Nexus, you know, big iron router in our core, and it's not inspecting anything, it's just routing. So we wouldn't know if HR was beating up a SQL server. So this is cool because you're gonna accidentally start to think about the gaps you've got in your own environment as you're thinking about how to build this one. But for the fidelity of the training environment, you know, have these sensors deployed so that they're inspecting everything because we wanna have that flood of information for our C-Cert to have to sift through to look for evil. So server farm, obvious, I mean, I don't really need to go into detail to you guys about what needs to be in a server farm in a corporate environment, but you just wanna have those important items like a domain controller, exchange slash mail server, right? A file server, some kind, SIFS, NFS, Samba, whatever. And, you know, in my environment, we rolled a SQL server, you know, because most organizations have large databases, and so we wanted to have one too. Branch offices, self-explanatory, but I'm gonna go over some cool tips and tricks here. So back to that organic traffic generation, right? So first and foremost, you know, what do you put out here? What do you put in these subnets? Well, ideally, you're gonna take a copy of the golden image that your organization uses, you know, IT has probably some image that they use to image new systems, get a copy of that, use that one and put it in here. Because now, during your training, you're gonna be validating the security of that image. How secure is that image? Have you ever really put it to the test?  Well, you're gonna get to in this environment.  There's a Chrome extension called Chaff, and this thing is awesome. And it literally, my traffic generation script, it does that, it does it actually even better. And of course, I didn't find Chaff until after I'd spent, you know, like seven hours writing mine. But Chaff is actually really cool because you can install Chrome on each of these workstations. You can tell Chaff, you can tell Chrome to start it boot, auto-login, start it boot, and Chaff will start automatically. And Chaff will literally do everything a user will do. Browse the internet, download Dumb Files, go to YouTube, stream some stuff. I've seen Chaff end up in some pretty risky places of the internet, which is perfect because during your incident response simulation, you're gonna see IPS signatures firing that you're like, I don't know what that is. It's one of the Chaff users. And that's perfect because that's what real users do.  Which is what it takes in order to be effective at incident response. There's gotta be a lot of noise. So Chaff, that's one of my little secrets there. But also it's important that whatever endpoint agents you're using, if you've got an AV, a HIP, a HIDs or whatever, a log forwarder, you wanna get that on this image as well obviously because you're gonna need data.  We'll cover that in a second too. So get those endpoints configured. You're gonna need a mail client on them because remember, mail, gotta have it. You're gonna use it in these simulations the scenarios that we're gonna go over in a minute. And then obviously, you want all these guys joined to a domain, that's self-explanatory. Okay, so lastly for this environment, you're gonna have to build out your subnet for your incident responders. And so typically, I hang that off the same core router but it is a totally separate subnet. That way, the ACLs, it's permit, permit, any, any, all that good stuff, no IDS, no IPS because you just want your guys to be able to have a straight shot into the environment because later on as you mature here, right, after you get past the stage of detection, can you find the evil? Okay, great, you found it. Now go ahead and leverage your WEMIC, your PowerShell, whatever, reach into that box. Let's get volatile data. Let's get event logs. Let's get a memory image. Then you can actually start getting into active response on these systems. And so it's good to have them on a separate subnet so that you can permit them to do all of these things. So I'm gonna go over a couple more solutions that you guys will want to look into for building out your environment. And actually to be honest with you, these are solutions you can look into putting into your production environment. So if anyone here is familiar with Elasticsearch, right? Elasticsearch Logstash has a really cool log shipper called Beats. And there's two flavors of it actually. So there's FileBeats, which is just an agent. You can put on a system and you could point it at any log that you want. It could be a custom log that you wrote for custom application or it could be a Windows event log. It could be syslog. It could be auth. It could be any log you want. And FileBeats can actually ship those logs to a central repository using TCP with TLS, like all kinds of cool stuff. So it's kind of a step up from your simple R syslog forwarder, right? Which is UDP, it's plain text. It's what happens if your syslog server goes down? Well, syslog keeps forwarding. It doesn't care. It's gonna keep forwarding. Well, Beats is a little smarter than that.  So really cool tool. And it integrates really nicely until Elasticsearch Logstash. ClamAV, which most of you guys are familiar with, SourceFire. It's ClamAV under the hood. Who knew? OSEC, which is an open source HIDs agent. Very, very, very, very powerful. That means this is an awesome tool. Millions of customization options. You put it on your hosts and it's looking for root kids. It's looking for evil and it's reporting back to a central server. So I'm just going over some of the things that you can look at using in your environment that cost nothing. How about your incident response team, right? Maybe you're in an organization with a ton of money and you've got a really cool SIM. Maybe you're not. The good news is you're not out of luck. Graylog, very, very powerful tool. I mean, it is very close to an open source Splunk. Not to be confused, Splunk is awesome. Splunk is extremely powerful, but Graylog comes pretty close and the price is right. Free $100,000. Alien Vault OSIM. If anyone's familiar with OSIM, extremely powerful SIM. It's got vulnerability scanning, passive asset detection, all kinds of good stuff. You could put that, you could stand that up in this environment for free. The Hive, if you saw my talk yesterday, I covered the Hive. Really awesome incident response collaboration tool. So if you've got a team of analysts that are responding to an incident, the Hive is an awesome open source platform for them to collaborate on. Check it out. So I just wanted to share a couple of tools that you guys could use if you didn't already have these systems in place. Okay, so now that you have your environment, now you have to start scripting your scenarios.  It's gotta be a surprise. So I actually had my Intel analyst draft up these Intel reports. They're totally bogus. I mean, it's like 95% fluff and about 5% actionable Intel, just like in real life, right? So we dreamt up this scenario and it's important that you pick one that's applicable to your organization, right? Like, so you know what kind of threats your organization faces? I'm a law enforcement agency. We deal with hacktivists, you know? People that don't like law enforcement agencies. And so we dreamt up a hacktivist scenario where there's this group like Anonymous or something that really wants to embarrass large law enforcement organizations and we've got all their TTPs and some observables that have been discovered in the wild and all this stuff. And in all reality, we really are only gonna use one or two of the indicators that we provide to these guys.  But this is to test your team. Are they paying attention? Are they looking at the relevant Intel that's available for a particular incident? Or are they like, oh yeah, great, yeah, Intel. Okay, cool, no, I can find bad. I've done this before, I'll find it. So this is a test on are they ingesting the information that's available to them to make them more effective at finding a particular threat to the organization. Yes, and we'll go over in one of the scenarios ways that you can really weigh in on, you can test whether or not they're paying attention to things like this. So now I kinda wanna dive into a couple actual scenarios that I've developed that were really successful in training our team. So first foremost though, I wanna preface with this. You're red team injects.  You can't just wing it. It needs to be scripted, you need to have a playbook that has repeatable and measurable impacts because for it to be an effective training solution for every red team inject, there needs to be a blue team observable. So as you're going through and you're mapping out your scenario, you literally need to wear both hats. So okay, I'm gonna try this attack on that system. What did I see on this system when I did that? Oh okay, or what did I see in my SIM? What did I see in my logs? What did I see here and there? So you need to make sure that everything that you're scripting for your injects is something that your blue team can't even find. If they can't find it, why not? That's a gap in technology. So another thing, if you go off script during a red team engagement in an environment like this, well now your blue team, you're not gonna know if they found the bad stuff or not because you don't even know what artifacts you're generating. So that's why it's important to script this out. So that's what we have. We've got playbooks for different scenarios that we're gonna run through. So for instance, and I'll just kind of blaze through these, but one scenario that we have is we're gonna kick off our traffic generation. So we got a ton of noise. We get emails flooding into the network every minute. I mean, hundreds of emails. We got traffic generation. And then we're gonna socially engineer a DBA. We got a database administrator, poor Regina Buckley. She's been with the organization for years and not the most security aware, but she does all right. Well, we're gonna fish here because she likes talking about what she does on LinkedIn and Twitter and Facebook. And so we found her. We're gonna spearfisher. And we're gonna send her an email that very urgently tells her to install a Windows update from vindosupdate.com. Which it's funny, you can, you know, when you're manipulating DNS, you can create any type of DNS record that you want. So vvindosupdate.com plays really well into your scenario because a lazy analyst that's just kind of skimming through logs doesn't see the vv, right? The simple things that attackers do. They're playing on your lack of attention to detail. So when your red team injects, you wanna do the same thing. Do things that are gonna force your analysts to pay attention to what they're looking at, not to glaze over it. So she downloads this update, which actually of course is a reverse shell that's gonna give our red team attacker a shell on her system. So she's here, we're here. We now have a shell on her system. Escalate privileges, get her credentials. We already know she's a database admin so we already know what access she has. So what we're gonna do next is pretty obvious. We're gonna start enumerating the network. Now remember, this is why it was important that we have east-west inspection because if we just had a direct link to the SQL server and we started scanning, our blue team's not gonna see that, right? So we have inspection for east-west and we're intentionally gonna generate some noise. We're gonna scan, we're scanning the server farm for the presence of a SQL server. That's a blue team observable. You know, port scan for 3306. They should find that, they should see that. Well, we found it, we found our SQL server. So from Regina Buckley's system, we now have our creds. We're just gonna go ahead and log right into that SQL server and we're gonna start ex-filling records. Now here's the thing, anybody here, especially red teamers, there are intentional dumbing down of red team activities because again, when you're first getting started with those guys, your team doesn't know what's going on, okay?  So you need to do some things earlier on to enable them, to build up their confidence, to help them find, to know what to look for. So what we're gonna do is we're gonna grab all the SQL records and we're just gonna pull them over to Regina Buckley's system, even though we could have just pulled them right through our interpreter shell and been gone. We're gonna pull them to Regina Buckley's system. Through our interpreter shell, we're gonna drop curl onto her box. And remember, in our fake internet here, we stood up a paste bin clone called PoneBin.ru. From Regina's box, we are going to, via the API for PoneBin, we're just gonna post 5,000 sensitive records, right to PoneBin.  So I'm just gonna post 5,000 sensitive records, right to PoneBin.ru. So what additional observables have we generated? Well, we dropped a binary on the system in a protected directory. When we ex-filled data from the SQL server through the network to Regina Buckley's box, if you've got a halfway decent IDS, it picked up on the fact that 5,000 PII records just traversed the network in the clear. That should be noticeable. Then we did it again. We curled in clear text from Regina's system via an API, we ex-filled all 5,000 of those records. Now did we do it the same way in Attacker would? No, of course not. And Attacker would not have used any of those methods. But remember, to start, your team will be, you're lucky if your team's gonna see any of that, if your traffic generation's good, if all the noise is there, if your team sees the scenario I just described, you're better than most. So you gotta practice with the intentionally obvious scenarios first, let your team get a feel for what does abnormal stuff look like. Then you upgrade to the more sophisticated attacks. Which if you're not gathering this already, this is a ton of fun for the red team too.  And that's a ton of fun too. So that's one scenario that we've had. Yes? Absolutely, step by step. Literally copy and paste into the Metasploit console. We even pre-built Metasploit resource files to automate a lot of the things like, hey enumerate SQL database, well you want that to be 100% repeatable. So I might have one way of enumerating a network, you might have another, but we can't have those variances when we have a scoreboard that's expecting a specific answer. How many systems did the attacker scan while enumerating the network? That's why it needs to be 100% repeatable. So yes, the playbook is extremely detailed. Down to the command that the red team's gonna run for the inject so that we know exactly what the blue team should have seen. So as we start to migrate to more, I guess more complex scenarios, excuse me, dehydrated here. So, or I wouldn't say complex, but different, right? A lot of organizations have, like I said, web servers sitting out here in the DMZ.  You know, if you've got F5s, if you've got IDS tipping points, whatever, are they properly tuned? Are they configured to look for the types of attacks that your systems are most vulnerable to? Well, try it here. So you get a web server, right, that's facing the internet, but it has a backend connection to the SQL server. Well, maybe that's another scenario where you get an attacker that's beating up this web server, running SQL injects against it in order to do the same thing, to X-fill data, to X-fill data. Would your team know what to look for, know how to respond to that type of situation? Now, this is where things can get a little more complicated, right? When you wanna start really upgrading to more sophisticated scenarios to put your team to a tougher test, that's where you do start using things like RC4 encrypted payloads that are not gonna set off IDS IPS.  But then how do you throw your team a bone to see how closely they've been paying attention, right?  But two scenarios later, one of them is, but they haven't even looked at that Intel report in hours, you know, it's in the back of their mind, they're not even thinking about it.  Does your team ever connect the dots to say, wait a second, the threat until we got six hours ago that wasn't relevant then is relevant now, we've had C2 on this system for all day long. Connecting dots like that, that is analysis, paying attention to the details, not forgetting about previous indicators just because maybe they weren't relevant at the time, that's another effective way to test your team to pay attention. This was a really fun one actually, this one I'm describing, what we did was we wrote ransomware, and we had a persistent beacon on one system, we dropped the ransomware executable onto the box, we waited till things got really quiet, the incident response team was distracted looking for rabbits in the rabbit hole, and then we went ahead and we hit the file server, we encrypted every single file on it, and here's the thing, we had given them intel of exactly what we were gonna do. Obviously, you saw the intel document, it didn't look like a really friendly, straight to the point manual because they're not, but we gave them an intel document that said exactly what we were gonna do, they didn't see it coming. What if I walked up to you and said, hey, I'm about to punch you in the face?  Are they using the information that they're given? Are they looking out to gather additional information? You could go so far to use real indicators. So go to OTX, Open Threat Exchange, and find some IP addresses that are tied to malicious activity, won't be hard, and then use those IP addresses in your environment. Now make sure that your blue team knows that hey guys, just so you know, the indicators that we're using are real world, so now they know that they have an option to go and research on OSINT sources. Okay, I think this activity's malicious, I'm not sure. Let's go check virus totals, go check OTX, let's go check Threat Connect, whatever, and now they can actually enrich the intel because you're using real indicators. So I mean, put your folks to the test on some of the more advanced capabilities that you need on an incident response team because just simply playing IP address whack-a-mole and blacklisting things all day, that's not effective, that's not solving problems. So you get these scenarios written and you've tested them, you've validated them, like I said, for every red team inject, there needs to be a blue team observable. How do you measure, how do you validate? And then at the end of the day, how do you make this fun, right? Because if it's not fun, no one's gonna wanna do it, you're not gonna wanna run it. Guys, this is a ton of fun. So when I built this, I built it actually just for my own incident response team. Then B-Sides Austin reached out and said, hey, we want you guys to bring this platform to B-Sides Austin. I'm like, okay, yeah, sure, I'm not gonna say no to that. We brought it to B-Sides Austin and we had over 40 teams come and play through our scenarios. We've had rapid seven guys play through our scenarios and they loved it. We've got overwhelming feedback about it because one, they said, hey, this is awesome because this is real world. This is exactly what it's like being in a sock, looking at incidents versus a CTF, which don't get me wrong, CTFs are a blast, but a lot of CTFs, puzzles, the games, the challenges, the decrypt, all that good stuff, that's fun, but not directly applicable to what we do on a day-to-day basis. Whereas this challenge was meant to be exactly that. It's meant to be directly applicable, but we still want it to be fun, enjoyable. We gamified a little bit. You pull down an open-source CTF scoreboard, like this one here, this is actually Google's. This is the Google CTF framework, and you gamify it.  You say, okay, the first phase of the attack was the infiltration, then there was the enumeration, then there was the attack, well, actually attack came before that, but then the exfiltration, right? So you break it out into those categories, and you make them run through the entire scenario, and you give them little breadcrumbs, right? Because again, it's all about enabling them.  So when they're learning how to walk, you show them how to crawl, point them in the right direction, and then get progressively more difficult as you go on. But like I said, the CTF scoreboard, something like that, it's gonna gamify, it's gonna make it a little more fun, but it also, as someone leading the training, it's gonna give you a true measurement of how well is your team doing? How far are they getting? If they're doing it as individuals, or if they're doing it as a team, you need to be able to measure the impact that you're making. And that is all I have. So I will open it up to questions. I know that there's a lot of technical information in here, and in the essence of time, I have to kind of skim over a lot of things that were actually six months' worth of development and implementation. So I'm perfectly happy to have a conversation with anyone that wants to know more about how to build this, how to run this. I'll even share my playbooks with you, so you don't have to reinvent the wheel. Just let me know. But any questions? Yes. So are you talking about like, communication channels to management of the organization? Or, oh, okay, absolutely, absolutely. So, no, his question is good. It's like, are you including management in these scenarios? So if it's an incident response of a large scale, obviously, there's more than just the incident response team that's involved at this point, right? You're dealing with executives, you're dealing with legal, you're dealing with HR, and so on and so forth. So honestly, that's not something that I've baked into our simulations yet, but 100% should. Absolutely. This is something my team does on a quarterly basis, but we've only been doing it for three quarters. So we're still growing, we're still evolving, but you're 100% right that to really test like, what would we do in this scenario, you're gonna be including your executives, you're gonna be up channeling. So that's kind of part of the organization piece. You know, this isn't just a skills test, guys, because I don't doubt the skills of anyone in this room or any of the people you probably work with. This is a test on when it hits the fan, Katie, you remember the basics, like I don't know, talk to someone, document things, right? Or do we all just fall into that silo and that rabbit hole of, oh, gotta look at the logs, oh, I think that's bad, oh, I think that's bad, oh, I think that's bad, but I haven't written down anything, I haven't documented anything, I haven't told other analysts what I'm looking at, what I'm doing, and I haven't up channeled to the incident response lead. Hey, you know, I think I see indicators pointing this way, and I might not be on task from what the incident response lead asked me to do. So yes, you've gotta factor those things in to really test how effective an IR team is in the midst of it. Yes. So that's something that we're still measuring. So obviously the major investment here, for me, was time, right? To build this, yes, it was a sizable investment in time. I'd say there were about three of us working on it, took us about six months. We were working on it obviously on the side, right? We have daily operation to maintain. So for about six months, we had to build this out. We had to learn a lot of hard lessons, we had a lot of troll-shitting. Once it was up and running, it actually became very easy because now all we're doing is we're changing the scenario. Hey, instead of a DNSX fill, next time we're gonna do this, we're gonna do that. So writing the scenarios now is the upkeep, and that's actually probably the most fun and one of the easier pieces of it. So the upkeep element hasn't really been an issue. Like I said, the hardware, I'm using recycled power edges. These things are nothing special. So it was a very small investment in hardware, and like I said, if you have a good relationship with your vendors, you can go and you can get OVAs or probably even bare metal like lab units of the appliances that you're using in production and get those in the environment for no cost at all.   If you don't have the resources on your team, you know, with solid networking skills, solid sysadmin skills and things like that, okay, you might have a tough time.  But I think I could say pretty safely that a lot of security folks came from other areas, right? Sysadmin, network engineering, things like that. And so again, I'm fortunate I have a pretty, pretty wide skill set across my team. So I have network engineers, I have sysadmin. And so it enabled us to build this without any outside help at all, but your mileage may vary. Okay, so that would be tough for me because I have 12,000 users. I don't have those types of resources for this environment, right? I cannot simulate 12,000 people. So no, obviously I'm definitely scaling down a lot, but I have found that as long as your traffic generation is organic and it's good and you're, put in some other things. Like I think in addition to the HTTP traffic, we also put some headless Python scripts that are like doing all kinds of other weird stuff, you know, pings, ICMP. And remember we had a file share in the environment, right? So one of the GPOs that we put in the domain was at boot, all those systems mount the file share. So now you've got a lot of east-west, you know, SMB traffic. So you wanna make sure that you're generating as much normal activity as possible and that takes a little bit of ingenuity. Because if you can do that, you'll find that at least we found that it made plenty of noise. And I think we emulate about 35 or 40 user workstations, it makes plenty of noise. So it's effective for what we're doing. Cause if you can effectively filter out 40 active users, it's the same concept to filter out 10,000. You know, it's the same filtering principles, right? So we found it to be effective. What time it is? Oh, absolutely. Yeah, so from just a good security posture, you should have sensors that are detecting user behavior, you know, abnormalities, right?  Okay, guys, it's 1 a.m. Regina Buckley is beating up the database. Now, Regina Buckley is a database administrator. So at first glance, that might be normal, but not at 1 a.m., right? So absolutely, factor these things in. The things that should be unusual, that should be anomalous, that's what you're testing your team on detecting, right? Another one that I like to use. So we have a scenario that we called no worries, AV caught it. I love this one, I love this one, because here's what we did.  And we made sure that it was gonna trigger AV. As soon as it hit the box, AV was gonna squelch and be like, nah, you're not gonna run. What is your, in your sock today, or your IT security folks, or whomever's watching these types of systems, what happens when you see something like that? No worries, AV caught it, right? Who cares? Moving on, but let me tell you what the second half of that scenario is. If you would have taken the sample from AV that it caught and mitigated, if you would have taken that sample, it was just visual basic, you know? You know, you would de-opfuscate it, you would see that it had hard-coded C2 beacons in it, hard-coded URLs for C2 beacons. Those beacons had been in use for multiple scenarios, undetected. So what we staged was as if an APT had already had a foothold, but they were trying to expand it maybe to another branch office. But they got sloppy, they reused a payload that there was already a signature for, et cetera. These things happen. I'm testing my team to see how quickly do they disregard something because, oh, it appeared to have been mitigated. Yes, that was mitigated, but you failed to look deeper to see if you could learn something from it that might help you find something that wasn't mitigated, right? So I mean, guys, the sky's the limit. I probably could have just talked about all the different types of scenarios that you could do in this environment, but I mean, I'll leave it to your imagination, but these are the things you have to think about. What would our team do in a really freaky situation like that? Like, would we be prepared? You won't know until you try it. You won't know until you test it. You won't know until you've got the muscle memory that it takes to do these types of things that don't happen every day, unless you're RSA. Just kidding, kidding. Yes. Okay, that's a great question. So now there are technologies that can do this. They can interact with like the DOM and like click on things automatically. I'm not that cool. I don't have anything like that. So what we do is what we call, it's a white cell task. So it usually falls on the red team guy or maybe if you got a CIS admin that's helping run the range during the environment, during the exercise, it would be a task item for them. Hey, we just fished Regina Buckley. Log on to that system, open the email, click the link, run the executable. There are ways to automate that. I just, I'm not there yet. There are really expensive technologies and platforms that can do this. Some of you may be familiar. I'm very familiar. So my history, I didn't really go into a good bit of history on this, but my time in the Air National Guard was spent largely in part going through simulations. Once a year, I go up to Virginia and run through an international cyber warfare simulation. And the things I've seen there are what prompted me to build this because I got there and I noticed that, hey, what do you know?  We know what we're doing, but we don't do this but once a year and it shows. I wanna do this more often. We need to get better. This needs to be more of a comfortable feeling for us. And so that's why I built it. But environments like that, they've got all those cool tools. Lariat, there's all kinds of really cool user automation to simulate these things. But see, I'm a state agency. I've gotta figure out how to do it on a budget. So, chaff, Chrome extension, Python. Gets it done. Gets it done, trust me. And then when you do have to do that one thing, like open Outlook to click an email and open an attachment, that's a 30 second inject for your white cell team, yeah. Oh, but that reminded me of one point though. There's one rule that I stick to when I'm writing these scenarios and I would advise you to do the same. Do not give in to the temptation to game the system a little bit. Everything that you do should be organic. You should be using the same tactics that attackers use. Because if you do something out of the ordinary and you might have a technical limitation that means, oh, I need a side channel. I just need to like load a CD-ROM on the virtual machine of this workstation so I could put this binary on it undetected. That's unfair. It's unfair to your incident response team. You didn't even give them the opportunity. So you need to up your skills and how are you gonna deliver this attack? Because if you start side channeling and gaming the system, it's gonna drive down the value of the training. It's gonna demotivate your team because they're gonna say, really, you did it that way. We were never gonna find that.  So keep it real, keep your scenarios real. But Ming's coming to kick me off stage. Everyone, Bob Newhart everybody, Bob Newhart.