Hi all. I'm Tim, or Vulcan I guess more often during this weekend of the year anyway. And if you were here to see the CFT, or CTF, I talked with the last 20 years of organizers which incidentally is all the years. So we have from the beginning all the way to the ones that are currently organizing up here on the panel. And I believe the current organizers are, this is the last one. So stepping down this year and every few years the current organizers step down and there's some process for passing the baton. And one of the reasons that this talk is on the docket this year is to answer questions of people that think they might want to step up and take the baton and also to have sort of an archival reference for when this happens in the future. So again, I'm Tim. I'm going to do my best to moderate these folks up here when we get to the panel time. But I'm also going to sort of lay a foundation of sort of the basics of what CTF is. I'm not going to try and make too many assertions. Now I am sort of inherently biased being one of the organizers for a few years and also being a participant for a few years. But I'll make every effort to sort of remain neutral up here and pass the questions off to our panel of experts here. Now the talk is geared towards organizing CTF. So I'll do a little bit of explaining about what CTF is. But it's from an organizer perspective, right? And some of us even do CTFs from time to time professionally. But in most cases these slides are geared specifically to what CTF means in the context of DEF CON. So what is CTF? Right? So we're not going to cover what is CTF in depth. But at the basics it's a cyber security based capture the flag contest, right? So it's just like flags when you play with hills and kids and stuff. Except the flags are digital. It's also called an exercise or an event or a game. Typically it's all geared towards demonstrating some minimum bar of proficiency or ideally some bar of excellence in fields of cyber security. So there's different models for organizing this. And we'll go over a couple of them in a bit. But they sort of stress different things. And as an organizer you get to decide how you're going to stress the different sort of areas of excellence. CTF exercises are becoming much more common.  So there's a circuit, so to speak. And there's also a multitude of contests that sort of just tack on CTF at the end where it's very domain specific like the social engineering CTF or ICS CTF or something like that. And some of those bear more resemblance to the game that we're going to talk about today than others. But in some cases this really just means contest. In other cases there are actually flags that could be captured. So going all the way back to the basics for capturing the flag, we need to talk about like what the flag is, right? Well it's the thing that needs to be captured. But from an organizer's perspective that actually has, there's a lot to sort of suss out there. Is it just random text? Well if you just use random text then from a participant's perspective it's difficult to understand whether you've actually found a flag or whether it's just actually random text, right? That's encrypted or compressed or something like that. So it's somewhat difficult to know when you're done and when you've found the flag. Also from an organizer's perspective it might result in a lot of guessing because the participants are unsure. So you end up with this sort of denial of service sort of situation where people are just guessing repeatedly hoping that they found the flag but they're not really sure. So then the next thing you do is you try and add some structure. You know it's a certain size, it's a certain format, it has a particular prefix. All of these things sort of limit that space of brute forcing, right? So now you increase the likelihood of guessing correctly for people that do want to guess. You also open the door to all kinds of different defenses like consider writing an IDS signature for the prefix that you know is going to be there because the flag has a specification. So there's these trade-offs that happen when you even define something as basic as what is a flag. So even beyond defining sort of the specification for what a flag is there's a lot of other mechanics that happen around the flag.  So you have to go back and assert to the organizers that you successfully stolen it and over the years that has really matured over time from verbal or writing on paper to email to now you have sort of rich-ish web APIs, right? Solid services where there's structure and protocols around how to sort of automatically or in an automated fashion submit flags to web APIs or scoreboard servers. And then there's all these sort of game decisions that have to be made. Some of them made by the participants where flags might be shared. You have collusion between teams for potential collusion. Do flags have a preset predefined constant value or does it change over time? Maybe being deluded as more people have stolen flags from easier services? Do flags expire? If you steal one at the beginning of a 48 hour period is it worth the same amount at the end of the 48 period? Is worth anything? All of these things sort of go into the mechanics of how that particular game is going to operate and adjust the game strategies that the participants are going to adopt. So then what is DEF CON CTF specifically? Well there's no shortage of quotes. These are just ones from the top hits when you do a Google search. But the assertion that I think most of the people up on this panel would say is that DEF CON is one of the highest regarded CTFs out there. And it's kind of grown over the years from being one of the oldest and sort of one of the only to being the world series or the best of the best. So typically the phrase DEF CON CTF, if you hear somebody say it, they're referring specifically to the on-premise version of the game which starts tomorrow morning at 9 a.m. and goes till Sunday. It goes throughout the weekend. So in the beginning there were the goons or specifically there was miles. And so DEF CON CTF is one of the oldest and longest running. There are some others that have been around for a while. Like the UCSB's ICTF comes to mind. But even ICTF is fairly restricted until just this year was sort of academic only, right? And they opened up for you guys. So as a pre-qualifier for this year they opened it up so it's not academic affiliated. But DEF CON has got a long history of being very approachable and very open. And it's one of the longest running CTFs. It's also one of the oldest contests that's still running at DEF CON. So back at DEF CON 4 was the first one. It was just called Capture the Flag or Network Capture the Flag I think a little bit later. And it has certainly grown over time. So while this is DEF CON 25 it's CTF 22. So if you look at a quick timeline again in the beginning there were goons. Sort of later on teams more formally organized and sort of had a multi-year approach to organizing. We may switch different computers yet. So around the year 2000 it was clear that organizing needed to be a team sport. Let's go back one and see. It's less epilepsy. Now it's less epilepsy. Sort of around 2000 it was clear that organizing needed to be a team sport, right? There were other hacking competitions. Hope for example had something. But there was nothing that really had sort of the network component, the defensive component. One more see if this will last. All right. So in the way of a road map which might stay there for a few seconds. Nope. Let's go back and forth here. Quick look at it. So in the way of a road map there's sort of different milestones that happened over the years. In the beginning I interrupt you for a second. Just me. Just wanted to say he's had two problems here that are totally classic. What happens every year when you come with new technology. And every year with organizing these contests. We don't know necessarily who's going to be in the audience what words to use to talk to make things work right. And we definitely don't know what cables to bring. But they do provide testing harnesses in the speaker room which work fine. You have to go kick the cable apparently. It's unfixed. There's so much in between here and there. Yeah, we'll just skip over it. There's sort of different milestones that happened throughout the years. There's the realization that teams are going to be competing even though it wasn't necessarily organized as a team competition. There's the realization that teams are required for organizing. It's too much for one individual or a couple of individuals to handle. Fast forward to there's so many people that want to play that there needs to be some sort of qualifying round in order to gate the people that can actually make it to what's now known as the finals or the DEF CON CTF that's on premise. And then the sort of different things that happen sort of along the way like the inclusion of the game outside the game with the meta game and moving to IPv6 and badges that have game code on the badges. And then last year was actually the first year and perhaps the only year that the architecture and the format of the game were announced like long ahead of time so you could actually spend time developing tools as a team and come in with tools that you knew were going to work. We actually announced last year that we were having a custom architecture. It's like a nine bit middle Indian thing monstrosity. Middle Indian is not actually a thing by the way. It totally is now. Yeah it is now. Now we actually have a custom architecture that the team just got this morning. They just got this main authority. So preparation before 9 a.m. is perhaps of limited utility. So getting back to what is capture the flag if we kind of grossly break them down into two categories this is again been covered in other presentations so you won't spend too much time on it but the two broad categories are attack, defend, and game board or jeopardy style. And the big thing in attack defend is that the participants are directly connected to each other. So defense actually becomes a big component of the game. Also service level availability or SLA becomes part of the game. The parts that need to be vulnerable and able to be attacked have to be there and running. So everybody is typically level set. Think of like a VM that has custom software and it's sort of distributed to all the teams. There's different ways to do that but that's a good analogy. And ultimately it's composed of a set of challenges and it's another thing that matured over time. The concept of challenges or services. And these are sets of custom software that typically run with some amount of concurrency. There's debate on how many challenges can currently make a good CTF or not. So conversely the jeopardy style is much like the show. You've seen the show. It's a grid of questions. Participants are not connected to each other so defense is typically not part of these games. And you solve a series of challenges where the order might be determined or might be controlled by the leader, by the organizer. And the categories might be completely arbitrary or they might be designed with certain learning objectives or goals such as getting for CTF, demonstrating proficiency and subjects that would work well when you get here for the weekend. There's also hybrids and things that don't quite fit but broadly these are the two. So jeopardy style worldwide or just proportionally in the number of CTFs that happen is by far the most common. It's possibly more diverse within a single event because defining what a question is is very broad. Whereas you define what a software service is in an attack to fend game is perhaps relatively confining. And it's arguably easier to organize for several reasons. So today, DEF CON, when you say DEF CON CTF, it means the thing here. There's typically a jeopardy style one that dates everybody as a qualifying round. So this was introduced in 2004, Caesar. You did qualifying, right? It turns out, yeah. We were just debating that. No, Caesar was like, we didn't do quals. I remember playing in your quals. So the last year, on the show, you can't see it because it causes epilepsy. But I think the last year, the director started quals. That was all DD back in the day. That was literally like force of one. Although Miles brought us along, DD implemented so much stuff. You really would have to understand the guy to know how amazing he could pump out user-mode Linux router system with double NAT and firewalling and stuff like in a night, in an evening. They also had some pretty crazy constraints in their quals, too. I remember specifically playing and not being able to actually make any kind of connection into that box or out of that box other than the literally just terminal IO. So we had to write UU encoder and decoder routines into like telnet so that we could move binaries to the box. Their first qual was just a race to finish eight levels of a challenge. Yeah, it was totally linear. Yeah, exactly. So we're having a sad realization up here that anything with pictures appears to anger the projection system. Which is unfortunate because there's a lot of pictures in the messages. So you've seen the board. This is what the board looked like when DD tech or sorry, when Kinshoto had quals in 2006 anyway. And there's a starter question. There's a hint that kind of points off somewhere, either download a file or go to some web service. And they should try and switch computers here. But unlike Jeopardy, so the leader controls the next question, but unlike Jeopardy, other teams can sort of catch up and answer the previous questions. I might just try and switch briefly. We can probably fill the space. Yeah, we're going to talk about stuff while he works. That's usually how organizing CTF is. It's pretty ad hoc to begin with. Basically something goes awry every time. This is its own sort of micro example of how CTF actually works in real life. Because it turns out nothing goes to plan. It's the old like, you know, planning is useful with plans. And I'll point out that Tim was a part of DD tech and DD tech's CTFs were famously broken. I think you completely erased the scores from one day. We're upset with that. Yeah. Playing CTF, most of my CTF experience was under DD tech and my two black badges is from winning their first and final year. But funny story, actually we were playing quals, organizing quals. I once wrote a mock web server with a CGI script underneath it. And the bug was in the CGI script. But all the web script kitties were like, yay, a web challenge. And then they started uploading like PHP shells and Python shells. And one team in particular uploaded a Python shell, connect back with a hard coded IP address in port. Let me play with this. So I killed it since it wasn't actually part of the game, or necessarily part of the game. And they did it again. All right, this is what's going to happen. Killed it, just immediately started connecting back to them. And then finally I got to connect back and they started typing commands. And me trying to L.S. like slash. And then cat something.  Eventually I realized they were talking to somebody. So much fun. Self control to not mess with everybody else's exploits essentially. People would land stuff on boxes and watching it happen you could connect to it faster than they could. And so not going and grabbing everybody else's shells when you're the organizer just for fun was actually like a massive, massive self control. The reason I did it was because it wasn't necessary to land a challenge. That was actually here. So I don't know if you guys know Geohot. He popped the service by figuring out that that bash RC was writable by general users. So he wrote cat flag out to slash tip, slash something. So he just waited until somebody else exploited it because they were just going to run bash because it wouldn't run bash. Whatever. And so he just waited and just saw that file pop up in temp and got the flag. It's a way to do it. We actually had a whole, we had a whole like sub focus area that we called the dirty tricks department. Which was basically like trying to figure out all the errant conditions in everybody else's exploitation methodology. So like I remember one year there was a team that was doing really, really well but they were always like their payload would actually cat the flag to a file in temp. And so we could just like walk around behind them and mop up all the flags that they were getting to. Nice. One thing I found that was gone. Be ready? I'm ready. I'm going to finish your pot. No, no, go ahead. Too much of a thought. Yeah, you're in the middle of your thing, go ahead. I will say that I walked into the speaker room earlier today with my laptop that's running Linux and plugged it in and everything worked. And then I didn't tell anybody. I was like, oh Linux and it finally works. It understands how to do this. It's the year of Linux on the desktop. And I tried it again and I worked again. And I got up here and I'm like, oh no. And now I have an act plugged into the projector and it's working fine. I guess it's still not there. Go back and show your timeline. I thought that was a good idea. You should put back to your timeline. I agree. Oh yeah. Yeah, nobody got to see that. A few slides down now. Great. So now we have timelines with no epilepsy. So I kind of went over a few of the parts and this is by no means exhaustive. It's just sort of like the things that stood out in my memory and sort of worked well for the slide deck, right? But it definitely has a lot of the pieces that matured over time with the qualifier introduced in 2004. And then the tradition that continues now with returning the returning champion. So Ken Shoto would be inviting the returning champion back for the first time in 2005. And there's sort of these, I don't know, dirty tricks or twists or things that sort of make it hard to come in prepared, right? So like one of the things that this year makes it hard to come in prepared is that it's an entirely custom architecture that wasn't announced until 9am this morning, right? So that's like a way that preparation is difficult. Some other examples of that going back in 2006 you had to submit keys over a DTMF. We were just talking about that too. I remember basically everybody. So we had deployed like an actual, I think it was an asterisk. And so basically all the tokens, all the keys were just really big, entirely numeric digits. And so we were making everyone, like they would pick up a phone and it was like, press one to submit a key, press two to talk to a member of Ken Shoto.   Yeah, exactly. And then, yeah, and you couldn't submit more than one token per call, so you had to actually hang up the phone and do it again. We had so many tokens, we exceeded the available bandwidth to submit our tokens. We couldn't submit a fascination. Yeah, so everybody ran out, everybody ran out to fries to buy, to try and find modems. And like people were like, how do I connect something over serial to my like modern modem?  Which also has like interesting ramifications because then next year, like teams would buy out all the modems. Yeah, and then we didn't use modems next year. They have good return policies. They do. But there's other things that sort of go through the times. Multiple hosts, IPv6, a lot of people were sad faced when it was all IPv6 and they had a lot of IPv4 tools and shellcode and so forth.  So MSP430 architecture on the badges sort of midway through legit BS and then again this year with the custom architecture. So these are sort of some of the milestones by no means exhaustive. But as I already said, the one that I want to focus on now is the qualifying round because that gets into the style of game. Before you jump off that slide, can I mention one other thing? I think there's an interesting pattern involved with this slide, which is the burn out cycle of hosting CTF. If you notice, almost all of those are the same-ish, like maybe off by one year here and there or whatever, because the just Herclian level of effort that goes into creating CTF in general literally causes internal drama and burns out teams over time. It's a second job. As an example, during Kinchotto CTF we had full time jobs and actual real life stuff to do besides CTF and then had an actual development cycle every year for CTF and by the end of it we were just like nope, not doing this again. How many hours a week did you guys work? Oh my god, it was literally a second full time job for almost the entire team. We had Didi, so we went four hours a week.  What'd you do this week? What's our contest like? Lightning who wrote the architecture for this started in 2015. To be fair, it took until a lot longer. So if you kind of fast-forward the previous slide showed a Kinchotto board from 06. Here's another one from 08. I think maybe even the same code or the scheme style was used in 2007. It looks very similar anyway. But you can see that the board is sort of staying the same static as Kinchotto evolved. You do get to define sort of the areas of excellence and the categories. There's an aspect that the organizers have to decide of how approachable do you want the open qualifier, anybody can sign up. How approachable do you want it to be for novice players? Typically that's solved with a trivia category and also the lower point value ones where they're very approachable. Approachable enough that for all the years we hosted at the trivia 100 had the same answer. Well, it was there you go. The last version was blank, blank, blank. Yeah, they submitted a version blank, blank, blank and people just submitted hack the planet. We actually have a baby's first category now. So it's all like baby's first heap explanation. So through the years, there's the next set of organizers, DD Tech, obviously a different look and feel but very similar, very similar semantics. There's still categories. You still choose an area of excellence so to speak. When you go around to other talks it's sort of all the same things. Crypto, cross-site scripting, buffro flows, heap overflows. The box is open to any of the security related fields. And then the current organizer have a slightly more Japanese game show. This is one of the earlier ones from our team. But the concept of qualifying the teams through this gate has persisted since it started. It really filters down to the handful of teams that are going to be present here.  Okay. I should have said at the beginning that I made these slides and they haven't actually seen them so there might be some disagreement and trying to be honest and faithful I've kept everybody in the dark. So the basic organization of the game is the same. Has remained the same. And other CTS around the world sort of take a similar approach. They modify like how you get through the board and so forth. So one thing you might have noticed the astute observer through the boards that we've looked at some of the same teams keep showing you know that but you see the sort of regulars the familiar faces that come back and teams have really started persisting over time back even when the ghetto hackers were playing and then organizing and these are teams that sort of prepare and practice and build tools and test tools and things year around so it's on cycle. Not the ghetto. The ghetto did not. Sort of I mean a few hours a week. We drank a lot. You had DD. I said that teams exist to do it. So they develop and maintain tools and teach processes and so forth and you can sort of track them. These are just points in time so it doesn't say anything about how they're final placing in quals but you see the same teams kind of over and over both in quals and also in finals. So so much that it's actually tracked somewhat formally now so this is a website called CTF time. There's other ones but this one is sort of emerged as the predominant one and it's sort of mostly opt in so it's you might not be as tracked as you want to be but when the results are posted publicly the teams are known and based on you know pattern matching on the IDs they're sort of matched together but individuals can sort of affiliate with the team and opt in but these are tracked over time right so these teams persist in job kind of situation there's formulas there's APIs you can use to extract the information and verify that your rankings are correct even so still subjective right somebody's still saying oh that CTF was harder than this one by some metric right and the formulas still have to be plugged in there's still interesting opportunities for cooperation and collusion and what happens when you switch teams and all that sort of thing so how many will participate so here's we go back to the timeline and you can see in the bar graph I don't know if you can read the numbers very well but the largest bar is 414 and the smallest bar is 162 I didn't have numbers from the early Kensho today's I didn't have them and then I couldn't find them but you can see there's a bit of a trend line right and the teams and it's important to know what you think Vizzy was getting at is how many teams are actually participating which is vastly different than how many teams registered right so I think last year there were 1,500 teams registered I think Vito said so 1,500 teams registered 276 actually submitted something right which isn't a perfect metric might there might be some people that like tried to play and just couldn't figure out hack the planet right and they didn't submit anything right it's possible so it's not a perfect metric but this is a good approximation of how many teams are playing right not how many people teams can have many people they probably have at least 8 in most cases but we're like 50 if you're school of root we don't discriminate there are rules in CTF opportunity so you can see it trending up there's sort of other caveats in that registering shadow teams might have some advantage depending on how structured and how flags have value so having extra teams on your team might be useful and so forth you can even in some cases dilute flag values strategically so then the other flip side of how many will participate that's how many play in open qualifiers but how many are actually going to play when you get to Las Vegas there's logistics in setting up the right amount of tables and how many prizes and what kind of orchestration you need to run when the ghetto hacker started it was 8 and it was always 8 and where did 8 come from I think we have that's how many teams we could fit tables in the room it was fire code and lines of network we built some really nice PVC structures to run the networking cables that's where 8 came from my working theory was 8 port switch you guys never had I think we ended up being the first ones to implement a team table limit I don't think you guys had that because we also saw the instances of teams 8 or 10 or however many teams were in the ballroom at the time we would see one table with three dudes and another table with people crowded around with laptops and stuff there were all star fire code considerations there too anyway you can see that one of the things that DD Tech wanted to do as organizers was expand the competition across several dimensions and one of them was size so it kind of ramped up a little bit the first one should probably have an asterisk there weren't actually 10 viable teams there was one sort of deceptive team there were really only 9 teams in the first year and then the last year that DD Tech ran out there were 20 we were sort of gently pushed because it was DEF CON 20 to have 20 teams and then legit BS maintained for a little while and settled on 15 in the most recent years so how many teams you'll support is another question that you'd have to ask as an organizer alright so this competition once you get to Vegas and these 10 teams or 15 teams get here how are you going to implement this attack defend game right so fundamentally you're going to be concerned about scoring there are four I don't know here there's four basic components but there's going to be there's offense there's defense if you have defense you probably need some amount of SLA and then there's this concept of like other points with like bonus points granted through a breakthroughs or other stuff so offense is going to be stealing flags or corrupting flags right you can you can like take one or you can burn one or whatever and these have to be combined in some way to come up with a score that you can then use to declare a champion so score might be offense times defense times SLA that has pros and cons right if any of those is zero your total score is zero so you can drive to zero real quick so if you have no offense you get no points you have no defense you get no points maybe that's the kind of game you want to run maybe you do more of a summation right if you add them all together now a zero score in any one of them doesn't drive you to zero but it also doesn't have that big of an effect so you can have slightly more complex formulas where SLA maybe is the multiplier meaning you have to allow your opponents to play in order to score any points right if you have zero SLA it means nobody can attack you nobody else can sort of play the game so you don't you want to encourage the game to be played and then these need to work over time right we have the concept of rounds you're going to play all weekend so now this has to be added up over these fixed or variable length 5 minute 15 minute rounds whatever so now your round score is the summation of these the round score is the formula and the game score is the summation and then you might have multiple services these concurrent services these 10 vulnerable things that are running in your host that need to be added together so maybe you end up with a formula that looks like this down at the bottom this is by no means like the right formula or even one that will really work well but you kind of get the idea that there's some mechanics to go into devising the formula that you need to have the score and there's value in having this well defined ahead of time many other methods once you even have these formulas figured out you need to measure it all so you have SLA so this might be a port scanner but now they're like much more robust polling and that's very service specific testing different code paths if a flag is corrupted for offense the organizers need to tell somehow there's a lot of sort of game strategies around sort of permanently overriding other people's flags and things like that so modern CTFs employ custom kernels and hypervisors and traps and all of these things in order to detect that our override has happened similar protections for reads and then defense is sort of the absence of offense you must be doing good because nobody successfully attacking you so now this is set in stone every year there's new opportunity to revise and come in with new scoring methods there are more questions here that will need to be answered will the scores always increase how important is offense and so forth some of the important parts about what you're saying some of the formulas he was just talking about if I turn off my computers and start hacking then the game's over right it's no fun it really is difficult to convince security people to do blue team when they're here for a red team contest that figuring out how to game things has been the evolution of the game really making it fun that's the whole dirty tricks department we had dudes basically going so when we were playing our dirty tricks department was doing stuff like we had a guy that did physical pen testing for a living he was the guy who talked his way into data centers and did all that kind of thing and so he had a high-intensity he was ex-military so he would roll around in camo camo pants and he would actually never ever talk to us we would send someone away from the table to make drops and meet with him he was rolling around brushing burned CDs off of someone else's table and he had duct tape rolled in a loop on the bottom of his shoe and he would walk with a limp so he wouldn't stick it down but camo guy walking with a limp no big deal so he would literally go by tables and then there's many so these services that are running currently you need to define how much you're going to define them is there a spec for this what kind of a box are they going to fit in the more of a specification you have for a service the easier it is to to automate and to test and deploy and to redeploy and to rebuild and also to outsource it turns out because if you have a spec come back with something that's really hard to organize around but lots of decisions about how many are going to run what's their point value and so forth you need to decide how teams are going to interact with the infrastructure so they're connected together there's some way that they're interacting with each other but they're also going to have to interact with all of the bits and pieces that you've assembled and created so will they operate their own defended host is it a VM will they route through the table do they get a tap tap the way and so forth there's lots of decisions that have to be made about how the game is going to be represented and how it's going to be interfaced with one thing that I think everybody another thing that's continued over time is having an immense desire to protect the integrity of the game there's lots of ways that you can take that to heart specifically it needs to be just hackable enough right it's inherently I mean it's already a hostile network it's already hostile people it's designed to be vulnerable but you want to have a fair game you want to make sure that when you're counting a champion you're sure that that's the champion that was the right person you're confident in your scoring you're confident in your infrastructure so there's lots of different pieces this is another slide that's by no means exhaustive but you want to take at least up until now there's been extreme measures taken to protect the integrity and the non-technical side there's even things about table positioning is it unfair to put one table with their backs to the door when you open in the room is it unfair to give one team a particular ID that might result in nulls in their shellcode because of their network subnet and things like that so you really want to strike who did that exactly when we designed that custom badge unfortunately we gave index numbers to people who placed in our qualifier and PPP got first so we gave them the zero index unfortunately the zero index was not exploitable on the RF network so that's one instance like everybody's got their instance so plan for failure right, like in this case bring a Mac that also can do your presentation but this is the kind of thing where I'm sure we can just go down the table and everybody can talk about the different failures but one of the failures that happened in this picture I'm pretty sure this is the right picture was a sort of a central scoring database had a hard disk just totally tank right in the middle of the competition so you have to have ideally some resiliency some like fail over some back up plans the game must go on mentality so what will the rules be so I think typically historically the rule is this is really to keep people from physically being hurt this is actually there also don't cut our cables because we had to do that, that would cost us money so don't cut our stuff so there's other things those were not the rules when we played it sort of expands into things like don't mess with the infrastructure don't destroy personal property messing with the infrastructure used to be like part of the game that was the game part of the scoring system I think it was Dede that came up to the table and he's like that's really cool knock it the fuck off did you receive bonus points for your that was always the thing you kind of want to reward people for doing something novel and new I remember the year that we had again the de facto rule don't cut cables because obviously you could walk around with a razor blade on your shoe and cut cables all day long one of the things we had said this isn't actually that cool if you're trying to do something cool and you mess it up we'll have a little bit of forgiveness so I remember I think it was school roots captain comes up to me and he's like so busy we were trying to do something really neat I was standing kind of the tables were like this and actually this was a lesson learned but we had the sort of ring of tables for the people running for kenshoto in the middle you know like the hotel does and so we're standing there next to the table you remember this story now so we're standing there and this was actually the DTMF here so we're standing there and he's like so what if a friend of mine theoretically was trying to do something really cool but kind of messed it up and now maybe a team needs a new cable and I'm like what'd you do so he's like well you see and he lifts up the skirt of the table like right next to me and he's like well we haven't been able to get him out for like two hours because we managed to get him under there so they had gone and bought a 900 megahertz phone and we're trying to wire it into one of the other teams POTS line to literally broadcast whatever tokens they were submitting over like 900 megahertz analog so they apparently messed up splicing it in you know and like happens but then they also couldn't actually find a window under our table covertly and so he had been under there for like two hours and so like John boss lifts up the skirt he's like so we've got this dude under your table and I'm like that's about it right something like that oh yeah there it is so like so basically we were like yeah that's actually really cool and well played like I'm getting you know people under our table so we just ran a new line and didn't cause a problem but basically that whole dirty tricks thing with the infrastructure definitely was part of our game I'm totally unrelated fact when infrastructure moved to dedicated rooms that may or may not have been in balconies the rooms locked at some point but the balconies didn't yeah who wasn't the climbed across from one of the balconies to the other to get into the kinshota skybox one year I think that was school route 2 we're talking about cultural influence actually right before the talk or some bits of culture that are missing from the slide so again this is sort of an incomplete slide but culture is interesting in that there's a lot of barriers right especially as this becomes an increasingly international competition right there's teams from all around the planet to play not only in quals but also in finals these days and there's clearly the language barriers clearly a popular culture like the hacker movies from the 80s like some of them weren't as internationally as popular as others like those questions don't resonate as much and those sorts of things time zones and holidays there's different things to try and keep in mind they tend to show up during trivia and question starters and things on the right hand side just generally when interfacing with the teams and when designing services that have ASCII protocols and the words sometimes have meaning to different cultures there's also the culture that's influenced by CTF where the community sort of leaves its mark in various ways so generals and producers and actors and so forth actually come through and visit the CTF room and want to have sort of like their own individual briefing and their own sort of explanation and you have no idea what the ramifications of those conversations are going to be and how that sort of spirals out so there's a team mentioned, a school of root mentioned in the HBO show another sort of thing that needs to be approached by the organizers is how are you going to engage everybody not only the participants but also the folks that are walking around in the average human attendee so should there be ambiance should there be distraction should there be music at all should there be videos should there be scoreboards there's probably contention up here about that but otherwise if you don't consider these things it just ends up being a bunch of people in a fairly dark room staring at computers and the teams don't really have any physical interaction with each other the attendees don't really know what they're looking at so handouts and so forth so this gives way to visualization so back in I think this is about 2002 I don't really know I think this is a ghetto created scoreboard yeah I think it is but we had scoreboards projected right onto the wall one of the problems with showing scores is it can be really easy to get away with a win in the first day and you might have teams start to lose interest this is part of do scores always go up this is a zero sum game and you can see the teams on the bottom have a little bit of a red bar and the bigger green bars or the top top games at some point you're trying to keep people interested even if in reality there's no possible way they can win any more so the first day we show scores live scores the second day we only show relative positions and then finally the third day we just play back the previous I think there were sort of a refinement process we had something where like I think for the last four hours we didn't show the scoreboard something like that too yeah yeah I think everybody has their own variation on that being I think you're a little bit more extreme than has been in the past so here's some here's a kenshoto board with the ordering and the team relative placement on a line chart over time a little bit later still a very similar scoreboard even later in the kenshoto time we still have a similar scoreboard but it's also rotating with ones that have I guess innovative in some sense like showing different types of information so the one down in the corner is kind of showing what's left on the table these are the points that are still back to services that need some more attention if you open your data you get visualizations from others so here's some quals data that was taken by some some sort of non-organizer folks and graphed over time and it's sort of easy in different types of those relations to see how the scores progress so here you can see relatively quickly a lot of teams spike up and one sort of is in the lead for a long time but then there's still a relatively dramatic comfort behind a few hours before the competition where the red line cuts up so it's quals so it's a little bit different but this is an example of that come behind scoring that you'd like to embody in the game if you want to encourage the availability for a come behind teams there were also teams that were like fairly heavy into like using their lead psychologically during the game like there was a certain team that spent the last two hours of CTF one year very publicly and very sort of flagrantly having all of their people just play guitar hero mostly just as like a you know you don't even need to try anymore because we've got this whenever we say a certain team we mean Chris Eagle pretty much a certain team is always them so there's a new new organizer to go over right the scoreboard is still relatively basic it's displaying the same sort of information this is sort of one of the things that the game ends up being more important for the organizer so the visualizations end up sort of being backburner than backburner and then like cobble together at the end right and that code doesn't persist generally from organizer to organizer so you see this trend where like a new organizer take over takes over and then it's like sort of back to square one and it kind of builds back up so DDTex early boards look like that then you know later on they had much more appealing graphics that displayed some of the same information but also like that's a placeholder screen on the side and then in the end you sort of see the the more appealing side and and some attempts at trying to display types of information that are kind of hard to consume like the bubble chart where it shows team versus team action this one is particular in the number number of rights per service so that big blue circle is showing that like one team is like massively overriding on one particular other team instead of evenly distributing their attacks across and then you know like laser gadgets and stuff like that and then another new team came come takes over still relatively basic they've taken a different approach where it's sort of a graph based thing where the edges display types of information but it's still sort of basic blocks particularly appealing that advances over time sort of similar information but much more appealing also the crossword live that was our first iteration you just saw the next one it's got much better that's all that's sort of the point right it sort of evolves over time and as an organizer you have an opportunity to decide how you're going to try and convey information and what information you're going to convey right because you want to engage the audience but in some respect it actually also informs all of the teams that are sitting there present right so in some way this acts as an intrusion detection system you could tell somebody's opinion right so that's why you have the stage thing where it's like live and then it's like delayed and then it's like gone and however you phase it out but these are considerations that you have to have as an organizer so then there's the I'm just going to rip through these pretty quick so we can get to questions there's the expansion into physical space we talked about this a little bit already right with the going into the custom badges that are actually scoreable the metagame lock picking is super common in lots of CTFs like as a physical aspect they get to your passwords of your keys or start the game or whatever another physical thing that was tried as a service that was in the game actually controlled physical things outside the game so like this is the robotic chicken fight that was a service in 2011 and that was sort of an ancillary to slide one of the teams actually went out and got like game controllers and had an adapter interface to the service where they could actually automatically exploit the service and then control it with their PlayStation 4 controller or something interesting things right so tradition so future organizers I think there's a lot of aspects of tradition if you're going to player or close to the organizers you get some of that trying to like document some of that here but there's there's certainly a desire to keep DEF CON CTF the best in the world there's certainly a desire to keep it fair and fun and innovative always desire to engage the audience you never know what the next generation of players is going to be and where they're going to come from and you really really want to have that open to everybody there's logistics there's game banners and team banners the winner typically gets to bring the game banner home as part of their spoils their swag there's there's t-shirts and stickers and so forth typically announcements are happening on April 1st right that's just a thing this happened on you no one calls is going to happen fortune cookies started I think I think kenshoto started that because they had a balcony and there was a desire to throw something off the balcony so there was the Sunday tossing of the cookies that's why we have the they're much bigger than they heard we had like the annual Sunday tossing of the cookies right yeah so some of those are here back in DEF CON 15 some of these are dbtech ones dbtech incorporated some of the challenges into some of the fortunes so they're stickers right the hacker thing there's laptop stickers and everybody's got their stickers there's coins as Ashi mentioned not necessarily one of the longest lasting traditions but there's one for every year going back a few years and there's shit really and actual stress cheap and so forth teams bring stuff you can't stop them they bring stuff they do stuff particularly like this slide should have been titled will there be sheep there was a they modified the stress sheep to have the LED eyes so teams have to prepare I've heard a couple people have already brought up that this is sort of a second job it's a multi month it might be an all year around thing like when do you start planning for next year as soon as this year's over kind of thing so there's a lot of preparation there's a setting of the servers the configurations building the services building all the infrastructure if you don't and then if you do like the metagame stuff you have to set up all the metagame stuff you have to program the badges we did DD tech interface with all of the human badges so I think if you have a human badge from DEF CON 18 you actually have CTF code in the firmware so that's like a whole another layer of working with other parts of DEF CON did you guys end up having the thing where like every year you swore off doing it again like every year we were like we're never doing this again and then like a couple months later you know if we don't do the thing we did this year next year it'll be so much cooler like even now like maybe you could do it next year no it's written down we're done yeah it was like the year that we stopped the year that we stopped hosting was literally the year that no one stood up and was like yeah we're doing this like everybody was like we definitely got drunk at Qualls and like yeah let's do this we got this next morning nope how much effort there is like you're around when you get there on site there's an amazing amount of things that have to happen behind the scenes and I say proportionally most of what you're going to be looking as an organizer is something like this you'll see like the empty rooms with no participants in it you'll be watching some of your buddies terminate network cables across the room as you're trying to make sure all of your services work and this is the view that you get or you get a view in the back room your infrastructure servers or something like since you're on the topic of preparation it's also pretty important to point out that the organizers of CTF the entirety of the infrastructure and the code and the logistics has always been completely something provided by the people actually running CTF like our actual like Kenshodo's logistics bill every year out of pocket for us was like 20 grand that's why I say don't cut our cables cause they're our cables so if we're talking ship all the stuff out here shipping costs alone the pelican cases and all the crazy nonsense if we're talking proportionally what the teams see or something like this cause again like the table limit for fire code or due to prizes there's only eight black badges the teams are much larger than that or some teams are and you see extra hotel rooms with wiring all over it and you'll be staring into Ida screens and debugger screens and have your persistent servers that have UPS cap to the bottom of them so you can wheel back and forth as the game goes live so so why do people play right this is like these last couple slides sort of like make you wonder like why right so why do people play well there's challenge there's some prizes especially around the world some of the CTFs are getting like $30,000 $50,000 prizes right it's not quite like you games but like you can't make a living out but you can what if you just couldn't get into talks at desktop right so you just stuck there I was going to say I think that last bullet glory is the reason you play so the real reason and that's why we started catch the flag was to have a chance to go head to head it's a black badge and a jacket that's why you play oldest black badge like so why do organizers organize so this is actually the question this is the question this is the first question I want to hand off to the panel so for you that just very quickly you roughly know who's up here but to introduce them sort of officially we have Vizzy wave your hand I crowdsourced my bio because I'm shit at writing bios so this is all the twitter responses to what should my bio be so he crowdsourced the bio main or chief organizer for for Kim Shoto Chris Eagle down there on the end so he was a player and then an organizer and a player again lots of CTF experience from both sides literally wrote the book nice dude hook a brother up I got you don't you worry Riley we're rebooting sorry our monitors are not working see how so he's the part of the ghetto hackers who won three times and then became the first formalized organizer and really brought it to the next level also known for for Cesar's challenge sort of an annual thing 21 years I have thrown a party on Saturday night because being a masochist and doing CTF wasn't enough find me at any time and I will hand you a puzzle and if you can solve it which is not very hard this year if you can solve it then I buy you drinks on Saturday also excuse me a two-time champion unlike the other champions we have two different teams two different teams and one of the current organizers so 2013 so we're not really sure when they are going to stop this year definitely yeah we're not sure when they're going to stop and and Miles right Miles is responsible he's like the reason that we're all here because this wouldn't have happened if he wouldn't have started it so I know that our presentation is not going to do it but I'm going to interrupt and say it's time for everyone to give this guy a big round of applause hacking is basically a modern watchable sport I mean there are twitch feeds now for this kind of thing it's all because of the bucket that you started kicking and with that I'd like to kick it back off and I think the first question will be so why do organizers organize and let's go in order in chronological order and then this room didn't get set up well we don't have mics so if you have a question you want to ask you have to come up here and you can either take the mic or I can repeat it and then also I'll put up a I'll try this thing where you can put up a URL and send questions to the podium we'll see how that works I would not trust a single person it's a room for that it goes through Google stuff so why why organize because there wasn't a way to figure out who was best and not practice out on the live internet where the con would get shut down we forget so you're saying it was harm reduction harm reduction totally and you forget that like at first I went by a NIM because I was seriously worried if my employers heard about capture the flag I might be fired and I worked as a security guy and this last year I was in the Smithsonian there's a black badge there I mean this is really different so yeah it was harm reduction it was head to head it was a chance also if you had controlled the environment that you could start throwing in some stuff that explains it to the general public well because it's really cool there's these things and it's a puzzle that they're solving so it was clear to me that we needed something with a bit more showmanship and also a chance where attackers and defenders could go head to head and that it didn't break the rest of the network and throw out the con yeah for us we came and we felt like we knew enough about security to not necessarily need to see every single talk so we'd catch a couple of talks and enjoy it and then we'd kind of wander over and see Miles' game and say so I just sat down at a table every person who was at that table today is a dear friend of mine today and we go out together all the time we live near each other we've all moved to live near each other so my social life is Miles' fault the ghetto hackers were formed because we didn't bring pens or paper or notebooks or hardly anything so we had a napkin and a mascara and we wrote IP addresses that were available on the network our group of friends came out of us sitting around each other and for us after the first year we just kind of tried and saw how it was and then after that there was kind of this fire of well you know if we just stored every exploit and that was the contest back then was like how many exploits can you bring in a searchable usable format and it just became this passion to actually just get better so for us we didn't necessarily try to win as much as we tried to get good like get good scrub we were scrubs and we came and you kind of made a crucible and a bunch of hackers popped out and then why did you run it why did we run it we ran the contest after the third year somebody in one of the teams said that they hated us and we were cheating bastards and that they didn't want to play hackers hello we took it as a great honor and but there was something a spark among us that made us jump up and volunteer to take the game to a new level and we got up on stage and talked about coming back the next year and making a new contest and Miles was part of that contest as well and that was how we kind of came to the scoreboard and sort of all some of the trappings that are the beginnings of what these guys who won the contests that we threw and we handed the reins to them that's actually kind of been a tradition I know it was kind of touched on early but you end up feeling this sort of sense of belonging and ownership to the game when you win it also and you have your own designs on how you want it to be or how you think it should be so like transitioning to the reason that like Kenshioto decided to run CTF for a few years I think there's a number of facets but for us watching the ghetto hackers and watching the coalescence that it caused in the hacker community specifically and the crucible effect is super huge like we would watch people literally have a reason have drive to learn all these new exploit mechanisms and all these like details about pulling packets apart and whatever so for us actually I think that the biggest thing that that caused us to actually run CTF was we were concerned initially that it would be taken over instead of by someone that is about the hacker community that it would be taken over by we were very concerned about that initially turns out you know off we go modern times and all that but the other reason the other reason that Kenshioto stayed in the game the reason that not the first year but the reason for the years after that was we really felt that it was sort of an unmeasured like an unmeasured resource right so as an example in our game we were sort of one of the first people to be like don't bring any of those exploits because they're not going to work with challenges that you have to reverse engineer and exploit and land payloads against like in that weekend and for us that was really important for us because we felt that that was something where people hadn't stretched far enough yet the bar wasn't high enough and we felt like we wanted to continue to be able to push that higher and higher so that eventually not everyone was making it so as an example actually there's also what do they call it I think they renamed amateur CTF but it's called something else now project 2 but like that as an example subtle humble break but that essentially formed out of the fact that we pushed we tried to push the game into actually measuring the real red line of what people were capable of in a weekend and that really drove us for the years after the first year that we hosted the first year was because we were super concerned that it was going to become this like corporate sponsorship kind of thing and after that it was because we thought that the performance that we had seen was a further measurement higher and higher so and that is a perfect transition to him to Mr. Scholaroo because we had a lot of the same reasons so we had played in Miles' game we played in Caesar's game we played in Visi's game and we loved every iteration I teach and it turned out that the game as it evolved was a great microcosm of the security space in which to conduct teaching and learning my students got really excited about it so we loved the game as it was I loved the game the last year we played with Ken Shoto and when they stepped down we were also very worried that some company would come in and commercialize this thing and we looked around and said who's going to run it we couldn't imagine who might run it I think there was one other team that might have run it we didn't know what they would pitch so we made our pitch and we basically but you lost our pitch was basically if you don't give it to us it's going to suck ours was if you give it to them it'll suck and we really liked the game and wanted to see it continue in much the same vein of course it didn't occur to us that we then wouldn't get to play for four years playing is way more fun than running why we kept running it was more or less well we screwed that one up maybe it'll be better next year I think we might have got it we got it close to right the full time it actually started on time you didn't delete all of our scores before you stop how did it change your teams experience your schools experience to go from playing to running it to playing do you play differently because you ran it playing to running is a really tough thing it's an entirely different mindset to become an organizer right so if you're doing you know, Valdev type stuff and then now you've got to turn around and write it you've got to write with an entirely different mindset writing a challenge is no easy task it's easy to put one bug in the challenge of a very specific nature it's a little more difficult not to put other bugs in that you didn't mean to it's really hard to write secure and insecure code just exploitable enough and how many of you have found exploits in a code you were adapting because I know I was finding exploits out there it's like wow that's been vulnerable since 90 we actually had two challenges that literally just incorporated a library that we knew had an ode in it yeah and we wrote challenges like that too you take things from real world and you try to bake it into this challenge we didn't want to drop mountains of software like okay here's a patchy find the ode which was easier you know ten years ago so everything was pretty stripped down and you try to build a bug you want people to hit and you try not to have other bugs so from running to organizing that mindset was hard for some people to adapt to so we did lose a lot of people actually from the playing side that's what they want to do and they want to keep on playing I don't blame them but we were a much smaller group during the time we spent running it that tells into another question this thing's actually working so we're getting questions that are coming in so you want to answer H.J. and I'll follow into the next question I think a short and sweet was anything you can do I can do better we played a text game and you got two black badges out of it we played their game got two black badges we saw what they did and knew how we were assholes and how to beat ourselves and decided we could make an interesting game I was just going to say we make people upset about how we beat their defenses anyway same reason we played their game and thought we could do better or do things differently and take it in an interesting direction for the community and they've done an amazing job the only thing I'll add is the only fun you can have an organizer is trying to fuck with the players totally true whether it's a challenge you designed or some new twist the badge challenge or challenges that were actually purposely not actually exploitable there was a bug but it couldn't be landed that's our history you thought there was a bug and now you want to say what's that that was an action for one team it was exploitable so the following question which perhaps I'll ask to Chris since you already started answering it how big are the organizing teams and you can take that with the transition from player to organizer we went from about 900 players down to you probably a core that topped out around 10 but there was even a subset of that that was a little more active we ran it we're around 10 as well we had a couple of outside contributors people who write challenges people who organize a guy who's actually the head he's more like having to deal with DEF CON stuff we've got one guy who's just an amazing network guy no joke he is a person that Cisco calls and they can't answer a problem not even joking we were as many as 14 but we did but it was really only I think six or eight people that were the core of our team you were one Miles was the army of one at the end was it still one or you had drawn into a couple of people it was one and that's why it was time to hand it on you lasted longer you're a good man Charlie Brown so it turns out that this thing allows you to vote so there's actually like most popular questions what is the most unexpected way someone solved the challenge there's like an intended path and then there's like an unintended path they used a free BSG jailbreak to get out of our jails we watched them do it and they were so inept when they got out people bringing odays and then completely messing up using them correctly would be a good one my favorite one was I had managed to let's see Quake was the hot game in Defcon 4 or 5 so I managed to talk id software to donating Quake licenses to the capture the flag contest like Quake servers and a couple of stations set up so that people could play Quake and then I got free video games out of it so all good and so someone came up to me and said you know we have dialed servers but it's not on the network we can attack the Quake server one of the people gets so pissed off we have to reboot so we can run our RC script I said go for it we convinced a team to surrender and give us all their points by telling them that we were about to win even though they were ahead and so they joined and all became ghetto hackers actually I'm gonna relay something we were talking about earlier an anecdote we were talking about specifically in the ghetto hackers playing so it wouldn't be an instance of Caesar having seen this occur as a novel way to score points or win the game but having executed a novel way to win the game they basically convinced the CTF organizers at the time that they needed to they needed to store a half rack of equipment that they had so they had you know your little like what would it be 20ish you rack and they had completely built out entire face plates of real servers and stuff and they like stuffed a person into the half rack and actually got them to store it in the room with all the rest of the CTF equipment and infrastructure overnight cause they're like well we need a place to put this right and so obviously out creeps person single roots all the machines single roots all the boxes puts them in single user mode roots them all we come back in the morning having not prepared or worked or done anything the night before we've been drinking all night the contest is over so let's all just go home and we got a whole bunch of points and it was pretty good contest was not over I already mentioned the Bash RC one which is fantastic when we were playing we once snuck the root search from one of the teams and got first blood on every service that was their game this works pretty good I can just like go to the next most popular one so how do you come up with unique flags or challenges I think there means challenges every year aside from following CVS or repeating content that was actually a serious source of exhaustion like mental exhaustion of attempting to come up with services and we would literally scrap 2 or 3 services every year because they wouldn't get deployed correctly or they wouldn't be done but coming up with unique ideas of like here's a web service that helps you make a sandwich or whatever and now with 20 years of DEF CON history exactly now you want to do something original and not just DEF CON history every other CCF 100 plus probably per year and how do you actually be unique you can't really be unique anymore we would actually basically the real truth of the story how we produced our services was mostly about getting a bunch of people in a room getting a bunch of whiskey in the room too and like just bullshitting until funny ideas came out and then codifying that list into services and then going and deciding what kind of exploit goes into it whatever but every one of those services had a little for us what we've done is we've really introduced multiple architectures three years ago I think we had five different architectures in one game and it was X86 and MIPS we even had a Windows IoT arm challenges running a power PC shell or something and this year of course like I mentioned we have the custom architecture that we built but it really is we've had to push it nobody has quite explained this system for storing bits in electronics so imagine like they implemented a CPU processor a VM all up on top of nothing it's whole clock lightning actually did it but single handedly yeah that's pretty good and the caliber of people who are playing are just phenomenal they frighten me now people like Loki scary good but the challenges the level that we have to do for the difficulty has astronomically increased the medium challenges back in the day are now easy challenges for everybody tools have gotten so much better too you don't want to write a challenge it's going to be auto solved by a tool under a minute you have to keep up with the state of the tools and try to find the weaknesses in the tools so that you still get the human who's got to do the deep dive that's a good segue into the next actually set of questions there's a sub related to CGC and sort of automation lessons learned from cyber grand challenge it's been a year was it a success or obsolete was it a success or failure I'm looking for one I sorted these and now I can't find one I saw earlier there's this one mostly for it is how many years until a computer wins DEF CON CPF right so the thing that the computers have trouble with is the dirty tricks department and so I think it will be a while unless a game is designed specifically for the computer system that's really kind of where we're at right now I think CGC was a really good example of this just massively forward in technology but at the same time you have to keep in mind that it was a reduced instruction set and all these other things that add a really sufficient level of complexity that I think there's still a lot of room a lot of wiggle room for like the human dirty tricks department kind of mindset so I think it will be a little while yet but it depends kind of on the structure of the game so if elements of the game require creative thought and unique approaches I think that will continue to kind of be an arms race I only think it matters if they have arms if you have to actually go rock pick something that that's a really good example because you guys were notorious specifically for incorporating into CTF a whole bunch of side games where those side games were like every team gets issued this ancient hard drive that was like gigantic and that hard drive is painted your team's color and it's going to be out at the DEF CON shoot which I think still happens right yeah yeah the incorporating the DEF CON shoot was mine it turned out to be a really bad idea so basically I think a team got some bonus points for like having basically punched a hole with a bullet like through the most center point of the drive or something like that computers aren't going to do that for a little while yet so the other aspect with computers is you have to freeze the game chess doesn't change chess has been chess forever and the computer knows how to play chess how would the computer deal with this new architecture so we want to see something different every year we want to throw curve balls every year unless you advertise that some amount of time in advance like the CGC version that you guys did last year I think in DEF CON CTF in particular I don't think we'll ever see it because that's not the game we humans want to play all we'll ever need is 64k of RAM right so what probably wasn't clear earlier but four people up here were involved in some way and then for those that aren't aware DEF CON CTF last year was mostly more or less CGC compatible right and the winner of CGC the machine was a player in the CTF competition last year which kind of ties all these questions back into the panel so it's sort of a related question that I won't direct anybody in particular but so it's been a year right CGC was last year and the CTF that had the computer was a year ago was that a success sort of a end-off in history I would say it was definitely a success I really am a fan of the fact that Anger was really open source I think getting these tools getting these tools out to the general community is fantastic the fact that anybody can go and use these and learn how to write software, write tools to automatically re and go towards exploitation is fantastic lower the bar to entry it really makes it that much better learning how the teams did their job last year is probably the best thing that anyone in the audience any of us could imagine doing for their career for the next year and to be honest even future RE just in normal careers is going to be automated software is becoming so incredibly difficult some amount of automation is going to be required just to get even the low hanging fruit anymore we have to incorporate this idea into CTF so that we can still be representative of the wider security community I think the mistake people make when they look at CGC is to think that the goal was to build a purely autonomous system and that wasn't the goal the goal was to advance the state of the art in software analysis and what we saw last year, we'll see it this year is that software automatons making humans better at what they're doing is probably where we're going the best chess playing systems in the world are hybrid systems that pair computers with chess players not purely computers I think we've seen that too in even other CTFs now with modern things like anger being applied in an automated way to catch some of the low hanging fruit or point out areas of the code that's like this probably so we've actually had challenges that required anger automated reverse engineering I think it was called a thousand cuts by Vito where you were given a thousand binaries really fast and you had to be able to exploit them in computer speed not human speed there's even a challenge on ponable.kr called AEG where the requires you download it and it gives you a random binary every time and you have to auto-RE it and write the exploit so how international has CTF become when did international teams start showing up and do Americans still stand a chance? One of the first things that I remember happening was Dillon Canaveron from Hack in the Box reached out and asked if he could take CTF obviously do it yourself I mean do whatever you want that was taking CTF to Kuala Lumpur, Malaysia in I think 2001 so it's been coming for a long time it's been diversifying and spreading rapidly it's I think one of the most true and honest ways that hackers can really be better than each other because we're not very good I think otherwise at knowing like am I good enough and being able to say well I'm better than that guy is a huge huge foundation to stand on we have competitors from all over Korea and Taiwan, Taiwanese and Chinese and most major areas of the world are representative and actually two years ago Defcore from Korea actually won our game it was one of the first times that I can remember the nonopsided one and they were mostly Danish that they've won CTF I remember actually it kind of goes back to some of the slides you were talking about earlier with language barrier stuff I think the first fully non-English speaking team was a couple of the Korean teams that were fielded early on in the Kinshoto CTF and we actually had to really specifically sit down and think what are we going to do about this because we don't want to create a game the game isn't about English the game is about bits we actually tried to create we actually went through several iterations of attempting to create pictographic representations like stealing a key submission and things like that with stick figures and stuff like that because the language barrier was so significant that I remember actually one of the the captain of the Korean team at the time they did amazingly well in quals they like warped everyone in quals that year they showed up at the actual game and they had a lot of trouble but it was mostly about understanding what was going on and the actual mechanisms of the game and the captain of that team came up to me like I think this next year we're going to work on hacking English nice some of those stickers I think that might have become stickers later on at the top there you can see actually no that's exactly what I'm talking about we were like so I clearly didn't make those right then but that became a thing for later so I guess speaking of that what's with all the Japanese or Asian references right so like when Miles ran it it had like the big the red did you have to go to Miles the Japanese thing no I think that was actually we decided that we wanted to give away a championship belt so we gave away the root food championship it was like a boxing belt like a literal like WWF like belt yeah yeah so we made up this whole concept of root food and it was going to be a measure of how good you could compete against other people and it basically turned into hacking and the whole thing fell down but as a theme we had all the cultural references in Blade Runner and hackers I mean the connection to yes Iberpunk Chatsubo as were a whole lot of bad ideas and things that were growing past so we don't have to live in that world forever but I think it was just the way things everything grew up then we kind of just followed suit I mean Ken Shoto actually like the name is in Japanese right so for us there was that tie-in but it was mostly out of the influence of that same sort of cyberpunk picture that you guys had created and one of our founding members was a Korean woman who fits in a half rack it turns out and she and Data Angel did all of the what we call the ghetto news network we did news videos that were kind of in the Blade Runner if you remember the movie Blade Runner up on the screens there's some kind of Asian commercials I think over like a soda we kind of took that image and we just blew it out and made a whole fake news system they had her basically congratulating teams on not fucking up the network as I recall correctly but yeah so I just have some backup slides to show some of that I can't remember what that's called but it's like the Japanese thing for something religious the red thing in the background there and like certainly the Japanese stuff and things over time so what's the oh I wanted to this one in the background and the challenges between the ones that are too easy and the ones that are too hard everything is too easy hackers are so much better when they're under pressure than we are when we're sitting around trying to be cool we had to modify our scoring algorithm though we do it for quals everything starts out at 5 points except for babies first and the more people solve it the less it becomes worth so it's kind of they were self correcting it's funny if you write a challenge it's really hard to judge how hard it is when you write it yourself so it's 5 points when I get it but then so many people follow along that I lose my points yeah that's fantastic we also made other people in Kinshoto actually do all the challenges with no prior knowledge of them testing because the problem is it's really easy to implement a challenge where there's some leap of faith that you didn't realize you've made and that other people wouldn't and aren't part of a logical analysis progression so that being the case like we had found challenges actually there were one or two in your guys's game too and there have been others and specifically and usually in quals rounds where there's some leap of faith required actually I think the best examples are listen to the crazy ass explanations of the DEF CON badge challenges where they're like I realized that these dots were actually Morris code of geolocation things that were airports that then if you arrange the airports to enforce their letters and just all these things that you're like why did you think to do that in the first place and so like we didn't want to have challenges that were why did you think to do that in the first place we wanted there to be this sort of like thread through them and essentially the way that we enforced that was just entirely here you go other person on Kinshoto like go break this and solve it it's also really easy to make questions that are easy for your top 1% and so it's like no one gets it in 5 minutes and then where's the rest of the comments so for the real game that's what we were shooting for because we were basically if anyone was topping out that bar and solving all the challenges they weren't hard enough right so for us we considered it a failure in any year if someone solved all the challenges and so I think that the pushing of that scale to being higher we didn't really concern ourselves in the real game with that approachability or middle ground because that's the thing that everybody gets to play everybody gets to be inspired and do something one of the things that we did for Qualls was that we really, prior to us they had things like forensics and trivia and because those weren't really necessary for the actual finals game we only did ponables, we did like gorilla programming things that you have to know in order to play and succeed in finals so forensics where you have to go and take the first bit of every sector that was terrible that was also before you did flag open brace, closed brace that's when you were guessing we left us in because we wanted Qualls to be accessible to a large number of people it was a conscious decision that's why we added babies first because we wanted that same approachability without still be directly be directly applicable to the game play school, my first TPEX boy yeah have you guys ever seen evidence of collusion? we literally collated with teams all the time we only won by collusion where's the young is he here somewhere, there he is so I think at this stage in the game for us, most people have at least begin treating it like a gentleman's game we genuinely want to see who is the best and other teams feel the same way for example, a few years ago there was a team for some reason who put a wiki up on the game network and that wiki had their passwords on it and one of the teams actually the young ops letters came up and said hey guys, we saw that these guys had a wiki up you should probably tell them and that was during finals where they could have gone and wiped all their flags and just ran the board but it's really become it was like in our game we were owning other people's laptops and stuff too if it was connected to the network so that's sort of breaking outside of the intended path but that's not really collusion but there was legitimate team collusion at least in one of the ghetto hacker wins and I mentioned and did drive and stuff like that from back in the day because that's actually a really hard thing from a game mechanic to detect so I mentioned the idea like haha to like mess with the teams we had services that weren't exploitable but they were actually canary services because if someone submitted a token from that from another team either the game mechanism was broken and they managed to get tokens they shouldn't be able to or they're colluding so we were very concerned about that because it's almost impossible to totally prevent if you're just like slide a couple of tokens by here and there between two teams so that they can like get way far out ahead you make it so that it's a race between more than 8 or 10 right yeah for us the biggest problem with collusion happens in quals you have less control over quals than what's going on in the game itself because it's not on your own infrastructure they're not in a room I actually think our game mechanic that we do where if you submit more than like submit the keys that score drops for everybody you'll see so back when I actively played I had a team myself I would go and test the keys see if it worked or that's actually a bad move now because if you were to do that you actually dropped a score for everybody you're including your own team so if you start colluding there's a possibility that somebody will actually pull ahead of both of you so yeah because you end up cutting those because now the people that are you can dilute flags that are more important to other teams but if you're on top then you're on top so that's not going to affect too much because we had unique challenges were tweaked in a small way or everybody had their own service monitoring source IP addresses of submissions people would say I can't log into my account that's because you're using somebody else's creds and you're not supposed to see those creds unless you've been sharing the keys were different per team so we did set up some services that were specifically to detect teams that were sharing it it takes a massive amount of work and as an organizer you're increasingly playing game theory we're playing the game everybody else is doing an activity we make an activity and for us it's a game maybe that's why we like to talk about this so that's why you're organized because you want to play the game kind of yeah well we know the tricks for the game so we try to beat them yeah for sure so here's a perhaps interesting question so why is the finals attack defense style why don't you just continue jeopardy style not hacking because reality because that's bullshit if we wanted to be a puzzle game it would be a puzzle game we wanted to be a hacking game there's almost zero pressure in jeopardy you sit back and you answer questions in your living room attack defense there's a reactive nature to that and that can cause people to shut down and we've always wanted to make a contest that everyone could start everyone could start down the road to winning death guns ETF we needed something that's hard enough that it actually slows people down who aren't ready so it adds another game mechanic in terms of for example defense like patching I can patch my binary but there's a chance that it will now fail poles so I'm a whole SLA discussion from earlier we've also added a game mechanic in our game where we have a concurrency for patches so if you patch something and all you do is add 40 hex 40 to the side of the stack so everybody can see what you just patched if you take nothing else away from this conversation that is the piece of sheer genius that we've been missing this whole time since we started playing is when you add defense everyone else has an opportunity to do the same that turns it I think really into a game finally for the players because now do I accept your patches blind? Oh PPP, they definitely backdoor patches they'll patch something go ahead use our patch yeah and like purposely deploying patches that then cause other people to try and analyze those to figure out what was going on purposely deploying a harmless patch that doesn't actually necessarily fix anything but sufficiently complicate something or you use resources of the other team having to try and reverse engineer like what the hell does that patch mean I've definitely heard of patches with QMU bugs in them yep you can create an obfuscated code contest subcontest as a player now while you use this system which really ultimately becomes like a team captain leadership triage contest who came up with this idea so it was actually the guys who did CGC it was their idea just the most important thing that's happened what's the can't find the exact wording what's the hardest question or service that each of you have ever fielded in your tenure that's super tough yeah no preparation I would say every new CTF is so much harder than the old ones that listening to anybody about Hawaii Joe is probably a waste of time cause the new contests are just so much harder I invented a CPU in order to write bugs on it's such a high bar one of our guys, Salir he wrote a he wrote a mud a full featured mud where you had to go in and collect certain numbers of items from each from NPCs and then that's your shellcode so in the place then you actually cause you trigger the overflow and the number of items you have different types becomes your your shellcode so we had a mud a lot of times and then hit enter and it would crash the game is not the same thing what was really fun was he made us all god mode characters so we got to walk around in it and it was really funny because if somebody attacks you you can't stop attacking back it just happens automatically eventually you realize alright we don't hit these guys cause if you hit them I just die we had a mud a couple years but it was all just about the aspects of quals it was where everybody would talk and hang out and stuff during quals but effectively we ended up killing all the players a couple of times by accident like some monster one of the other kinshoto people was creating like got loose and we didn't I don't know so sir goon the guy who wrote the mud is also the guy who developed the badge that you saw up there so that badge he designed and we floated ourselves on a hop plate we put it on there and that was you had to exploit it over RF you can send them all text messages back and forth so a slight twist question that came in from the audience what about the most elaborate challenge that you were able to dream up that I couldn't quite pull off I was like still on the table let me make sure I can act I can actually talk ours was the world series of poker your printer challenge oh yeah I remember that one we started to destroy that thing it's not our printer one year at the Rio we rolled in right after the World Series of Poker had ended and in the back rooms were all the printers that they had used for the World Series of Pokers and we just wheeled one out and put one on every table everyone had an Ethernet jack we ran cable to every table and we wanted to work that into the game yeah some of them got fully disassembled and never reassembled there's actually some significant analysis on the internet definitely ended up having to pay for a few printers I think Guino are you here? I'll try to ask if I can mention his challenge but I guess I'll go ahead so he came up with an awesome idea GDB over phone so he dialed in and says this is your prompts and he actually got a decent way into the challenge I press one for R these are your registers step the bandwidth of throughput just wasn't enough operator real person and then the teams have to implement audio language processing in order to automate at Goa at Nolcon I was really proud of one of the kids there he made a DTMF only attack for IVRs he found a sequel injection attack in the default template for IVRs running asterisk so he had a compressed DTMF string that he could play into an IVR and get it to read out the username and password of all of the accounts allowed to edit the system and the phone came out he used his own voice so it was hard for me to understand but nothing to do with our game but you can make DTMF attack turns out it takes only about 50 milliseconds to get a digit through when did black badges start being awarded was it always 80? we got the first ones I think they were I don't know actually when black badges came out third time the team was huge they were when the team was I think the largest ever team 20-odd people and DTM standing there with one black badge so he went and grabbed and he could like spare 8 total 8 of us got black badges the rest of them did not we picked 8 people for our team for the first year because you guys did and you probably picked it because at some point he just had 8 badges 8 teams 8 players that was also based on the table layout at the time there were 8 approximate seats without whatever so we basically decided you couldn't have more than that at the table I think it's all the same we limit the size of the table just because of managing it all which doesn't really prevent large teams but what it does do is give large teams the same problem that large organizations in real life have communication problems and organizational problems because like 20 people sitting at a table can still kind of coalesce and self-organize but 20 people divided into two groups sitting in two different rooms can't in the real world you can buy a fractional T3 or buy a private line and have it run and have somebody put up some VPNs and buy some keys and walk them over there and all these things here you got 48 hours to end the contest from the time you know where you're going to be and getting a remote team connected is limiting the effectiveness of a large team was one of our driving ideas and the way we've done that is we actually don't when the game first started during duty tech they would just give you all the services immediately except for maybe a few handful of exceptions now we'll drop one service so sure you have 50 people let all 50 people look at one service good luck, have fun limiting parallelism is what we do and then maybe later we'll do two more services and we never thought of anything like that honestly I wish we had because limiting team size was a huge motivator for us too for a certain team two actually you and shellfish it's sort of interesting because it also plays into the distribution and turn inside of a team like how many rock stars are there and how fast does it drop off we also started implementing what we do we patch a service we'll implement the basic functionality of a service and then maybe midday on Saturday and the original bug is patched so now you have to pull the person who was originally the facto expert on that service and they now have to do this new thing because do you really want to ramp up somebody else on the same service or pull them so yeah we really spend a lot of time thinking about these game mechanics and how to make it more playable useful for the real world so how many women have participated I know that more smoked meat chicken at least had female participation unfortunately not enough we had more than 25% every year that we played we did not try to make a conscious inclusion of people we picked the best people we knew and in Seattle that mixed group we had women with us we had women with us every year with school root imagine they just don't want to smell us we start stinking after a day participation participation wise there have been a lot of mixed teams but by far it's obviously been lopsided what about as organizers so lighting the person who wrote this again just a full style like custom architecture implemented by a woman absolutely we're not looking for a winner no you are looking for it but we found it we found the winner of CTF I feel like just putting this up here even though it's clearly a troll but Frank wants to know he's hacked the Gibson and only got a partial download of the garbage file so can you help him look through it I think there's a person in the audience and we just found this out right before we walked into this room hacked the planet blank blank it's like the only question that's been posted that wasn't anonymous so I felt like I had to actually put it so here's a curious question at least to me did you do dry runs and how many people would participate in a dry run we never had time time plus we can't do dry run we did dry runs of individual services right so the idea of the dry run thing of your other member of kinshoto or whatever try and solve this challenge but other than that internal beta testing I guess you'd call it unit testing and not integration or systems testing the system goes online and works on the real game test them and we had to make changes like last second all the time like every year we did it as well we did it for all of our automated polling and all of that stuff the second part of the question doesn't make sense to me how did you stop any input that's an upset thing well that's why we didn't allow a dry run assuming that there was no folks on beta run or something it was only team it wasn't like we opened it to public beta it was only our internal dev team it's probably one of the reasons code hasn't even been transacted from hand to hand really is just because we all feel like at least I think we felt like we wanted to play Miles's game and then once we kind of got on the inside we didn't want to be tainted we didn't want to taint anybody else we wanted all of our friends to be able to enjoy the game that they love yep totally okay the program says another nine minutes but we can be done if you want to my beer is empty mine too last question then what teams do you admire most in the teams that compete? fury there are teams that just throw down and they don't do anything else they don't sleep, they don't eat do you remember the meds? I admire no whining yeah as an organizer that becomes a big problem I agree that's a big issue but just the fury the passion of it the fact that you can go and you spend 12 hours in front of a pc and then you go and spend all night in front of doing the same thing it's pretty hardcore yeah the endurance it really is at some point you know what's funny is the thing about that is right now the cost entry they get free entry now it wasn't when you guys was the case but they get 8 badges for per team but before when you had to pay when you go and you come here especially people coming from far away and they spend their entire weekend in front of a computer the exact same thing they do at home but they come in here and do it aside from that the thing that always shocked me as an organizer time to solve is the thing that I always looked at something drops radically faster than you expected that's like an instant I agree and the level it's not just the stamina like I can drink a bunch of Mountain Dew it's the I put in the 10 months getting to this and now you have to bring custom stuff for example when he was mentioning that if you pull a flag and you don't encrypt it it's gonna get caught all the teams have access to the packet captures so they see all the data coming across the network and they control what's coming in and out of the box so if you don't encrypt your your data back and forth and your shellcode your shellcode has to implement this encryption then you're gonna fail you're a gentleman just one of the things in time to crack I don't know if it was you guys one of our routers leaked one bit of information and the scoreboard had a different TTL off by one I remember that was us the TTL we could tell apart the packets that were your engine testing that a service was online from everyone else so they firewalled only the players 100% SLA 0% defense and they did this in maybe I don't know maybe seven minutes from the time we handed them their disks it was amazing we profiled TTL and your user agents on your yeah we missed that one we've got the cut off so the slides will be posted and the video will be posted thank the panelists thanks for coming