Hey, I'm Jonathan from AJ and Smart, and in this video you're going to learn the perfect one hour problem solving and decision making workshop that you can run for teams of any size. It's a really killer workshop. You can really use it for anything. In this video, I'm going to give you this step by step guide on exactly how to run LDJ, one of the most powerful workshops you can ever use for almost any problem solving scenario. It's super, super flexible, and I really, really hope you love it. Before I get started, just to let you know, you can get this booklet down below. There'll be a link so you can download it. So you'll have a step by step guide as well that you can print out if you want to have that in the workshop. But like I said, you can use me right now as your facilitator to run you through this exercise step by step. All right, let's dive right into it. So what is Lightning Decision Jam? Let me just put it up on the board and give you some examples of how you can use this, what it is, where it came from and all that before we jump in to the actual workshop. And down below in the description, if you already know what LDJ is, if you already know everything about it, then down in the description, you'll see like a timestamp where to skip to if you just want to go straight to the workshop. All right. So Lightning Decision Jam. So that's the name of the workshop. You can also call it LDJ. LDJ is about, you know, a three-year-old workshop that we came up with at AJ and Smart as a very quick way to help teams solve problems and make decisions. And that's the two main things that you'll be doing with this workshop. So the main things you'll be doing are solving problems and making decisions. But within that, you'll also be doing a lot of the things that most workshops kind of require. So you're going to be also doing brainstorming. You're going to be doing, you're also going to be doing idea prioritization. You're going to be doing problem finding and a lot of other things that you're going to need to bring a team together and make decisions fast. The workshop lasts about one hour, and you can do it for up to seven to eight people at one table. You can also have like hundreds of people in the room, but try to keep it to like seven to 10 people per table. But you can have one facilitator facilitating an entire room. For example, if you're doing like an innovation hackathon or something like that. So like I said, this video is going to be me running you and your teams through this. So feel free to use me as the facilitator. So I want to tell you about the supplies you're going to need for this workshop. Number one, get yourself a big chunky timer. This is a time timer, something that the team can see so that they can see just how much time is left in each exercise. Okay. In this video, we're not going to do every exercise in real time because then it would take a full hour. This video would be an hour. So you'll be pausing as the exercises start. So do have a timer like this. So when I tell you to set a timer for 10 minutes, you can just do that. And the whole team can see it. So that's the first thing. And this is a time timer. Next thing, give everybody a sharpie or a different sort of thick marker so that they can't write in detail. But you can see really clear to what everyone's writing. So I wouldn't go for a thin pen. I would go for a marker with this sort of thickness. Next, two different colors of sticky notes. We're using post-its here. You use whatever you want. These are square. Next, get yourself some rectangular sticky notes. Finally, get yourself two different colors of voting dots. And the last thing you'll need really is just some sort of space like this, like this flip chart or a whiteboard to stick things onto. So I usually recommend like just a wall. Doesn't really matter, okay? So just a space to stick stuff onto. If you don't have a whiteboard, then I really would recommend one more thing. And that is magic paper because magic paper is this like adhesive paper that can stick to anything and turn anything into a whiteboard. So I could stick this on. So you can stick this onto anything and it turns into a whiteboard basically. So this is adhesive paper on Amazon. It's called magic paper. There's probably other brands, but this is the one we use. It's great stuff. I always bring it with me when I'm doing a workshop. Now you just need one more thing before jumping into the workshop and that's a broad challenge. And I'm going to give you an example of a couple of broad challenges before we actually go into the workshop. So we don't want to go into this workshop completely empty-handed, even though you could. So you could go into this workshop if you're just doing like a team bonding experience and there's no real specific topic. You could use this as a retrospective, but it's usually super helpful if you have something that the team can work on. And so for example, you could say, OK, so guys, you know, we're having some problems with collaboration in our company or teamwork or the office environment or, you know, how we build products or our processes or something like that. It can be something extremely general, but what you need to do, what's great before this workshop starts is to turn that super general kind of challenge or problem into what we call a how might we statement. And this how might we statement is just a way to phrase a challenge as an opportunity. Let me give you a really clear example of how that looks. OK, so let's say I'm just going to write a few like broad kind of things you might want to work on in this workshop on the left hand side. I'm going to show you how to turn those into a how might we statement and every team should just have one how might we statement to work on. And often when we're doing a big mass workshop, everyone in the room, every team is still just working on one thing. So here's a couple of examples. So people think the office is too noisy. Collaboration isn't working. It takes too long to make products at this company to launch products. Another one might be we want to make the office environment a bit cooler. Literally, that might be one. OK, so these you could start with just these, but it's better if you turn them into how might we statements. It's easier to work on. So let me show you how you would turn these into how might we statements. It's super, super simple. The office is too noisy. What would that be as a how might we what I would say is if the office, if the problem is the office is too noisy, I would change that into something positive and broad, like how might we make the office a better place to work in. So office is too noisy. How might we make the office a better place to work in? I'll just do one more. OK, so. It takes too long to launch products at this company. What would you say the how might we for that might be? Yeah, how might we launch products faster? OK, so there you go. So let's just choose one of these to make it very simple. And let's just say that for this workshop that you're going to do and it doesn't really matter. You're going to choose your own how might we if you're using this as sort of a guided facilitation. But I'm going to use this first one as an example, just when I'm doing these exercises. So I'm going to be working on the office is too noisy. So I'm going to be working on how might we make the office a better place to work in. OK, so that's what I'm going to be focusing on. How might we make the office a better place to work in? OK. So let's actually now you've got everything you need to actually do the workshop. So from this point on, we're going running through the workshop step by step. And I'll be occasionally referring to this booklet, which you'll also have access to down in the description. And, you know, we got to say it. If this video, if this is something that you find helpful, if you want to see more things like this, do let us know in the comments. If you've any questions about LDJ, let us know in the comments and we will answer your questions. All right, let's go. So the first thing you're going to do is remind your team, hey, this is the challenge we're working on. So if you're working on the office is too noisy, I would say, hey, everybody, the challenge we're working on today is that the office is too noisy. I've rephrased it into an opportunity statement, which is how might we make this office a better place to work in? That's what we're focusing on. How might we make this office a better place to work in? So that's the thing that you will need to tell your teams before they're guided through. So what is it they're actually working on? Right. If you didn't choose a topic and it's just like, let's see what comes out of it. That's fine. But generally, it's helpful if you start with how might we and just tell them what the challenge is. OK, so the first exercise is called the sailboat and it's 10 minutes long and it's super, super simple. So the very start of this exercise, what I'm going to need someone on the team to do is literally draw a sailboat. So essentially, I need someone to draw something that looks like this pen is running out of ink. It's getting a little unsatisfying. That's supposed to be an anchor. OK, so what you need to draw is a sailboat with a sail and an anchor. That's a really bad anchor. I've drawn a terrible anchor here and some water. That's it.  And step one is start with the things that are working. So the first thing you're going to do is give everybody a block of sticky notes and give everybody four minutes to write what is working well. So what are the positives when it comes to the challenge that we're working on today? So if we're working on how might we make this office a better place to work? The first thing we want to do is ask everyone to silently without talking one post at a time, write down what is working well. So what are the things, even though we know this is a challenge, what are, first of all, the things that are actually moving us forward? What are things that are working well at this company? Let me give you some examples for the office environment challenge. So if I was thinking, hey, what's working well? I have to write as many of these as possible in four minutes. So what I'm going to write and remember one thing per post it. So everyone on the team is writing these completely alone. So first thing I'm going to write is, well, open kitchen is really cool. OK, so that's something that I like about the office. We have great natural light here and on and on and on and on for four minutes until I have, you know, up maybe 10 or three or four. It doesn't even matter just until I have a bunch of these positive things. Now, once the four minutes is up, what we're going to do is that each person on the team is going to spend one to two minutes sticking up their post-its. So one person at a time sticking up their post-its on the top of the sailboat and telling the team what they wrote. So this is how that's going to look. OK, I think the open kitchen is really cool. No discussion. I just say the thing I've written. I think we've got great natural light in this office. And then everyone does it and they're all stuck up on the top of the sailboat. And what does that do? It just gets people warmed up. It gives you kind of a good vibe for how to actually do the next steps of the exercise, which are a bit more negative. So we're looking at what are the positive things when it comes to the challenge we're working on first. And that whole step should take 10 minutes. Let me recap what's actually going to happen here. First, give everybody a block of post-its. Second, they get four minutes to write as many positive things as they can about the topic we're working on. Third, they're going to stick these up on the board one person at a time and say the things, but not discuss with anybody. And that's step one complete. Now, the next step, we're going to get a bit more negative. Everyone on the team still has their block of post-its. And now we're asking them to write, OK, when it comes to this topic and in this case, when it comes to the topic of making our office environment a better place to work, what's holding us back? What's bad? What's negative? And that's sort of what the representation of the anchor is here. So we're going to ask everyone to take four minutes to write as many negative things as they can on a post-it. Again, they're anonymous. So nobody's talking to each other. You're doing this completely anonymously and each person is writing as many as they can one per post-it. Please watch your handwriting because you will not be presenting the next ideas to anybody on the team. These are completely anonymous to everybody. So the next step is very simple, four minutes, write as many post-its as possible on the topic of what's holding us back.  Now, remember, we're not focusing on the noisiness of the office, but now someone's obviously going to write that. I can't concentrate in the open area. OK, that's something, something someone might write. Hard to make sales calls. OK, whatever. Internet too slow. So this is not related anymore to the problem of the noise. People are just putting out the things that they really find that they have issues with and et cetera, et cetera, et cetera, et cetera. So we ask people to write as many as possible. Now, whereas in the previous step, we asked people to put them up one by one and talk about them in this next step. Everybody just randomly sticks up their post-its as quickly as possible. Randomly, everyone at the same time.  And they're all anonymous because you don't know everyone's handwriting, right? And you don't have time to think about that. So what you're doing in this exercise, the final step is just ask people once they're done with the four minutes to stick all of their post-its up at the bottom of the ship and just spread them out. Don't try to put them all together and don't try to cluster them right now either. Right? You don't need to. If this is the same as this one, you don't need to cluster that right now. That's the next step. So don't worry about that. So all you're looking for is really to just, you know, get all the positive things on the top, all the negative things on the bottom for the topic that you're working on. All right, let's move on to the next step. OK, so the next step is you're going to give everyone on your team three sticky voting dots. Doesn't matter which color, but just red is what we generally use. And you're going to give them three minutes to silently vote on which negative chat. Forget the positives. Those were just for giving a good vibe, but they're going to focus on which things that are holding them back cause them the most irritation that they personally feel are subjectively the biggest challenges. So you're not asking everyone as a team to talk about which ones are the most tricky. You're asking, hey, vote on the things which you consider to be the most distressing, the most irritating. Now, what do we do about duplicates? We tell everyone on the team when you see a duplicate, do not discuss it. Just take it and cover the other duplicate. OK, that's it. You're not discussing. It's not a big deal. You take it, cover the other duplicate. OK, and if there's a vote underneath here and two votes here, that's then going to add up to three votes. So people get really focused on the duplicate problem and when they get focused on the duplicate problem, I can tell that you're someone who's way too focused on the details and not focused on the whole point of the exercise. So don't get too into those issues. So you're asking people to vote. They have three votes. The voting rules are yes, you can vote on your own. Yes, you can put two or three dots on one idea. If you like one challenge, if you like, that's up to you. The whole point is this three minutes should be silent, right? People should not talk to each other. They should not discuss. You're silently voting. And once those three minutes are up, once everyone has gotten rid of their three dots, what you're going to do. So we need someone on the table who's almost like the kind of co-facilitator. That person is just going to take those votes. And, you know, in this example, if this has two dots and this has one dot, the facilitator at the table or the assistant at the table should just put them to the side in order of priority in terms of this one has the most votes, this one has the least votes. Super simple. We'll put an image up on the screen of how that should look when you have multiple things there. Actually, let's make it a bit more realistic for you. You'll probably have like something that looks like this. You know, I'm going to put the dots on here to make it a bit more realistic. So like this, you'll have something up there. You might even have another one that has the same amount of votes up there and has four votes. So you have you'll have something like this. Your prioritized list won't look super neat. It'll look something like this. Now, what we're going to do because this LDJ exercise is all about speed, making decisions fast. We're going to just choose one thing to work on for now. And if your team has another hour later, you can choose the second thing to work on. So if it's super scientific method, if we're looking here, OK, so we've two things up at the top. We've two top voted challenges that we want to work on. There's scientific method for choosing the right one is to choose the one on the left. And that's the end of this step. That's the end of step three. So this team has decided that one of their biggest challenges is remember, we started with the offices too loud. Then we said, let's think about how we can actually improve the office space as just as a frame. And now we've gotten to the biggest voted pain point. And it's it's it's written in the words of your employees or your team. Open plan office is not good for concentrated work. OK, so that's what someone wrote. So the for when you want to do concentrated, like really focused work, the open plan office is not a perfect space for that. OK. So that's the challenge or that's the negative thing that got voted up. And what we want to do now as the as the person on the team who's kind of taking charge of being the assistant facilitator, what I want you to do is simply turn that challenge, that negative point into a how might we and as we saw earlier, it's pretty simple to do that. You're just trying to say, OK, how do we make this into something we can work on? And if we're looking at the open plan office is not good for concentrated or quiet work, what we want to write then and we're getting a bit more specific now is how might we create an environment that supports concentrated work? This is what the team is calling it. That's what the team at AJ and Smart calls it. But let's maybe just call it something like how might we create an environment that supports quiet work? OK, this is a three minute job. You shouldn't spend more time on it than that. And what we've done so you might think, OK, so now we're basically back to where we started. No, we started with offices too loud. Then we got everyone to just let everything out, let all the irritation out. Also kind of share some positive things. And now we have gotten to the point where we're like, yeah, OK, so we need to find some way for the office to support quiet work. But often in these exercises, you go in a totally different unexpected direction. I just wanted to keep it simple for this exercise. It could be that the internet is too slow was actually one of the biggest irritating points. So now we have a how might we that we're going to work on. So we're going to really create solutions for how might we create an environment that supports quiet work. So this is the thing you're going to be focusing your team on. Just to be super clear, you're only generating one how might we. You're just creating one how might we for the top voted challenge. If you have extra time later, you can turn all of these challenges into how might we put them in a backlog for later LDJs. And then you can start the LDJ already at this point. If you're if you're that clear that this is really the challenge. Right now, you're just bringing one how might we forward to the next step. OK, now we've got our second how might we of the day. So this how might we was generated in the LDJ workshop. Just to be super clear, the how might we you generated at the very start of the workshop is now no longer relevant. That was just to get things kickstarted. That was just to have a frame. This is now what we're 100 percent focusing on. So we're focusing on how might we create an environment that supports quiet work and that just might be, you know, whatever that is for your team. That might be completely different from the frame at the start. And that's totally fine. So now what you're going to do is you're going to give your team a different color of sticky note just to make things exciting. OK, just make things real exciting. And you're going to get them six minutes to create as many solutions as they can think of for this challenge, OK, as many solutions.  It doesn't matter how crazy they are. It's how might we create an environment that supports quiet work and we want them to write as many ideas as possible. OK, so what could be some ideas? Again, it's one per sticky note and it's completely anonymous. So what could be some ideas for something like this? Having designated quiet hours. OK, maybe we are building on top of having designated quiet hours. I might write quiet Wednesdays. Why can't I spell quiet? OK, look how I spelled quiet here. Quiet. What else? What else would help? Do not disturb if wearing headphones. Another thing could be buying people who need it. Noise canceling headphones. OK, so that's just an example of a couple of ideas that people should write. But they should keep writing until the six minutes is up. And then at the end of it, just like you saw in the earlier step, you're going to have everybody stand up and just quickly stick them all up randomly. No order. Just spread them around and now everyone's ideas are up on the board. There would be like 20 or 30 up here. I was just fixing my hair. So that's six minutes. And all you're looking for is to have a pile of different ideas. And you're going to follow the previous procedure that you learned. We're not going to deal with the duplicates right now. We'll deal with them in the next step. And that step is step six. OK, we're on to step six. You're giving everybody six dots, six voting dots, and you're giving them six minutes this time to go and look at the ideas that are on the board, the ideas that are on the wall and vote like they did before, vote on their own ideas if they want, put three dots on one idea if they think that's the best idea. But what they're voting on is what they subjectively feel like are the best solutions to this challenge, which could be the best answers to this challenge. So what they're doing just like last time is really just voting based on their own subjectivity. Now, what happens if they see a duplicate? What happens if someone sees that this one and this one are the same? What do they do? Yeah, you just do this. Do this without talking. OK, so duplicates should be dealt with just by the people who are voting and notice that there's a duplicate. Just take it off and move it and do not have a big discussion. Someone right now who's pedantic in the back of the room might be asking, oh, well, what if they're not identical? Please stop thinking about these problems. OK, that's not the point of this exercise. The point of this exercise is to really quickly have a few solutions at the end. Doesn't matter if this one is a slight variation of this one. Stick them together. Doesn't matter. Does not really affect the outcome of the end of the workshop. Why do I keep talking about this duplicate thing?  OK, so guess what? At the end of this exercise, you're going to have something that looks a little bit like this. Let's do it again. So once again, you'll have your prioritized list of ideas. They might not look as neat as that. They'll be a big mess. And that's all you're trying to get to in the six minutes. You're just trying to get to this point where you have a prioritized list of interesting ideas or at least ideas that the team the team think are interesting and you're going to run with those. Now, let me tell you something. If this had been a normal meeting, we would never have gotten to here in such a small amount of time, because everyone would have been discussing. You wouldn't have been able to visualize these ideas. That's something we talk about a lot on this channel.  You're working together alone. That's something you should really lock in if you're thinking about doing another meeting, just this idea of not talking to each other means there's even a bigger variation of ideas here. OK, we do now have a list of ideas, but now we need to execute on these ideas. We don't want to just say, OK, that's what's happening. We want to execute. We don't we don't just want to go with, OK, this has the most votes. So we're doing that. We want to see which things are actually practical, which things are feasible and which things are actually worth doing to do that. We're going to move on to step seven, where we're going to actually figure out which of these are worth trying out. OK, we're moving on to step seven. And for step seven, I want to get some magic paper just because this paper is a lot more expensive than this paper. Oh, that's bad. Oh, no. I've caused some bad things to happen here. It's all good. You got it. All right, so I've got all my ideas on the left here. They'll probably be a lot more in your workshop. But what you can do actually to limit it is just ignore everything that has less than two votes if you have a lot of votes. But I'm just putting three ideas up here, imagining that we have a very small workshop. What I'm going to do right now is I'm going to draw a very simple scale. Anyone who's worked at any sort of consultancy firm will have seen one of these a million times, but they're super useful. It's called a called an effort impact scale. And what this scale is going to help you to do. Obviously, it's extremely accurate. You can see the extreme accuracy here. What this is going to help you to do is figure out which ideas are worth trying. Let me explain what the effort impact to scale actually means. So everything in the top left here is something that we think would have a high impact on this challenge, meaning we think it would be a very effective solution if we were trying it on this challenge. But we also think it would be pretty easy because it would be low effort. Wow, that's really fading away. It's a beautiful. We think it's low effort. So this is the do now category. So everything that falls into this top left is like do now. That seems like something that's worth trying. Everything over here is still high impact, so we think it could really work. But it might be something that is big enough and complicated enough to be a project. So this is something that's make it a project. Down here, we're thinking maybe it's not going to work that well, but it's going to be easy. So this is something where we say make it a task, just put it into the backlog for now. And beautiful handwriting. And here we're going to say forget it because here is something that we're thinking is low impact, but also high effort, these are things we want to ignore for now. OK, so wouldn't it be great if we could have all of our ideas on this scale? So therefore we would know exactly what we're going to do as a team. Remember, you're going to have like maybe 10 things here that you need to get on the scale.  OK, impact means how much impact would this idea have on creating an environment that supports quiet work and effort is how much effort that would take in your company. So what we're going to do now is I'm going to show you how this works. And it's a 10 10 minute exercise. So it's one of the longest exercises of the entire L.D.J. Just because it causes some discussion and requires some thinking. OK, so what I'm going to do is the facilitator. And if you're on the table or you're the facilitator, this is what I need you to do is first straight away take the first top voted idea, if you can get it off. This idea, first of all, read it out. So this idea is do not disturb if wearing headphones. Let's make it a rule in our company that if someone's wearing headphones, they're not disturbed. So we don't need to think about how we would do that yet. But just if this were true, that it was a rule in our company that do not disturb if wearing headphones. How much of an impact do you think this would have? So as the facilitator, I put it in the center of the board and I just focus on impact and I say higher or lower impact. What do you think, guys, behind the camera? Higher impact around here. OK, so what I'm doing is I'm waiting for the team to sort of give it like a OK, yeah, so this is something new for LDJ 2020. We used to now immediately go to effort, but I want to show you a quicker way to do this. We're just going to focus on impact first. OK, so the next thing, noise cancelling headphones. If we bought noise cancelling headphones for everyone, how much of an impact do you think that would be in the company? I think it's slightly less than a year. Here? OK. So what if we had a whole day dedicated to quietness, quiet Wednesdays? Like higher impact. OK. And what if we had designated quiet hours during the day? OK, so around similar to this. OK, I'm just going to put it here for now. OK, let's look at these in terms of effort. If we were to buy noise cancelling headphones for people, how much effort do you think that would be? Not much effort, OK, so we're going to move it down the scale here, around here, just for now. Everything is sort of related to each other so you can move them around. So and by the way, this is how it should look on your team. It should be very simple and you're just asking your team. You're avoiding big conversations, but you're just asking questions that are very simple answers. OK, do not disturb if wearing headphones. How much effort would it be to implement that into our company as a rule that people follow? You don't think so? Well, actually, I know so. OK, so I know it's not really people. Well, the thing is, I mean, this is actually interesting and this is the conversations you should have. This requires a behavioral change in the company. But, well, so what do you think? Significant effort, what do you think? To get people to stick to a rule if you're wearing headphones, you don't contact them. OK, so quiet Wednesdays, a whole day dedicated to not making any noise. Implement them this. OK, where do you think it is on the scale? Higher or lower than where it is right now? I'm seeing lower. Lower, like here? OK. So having designated quiet hours during the week. OK, so just a quick example of how this actually works here. So now at the end of your exercise, you'll have maybe 10 or 11 things on here spread around the entire effort impact scale. But what you can see is how much basically which ideas are falling into the quick wins, which ideas are falling into this could be really great, but it's a project and which ideas are falling below the line. None of them fell below the line here. This is so ugly, it's unbelievable. You'll see in a graphic how that actually looks. You're taking 10 minutes just to get these things onto the effort impact scale and that's it. So that's all you're doing in this step. So we're on to the final step of this LDJ session, step number eight, and this step lasts five minutes. Very, very simple. All you're going to do now is you're going to take the top everything that falls into this do now category and you're going to turn it into an experiment, an actionable experiment, a time boxed actionable experiment that your team can try that someone on your team is going to be assigned and they can test out doing that thing and see how effective it actually is within the company. So let's take and how this is going to work. You're going to choose just for the sake of simplicity. You're going to choose things that fall into the do now box, but are on the easy side first, right? We're just starting with the easy things. So this happens to be the easiest one first. The buying people noise cancelling headphones. We want to think of an experiment, an actionable experiment. So what I do is the facilitator and what you're going to do with your team right now is you're going to say, OK, so noise cancelling headphones. What's a two week experiment that we can try that would help us see if this is actually worth doing or not? What's a two week experiment? OK, and so you're going to write on here two week experiment and then you're going to write the bullet points of what that experiment is going to look like and keep it super basic. So we don't want to buy noise cancelling headphones for the entire company. What we're going to do is we're going to buy noise cancelling headphones for maybe three to five people who need to do quiet work so that you can get people, you know, get people to volunteer who wants to volunteer for this. So actually, let's keep it really simple. So we're buying three sets of headphones. We're going to use them for two weeks. And then those people who volunteered when you get back as a group together are going to talk about their experiences using noise cancelling headphones and talk about how much it helped solve this problem. So you choose an actual date and you choose the actual people. So on March 29 back in this room, Dave, Sarah, blah, blah, blah, will come together and talk about their experiences. So that's the experiment for this one. Now, sometimes we will stop there. Sometimes we'll take the easiest one and we'll say, all right, we're just going to see how that works and we're just going to see how it affects this and then we'll come back and if that doesn't work, we'll take the next one. Because what can happen is if you're too ambitious, sometimes you just don't get anything done and you muddy the waters. So if you try to do noise cancelling headphones plus quiet Wednesdays plus having a designated having designated quiet hours, you might actually not be able to control whether or not that worked. But I still create actionable experiments for each of these. So that basically, oh, no, so that basically if this doesn't work, if this experiment doesn't work, we move on to the next one, which is closest to the left in the do now category and try that for two weeks. And that's basically the workshop. That's basically how you go from having a broad set of ideas to an actual actionable experiment that you can execute. We use LDJ every single week at AJ and Smart. If you would walk around the AJ and Smart office, you'll see these things all over the place because people are running this exercise all the time. Whenever we want to come together as a team, make decisions, solve problems, come up with ideas, come up with experiments. We use LDJ, we use Lightning Decision Jam, just because it's so simple and just because you can run through it without having to think about every single element of it, not having to do an open meeting where everything gets lost. You can just go step by step through it and know that there's going to be an outcome. LDJ is super powerful. Companies all over the world use it since they found our first YouTube video about this a couple of years ago. People have been replacing their meeting structure, replacing their like open brainstorming, replacing their like innovation workshops with this sort of super structured approach. And LDJ really, you know, it's super powerful. It's free. You just use it now because you've seen this video. You can get the download down below the e-book of this or whatever you call it. And really you can run it yourself to date. There's really no challenge in running this exercise. It's super simple, super powerful. Really hope you loved watching this video. I really hope you enjoy our workshop videos. Do let us know in the comments if you've run this exercise, if you're going to run this exercise, if you found this helpful, thank you so much. We really do appreciate it. Make sure you check out AJ and Smart on LinkedIn. You can check us out, AJSmart.com. We also have a website dedicated just to workshops, which is workshopper.com. So we're all over the place. Tons and tons and tons and tons and tons of free content. If you like this video, if it's something that really tickles your fancy, if you love workshops, make sure you subscribe, make sure you hit the like button and do hit the bell button because we have new videos every week. We have new workshop videos every week. You can take these, you can make them your own and run these for your clients and run these in your companies.  Because if you can help teams work better together with less stress, it's really just a killer career advantage. So make sure you check out this YouTube channel. It's really every week new videos. Hit the bell, hit the like, leave a comment. It helps us with the algorithm. Thank you so much.


Well, hello, thank you. We're really really grateful for the opportunity to come hang out with you guys This is kind of a passion for us and something that we really like to do for fun and also Thankfully for work. It's a good blend This is a presentation on security training at the harbor or running capture the flag with Docker I want to put a little bit of dev ops spin into it because a lot of this capture the flag Material is a lot more cyber security oriented, but obviously hosting a lot of the infrastructure and creating a lot of the Everything at scale right everything that we try to build we try to make it in a Stable way that is also automated and can be kind of encapsulated with code and and and dev ops, right? So what we wanted to bring is a conversation on our experience Hosting a capture the flag on a worldwide stage an international event and competition that we put on so We will dive right into it. We'll get through the obligatory introduction Kudos and thank you to Trevor for those wonderful words. He said it right my day job. I'm with the Department of Defense cyber training academy I'm a trained developer there and I used to teach with them I also do some kind of red team penetrations testing stuff on the side and in my free time I do like to play and create some of these capture the flag learning environments and training exercises I also have a pretty silly YouTube channel It showcases a lot of this kind of stuff with cyber security and capture the flag competitions and events and That's grown. It's a lot of fun. It's kind of a pet project But it's just my name if you have any interest you can go look me up and I'm sure you can track me down So John is also a government contractor. So he is a sub case you guys were curious So I'm actually active duty Coast Guard. I just graduated from the Coast Guard Academy this last year in 2019 So I am at Coast Guard cyber command as Trevor said I am a blue team lead. So most of my Day-to-day job is in like compliance and vulnerability assessments And I do at night. I do have free time. I play in cts With John sometimes by myself, I'm not as good as John is so I'll play in a lot of the beginner ones or me Intermediate John plays a lot more of the advanced, but I do also do some development for cts, which I did help with development for the ctf and I am I do some web development on the side. That's actually kind of like what my fun passion is so Yes, we are not six feet apart because John's camera cannot Cannot actually extend that far. So we are literally sitting right next to each other I don't know you might be on my side Okay, so Uh, this these are the topics we're going to run through this goes from like the background of the capture the flag itself that we ran as well as how we built it and also infrastructure That we implemented as well as infrastructure that we hoped to implement In the future because we are actually going to hopefully run another ctf in june so this talk and presentation is meant to be Both a story right us kind of hey explaining our experiences what we learned how we grew from it, etc And still kind of talk about some of the the tech behind it So we'll throw in a lot of terms between hey docker ns jail Some of those stuff that you saw in that little outline or agenda But bottom line is we're here for you There's not a whole lot of incentive. I guess We're not doing this for us. We're totally doing it for you We want to make this informal and casual and just kind of spend some time with you guys So please don't hesitate to ask any questions I know it's strange doing this virtually or digitally, but We're just happy to have the opportunity and spend some time with you disclaimer we We don't come to you with with gospel right some of the stuff may have worked for us and Worked again is still kind of an interesting word because maybe some stuff was successful. Maybe some wasn't Disclaimer if you had get any insight from this or you're interested in some of those concepts We can't promise you immediate success So your mileage may vary with some of this stuff, but we wanted to at least put that stamp on there just in case Why are we talking about this? So a lot of the stuff we did for the cts city This is the first large ctf that john had hosted and he had asked me to co-host with him So we made a lot. We put a lot of work into infrastructure and automation So doing that allows us to not rework or not We don't have to like make the same thing for every challenge You can just use one thing for all the challenges. You can use one thing to host everything Do you mind turning up your voice then your volume a little bit, please? Do you move to this mic? Is that okay? Yeah, I'm just going to change mics because apparently my mic is like causing issues. So it's fine. So We did have automated infrastructure for this and we For the most part it was stable throughout the entire time Also, we did learn some lessons with automation And that trusting players is not is not always a good thing to do with cts the whole point is to have fun and For everyone to learn and there are always going to be people that don't want that to happen So they're just kind of we're going to talk through some of the problems We had with specific instances or infrastructure that we had that maybe caused us to have some issues with certain challenges or wants us to Things that we can improve in the future things that we're going to implement That we're going to use in place of stuff that we used for this past ctf So a little bit of background to kind of set the stage for really the vessel of of the talk here A few months ago. I was asked to help host a virtual conference similar to Right what we're doing here the whole dev ops dc meetup and just doing some of this cool stuff online That was going to be hosted by ben Nomsack over at hacker one. They do a ton of bug bounty stuff And the cyber mentor or heath atoms and he's another kind of online personality for showcasing and training cyber security We wanted the event to be I guess wholesome right or for good, right? So There was no cost to attend. There weren't any ticket prices But if you did opt to buy a ticket all of the proceeds that went to the conference was going to be donated to charity Everything that came through either through sponsorship or through any ticket prices that people were willing to buy that would all go to the leukemia and lymphoma society So that was really cool for one thing because it set Just a really good tone for the conference and everything that we really wanted to do Even in I don't know the whole aspect of the capture the flag or the game or the competition that came with it So it was going to be a conference. So we had speaking tracks, right? And there were some time slots for speakers There were about 11 speakers that packed throughout a two day event This ran over a weekend and of course there was one Capture the flag and that was where we came in we were invited to go host that competition So, uh, as john said mileage may vary. So you want to kind of explain what a capture the flag is So capture the flag ctf same thing. Um, this is what we hosted for the conference itself So it's essentially a cyber security competition. It gamifies learning new tools new skills Or ways to think about how to solve a challenge Challenges can vary and they essentially will come in they'll come in a very a variety of forms You can have a service. That's an active thing that you connect to It can be a file that you download an image of pdf anything along those lines and It requires you to use either some hacking or cracking I guess or Possible defensive computer defensive computer systems to actually get a flag You want to get some form of it's usually an ascii text Flag usually we use tags for ours like jctf or verset con ctf was like our flag tag Some categories that we had in the ctf We had all of the ones that are on the slide with the exception of the oscent We didn't really have any need for oscent ones, but Some of the ones that had the active Services that you would connect to would have been we had web security So like connecting to a web server some of the Uh binary exploitation ones and I don't know if any of the other ones were most of the other most of the crypto We're going to be downloading a file or an image such as steganography or forensics But that dichotomy or the difference between kind of a static downloadable file Or a dynamic kind of an active service that you connect to Brings us to really the next topic and how we can kind of pour more into that dev ops sense The dynamic services and the challenges that the players would need to connect to and actively interact with Well They're going to have Vulnerabilities in them right because it's supposed to test cyber security The player might be doing some more offensive or kind of emulating an adversary and a lot of their techniques so If we create challenges or problems that have Vulnerable code or there's a real weakness or flaw in there Well, we don't want that running on our Actual computer right if i'm trying to host a website for someone to go hack into and run code on I don't want that on my legitimate laptop So we opted for one technology and maybe you guys have heard of docker Or you might already be super duper familiar with it, but docker is Incredible it is a technology that allows you to Essentially bundle up an application or kind of package it so that it can be shared and ran anywhere and it's encapsulated It's not so much a virtual machine, but it's a little bit smaller than that It's lighter and more compact than what you might expect in a virtual machine and for our purposes This is fantastic because we get to put them inside of a jailed Encapsulated environment where those vulnerabilities what the player is exploiting isn't going to break Our actual computer or our real infrastructure. It's all written in code, right? You can see some of the command line stuff there Creating a docker file or spinning up an instance or an image or a container It's all really easy to use and really easy to develop So we certainly use that for all of the challenges that we went ahead and worked with And when I say All of the challenges that means that we have a lot of them, right for every single challenge They all want to be in their own instance. They all want to be inside of their own container so not just one but 50 of them or more and they're all supposed to be Available and accessible to every player throughout the entire competition. So that means two days 48 hours All of these services need to be up and available The way that we did that is we utilized a docker compose Which again kind of a little sideline like carried with docker technology that allows us to Orchestrate or really kind of keep a symphony going of all of these different Instances and services. I can specifically say okay docker start All of them Let's bring up suddenly all 50 or the whole mass of services that I need to bring up or down and I can kind of declare and specify Hey, I want this network service to be available I want this port to be reachable by the player and accessible So docker and docker compose allowed us to do that and host the services On the back end really running that kind of the end of the competition event that the players would end up interacting with When they're working with challenges when they're working with the services So some of the issues that we did have with docker were Some privilege issues for users specifically configurations that we had that users were able to Change a flag edit Text different files add things delete things break things So that whole don't trust the user mentality We kind of had to actually take down one of our challenges because of that during the competition Which sucks, but eventually has we have to think of what is fair and fun for everyone because if a challenge just keeps breaking It's there's no point in keeping it up So this is when we kind of turn to nsjl as a possible improvement to that so John actually got invited to come to the google ctf finals in london due to his youtube channel he Talked to the people that created the google ctf And this is something that some of the developers over at google as an unofficial google product actually use for their challenges there For those services that are running or the web services that are running So nsjl is really cool. Um, it allows for process isolation and by default It's a read only file system So immediately the user is put into an area that they are not supposed to be able to change anything or do anything Other than what you intend for them to do So you can see there's a snippet there that kind of has Uh choices that you can give the user or what you're allowing as a default in there It does allow you to do customize configuration per connection, which is another really cool thing And nsjl, uh, is super easy to set up. It does have a github that I think we'll show at the end. Um I actually used this for a couple of new challenges that we made really cool infrastructure for just doing, um Like instant containers. So this is basically going to be a replacement for docker for us But it does allow us to integrate with docker containers that are pre-made So things that already have like a sequel database setup Or services setup that we want to use nsjl allows you to use those characteristics of that docker container But in a more secure fashion in nsjl's container outside of that So for the for the ctf, uh front end that we actually used, um, we use the framework ctfd So ctfd is one of the more common frameworks that you can use For a front end for hosting a ctf So pico ctf is a common one. You can also always just make your own But ctfd was a really easy one to do automation with uh for setting up And it allows you to have an admin panel that we could only look at and we could hide challenges We could have challenges ready to put out, uh, like at our periodic times So we could give the players more stuff to play with Uh, it also allowed for us to have stuff that you can have the dynamic or static challenges that we talked about earlier in a ctf As well as a lot of great visuals for players to see as well as for us to see So having a framework is really nice and an easy way to actually run the front end of a ctf Digital ocean is the cloud service the cloud infrastructure that we used to host all the back end and kind of the infrastructure for our ctf Um, i'm sure some of you are familiar with digital ocean digital ocean or aws has instances Digital ocean has droplets Uh google cloud and microsoft is there are two of the other ones that i people are probably familiar with but Primarily even for my personal web development. I used digital ocean droplets for stuff and That was john's choice for us to use for the ctf and it was really useful So for the front end, we had the ctfd service i talked about we had The docker services were the back end and we also had load balancers that john actually John actually set up all the load balancers I mostly helped with the back end servers and managing the ctfd and the discord that we had set up to Know when i needed to go check a challenge restart a challenge look at something so For we actually had seven of everything that way that when for example one challenge a user connected to the instance five A challenge went down We were able to have six other instances that if a another user connected the challenge wasn't down for everyone It was just down for some users So you can always have those backups and we were able to Isolate the one challenge that was down just have the other six be up and work on that challenge to try and fix it So then we could bring everything back up and everything would be operational So it was really helpful for us because we were the only two people running the ctf So a lot of times i would be working on one Or and or john be working on one But players could still play because the entire ctf wasn't down since there were other instances to connect to with those load balancers So something that we had to actually maintain the ctf were some scripts that we used That utilized ssh and ssh pass to actually connect to those back end Uh droplets i was talking about to like check challenges or to go in and actually Um edit challenges if we need to when someone did remove the flag or mess with the challenge We also had a bound script and that was really cool. So The idea of being able to like check a challenge or see if it was up or down Is really hard when you think about like well what how many people are going to join the ctf We had no idea when we had started developing the ctf How many people would actually sign up for the conference originally? I think it was we were thinking like oh just a couple hundred people But as we saw The registrations come in we were reaching like the thousands and we're like, okay Well us going individually to check seven instances of 40 different challenges is probably not ideal so John um started this writing this bounce script that allowed you to stop and start a service like on a dime So you would just run that if you there was a challenge people were saying oh, there's an issue I can't really get in so just to see if maybe stopping it and restarting it fixed it and it was just a weird Connection issue and that usually worked. Um, I would say for almost all the challenges the exception of two That was the fix nine out of ten times So another idea that we had To do that to kind of expound upon that stop and started service was a health check So we both worked on writing a script that actually would connect to those services To verify that it was running And then run that bounce script that we had talked about to actually See if it doesn't if it didn't respond to actually stop it and then restart it This was really helpful because As john said the challenges ran over 48 hours So we you know need to sleep eat do other things other than just like respond to discord messages and like check a cdf So having that health check actually Really helpful to know whether or not it was a user That was just having a weird issue if the challenge was legitimately down And we needed to go look at it more because if the bounce script Stopping it and then restarting it and then going immediately back down with the health check again Was a trigger to us like okay like stopping and starting it's not working still And there's got to be something else wrong with the challenge itself The issue that we had that we ran into Kind of this poor man's maintenance is what john galdick since it was just running scripts or Manually connecting to stuff a lot of the time or running a simple Bash script to do the bounce or the ssh logins was we had no ability to check Challenge solutions, so we weren't checking the flags on the challenges And we didn't have an automated way to like apply patches to a challenges or Like be able to see if it was one specific user that kept Dawson a challenge or that because we had one challenge in particular that we were pretty sure it was like one person Just hammering for no reason but was doing it and the challenge just kept breaking But we had no way to identify What was causing that so we eventually just had to take that challenge out Which is a bummer because it does cause issues at the end with teams that are like we solved everything but like They didn't end up at first because we took out a challenge that they had And all in all this was for fun So as organizers, we didn't really see it as that big of a deal There was no cash prize. This was just for a foundation But something in the future for us to look at and think about because if we are ever asked to do this for Something of monetary value like you have to keep that in mind competition and people being competitive with challenges so After the fact right After the weekend after 48 hours in a two-day competition where we had I guess you can see some statistics There are 67 challenges running All across seven front-end servers seven back-end servers balance from the load balancer Uh It was still a success It was actually really really well received. So we had just over I guess 2,500 players that registered Um, they asked hey, can you please keep the challenges online? This has been a lot of fun. We got a lot of training value out of this So I think even now I think that number has gone up to about 3000 participants, which is Just surreal and kind of crazy to me But when we asked for feedback a lot of it was positive everyone said hey We didn't experience any downtime whatsoever. The game was always available And probably the biggest staple or indicator that hey, we did a good job is that they invited us back So, uh, hopefully we'll be able to put on another event in competition like this Next month. I think june 13th is when we're planning another nom con and we'll have a whole new range of 50 challenges and 50 gamified uh Plays uh exercises to work with With that said, uh, we did have some success, but it's not all sunshine and roses, right? There were some things that we Got wrong or we could certainly do better with so We mentioned some of those dynamic services the ones that would allow the player to interact with them Some of them got vandalized, right? They could leave some files left over They could tear down some of the files that were there or remove the whole flag or the solution to begin with And that would just ruin it for other players Learning from that Well, we should make those docker containers read only and nsjl as an option Might make that a lot better and still kind of lock down some of the other processes Or memory or resources that might be necessary to hold that Another point While we had I guess over 2,500 participants a ton of teams that were registered and some people playing There were just about a few people I guess five or so participants that had solved all of the challenges And That's not a bad thing, right? That's good. It's it's awesome that they were able to roll through everything and But thankfully it wasn't super duper early on in the competition because that would probably bum out a few players like Well, I'm done with this game. I'm bored now But thankfully it was just before the end of the competition about a few hours left every uh, some of those teams were able to solve everything From the content creator and the developer standpoint I'm bummed at that because I want to keep providing That training value. I want to keep putting them into new exercises and having them learn as much as they can So I want to be able to make a game where no one solves everything No one should be able to 100 but that's hey some lesson to learn from next time We'll keep putting some more fire in there and as katlyn was saying One of the challenges was just too problematic and we had to end up pulling that one from the rotation So anyone that had solved that challenge, I guess the few people that had before it somehow got broken beyond belief They All the points that teams had gotten from that challenge were removed and it would it kind of shook up the score board Right because at the very end of the game those people that had solved everything They're really competitive and they want to say hey, I want that first place spot Because you remove that challenge it alters my standing in the scoreboard and that was a little bit of a storm but Okay, we'll know from next time if we release challenges throughout the competition or if we end up pulling anything We really need to be a little bit smarter on on that and how that experience is for the competitor and for the player So with a lot of that in mind How can we kind of do this better when we put on the same show next month? Truth be told a lot of the infrastructure that we created within digital ocean Some of that was by hand, right? We needed to have seven instances of the front end seven of the back end the load balancers and Of course, we have a database server and a redis caching server and some other kind of decorations off on the side We would create one instance Maybe of the front end and then clone it and duplicate it So we'd have seven of them to work alongside that load balancer Still we're manually creating that first instance or the database server or the caching server How can we remove that manual configuration? We want to automatically deploy all of that cloud infrastructure So i'm learning about and getting smart on terraform And that would totally replace all of that manual interaction We could automatically create that droplet or that instance in the digital ocean cloud infrastructure And then we could decorate it and provision it with the database installed with okay Configuration known to accept connections from whatever ip's those in the load balancer pool, etc, etc And again terraform is kind of like docker and that hey, this is a text-based thing You're gonna be writing some code or some configuration files and then it will Just speak it into existence and that's crazy cool It does have providers and it does have support for digital ocean So i'm really really happy about that you probably see it all the time with aws or i guess google cloud or anything But digital ocean isn't always as advertised So i did a little bit of digging and our research a little bit But it can go ahead and just spin up droplets for us so i can Rather than clicking around and spending however, maybe half hour or hours just building those machines I got a simple command that i can dot slash and hit enter on and suddenly everything exists And i can tear it all down whenever i need to That creates all the services that builds the whole game for me but How do i get away from that poor man's maintenance that i was doing with those dirty bash scripts or kind of Running around back and forth into one server and the next just trying to fight fires We had that problem where if someone were to Vandalize a challenge they'd destroy it for other people or maybe even There's an unintended solution and someone just was able to grab the flag really easily Well, i need to patch that i need to fix that challenge Problem is it exists across seven other back-end servers How do i do that as quickly as i can so it's not disrupting the game for these thousands of people that are playing So another thing i'm trying to get smart on and learning about is run deck So After i use terraform to deploy all of this infrastructure I want to be able to fan out all my commands and all my changes To each instance so i can script and run code on all of those front end or back end servers That would totally replace Those dirty bash scripts that i was doing in my poor man's maintenance Maybe we could update the challenge source code like to roll out a patch or to fix something or redeploy a challenge right? bounce it start and stop the service or This is another thought that we had because Not just even the dynamic active services and the challenges that players need to interact with but even those static downloadable files Those exist on the front end server rather than the back end server So again, we had another issue because if a player is trying to download a file And it exists on only one of the front end servers Well, then it's just a matter of luck if the person happens to be in the right rotation of their load balancer to get that correct file so Rundek would even help spreading all of those files across all of those front end servers and back end servers Rundek could allow us to automate Spreading out that infrastructure and provisioning it and working with it as we need to Yep, that's perfect So those are the technologies that i'm really looking forward to a little bit smarter on and How we could apply them into fixing some of the elements of the game that we fell short on There's still some stuff. I'm thinking about right We had a conversation as to we're getting a lot of traffic We're getting 2000 3000 people hitting these servers And maybe challenges go down or maybe something's getting Beaten up with a denial of service attack But i'm totally blind to that. I have no idea What ip address out there on the internet is trying to hammer these services if anyone might be doing any malicious activity I want to see okay, how much traffic am I get am I getting where is it coming from what ip addresses Do I need to be aware of or what firewall rules? Do I need a place to stop those people from just trying to flood this infrastructure? I needed network visibility and I still need that I'm still looking for what is a good solution to be able to get my eyes on all that traffic coming in Another thing that we discussed with our health check scripts and those bounce scripts that we were Starting and stopping a challenge whenever we need to maintain that That was good to test if the service was available and if it was live The problem was I didn't know if that service was still in that challenge was still working as intended Because maybe someone clobbered that flag file or it got vandalized and it's destroyed and it just broke In a different way that simply knocking on the door wouldn't find out So I need to automate solving that challenge. I need automated solution scripts That will go ahead and find that flag for me and verify. Hey, this challenge is not only still alive and well But it's also functional and working the way that we want it to That's another thing on our to-do list One last bullet point that we're thinking about is that All of these docker containers and these instances exist All throughout the game and they're accessible to every player, but they all exist Like in the whole scope accessible to every player and that's why we had some of those issues where Okay, someone could break that challenge for other players one thing that we could do Rather than using nsjl or really locking down and making that read only is we could Give each player their own dedicated specific Copy of the docker container so they can break it all day long. They're just ruining the fun for themselves and not everyone else Um, we're trying to figure that out a lot of other frameworks like captured the flag hosting frameworks in the front ends We mentioned ctfd Ctfd is what we're using But it doesn't have built-in support to be able to spawn these containers and instances per player Pico ctf another option does and so that's why you want to kind of survey the scenes when you're choosing your provider But also learning and also poking around It sounds like we might be able to use a plugin that will allow us to spawn up specific containers Individual specific to the player and that would be really really awesome Because then we don't need to worry about the read only file system trouble And if they want to break the whole instance They're only ruining for themselves and not for thousands of other people So, uh, I we can work with trevor or that to possibly give you guys the slides if you want If you like don't want to like copy and paste all these links But these were all the resources that we use kind of for when we were researching stuff either before We were doing the ctf during the ctf as well as after the ctf when we were working through stuff just kind of some general guidelines documentations good Um github references digital ocean references that kind of stuff So the github for nsjl is really awesome. I was in that a lot Uh, and digital ocean does have a lot of great tutorials for random Different tools that you want to integrate with it kind of similar to what john said it was a terraforma digital ocean Yeah, digital ocean has like really weird random if you search digital ocean with something There's a good chance. You'll find a tutorial on that which is really awesome And Now it's time for questions. I don't know if trevor has any or if there are any in the chat Immediately that we can answer but Yes, and those are both of john and i's contact info in there Yep, sorry about that katlyn. There are a list of questions. Um, I know there was a lot Like a smattering in new technology. Sorry that was a little overwhelming Well with the audience here what we typically do is we specialize in provisioning secure systems And you're sharing how to provision in secure systems But you're also doing it with a lot of the techniques and concepts that we utilize in enterprise architecture Which was mentioned early on which is providing the service to users quickly and timely load balancing High availability like that would that was really good examples to hear from But moving on with the questions, um, Peter asks if you have a demo for those that don't know what a ctf looks like Do you want them to watch your youtube channel? Oh boy The the easiest thing Yeah, to not I don't know how quickly or in the weeds. I would be able to spin something up Let me try it in another window here. But the easiest thing Yeah, go take a look at some of the stuff that I have out on on the interwebs on youtube We actually did do like a stream on john's youtube channel about Running the ctf and actually showing, uh tutorials on how we build stuff and solve stuff. So, um John's youtube does have actual like for this specific Conference and ctf like how to do that stuff. So yes, we do. I don't know if he'll be able to pull it up right now, but I can give you just a quick. Um, I will share out information If you're totally cool with the quick, uh, short demo here is a capture the flag pico ctf That is normally kind of targeted at more the younger age or just to train people and kind of get into the scene Thankfully, I was able to help out with that guy.  Can you calculate this specific thing? Or determine one specific program with a lot of different categories, right? So pico ctf as you can see it's set up more for like an individual view Uh ctfd is more of like blocks and it does it's It doesn't look as more maybe personal as the screen for pico looks It's more of kind of just like here are all the challenges So pico is definitely the one that's john talked about just like having the individual instances That is kind of what pico is more geared towards Does that sort of answer your question? Sorry Yeah, that's um, great I see that can be used both for breaking things, but also just testing general skills. Um, that's cool. I Have to think about ways to use that Well, thank you. Uh, the next question The victoria asks if there's any paid product that you didn't have access to that would have helped you run the event None of the ones that we found were I mean digital. Okay, so digital illusion does cost money So we did we did pay for those Yes, um My thoughts it's funny. No one has ever asked that question. Like how could money help you? um I think about the network visibility as one Area and I think gigamon I think I think those are really really solid on kind of visualizing some of that stuff That might help Uh, digital ocean, obviously. Yes hosting all that infrastructure takes some coin I think I've actually I think I got a little bit of headway today actually and Getting a little partnership or maybe some sponsors sponsorships set up, which I'm looking forward to But so far I We're like a one-man shop or a two-person shop, right? So how do we do this with open source technologies? How can we do this without breaking the bank?  It wasn't like a cost the only cost we had was the digital illusion um cost but For the most part like other than our time, which I guess like, you know time is money But we enjoy doing this stuff. So this was all done Um using any everything else we used was not a cost and all the tools we talked about terraform in that Uh, they're all free that you can use and implement. So An nsgl stuff that we are going to use in the future. So yeah Thank you. Thank you. Oh, and then how much did the digital ocean actually end up costing you at the end of the event? Oh boy. Yeah, so Monthly it's like I can say from a monthly perspective if you're just using it like day-to-day stuff Um, it's usually somewhere like five six dollars because that's what mine is on a monthly basis But I don't know how much yours was. Yeah So I feel really bad about this because a lot of people ask in order to kind of encapsulate or get an idea of Uh, how much should this cost?  I had some digital ocean credits. I had about $2,000 in digital ocean credits Uh, and then I was able to use it without yeah I got that as a prize for a recent game and We started with that amount in the pool But keep in mind we ran seven front end and seven back end and then three other servers So suddenly we add up to maybe 15 or 20 instances running in the cloud um For the weekend for a 48 hour event I should have done a better job of Capturing how much that really cost and then after the fact when the players asked Hey, can you please leave these challenges up online? We're like sure absolutely Only then did I go take a look back at digital ocean and see how much that had eaten up? I would think It would I would think the weekend itself maybe would have cost 500 or 600 dollars I don't know and that's totally my fault, but we're close. Maybe hopefully Okay, thank you, um Brandon asks two questions uh Did either of you get into any of the darker hardening processes the the user space Yeah username spaces set comp app armor se linux anything like that We tinkered a little bit with app armor In some set comp I truth be told in some of the challenges. I don't think we needed to go that far It wasn't that Extensive in some of the some of the things a lot of the ones that would require Some of those on the system command line defenses were showcasing and highlighting a really small trick Like hey, can you find this hidden file nested in us however many amount of of sim links or can you Use just this single command to escalate your privileges and now unlock the whole rest the file system or something something like that Where we intentionally had a vulnerability So putting in some of the defenses like app armor or some of the username spaces It wasn't something that we needed But maybe some more complex stuff. We certainly might have to but nsjl. I think looks really good for that Cool, thank you the the second question Brandon asks is what type of monitoring logging or metric solutions you used for the infrastructure Whether tools like prometheus or elastic elk Yeah Again, again, that's like a fault of like well, we didn't have all that visibility for for logs and Even cabana or elastic. We just didn't have that in place Yeah, so I I've used Cabana Oh here, let me stop this so you can do that. Yeah um So I've used uh cabana on that before and that was something I had mentioned like the day prior to the ctf But um, it just wasn't something that I think we thought of again We kind of We kind of just like just said. Oh, yeah, sure. Like we'll host a ctf um And did very a very bad job of uh prior like thoughts in terms of like monitoring and logging Let me try to defend myself. Okay, sorry background When I like to do these kind of events I like to bring them to local security conferences or local events like like a b-sides or like an o wasp conference So we're expecting a room of 20 or 30 people. That's what I'm saying Yeah, and and this was the first time They said let's put it on the internet Let's let's open it up to the whole world So it was a crazy learning experience that I think we did a decent job keeping it afloat and just really awesome to learn How can we make this better for the next time? All right, thank you um The last question is you a lot to ask how do you fix the bugs in real time or Did you consider that off limits for yourself? It do you keep it broken and you just move on to the next thing? So no, we we did not keep them broken There was one challenge like we said we had to take down but most of the time fixing the bugs was We would see Users saying like hey, I think this challenge might be down or like I was testing challenges periodically Most of them we did have solve scripts for or I had gone in or john had gone in and Solved them already when we made them Because I had made a couple challenges john made majority of the ones for the ctf we did I've made more for the next one just because I didn't have as many in my back pocket john had done a lot of development years prior but um We would go in real time and and fix them. We we would uh run our script and see if it would break Go back in kind of to how Like notes we had on when we actually made the challenge and kind of go back through those if we couldn't figure out Immediately what was wrong and we would fix it and for All of them except for one it was successful. Yeah, um as I guess the guy on keyboard Seeing all the messages and notifications that would Personally like set me on fire Caitlyn was probably Doing a good job tolerating me just kind of mentally completely scatterbrained because I have 2 000 people waiting for me to go push a fix And once I if I could either say Oh, I know how to do this. I can I can I can crank this out and I can I can get this done I would do that and then we need to go drag it over to Back end 1 back end 2 back end 3 back all the way up to 7 So again, that was a limitation and we're looking forward to having run deck or some of the other automation to put that in place Cool. Thank you very much. Yeah, I just want to say thank you very much for coming and speaking with us. Um You know, we use the same tools to build secure services to provide to users and you're using those very same tools to build insecure Services to provide to users. So it's really interesting to see that that uh, That's spin of the take there Thank you so much for having us. Thank you. Oh, absolutely Peter would you like to close out? Oh sure. Thanks. Um, I'm as one of the people helps run cloud.gov and It occurs to me be interesting to do a cft just general challenges like Team see if you can as a you know at a conference or something say Come in and see how long it takes you to spin up this particular kind of service So people could build their their skills not as a cyber security challenges as a general Skill leveling up challenge. So, um, thanks for So there's all kinds of different ways. I'm interested in taking this. Um Anyhow, thanks everyone for for showing and participation. Um, we'll try to keep you all in the loop as to, um, when our next Meetups gonna be and we will get the word out. Um on many of the one lovely resources that you share Um, thanks all. It's yeah, don't have turn on your cameras and wave. Goodbye If you're like doing that sort of thing and maybe I can capture a screenshot that put on the um Put on the page Okay, waving waving waving All right Bye. See you next month You