Good morning. Good afternoon. Good evening and welcome to another edition of get ups guide to the galaxy today We're talking about tecton and ci and I'm gonna hand it off to the get ups extraordinaire Christian Hernandez How you doing today Christian? I'm doing well. Um, I'm now feeling the meeting fatigue that everyone's feeling It took a while for me I'm a I'm a people person. So I I like talking to people anyways But right it's so it took me a little bit a while longer to hit that but I'm on now I'm excited To talk about tecton because ci is actually something we haven't really talked about No, and it's a critical component, right? It's it's actually a big piece of the component for your get ups workloads, right? It's it's it's kind of like without ci Really, you're just doing infrastructure as code without get up. So um, so you're not even really doing get ups If you're if you don't have your ci process. So I think um This this would be a good show to introduce Uh tecton as ci right kind of explored a little bit. I'm going to be doing a lot of hacking I'm going to be I have a presentation to be doing a lot of hacking. So everyone sit tight Get your questions in we have questions. They're always great. Yeah. Yeah, this is an officer.  I might go a little long today. So for those who are, um You know, who have a who have a hard stop just re what, you know, you can re watch the The the recording after but just for a warning, I might go long because I'm gonna kind of go over tecton a little bit Then I'm gonna hack at it. So I'm gonna go from zero to one And then I'm gonna go from one to ten, right? So This isn't gonna be um, not even a crash course. It's gonna be kind of like I'm gonna throw you in the deep end a little bit but um, you know, we we can always, uh, Re uh revisit this topic. So without further ado, I think I'm gonna share my screen here Um, where is that share screen button? It's always it's always the same place. Green button in the middle. Green button in the middle, man It's it's always in the same place yet. I can never find it. Um, I just want to make sure I'm sharing my uh, the the right desktop right here. All right, cool. Um, let's go to um Let me present this here. Um Also, this will be this will be shared out, right? Um Uh, Chris, uh, we'll we'll share this out. I gave him a copy of this. So I just gotta find that link and yeah, so Yeah, he'll he'll put that in the chat somewhere. Um, and so I'll uh I'm gonna go over like what tecton is right?   Um, oh also hillary was supposed to be on chat She said she's gonna abrade it silver by the way and that that's like her um Her background is ci so she'll keep me honest. Oh cool.  And so, um, what differentiates Cloud native ci cd and traditional ci cd, right? Uh, there's this table here. I'm not gonna read. Um I'm not gonna read off the tape this table here. There's the links in the chat so you guys can uh, take a look at that Uh presentation But mainly is the traditional uh, what we call traditional ci cd was built before cloud native architecture. So it has Um, it has a lot of tech debt behind it, right? And that's you know, I'm not saying that as a Um to be grudget or anything. It's just that's just the state the state of the world at that time and you know, this just happens right and so, um And the why behind the cloud native ci cd is essentially it's like we don't want to bring over all that debt That's like that onto a cloud native platform, right? So some things like the sea traditional ci cd is really designed for virtual machines and cloud native ci cd is designed for containers and really I think really that's the core of it and then everything else kind of um Kind of falls into place after like you keep that in mind, right?  Yeah, no peanut butter It's gonna come Peanut butter and jelly is for ci cd. Yeah, that's right. Exactly. This is the peanut butter and jelly aspect of it, right? Is you know cloud native, um, a ci process, right and and really let's let's let's let's not um Beat around the bushes or I'm talking Jenkins versus tank ton, right? And so, um, and not not to say anything bad about Jenkins but even Jenkins itself, they know Um, this limitation because there's projects like Jenkins X, right? Yeah, uh, if you get so which is cloud native Jenkins, right?  So, um And we have, uh, Opuship pipelines, which I think GA today It may or may not today today. Oh my gosh. Yeah, like Like literally a few hours ago, right? I've been on air. So I haven't been checking my email. Yeah And so, well, I'm on I'm on the chat with the the engineer. So it's either Either just came out or they're getting ready to stage it to to sit up, but it's it's here, right and, um Opuship pipelines is based on tecton and so Uh, the tecton which is kind of weird tecton is a ci tool, but it's part of the cd foundation I don't make the rules. So I don't know that right, but it is what it is, right? Um, it's a governing body, right? And um, people like like cloud bees Jenkins, uh tecton, they're they're all part of the this this yeah red hat We're all part of this foundation. So google's there, you know, it's it's a uh the governing body for a lot of these, um Uh projects. So, um, Opuship pipelines built on tecton and so what, um I want to go over kind of the the concepts for tecton before, right? So, um, so one is uh tecton We're gonna hear me Talk about steps, right? So steps is essentially like running commands in a container, right? Like get clone Maven build that sort of thing right now docker push um, and a task is essentially a list of steps So, um, so tasks is nothing but a collection of steps to take, right?   So you have a pipeline will run one task tasks, uh, run on the other tasks run the other tasks and they could run, um You know those tasks in parallel or uh Serially, right? So It's kind of like it builds on each other, right? You can think of those little Russian dolls, right?  Um It's actually deprecated. We don't use that anymore. However The only reason I put it up here is because if you if you look up tecton on google You're gonna see a lot about pipeline resources, but we don't use those anymore. Um Fair enough. And so yeah fair enough Uh a task task run Um, and this will make more sense once I once I started drilling down But a task run is a invocation of a task, right? So a task is nothing but a definition And task run is instantiation of that definition So it's kind of like the difference between an image And a container, right container is a running version of the image, right? So it's kind of that sort of paradigm And the same thing with pipeline run right pipeline run is an instantiation of a pipeline, right? So you define a pipeline And when you run it you have to um, you know, that's a pipeline run, right the the whole Um instantiation of that so And there's things like triggers, right tecton has the concept of triggers Just like any pipeline so I've talked a lot today, so I'm gonna drink a lot of water I'm in the same boat man like I've been talking all day since like 6 a.m. Just for context folks. We've been planning summit uh Getups con which is uh, you know a day zero event for cube con Yeah, cube con itself Open shift commons gathering And all the office hours for cube con. So yeah, we've been a little busy this week So forgive forgive me if I pause the drink water because it's yeah, it's a lot of talking has happened especially today. Yeah Especially today has been um, so um, so steps, right? So here when I talked about steps like maven build is a step Um, you know, uh, you know parse the ammo is a step get clone is a step, right? So, um, they're container specific So when you are providing environment variables Volumes can big map secrets, right? You are You're providing that to the the step, right? So that's what what a step is So a task is a collection of steps, right? So it's essentially a unit of work um like You know maven install is a task, right? So that'll have one or many steps, right?   Well, what you know, like my previous example of get clone, right? Okay. Well, we're repo you I made a clone And then once it clones it that's a task run right at the very end. It's like the end result Of of all that right so in a task Run runs in a uh in a pod Essentially, so each task runs in a pod And so a pipeline Is a collection of tasks, right? So you can kind of see how I'm building this already Is I have their steps the task is a collection of steps and a pipeline is a collection of tasks Um, and then basically defines the task execution All right in the order so you can do things like serially, right like one two three four Um, you can have things running parallel right things that are that you want to run at the same time that don't really Depend on each other and then you can put dependencies in right? I want you to run task one after task Sorry, I want you to write task two after task one Task four doesn't matter when you run it. You can run it just as long as you run it after task two But in parallel to task three that you can get really really crazy with this as I as I've learned right and so just just kind of a heads up I I learned tecton by just beating at it for a while because it just was it wasn't making sense to me But then once it made sense I go, okay. Yeah, I could I could see the power In in in tecton and I'll and I'll cause I guys I'll I'll show you some good resources after this right so Kind of powering through this right so if every task runs in a pod You know, how am I supposed to chain these tasks together in a pipeline right? So if I do a get clone and my next task is a a docker build How does the docker build See your get clone and just the answer to that as a pvc right a persistent volume. So um Your your tasks are running, you know pods and pods spread around, you know all kinds of nodes in your cluster So all you need to use a persistent volume right so and if Kind of kind of a fun fun fact if you use a persistent volume that has block storage all your All your tasks run on the same node just because that's just like you can't because it's um block storage You can't share it between nodes since so it's kind of fun secret that I found out that may or may not matter to you but just kind of secret Yeah Well, I was overloading like my node wasn't scheduling anything because I was like overloaded. I was like, oh, I'm using block storage Yeah, that makes sense like all the pods want to jump on the on this one node um Because so if you're um, so just you know read wide read write once versus read write many use read write many if you can um So that way your pods schedule in different nodes, so pipeline run kind of like a task run is basically An instantiation of your pipeline right you define your pipeline and when you say pipeline run It'll uh, it'll ask you the series of questions, right the parameters, whatever you put in and um, you know When I when I start hacking away, it'll make more sense, but um, that's what a pipeline running is, right? So Uh triggers are pretty cool. Triggers are something I'm going to show as well. Um It's basically um Something that well the way the reason I'm the way I'm using it is that I'm using it Get webhooks, right when I make a commit it'll trigger a pipeline essentially Um, but it's just a webhook essentially so you can curl it you can You can chat opposite if you want, right? Yeah, no, I mean webhooks are awesome for their versatility and ease of use mm-hmm Definitely definitely and so um, so yeah, so you have a um the event listener has a triggered template meaning that When you do a pipeline run It'll ask you for the parameters, but you can predetermine that parameter, right? So when someone hits this webhook pass these parameters to this pipeline And you know, you can you can already see how things are starting to chain together. So, um Um So tasks You as you'll see you can pretty much write anything is like you can do environment bash Even write a whole shell script of what your task will do. You can do uh env write python, right write a python script um But why uh, but why do that right? So, um, there's people like, um, you know, even red hatters people in the community have a um Like a catalog, right of reusable tasks um, so Why write it when someone else has already written it sort of thing and oh tecton hub is also in beta, right? So if you go to hub dash preview dot tecton dot dev um Can I put that in chat? Someone put in the chat. Yeah, I'll do it Yeah, so, um, you said preview Uh, actually it's hub dash preview dot That's right. I got and so, um And so, yeah, and that's kind of like the tecton hub. That's kind of like the you know, uh, the the known good, um Tasks that you can use so So basically if you want to write your own cool if you don't just, uh, um You know search the catalog Steel, you know what like i'm i'm a i'm a i'm a fan of reusing someone else's work if it's good, right? If it's good Yeah Great artist steel Yes, exactly. All right. Like seriously Yeah, I well, I don't have time to also like I don't want to write my own thing that I need to maintain, right? Like why I can add on to what someone else maintains. Yeah. Yeah, which which you'll see, um Some of the stuff that I've done here. So, um This is actually pretty easy, right with the pipeline architecture You uh define a pipeline You know, it'll run many tasks once you have those then you can reuse it, right and for many things, right? So, um Technically speaking you can define a one pipeline But use it for many um Pipeline runs So this is kind of a paradigm that the jankins people You know those those who are really into jankins Um, this will mess you up because it's usually a one-to-one relationship, right? Like this is my project foobar pipeline This is my project baz Pipeline and is there two distinct pipelines actually in tecton is just one pipeline Right, like it's generic enough to where it's one pipeline. The pipeline run Is what differentiates those pipeline runs and so Okay, so you could you could essentially pipeline is a template and then you can use that over and over and over and over and over again so um This will this mess me up a little bit. Um And I still haven't quite figured out how to um make a pipeline template, but that's the idea. So I'm just kind of just giving you the uh, um The the nirvana concept. So, um There you go. So, uh so cool real quick. I didn't take too much time because um, I wanted to how do I get exit out of this? There we go. I wanted to kind of go over um tecton in general uh So here if you go to the pipelines here, I already have a pipeline, but I won't talk about this pipeline And specifically if I want to create a pipeline I can come here and this has like a like a GUI Saying like hey, I want to do a like a get clone Yeah, and when when I do a get clone, you know, I enter this url You know this, um, you know branch or whatever, right? You can put main Um sub modules blah blah blah, right? So and then you can start chaining them together So I'm not going to do that. I'm actually going to go drop down the command line And go over a very simple hello world to begin with. So I'm going to put this in chat And so this is One of if not the best tech getting on getting started tecton Um articles I've ever read. So cedric wrote it. Of course it is. Yeah, so I'm saying he was like brand new. So yeah Yeah, yeah, yeah, so this is actually so for those who you don't know cedric was actually one of the uh one of our interns He's coming back this summer by the way. Yeah Yeah, yeah, so he by the way everyone absolutely loved him He was he wasn't even an intern on our team. He was the intern on our dev advocacy team Right, but like and he but he was he did a show on open shift tv and everything and he did it phenomenally. Yeah He actually taught me some something so it's always great when an when an intern can teach you something Absolutely love him. Um, so uh, he wrote this Um, so if I'm kind of going to be loosely following this blog So if you want to fair enough take a look you got there, right? So I got um So, uh, let's take a look at a simple task, right? What do they call this? Uh, there you go. Um So this is a task, right? So what I'm gonna say I'm gonna have I'm gonna have one step The name of it is say hello, right and then I'm gonna deploy a container Yeah, you behind container and which is this was really pretty cool is I could just do a bash Dash c and just start scripting away, right? And that's nice. It does echo. Hello world, right? So let's um, let's uh, OC apply this Come on. Come on cluster. Very good. Um And so I'm actually like hopefully they don't delete this cluster as I'm using it. So we'll see It should it's less than 24 hours. So I guess it's good. Um, so now if I do um OC get uh tasks Trasks this is there, but actually you there's a tecton cli. I can do uh, I think it's task list Right, it it essentially uses the same thing. So um, so now I have the task loaded up. Let's let's run it, right? So if I do a tecton uh task start Uh, she'll log and I want to do hello Can you extend the lifetime? I have a command for that if you have the permission. Oh, no Yeah, no, she's uh, um, she's talking about uh Something no, I'm not I'm not using that that um that cluster using something else. Yeah. Yeah, I'm using something else. Yeah. Um, so here a tecton task start, right? You see It says hello world pretty simple essentially, right? Did you say hey run this task? Uh, if I do OC get pods, I think yeah, it's completed, right? It ran actually ran a pod there and that just ran that and it exited. Um, so if I do OC get um task run Right, it's essentially there. So if I if I wanted to run it again, right? Run it again it'll Right hello world beautiful if I do ot get uh task list, right? It'll have one task I do get uh tasks right one task, but if I do a task run Right, there's multiple tasks. So this is kind of like this will show you it'll say Every time it runs it essentially creates a new pod And runs that's a new instantiation of that. So Um, cool. So that's like the simple hello world Let's add some parameters, right? So let's do parameters So this file here. So it says So I want to be able to reuse this right? I don't want to just say hello world I want to say hello and then someone's name Um So here I'm gonna parameterize it, right? I'm gonna say or I'm gonna create a parameter called person And this is kind of description name a person to greet Um, and this is it's a type of string, right? So you can actually have types. You have string You have bullions, you know, you have uh int Whatever, right? And then the default is world. So if no one specifies anything, I'm just gonna say hello world And essentially here. This is how you specify the parameters. You can say param dot person It's essentially a uh follow the json path, right? It's like prams Person, right? So hello that straightforward So if I do Oh, let's see. Well, we'll see right famous last words Okay, so I got that Uh, tecton Let's do a task run Where am I? It's clear this screen here. Um, thank you Just get pretty far down there So now now when I start the task it'll ask me um Hey, you need a parameter. You didn't specify one the default world, right? So if I press enter, it'll just say hello world um and There you go, right if I run it again, I can say, uh, hello christian Right and you provide the parameter on the command line Great question. I love I love it when people ask questions That of the next thing I'm not about to show, right? So, um So you can say uh dash p for parameter, right per person equals, uh We go, uh, Bob. It doesn't matter Bob, there we go. Uh, oh Task I'll name the argument Yeah, no, I forgot to put hello the name of the task. Oh, duh Yeah, yeah, I deleted the task name for whatever reason, right? And so here it'll um You know, then you provide that in the command line. Hello, bob another cool thing Another cool thing is I think it's dash dash use uh Use uh prams defaults, right? Um, again, I forget the name of the oh, yeah So then you can do um pram defaults, right? So since I set up a A default of world have a default. Yeah, so that way it doesn't ask you interactively. You can just pass it long So very nice. Um, so those are parameters, right? You can pass that to a task so now, um Now let's create a pipeline Right. I have yep pipeline right see here.  I like I like cedric's um Names, right? Is it's yeah, he he he always keeps you engaged even in writing. So it's always fun. Um, right So it's it's something to say, right? Uh, so what should I say the defaults? Hello Um, how long I should wait, right? So it's kind of giving me A uh Something to sleep on right because right step here It says uh sleep pram Pause duration and then say whatever you're gonna say, right? So I kind of chain those together. That's nice So you can um, yeah, so you can kind of just see you know, asleep however long then echo whatever you're gonna say So in the pipeline just come down here Now I have two tasks. So task one. This is kind of hard to see. So I'm gonna highlight it a little bit. So this is task one Um From uh, 29 to 36. So task one I'm gonna pass. I want you to sleep for two seconds and then say hello. This is the first task Um, and the task rest Um, and the task ref meaning like, oh, I want you to apply this parameter to this task So the task we just defined above say something second task Second task. I'm not sleeping anything. It's right. So it's going to take the default which is zero sleep No sleep sleep. No sleep. No sleep for you. It's kind of like us. No sleep. We're busy. Um No, I sleep. Sorry You don't sleep I dedicated time to sleeping I Yeah, I provided eight hours all right to my to my task. So um, I say this I have to check this box to my list. Yes, right And then um, and then I'm I'm referencing the same task. So here you can see that you can have a Generic task, but you can use it multiple times in your pipeline. So Um, cool. So let's uh, first let's do oc get pipeline All right, there's no pipelines if I don't see apply And then get pipeline my pipelines there And so, uh, let's run this pipeline Uh, so that's tecton, uh Pipeline start And then oh, what was the name say things? There we go So say things and then uh, show log So this is going to fire up this will actually fire up two pods It'll fire up the first pod for the first task And second pod for the second task, right? And so this says um, hi this is the first task Right runs the first task And then Once that's done It'll fire up the next pod, right? And then run the second task. So Pretty straightforward. Um And then What's really cool is oops When you start chaining these uh together together here I'm going to say things in order, right? And so, um, I don't I don't have the task to find here because it's already I've already defined it Right. I already added that in Um, so first task I'm going to pause for two seconds say hello Right same as the other pipeline This pie the sex pipeline is also I'm going to pause for two seconds And then but if you notice here Lines 23 and 24 I'm creating a um Dependency here. I want to say Run this second task Right after the first one um So in the previous example and we we You couldn't see it because just because of the the nature of the thing But both of those pods fired up at the same time um Because we didn't build a dependency So the fact that the first one came in first and the second one came in second was just a coincidence um This here i'm creating a hard dependency. I'm going to say don't start this task unless the first one finishes Third task I have um Also dependency here on the third third task run after the first task But i'm not putting a dependency on the second task, right? Um It I see I have no dependency on the second task only the first task So this so this kind of says the second task and the third task will run at the same time However, I want you to note the value The third task for to sleep is one And the second one is two. So this will produce an interesting result And then um the fourth task I have a hard dependency on the second and third task. So um A little bit confusing especially when you get when you get down to this level Yeah, like And so, you know when I think of sequencing tasks together You know ansible immediately comes to mind and there is some you know syntax similarities here, but the yeah, yeah Fusion is a little different, right? Like Yeah, the way you make it happen. Yeah, exactly. There's Run afters and you know making sure the order of operation stays The way it is is interesting. Yeah Yeah, and and order in order of operation is important and I'll I'll sort of lie in a little bit. Um We'll see. So that's Yeah, yeah, it's always gonna be somewhat important Yeah, exactly. So here let's go. We'll see apply Uh pipeline order All right, so let's do a tecton pipeline list Right, so now I have say things in order Uh, let me clear this here So tecton pipeline uh start And then what is it? Uh, oops Things in order order show log, right? This is always a fun part, right?  Nice beautiful first task so, uh Both of these happen second task their task, right? And then um So these two run in parallel And then this happened after the second one. Yeah, and so, um, you can put dependencies on that. Um I feel like that's like that's a little bit cleaner than you know, like what I was thinking. Yeah, right like The fact you can do tasks in parallel so easily right like that's kind of an advanced feature for some tooling Like this is just right out of the box like we're gonna embrace this concept Yeah, yeah, exactly. And so, um, so yeah, this is kind of like the the crash course in tecton So I put that in the chat. It's it's actually I was struggling with tecton and and And I found out from cedric and I was like, okay, this is like this was the nirvana At least for for me to for me to get it Right That's doing complex things as funnily enough complex, right? And so, um And so how does this how does this fit? tecton fit with argo cd, right?  And so, um, first thing I'm doing is I'm cloning the repo And that's like one task, right? I cloned the repo. Um, and then I Set an image tag, right? So I define so, um, I'm not a big fan of floating tags, right? So floating tags for those of you don't know is like dev prod stage Right. Yeah. So like long live tags, right? Like yeah. Yeah. I'm not I'm not I'm not a big fan of those tags I'm actually more of a fan of like a version like v 2.1.3 or What I'm doing here is I'm just taking the the I'm chopping off the last part of the hash in my repository my my code repository that specific get commit. I'm taking that tag. I'm taking that hash And I'm just using it as an image tag. So I know what version is what? Um And so that's task two Test three is um has many steps, right? So here these, um, clone has one step Single step this one has three, right? I have a A build command, right? I'm using build a So that's um, I used to use source to image for now. I'm just using build a And uh, it does the build So once it does that it takes that the end results and then pushes it to quay. So I have uh, or for those Right, whatever And then what I do is I take those end results and I load it up in a variable, right? Because I use Every step I load what I'm doing into variables. So I can use it in downstream steps. Um I So here I have a parallel task, right? I have two tasks that are independent of each other. So I run them in parallel um So I do a scopio copy of the Task of the of the of the tag that I'm using to latest, right? So my my dev version is always latest. So I just do a scopio copy making sure The image is updated and then I'm cloning my deploy repo So once I finish, you know, once I'm finished doing Um my application, right? So if I come here and I show you yeah This this is my code repo. This is nothing but my source code, right? And this is what this is what I'm building here I actually have a deployment repo. So I have two repos for my application I have one for the actual source code and one for the actual get opsie Stuff, right, which makes perfect sense Yeah, you want to keep those separate? I I'm on the big proponent It trying to do it all in one repo is Heading messy at best. Yeah, really messy. Well, and also they have different life cycles, right? My application is going to be changing a lot versus my infrastructure not as often not as often as you think and so, um So I'm cloning that deployment repo, right the deployment uh repo where I defined A lot of these like right. I'm gonna deploy this. I have an overlay for a dev overlay for production, right? And I'll I'll show you how that works a little bit. So, um Well, watch it Watch it be no, okay. Good. I like watch it be destroyed and so I have um you know, I cloned the, um The deployment repo and I do that in parallel with scopio copy because they're they're not related So now I'm using customize to patch um the image Um inside my deployment repo So I use customize set image and that'll update my customized file in the git repo And then I actually commit to dev So since it's my dev environment, I just Screw it, right? Just push it straight to dev. I want to see This could be sandbox. It could be whatever environment you want But in my case, it's dev. So I just want to do a git push But that update that I made Next step is I'm going to patch prod um, so I want to say hey, um Production I want to um, I want to also update the image for production But instead of committing I want to create a branch And then the last step is create a pr right, um And so notice all of the steps here in tecton There's absolutely nothing to do with argo right Nothing to do with argo. Um Like at all, right all argo is doing and here Here's my dev environment. All argo is doing is looking at my git repo so, um tecton right does the ci Argo does the cd part and git is a center Of everything right and so, um, you know Chris we always say that git ops, uh, isn't a tool to practice right, um But if git ops was is is a tool. It's not argo. It's not flux. It's not acm It's actually git right that's like your git ops tool is git because The workflow is all happening git and so, um And that's why they're kind of not They're not interacting directly with each other argo cd or ci You know, they don't really necessarily know about each other Um, but they are working together through your git workflows and so let's um So I have this my source code repository. This is my, um Deployment right so I have welcome app and welcome deploy Um, I use overlays for my environments Um, this can spark a very deep discussion and this will be another show. This will be another show. Um, I but um, you know I'm a big fan You know the difference between production and development or like a dev and sandbox Is actually Not a lot, right? What's what's really different is right? Maybe the the image you're using and the secrets environment variables But the the base is and now i'm kind of talking about customized the base is usually the same deployment custom like all deployment service route It's the same. You're just kind of patching it. So that's why I do that. Um But we'll talk about directory structure that that's a whole another show by itself. Um So I have we do need to do that because that's usually like One of the first questions that people start asking me. So, you know stuff about argo, right? Yeah, I know a little How do you know the directories directories?  Well, let's go get some Then they eat real quick and we'll talk Yeah, we'll talk here. Um, and so yeah, so my deployment here Is uh, this is kind of where um, the magic this is the workflow here is kind of what glues it together So, uh, let's make a change, right? So I have my development here. It says blue In my how do I move this thing? There we go. Um My production says blue as well. So, um, they're the same currently. So let's make a change Let's go to visual studio code Um Where is blue? Let me just sue a search of that I'll call this green, right? There you go. Stand standard blue green deployment. May why make, um Make changes here. Why did something yellow? Yeah, I guess. Yeah, I guess. Yeah Uh Yeah, yeah, exactly. Uh, so get add that guy get commit and we'll do, um updating to green And then I have, um Sign commits now, so I'm I'm doing the cool crowd. Yeah, joining the gpg sign commits I've kind of given up on gpg to be honest with you. Oh Actually, I did it with key base and it was just like it was like a breeze. So well, yeah If you do it with key base, but then it's like it's like proprietary pgp Yeah, yeah, it's it's yeah, it's kind of weird. Yeah. So, um, I actually got this from uh from scott our friends over at we've worked He kind of um, I was like, oh, I think I'm gonna do that. So anyway, um, so I pushed that up Oh, I actually didn't push it up. So I do a git push With my code Right and so this Should trigger rise about to say yeah, what are we triggering here? Uh, let's go to pipelines Uh, so you see pipeline run it triggered that right and so, um So as that's running, I'll show you where that's set up. Um So here, uh, let me go back real quick. It's going to go through all those steps, right?  Um, and so I'll keep this up here, but I set up a My settings So since this is a cooking show, I did this beforehand. Um Webhooks, right? I set up a webhook here. Uh, this is what it's called an invent listener, right? And it's just a standard, um, webhook here If something happens Yeah, if something happens, so here triggers I set up the the invent listener trigger. That's um You know, one of the things there. So next where is my pipeline and so, um Yeah, here. So here I cloned the repo right doesn't get cloned kind of straight forward uh set image tag essentially, um I take the hash of the commit and then I am Providing it as the tag For quay. So quay.io welcome app And then just a hash, right? So I can identify a specific commit to a specific image Um This part takes a while right is build a essentially I'm just yeah build just gotta pull down all the layers and everything and All that fun stuff But this is how you can use your cluster to build your images, right? Like yeah, yeah and so and this is kind of the the um the idea about cloud native ci is that You know, it's kind of like a serverless type of thing is like you don't have a monolithic You don't have like a platform. I mean nothing.  You're leveraging your own cluster to do the builds Because everything's containers, right? So You know and You don't have to have things running all the time. You just kind of serverless type of thing where It spins up when you need it and then you know once this clone repo task is done that pod leaves and those resources free up for other things so Can you set up any caching along the way just out of curiosity? Yeah, so um, that's that's that's a good question. So um, I'm doing a clone repo And doing set image tag and you're like, how how how are you doing all that? If you go to storage and persistent volume claims, right? I have a volume Okay, and you can set up your pipeline to use the same volume So right now the way I have it set up. I it just creates a new volume each time. Okay However, you can just tell it to use the same volume. Yeah, you could totally be like, hey Just continue to use this volume. Yeah, you don't have to check that every time you just do an update Whatever.  Oh, yeah Cash if you want to cash your maven, right because you don't want to download maven build download the internet, right? You don't want to do that all the time. Yeah, you need to cash that. Yeah Yeah, so you want to you want to be able to cash some of the things and you just reuse the same storage and um And uh, what about external so here here's a question. What about external artifact repositories like jayfrog jayfrog? That makes sense. Yeah, could you pull from that? Yeah You can pull from that as well. So you can have not only caching locally Right by yeah or artifact or whatever you can like cash it you know on on the storage and um How do you clone a private repository? That's a good question.   We don't care about annotations. So you can provide um A what am I doing here? Oh, you don't provide it in the task. I'm sorry. You provide it in a secret I was about to say you is this a config map or secret thing? Yeah, it's it's a um, where's uh, I think it's under compute. I think is it is under compute No, no It goes to show you how how often I use the how often I go into the ui See here, uh networking work workloads. Maybe it's gotta be Secret there it is. Yeah It's gotta be someplace up top and then and I do get so I have a I have a token and an sshk and you just load it up um in Uh So this is kind of like the black magic. I kind of wish we did this with jafar, but this is like middle of the night with jafar um When you add a get secret. Oh, that's right. So you add a um now I remember so you add a the your secrets and when you are Calling them in the task No in the the the task runs as a specific um service account Oh called called pipeline nice service account pipeline. You just have those Yeah, so I I said secrets. Yeah, nice. Yeah, so the service account pipeline So first I create the secret and then I say the service account has access to this secret and this secret And tecton just knows to use it. It's mad. It's actually just magic at that point, right? So you say um It's just magic as in lines of go code Yeah, well, yeah So it looks like my pipeline finished. Let's look at the pipeline run Um, so my pipeline run all green beautiful nice task runs Right, you can see uh where the logs go so you can say so you can see oh you can see oh wow fancy Yeah, so I have you know, uh, we left off here building push, right? So step one build it step two push it step three um You know cat the information I need and store it until variable because I'm using it later um So I clone so these two ran at the same time clone deployment repo Tag to latest, right? So I'm doing a scopio copy Um, right. I'm gonna say quay this hash Is up to latest Patching dev. I'm running this customize command. So customize. Um, I was doing this weird yq um Thing in order to patch it and then someone said hey, you know, you can do that with customize. I was like thank god I'm like I I was like, uh, that's actually um, andrew pit one of the the canadiens that I Ah, got it. Um, like hey, you can do a customize edit. I'm like, this is really cool Uh, so basically what that does is it it updates your customize dot yaml file to say um update the image for this deployment um when you read it in so I commit to dev because I like to live dangerously, but um I using the get commit right command to just push the dev I patch prod right same same it's actually patch prod and patch dev are the same task except, um, what repo I I target uh, what directory I target i'm sorry Uh branch to prod right i'm creating a branch And then I do a pr right. So how does this look like? Uh, argo should have already Yeah, so argo if I refresh dev It says green but prod does not say green I have a process I have a process right I go to my pr And look at you see here magic fm right Magic here is the p is my ci process created this pull quest and this pull request is basically I go to files change right and um If you noticed here it says overlays prod customization. Let's look at the full file Right in this basically the section in the customize edit that you saw on there Essentially updates this this field here if I go back so you can see here So here, you know, you can you know, I can say this is dumb right like you can go through your You know start the review start the process That sort of thing right so um kind of just to Go back here. So what I'm doing is I I commit to dev directly But the way I'm doing the gating is essentially I'm doing it through a pr And using get using your workflows, right? My workflows pretty simple, but your workflows will be a little more complex, right? There's gonna be code review You know, you have protected branches.  All I have to do is I go, okay cold merge pro request You know merging my own pull requests. Um, I don't need this. I don't need this. I don't need this branch anymore. So I'll just delete it. Um And then the code is updated and so if I go to um Do a hard refresh on You see that argo says, oh, hey, there's a new image updated And argo automatically creates a new pod And so if I go to my application, it should say green now The route stale route Come on. There we go green. Um, so now my dev matches production because I just approved that pr. So Um, there is a question here. Let me see here One second I'm gonna need like hot tea after this. Um So do people usually create separate cluster for this? You probably not want this in your regular application clusters, right? so, um So argo, so it depends what what you um Yeah, so it depends what what you're doing, right? So it looks like braided silver also says it depends as well, but um You your your pipeline it just depends where where you'd be running it, right? So in my pipeline um I would do it all in that same cluster the deployment of that pipeline though is different, right the deployment Argo city can deploy to many clusters Right argo city can deploy to one cluster, right? This this is dev and prod in different namespaces, but it can be different clusters as well um But the actual pipeline is is running um on the cluster So there's not like a clustered pipeline, right? Like that's not the idea of of tecton is you won't have um, like a fleet of tectons running you would have One instance of tecton, but I can see why you would want that because you're coming from the um, the jankins world, right where it's like you'll have Because jankins has like what what are those jankins? Agents or whatever that they deploy and do their own work This is like another way of thinking of doing that Yeah, pipelines could spin up a test cluster as well, right hiller hiller is saying validate and promote your production. That's another another way of doing it is That's what kind comes into place, right? Right like that's where There's dev test separate from prod cluster a common pattern That's a good question dev test cluster. Yes But yeah, it is pretty common actually. It's usually pretty common. Yeah It's actually I would say like at the minimum I see customers have three They have like like their their the dev cluster like they're staging and then production minimum Right, and then sometimes they even have a Well, there's like sandbox, right? Right. Yeah, or or like a production mimicked instance that they get below balance over to or whatever for any upgrades or whatever sometimes. Yeah Yeah, I'm a fan and I've said this to a lot of people and they know I'm a fan of clusters as cattle right like if you're going to deploy um You know a new application you're deploying a new version of open shift, but that's just me I know I'm talking crazy. There's gonna be a lot a lot of people that says I'm talking crazy. Andrew says I'm talking crazy I understand that. Um, I understand customers aren't there yet. Let's just say that but I am a fan of clusters as cattle because You know get up fan over here. Um So Sully's asking can get up slash argo cd be used to coordinate build on one cluster and deploy to another Yeah, so, um So I think this is also the the idea of divorcing the idea between ci and cd, right? Yeah, so, um I I'm in the opinion that's not argo's job To coordinate the build on one cluster down the other that's actually that's actually A tecton's job. That's that that that relies on the lap of Of tecton tecton will coordinate that for you. So, um argo cd in this scenario in ci cd is actually Has a very small role to play. I would say it's actually, uh, uh, a simple but a simple slash powerful Design of argo is like, hey, I'm just a reflection of what you want to get Um, right It has other tool sets that make it actually Pretty neat. You can do things like precinct hooks posting hooks sink waves and that's actually a nice segue to I've talked about those before so if you go to red red dot ht slash get ups That I've talked about sink waves and things like that before so argo does have certain tool sets that can orchestrate a little bit of this But the real brains is in your tecton. It's in your ci process And so, you know, the way I would set it up To coordinate build on one cluster versus another is via c, uh tecton. I would have tecton Push build that somewhere else. Yeah push things in another so So, uh Cool Yeah, so so hillary says, uh, yeah ci merger code in an integration environment for testing validation cd deploy validated code to production So not everyone uses ci cd with those definitions definitions, which is very true I've seen that yeah sad and true. Yes Yeah, and and also I think this is um um This is kind of just more of the tech that we got from jankins because Uh jankins had a such a large footprint and people were just using jankins as a one big monolith that does everything Um, which at the time probably made sense um But they are distinct Things ci right continuous integration is more to do with code and anything else cd is really You've done all your testing you've done everything in um in your ci process When you're finally doing cd like it it should be like it should be like the same thing to do I know we're on where I know a lot of people aren't there Right, we're not all there as an industry, but the idea is The cd shouldn't be scary because the scary stuff happened in ci right it failed in ci. That's that's why you wanted to have failed in ci so I'm not gonna say the name, but it starts with the t I am using an approach very similar to yours and i'm getting some pushback from devs That their ci does not let them know whether or not the deployment has worked or finished Yeah, um you could have um You can have hooks built into place.  Yeah, whatever kind of notification thing you need A system of record, whatever your thing you want just have Yeah so Yeah, so this is also the uh a paradigm shift as well is that um, if you want notifications you have to Built in the notification right and it's not um, it's not like out of the box Right like it like hey after if step one and step two finished I'm gonna curl this, you know, you can be as as as as course as that there might be something on the Tecton hub. Let's just look real quick There might be a notification task. Let's look for uh, let's look slack Send to slack channel. Look, I just solved all your problems in one. There you go Solving all my problems. And so um, there there's there's definitely notifications you can set up Um, oh, they even have an ansible runner. Yeah, all kinds of stuff. Yeah, there's all kinds of stuff in there Yeah over that or uh email a send mail simple smtp Email relay. I'm pretty sure there's something for send grid I like using send grid. Oh, no But anyway, the telegram that's awesome Because I could just have a telegram group of my devs and boom off they go. Yeah Telegram since it's a telegram. There you go. Um Yeah, so they're um, so they're the messaging tag It looks like click check the box for messaging clear your little search field check the box for messaging It's cool because I don't have to uh, I don't have to write any of these Right telegram send webhook slack send a webx room send a telegram send a webx room. Wow That's impressive right like yeah, yeah needs carrier pigeon someone says I have that Neat carrier pigeon. Yeah. Yeah, it's a carrier pigeon.  So post posting hooks, right? So there is uh multiple notifications you can set So the the argo. Sorry the um the tecton right in your ci You can have hooks going in to send notifications when argo actually has a Post sync hook, right? So like once everything is finished you can say hey the sync was successful. Here's here's a hook Yeah, um, there's also argo cd hooks or um notifications I'm not a big fan of notifications in this aspect, but um One because it's like alpha, but you can have um A notification argo cd notifications come in Oh, the uh, you can commit the results of the ci testing back to get Suspended you can do that as well. Yeah. Yeah Commit the results of the ci testing back to kit. That's actually pretty neat. Yeah, you can do that as well I haven't thought about that So I'm just gonna sit call it now T's question uh was more due to the fact that the deployment is sort of asynchronous You just change the manifest and wait for argo to do its thing. So the pipeline itself does not necessarily wait Correct. Correct. Yeah, you could um and this is because I'm using a get ops workflow, right? You can use um event driven Right. So what's really cool about um Argo cd is that you can You can turn off the automatic sync and just use hooks and then it turns into like an event driven So you can actually still do event driven architecture Um with uh cloud native tools, right? So um And so yeah, also pipelines aren't linear as well, right? So it doesn't have to pause and wait for things You can do things in parallel. So um as we saw in my pipeline, right? I have I have one task that two things happen at the same time um And so yeah, so you can actually definitely use um, you know pipeline Trigger something in argo and your next task maybe to wait for something to happen in argo Get some, you know You can even do something as dumb as uh until curl equals 200 Right, you know sleep 10, right? Like whatever. Yeah, you could be really coarse with it But you know something analogous to that No, but you can you can definitely do event driven things as well. So yeah Cool So is there any any other uh pause maybe for a little bit any other questions pause or a second. Let's see. Um Not not I think we should have just had hillary on right like maybe Hindsight 2020 she should she should have came on to the stream, right? You know, maybe next time we're touching All of this maybe hillary should come on Yeah, yeah, that'd be that'd be really cool. Especially get an sre's perspective on it. So right exactly So, yeah 10 years of 10 years of ci city and being an sre She'll have a thing or two to say about it. So, um, yeah, so yeah next time we talk about ci Um, maybe we'll invite hillary on that'd be it'll be fun conversation Where's hillary sre? Sre dedicated or Dedicated. Yeah, okay. So she probably knows Chris Collins She knows all kinds of people hillary. If you know chris collins reach out to him about what he's working on with me Oh, he's doing. Oh, that's right. You guys having a ask sre channel. We're working on it. Yeah, I mean chris collins and I go way back Um, so he reached out to me Yeah, no, he's like six eight and i'm like six four. So like we both worked together The only person who is who you're shorter than yeah, right? We both worked at duke and Duke university and he was on the duke it side and I was working on the research side and We were both doing kind of similar things the same time. So we're you know feeding off each other So the idea now is that like he's been an sre for a while. He's all these different ways of doing dedicated now And let's bring that to the channel. So yeah, it'd be cool to get A bunch of sre from throughout that org Yeah, it would be for sure. Definitely So, yeah, cool. Awesome, man Well, thank you for showing us all this. Yeah I know this was a crash course um because I went from like Hello world to full pipeline, but um, I kind of wanted to You know go over the um, you know, how it fits in with getoffs and how argo cd fits in And how really your get workflow will dictate dictate all that that's the center of everything. Um, so Yeah, so unless there's any other questions, I think uh, I think we're good for this time. Oh one housekeeping totally forgot We are skipping um We're we're taking a break for the next show. So we're skipping Instead of the next show being in two weeks. It'll be in four. It'll be a month out because of uh, kubecon, right? So, um So you're talking to use going on So I have yeah, I have a I have a thing I tweeted it out and kubecon, right? I actually You guys kind of kind of got a sneak preview because I kind of go over this demo in that in that talk. Um, and two Uh, we're having get ops con, right? So if you haven't signed up for get ops con I don't see if cfp's were submitted. We we emailed out the congratulations. It's happening So I'll be at get ops con. I'll be at kubecon. So we're taking a break that week because there's gonna be um Yeah, I mean the pattern is full. So yeah, uh get ops con is on Monday the 5th Open shift commons gathering is on the 6th and Yeah, something like no 3rd 4th So get ops con Yeah commons is commons is the 4th The conference kicks off on the 5th and I'm doing a talk that morning On the 5th. So yeah by by wednesday. I will probably be exhausted And yeah, so we're taking a break. We're taking more days of kubecon going still. So yeah Um, yeah, so um, but please come check out Chris or the christians talk my talk everybody's talks need to kind of be seen here. So yeah Yeah, yeah, so go go check us out there. Um, next week is red hat summit.  I just dropped a link in chat. It's free if You know, let's say you can't make the first half, but you want to be alerted about the second half please Sign up anyway because that's how you get in the notification chain There you go. So, uh, lots of stuff going on in the next couple weeks. Yeah, so I figured we'll take a break on streaming Well, thank you. Um, well, we'll we'll still be out there. We'll still be streaming. Well, that'll be around. Yeah Yeah, yeah Well, just not not not not on openshift tv Yeah, there'll be there will be during kubecon. There will be office hours that we're doing Um, so I need to get those all booked out and put in the schedule, but I think I'm doing that tomorrow Well, maybe not tomorrow given looking at the calendar now. Uh, yeah Talking about spillover for meetings. Mm-hmm. Yeah today was everything spills over into tomorrow, but anyways Uh, thank you christian. Thank you audience. Uh, This is it for the channel today. We appreciate you tuning in throughout the day and uh, stay safe out there everybody for sure Yeah, yeah, see you guys later


Hello. Good morning everybody. This morning's talk is going to cover analyzing intrusions in intruders. It's kind of a deeper look at network incident response, intrusion analysis. Okay, so this presentation is significantly changed from the one on your CD, but I've already provided this to DEF CON last night, so it'll be on the website after my talk sometime today, hopefully. Who am I? I'm Sean Bodmer. I'm a computer science, criminal science researcher. I've spent a lot of time researching honey nets over the last six years. I'm not an expert behavioral profiler, nor do I work for the Department of Justice or have any background within specifically. Intrusion analysts by trade spend the first few years in doing penetration testing, exploit analysis, signature generation, FIDSs, IPSs, and the last six years specifically dealing with incident response, intrusion analysis, and honey nets. And I'm also building a thesis on attacker and threat profiling. Why am I here? I'd like to convey some concepts and methods that you all can hopefully walk away with and better protect your networks and better understand threats that are actively coming against your network. So basically this whole talk is going to cover the silence of the lamb meets the silence of the ram, in a sense. Okay, so everybody's challenged, right? How do you better understand the threats? You get hacked, you have an incident occur on your network. How do you understand what the motive and intent is? How can you take that information when you understand it and better use that information to put in stronger protections in your network? How do you effectively communicate what just happened on your network to your leadership? How are they going to give you more budget? How are they going to allocate you more resources to increase the security protections or training? Overall foundations of this talk is going to basically be founded in behavioral profiling. Yes, it has negative connotations in the media and the press, but it's been used for a very long time and very successfully. I think statistically it's like 70, 72% on average behavioral profiling does work. It's not 100%, it's not the silver bullet, but it does work. You can use this to analyze patterns of an attack, focus on the behavior, the skills and abilities. You can look at the code that's being used against your network and see if it's custom, if it's not. Motivation, what are they going after? Your finance information, your logistics, operational, corporate espionage, complexity of the attack, how hard was it, accessibility to the resources. This is the foundation I'm going to build on from here. I'm going to bring you all the way back about 100 years ago, 1888, Jack the Ripper, this is where behavioral profiling started. First case profiling was actively used. Without even ever finding Jack the Ripper, they were able to build a list of a profile of Jack. There's still today nobody really knows who he was, but they infer a lot of personality traits about Jack the Ripper. They have all his victims, they have the dates, circumstances of death, and how they were harmed. You can relate all this information to your network. What happened? What OSes did they hit? What services? What tools were dropped? What files were altered? What was taken? What presence was left? From an intrusion analyst perspective, you can relate all this information that's already there to what you're doing on a day-to-day basis. You can develop patterns and signatures of an attack if you can write your own custom host-based signatures. Okay, so with Jack the Ripper, they never found him, right? But they have a lot of suspects, right? They have all these plots. The royal plot, he was a doctor. How the bodies were mutilated. Some say he was, some say he wasn't. It's a still ongoing debate. Jack the Ripper's crimes were disorganized. Some of them thought that it was a passion. Organization, knowledge of your environment. So did he know the streets where he was murdering, committing his crimes? Does your intruder, who's hacking your network, does he seem to know exactly where your systems are? Does he know where your users are, where your files are stored, where your key assets are located, if you have a very large enterprise? Also, is he extremely skilled with tools? What kind of tools is he using? Stuff from open source, or is he customizing his own tools? That can help you build off that profile of Jack, right? This is kind of an overall basic profile of Jack. I pulled it from several different sources right here. So you look at Jack and you see, okay, there's all these assumptions about Jack the Ripper. You can make the same assumptions about your attacker. Try to narrow it down. And it's all at least trying to understand your threat. Are you dealing with some kids? Are you dealing with organized crime? Somebody who's a disgruntled employee or something even worse? So like I said, there's over 100 years of this experience right now that you can use as information security professionals. You can go on the internet and read. You can go buy books. You can research to your head turns blue. That information that's already been built, and the Department of Justice folks, the law enforcement types, you all, this is not new to you. You all have been doing this for decades over a century. And so this is all really for the security guys, network guys who don't really think about this stuff or haven't, and where you can walk away and put this into your program. So now the information systems are at a point where all the heavy lifting is done by your automated systems, your security systems. It leaves you time to do the heavy thinking, right? Well, you can implement that a little bit better with recursive learning, whether it be AI, automatic signature generation, managed security services, groups, and on-site contractors, right? So there's different ways that you can kind of build up in certain areas to help take all the leverage, all of your time, instead of worrying about all the logs, you can filter it all with different tools that are commercially available. All right, so get a little scientific, criminal investigative analysis. Yep, you can go on Wiki, you can go on the Internet and read all about it. That's not something that you specifically do, but it's a good resource that you can go read about and walk away with, right? The basis of it is reviewing crimes from a behavioral aspect, assessing the facts about what happened and interpreting what the attacker did. So what you see happen, if it's a very clean and organized attack, if it's very disorganized and seems slightly opportunistic, then it gives you two different types of threat profiles. Law enforcement is a pull-off of law enforcement code. A person's basic behavior, exhibit in a crime scene, it's also present in their normal lives. You can infer that any way you want to. But it will help you determine, especially if you're working in commercial space, it gives you the ability to see what's going on in your attack and go back out and do research on your attacker. Sometimes in code that you do, if you reverse it and go through the email addresses, IP addresses, specific keywords that you can go back and find a hacker group website or somebody's MySpace account that has all their information on just by finding a keyword in the code, right? And that right there allows you to better understand what is coming at you. Okay, so again, we'll technical threat analysis, threat modeling. This has been around for several years. So common components, you look at potential attacks, weighs in the threats of your network and the risk, right? So if it says vulnerability, risk, people's threat, whatever. Analysis, you start to look at what's going on in your network, what's trending, what's happened, and then what you can do to counter-measure that. What kind of measures you can implement in the future is stronger protections, which lead to future preparations. A common analysis approach, locate vulnerabilities, classify possible attackers. So if you're running a bank or financial system, who could possibly want to get into your network and take your information, whatever it may be? Identify the goals of an attacker. Okay, so you know your valuables and then you say, okay, well, these groups of people may want to get into my systems. You look at your vulnerabilities and say, okay, these are the ways that they could get in and you try to safeguard those as much as you can. And then create a resolution plan. If this happens, I know to go look at these systems. I know these are my weak points, right? And everybody always looks at the users as your weakest link. So people have looked at criminal science and people have looked at computer science. It's basically two separate worlds. And right now, I see a lot of IT information security personnel that I work with and I work for and consult to. They do a lot of post-mortem analysis, right? Something happened and they respond to that. They're not actively looking right now and keeping on track and understanding what's going on. They wait for something to happen. They pull the hard drive. They review it. They go through their logs, but it's already happened. So there's no way to, you know, how do you respond to that? Your data is already gone. You're already toast. So it is possible to be a little proactive about it and implement these other sciences into your security program to make you a little more proactive. So you're not always responding to something. You're actually doing something so when something happens, you're significantly more prepared. Behavioral profiling yet, negative connotation, but it can define one aspect of understanding your threats and understanding attackers that want to do you harm. Okay, so it's here. It's now. You know, my humble opinion, everybody, at some level you have thought about this, who wants to break into my network, who wants to do me harm, but there's really a lot of resources that you can go read about to help you better understand it. I have an entire bookshelf full of criminal, several books on criminal profiling, criminal sciences, and UK studies that has helped me relate a lot of intrusions that I've seen into, you know, from these serial murders and criminal case studies, really how they found them, how they identified it, how they enumerated who did it, right? Attribution. So you can create, you know, models of this. You can create go-by programs to help build stronger models depending on what kind of network you're, or enterprise you're working with. And you can understand, if you understand that, you understand your threats, you can understand where you may be going next, where the attackers may be going. The latest threats, okay, you know, this new exploit came out, okay, I have six servers that have this vulnerability. Yeah, I can patch it. How fast can you patch it before they hit you? And how much of a threat, how large and strong is the threat? How fast are they going to respond to that zero-date that just came out? Okay, so this, you can be Colombo now.  So when you do an intrusion analysis, you can guess about what happened, you can try to put pieces together and make a lot of assumptions, or you can really look at all the facts, bring all your data sources together, and try to really peel away the needles in the haystack and put them all together, okay, you know, this pattern happened over the network. If you have some session-based monitoring tools, you can look at the session of what happened and kind of tie it all together, and it should be this one attack, to maybe this one attempt was the same person over here in these five other places, right?    Assessment, right? So you see what happened over the network, over the host. You put it all together, and then you just try to build a profile, a threat profile, an attacker profile of the intruder or the group that just came at your network. And then you try to generate an intrusion report that has all this combined information in it and give it to your senior leadership to let them know how big of a threat just came at their network, whether it was some kid just using some tool he downloaded, or this is something serious, or, you know, I need, you know, more tools.  Okay, here's a sample investigative life cycle which we'll run through really quick. All right, so you have your incident, you know, awareness consultation right when the incident occurs, preliminary analysis, you know, image acquisition, recovery, you do analysis on it. Preliminary and final report. And either you're going to go to prosecutable or non-prosecutable. You know, even if you go prosecutable you still have the bottom containment and prevention for the future. They still kind of go hand in hand.  So where do you start, right? So an incident occurs and you have all this data, right? We've talked about how you can do, you know, behavioral profiling and intruder analysis, but when it comes down to it, you have all of these components of technology that you have to wrap your head around when something happens. Where do I start, right? Do I look for, you know, the hard drive to look at the apps, the network logs, the hard drives? What do I do first, right? And it's all around what just happened the body of the crime. So you have all this to go through and it's very inundating to figure out where you want to go. So I'm going to go through a couple of case studies throughout this, okay? Haale, Mojave, Mahart, El Griton, Julio, Ardita. So this is all open source information, right? This is a hacker who broken into some government organizations, colleges and some foreign countries. So he did some very nasty things, right? So, but bottom line, you know, they charge them at the very bottom for years probation, you know, had to pay some fines. Nothing too big, right? But when you look at this and you're watching this, if you were on that assessment team, right, trying to do the incident response, how would you have handled that? How would you have looked at the difficulty of his code? Consider the target, what he was going after, what systems he was hitting, where the data points were that he took the data. What was the outcome, okay? So did he take your financial information? Logistics? Did he take personnel records? PII, whatever you want to call it. What did he take and what does it all mean to you? How much did that damage your day-to-day functions, right? And how would you analyze that? Typology is basically typecasting or stereotyping certain groups together, you know, like the naive hacker, the organized crime, and I actually have a list of a few slides down. And victimology is basically looking at what servers he hit. You know, actually looking at the victim, the target, and trying to assess, okay, so what was the value of that target that he just went after?  And there are other methods you can use, you know, just using standard digital media analysis and trying to do some kind of incident response reporting, a damage assessment. But attacker characterization is very important when you try to do this assessment, right? So attacker characterization has two primary components, events and threats. So what has occurred by the act of the attacker and then the threat, the motives and intent of the attack, right? So generally, session data isn't available on an attack. You're looking at some, you know, some fuzzed up host logs. You're looking at some network logs, IDS logs, firewall logs that have more than likely been fuzzed. You know, somebody throws a lot of noise on the firewall to pass an IDS and tries to slip by, goes really low and slow and tries to throw a really slow fragmented attacks. You know, it could be, depending on how important you are as a target. Okay, so honey net technologies, if they're available and you use them, you know, and if you don't publish them, okay, so general consensus is that yeah, everyone's like honey net, you know, bad connotations as well. But if you use them and implement them inside your network and don't advertise them publicly, you can use them, right? It gets kind of shady when you go for prosecutable the prosecutable path.  Okay, so these are the common, we need to talk about typology and breaking your threats down into groups. Here's a, just a, the groups that we use internally for analysis team. And you see, it goes a lot down from naive, a novice to foreign intelligence, which, you know, depending on what kind of information you're protecting, it can go all the way there. Components of an attacker profile that you can start building, and we actually have, you know, some broken out forms that we use. First hour information, we try to fill all this stuff out. You have motivation, objectives, timelines, resources, risk tolerance, skills and methods, actions, attack organization points, numbers involved in the attack. You may see one attack, but it could be several people involved in that attack. One person scans, one person actually injects, one person runs the shell, one person gets in and pulls the data, one person cleans up. That can happen. It can be one person from cradle to grave, soup to nuts, you know? A knowledge source, right? When I say knowledge source, that again refers to if you're, you can have the ability to find out, you know, an email address an IP address, a keyword that you can go Google or, you know, search for on the internet and find out more about that person or that group. And you can figure, okay, what are they about? Do they hate, you know, my company? Is it a former employee?  But that's a good information source is trying to go out and try to find information about that entity or person. Common challenges in attacker attack characterization, the cost. How do you really effectively, you know, convey that to your leadership?   We have a lot to do on a day-to-day, you know, function.  Well, you can automate a lot of it.  Do you have to have extra backups of that? And what happens when you have, you know, disaster recovery? Is your leadership going to let you have a whole other hot site or cold site for that? You know, and legal. Most lawyers get really tense and pucker up when you talk about, you know, setting up, you know, any kind of extra attacker characterization or kind of profiling systems. That depends on how you word it. So, we talked a lot, you know, about implementing the technology, about the concepts of actually performing the work, when you're actually doing the analysis, you want to trace an attack to the insertion point. In your first hour, when you find out something's happened, you try to find the best, you know, that point of entry, right, and try to trace it back because that first device, like, you know, most ISPs, what they dump their data in 24 hours, so you have 24, 48 hours to find out where that attack came from before all your data's gone.  Acquiring all of your internal assets.  So getting all that information together really quick and storing it, then that's really important. Postmortem is reactive and not proactive. If you're not implementing technologies to actively monitor, you know, the anomalous behavior, you know, you're fighting a never-ending battle. It's completely uphill and you're just going to fall down. Datastream, Calboy and Koogee, okay, so 26 days of attacks, 20 days of monitoring. A lot of sniffers. So look at the overall damage and that's the damage just to the computer systems. That doesn't cover the costs associated with the time, right, to take all the personnel to rebuild all those systems. The information lost as with case study B, right, there's a defined, you know, monetary number with all that. That hasn't, you know, ever totaled. If you looked at that and you were trying to define, you know, how bad is this stress? So think you're all on the receiving end of this and you find out there's all these sniffers loaded on your network. There's all this data going out of your network. Okay, the tools that are running on your network, how good is the code? You know, is it very common? Again, is it open source? Common tools. Who would you consider the target? How would you consider it, right? Would you look at what just happened to you and try to figure out, okay, so what does this all mean? Are they taking like, if I'm building like an airplane, are they taking, you know, different pieces of it from different sites to all take it back and build a nice plane for themselves because they're starting up their own airline? Could you look at the typology and victimology, put them together and assess the situation? Could you look at, okay, so I think that this information is highly valuable and it could end up on the black market or in another country or, you know, specifically the black market and look at your target, your victim. What does this all mean to me? What service did they take? Again, you know, what applications, what files with information. So giving you a kind of a light run-through of when you see an intrusion, how would you kind of look at it, right? So you have this information and of your attacker and this is how, what I actually use to build, you know, profiles of after an intrusion. This is what I do on a day-to-day basis. Gender, content analysis, you know, can you see inside the code? Can you look on the internet and see what, you know, if this is a male, female, can you do research on that? I said on the internet, aged, older, younger, you know, middle-aged. Command use, keystroke use, if they're using some really old commands, you know, Windows, Unix, you can infer that they're older, younger. There's things like that that you can use.  They can do things like that. Typology, methodology, content analysis, looking at the p-cap of the attack, what actually happened in the flows. Race, ethnicity, can you tell if they're from a foreign country? Can you tell if they're local? What region of the country they're from? What part of the world they're from? You can do that. What's this one? Talk about command use and keystroke use, and that's related to honey nets and honey pots. So, you can also do level of intelligence and schooling. How well educated are they? Is your attacker? And that can also mean you can infer from the level of intelligence and schooling what kind of threat you're up against, right? What kind of, who wants to do you harm? Whether this person is highly funded and well educated, you really don't find highly, highly, highly educated people doing, I mean, you know, masters, PhDs, I'm referring to, doing a lot of major hacking. I mean, you find some people doing it, but as far as doing cross-organizational intrusions, unless they're highly funded, PhDs are masters normally at work. Political affiliations, you know, are they extremist group? Are they with some, you know, environmentalist group?  External data sources, if you find out that somebody else has been attacked by the same entity, by the same group, you can kind of put your heads together and go, okay, well, we think, because our two organizations have these, you know, functions that it could be this kind of group coming after both of us. Physical mental health, we've all seen the movie Swordfish, right?  And you can also see are they narcissistic, do they have any kind of, you know, manic depressive, bipolar disorders, you know, what are their patterns of attack?  History hotspots, nature of the information targeted, you know, again, victim system functionality. Vulnerability exploits, is there any disclosure history on the internet about that attack, or that attack or disclosing the attacks? M.O., motor stop brawn dye, motor operations, signature, content, patterns. Can you look at any of that, any attack, and can you correlate it across multiples? Utilization of the access. So, how often, when they got into your network, did they use it? Did they get in for a few minutes and just back away? Did they get in, and were they so comfortable on your network that they found other user accounts, logged in to other users as users, you know, overnight for hours and hours and hours, logged in to multiple locations and just took all kinds of data and, you know, wove it out of your network. How do you look at that, right? How well? And the data transfer technique, did they just FTP it out, did they SCP it out, did they use one of encrypted, you know, channel out, you know, and that can help you infer the skill level and motive and intent of your attacker. Logging alteration, you know, deletion techniques, did they modify the host at all? Did they take care to cover their tracks or did they not? How brazen were they that you can, you know, sort of build this information about your attacker, right? It's a little profile. So you're going to analyze session behaviors. Did I copy this one? No, I think I did. Okay, so if there's session data available, you can look at the knowledge of that attacker of your environment. System locations, right, system functionalities, if it knew where it was on that, you know, VLAN, it knew that that was the mail server, it knew that this was the backup file server. If you have a major enterprise and you have your own proprietary naming convention, right, how well were they in your network and how were that entity before they got in your system? Then they know exactly where to go or do they have to putz around for a little while and do some, you know, net views or whatever to get in the system. They know exactly what folders and files without doing any greps or direct researches, right? Exactly that your vice president was, you know, in that office, in that state or in that network. Whether you attack to the scripted or not and how often does the attack generate typos.  If you have session analysis or session analysis systems, you can really look at all that data and really build a good understanding of your, you know, attacker. Okay, so implementing session analysis, you know, again, honey nets, honey pots. So, I'm going to show you a few sessions that we've collected and they've all been scrubbed, but this information has been involved in a couple of things that I've worked on. So, I don't think you can see this too well, but it's on your CD and I've kind of cleaned up some of the text on the CD. It took out some of the MySQL statements. But, basically, you see the person, you know, I peek and fig, look at another box and went down systematically and went through look at the domain, directory to some tools, TFTP to some stuff out, you know, echoed pulling stuff out to a file, right? Outputting all this information from the server to a file, took out the file, renamed his tool to, a nice tool, went down, panked some other system, did some discovery and some other systems on the network and knew exactly where to go copying some of these files in, right? So, bang. How often did this attacker really dig around and look for a whole bunch of systems, right? This actually happened very fast, about a half an hour's time. So, this person, this entity, knew exactly where to go and just testing to make sure some systems were up and logged in. But it was very quick, very fast, highly educated, you know, don't know about the resources of that entity, but it was very fast, highly educated, they knew exactly where to go. So, but you can see that they went down, found some passwords on the local system and kept digging, burying deeper and deeper in, right? Found some passwords, looked at some more systems, copied over some more files and some other systems. So, you can see they're just pulling out the data, pulling it out, pulling it out, right? Very fast. So, after you look at this, right, and this is actually about six pages printed out of a honeypot capture, how would you have analyzed or evaluated the attack, right? You can look at the network data and you can look at the signatures created and kind of put them together and then say, okay, here's the session data and match all that up and build a very good profile and understanding of your attacker. Was it sophisticated, motivated, targeted, opportunistic, organized, disorganized, it's a good criminal science term and then automated a lot. Was it scripted? Did somebody prepare all this? Did somebody prepare this kind of information, this attack very quickly? You have to have an extreme knowledge, a very deep knowledge of your internal network, your day-to-day operations. Okay, honeypot capture two. Okay, so basically the person, this is kind of a pudge around on a DMZ. Somebody was looking around for something. So, we see that they're just kind of digging around, looking for something and they're not really doing too much. They're just trying to do some discovery and it's not that very long of a capture. Two and a half pages. So, you can see and I will say that I actually messed up on this one when I set up my honeypot and actually left the, in the recent documents, link I left, the little cvac.zip file in there, so they discovered it pretty quick. So, don't do that. So, they go through and try to dig around the system, dig around for some domains, right? Nothing too fancy, nothing too good, but they did try to pull out some data to an external IP, which is fuzz. And you can see that they tried to make a lot of noise on the network, to pull out the IDS logs. So, you know, not too sophisticated. Somebody's poking around for an empty box to pivot off of. You know, like I said, not too sophisticated, motivated, and they never came back after that. It wasn't very, it looked like opportunistic, not really targeted, because that IP or that presence really hadn't scanned like that or hid that service on that honeypot before. Automated, it was live. There was a live person digging around that network. They just kind of faux pas like that, and made those typos. So, session capture three, this is actually much shorter, but you can look at it. Somebody's just digging around, looking at a domain that they just tried to get into. And this was a web server domain, and hit a server, tried to look for some documents, recent documents on the system, tried to do some other some good queries, and made a lot of noise on the network. But it wasn't very sophisticated. No. You look around, and they left after a while. And this right here was about an hour of digging around. Session capture two was about an hour and a half. So it wasn't on, they never came back, they never did anything more with this system. So when it's sophisticated, wasn't very motivated, because they never came back, that presence never hit that honeypot again. Didn't really seem targeted. They were kind of wildly searching, they weren't searching for anything specific. Seemed kind of disorganized. They didn't know exactly where they were going, what they were going after, and it was live, all the typos. So Carlos Smac, Salgado. So here's another hacker, ended up with a large fine, two and a half years in federal prison, made about $200,000 in sale credit cards. So if you're an online financial company, you have threats like this every day, you can go on the Internet and IRC Underground, find all kinds of credit cards, you can buy every day for $1, $10. So if you had been a part of this on the financial side, trying to figure out what was going on in your network, how would you consider the difficulty of what he was doing? Did all this information was easily accessible? Maybe a few security controls in place? Would you consider the target? Very easy to get. So this is financially based. Is it financially motivated? You can sit at the outcome, you can watch me hold my credit card information, so you can deduce, it's probably going to end up on the underground Internet, so you can probably get on another box and try to get off on the open Internet and try to find your credit cards or any of those numbers out there fishing around. Typology, what type would you have mixed them in? Of all those common attacker types that I threw out there. That financial data that was out, you can look at that and e-processing server, all my credit card data was stolen, what app did he get through. You can put that information together for your official report. I can't go over all the theory. I mentioned some tools. In your toolkit, in my toolkit, we use network IDSs, host-based IDSs, firewalls, antivirus, router logs, honey pots, honey nets, host-based analysis and system event logs. You'd be surprised how many attackers, as you forget to delete certain web logs, certain event logs when they're trying to cover their tracks. In order to catch someone crafty, you need to be crafty. Honey pots and honey nets. How many attackers really expect you to deploy a few honey pots outside your network? Not many, right? I know in the past few years it's gotten more and more prominent and people are starting to look for them. Not as many as they used to. They still don't look for them. I rarely ever see any honey pot that I've helped set up, look over their shoulder and try to dig for a honey pot.  Security of resource lies and your ability to be able to use it. You want to make your honey pot look as close to your current environment as possible. Host name, naming convention, user names, take some fake data, some used data from your network and dump it on that box. When they're doing that directory grep search, they can find the docs and spreadsheets and PowerPoints and any kind of recent temporary internet files. Make it look a little realistic. That's a huge thing. If an attacker gets on your honey pot and sees nothing within the first five minutes, it's right here because every box in the network has a whole bunch of data in it. So you really have to make it do something. So honey pots, advantages. You can collect a small amount of data for only what hits it, but everything that comes to it, you can infer it's completely malicious or suspicious. Reduces your false positive rates and helps you correlate looking through all the haystack. All you see is needles in a honey pot.  I'm seeing this here and looking across your entire enterprise really quick and go, where else am I seeing this? And you can go back and say, have I seen this a week ago? A month ago? A year ago? If you have logs that long, see how low and slow is this person. Catch new attacks. Works encrypted. IPv6 environments. Easily. Simple concept requiring minimal resources depending on how you deploy them. Disadvantages. Limited field of view. If you set it up incorrectly, they can use your honey pot to attack the rest of your systems or other companies or other countries, whatever it may be. Automation isn't perfect, but if you put some resources into it and buy some commercial tools, it works out a little better. Many types available. High interaction honey pots, which are full blown systems. They don't scale too well, but requires a lot of resources, including host based analysis, network analysis on full blown systems. Low interaction systems, they can scale very, very well, but their specific purpose and most of these low interaction honey pots, they don't trick an attacker to actually get in the system. It's just for scanning and early warnings of a network. So here's a few screenshots of actually the RU OS, which is the latest version of the iteration and you can see, this is all sample that was pulled off the open off Google images, but basically you can see the attacks, the times, process IDs of actually what occurred, the directories, what the person is doing. So all those keystrokes that I showed you would be in the bottom of this window and you can easily look at really quick, okay, so what's going on? What process is he using? Where did he come in? You can look at the information that we do a lot of. You can sit there and correlate the time, the event, source IP address, destination, and custom signatures. The signatures related to it build you a network profile of your attacker and then you go look at the host base part of it and even your production systems, not even your honey pots, and you can match up and the event logs with the honey pots session capture and go, okay, so he's doing all these things where she or that entity is doing these things. Okay, so you can infer that what's going on here and you can try to see the skill and intent. Again, this is another one. What I want to talk about, a person at the bottom here, you see a delete, delete, delete, delete, right? So that person just kept trying to, so if they have a fat finger and keep hitting Q, that's also a pattern in the signature that you can put on somebody right there. Here's another one, so process.  You check the environment of the box. You try to set your own environment, set your own shell, however you try to play with it. But you always go through some specific services that you want to get to, right?  These two attacks are completely different. Different entities, different personalities, different methods, different modes, right? They jump from one to the other. So here goes some nerd stuff. I'm going to start with trans, sorry. So you can spend more time analyzing attacks and spend more time performing analysis on these intrusions, right? Just don't rebuild your hard drive and keep going. I know most senior leadership, they want you to take out that hard drive, rebuild it, smack it back in, and just keep marching on, right? They don't care about what happened, they just want to check that box day-to-day operations, bring that money in. But that's not going to keep the threat out. Just keep it persistent. You can perform yourself victimology and typology on the attacks, which will help you have a better understanding of what you're up against, and what kind of protections you need to implement. You can build a profile of the incident. You may see crossover. If you build a profile or an incident report with a profile of every incident that occurs, you can easily, one day, advance across other events. Okay, so these five right here may have been the same group or same person and sort of correlating that and helping the community better understand what this one hacker group, how they behave and how they act. And use the lessons learned to add stronger policy, to add better signatures to your network, to implement better countermeasures at your weak points, at your shop spots. Professional recommendations, execution, you're definitely going to want photos.  If you already performed any kind of analysis steps on any of your production assets that were hit, think before you act, please. If you're working in prosecuting intrusion and get your lawyer involved, your corporate attorney, you can always describe every detail possible and as much depth as you can. You never know when it'll be important. You never know when you're doing your analysis, when you're writing down a detail, you get to some other systems it may pop up like, oh, yeah, that was important over here. And then you actually put two and two together and you get five. Take more time to study non-cyber-based case studies. It helps you relate the methods and how they were able to find the criminal, how they were able to relate it back. Sometimes you can relate that to IT security. Document everything. In short, attempt to better understand your threats on your network. What may actually be a value to anyone that wants to do you harm. Define your assets and valuables. You can do a whole risk management, risk assessment of your network. Study non-cyber-based, mention that and increase your ability to correlate with more events across your network. You never know if these two client side hits, these emails, phishing emails, if they were both going to the same entity, the same organization, document everything because it can all leave back. When you study more non-cyber-based case studies, cyber crimes, cyber criminals, there's a ton of them on the internet that are published now because there's enough of us that have been caught. Serial murders, habitual offenders, there's all kinds of data sources on the internet that you can learn, you can research and find out some usual habitual methods that these guys fall prey to. And they help provide understanding of the resources they may use when you start looking at an attacker, start looking at the tools they may be using against you. You start better understanding how they're getting in, what they're doing, and that helps you understand what you need to do to keep them out and what you need to buy. Keep up to date on the latest security trends. Maintain an active record of your environment. Be aware of your network behavior. Know your network people. I'm saying that none of you do, but I have consulted two groups who have no idea where this system is. The host name, I don't know, I think it's on the fifth floor and it's not even in that city. People who don't know when there's a daily crime job that transfers an R-sync to another database in another site. I didn't know that happened. I didn't know that was something going on. Just please know your network. That'll help you reduce a lot of problems. I added this last night, some of it, so I wanted to be able to give you awesome information to walk away with and do some reading. This is just some of the books that I have. The cyber adversary characterization is really the only good cyber adversary characterization book out there. It's a lot of fluff. It's a lot like the Microsoft press from about 8, 10 years ago, the beginning of some processes that you can use. Then the criminal sciences books. I always talk to lawyers. I talk to law enforcement types. If you're in that type of world, you can reach over and knock on their door or give them a call and say, hey, I want to learn more about this type of criminal sciences kind of study. Where should I go to go learn? They can recommend you go to some certain places. Honeynet.org com. Very, very good resource for criminal sciences, behavioral profiling, offender profiling. Very good resource. Wikipedia, of course. It has several dozen links on each page. Dartmouth University. They have a lot of good stuff too. Yeah, here we go. I'll close this off with some famous dead guy quotes. You can't go anywhere without those these days, right? The first one's very applicable today. I think most of us who are on the defense side and I think some of us here are really close to the second one as well. All right.


For a swift response, we're out here working with our host nation partners here as well as the 173rd. We're out here doing our joint inspection, which is a pre-inspection for all cargo going onto military aircraft. With that, we do a couple of things to go over the documentation with the users, which are the Army, and we are clearing it for safety of flight, whether that's ammunition, gasoline, diesel, or fire extinguishers, and he has that, is ran through us. So the joint inspection is important due to the fact that this is the last check before this cargo will get onto military aircraft. What this prevents is any time that the aircraft goes up, we are making sure that nothing goes wrong while it's on that aircraft as far as what the cargo possesses. I think our guys are handling this exercise very well and very professional. When you step out of your realm of just being one branch into a joint branch situation, I think a lot of things come into play, and I think both on our side and on the Army side, that we're coming together very well to execute a mission that is multi-national and multi-branch very well. So I think both parties, along with the Air Force and the Army and Host Nation, are doing amazing to get this job done.