Tom here from Lauren's Systems and I'll be joined by Amanda Berlin, lead incident detection engineer and Blue Mera. Being heard is going to discuss how she utilizes Sysmon for threat hunting, testing detections by looking at real world examples and detecting malicious behavior in the wild. She's a longtime cybersecurity professional and she's going to share with us how she looks at the world, essentially through the eyes of a threat detection engineer. So we're going to talk about anomaly detection and then we'll get into some of those more fine details and the tools that we're going to be talking about. You'll find links to down below. And for those of you wondering if you can do this yourself, absolutely. I have a video showing how we can use the Sysmon modular and even export all the logs out of your window system or view them within windows itself. You'll find that video link down below if you want to try these yourself. And these are the same rules and threat detection that she uses. So they're open source, they're available. This is one of the things that Amanda participates a lot in the community of sharing all this knowledge and threat intelligence knowledge because that's how we get better. And that's why I'm doing this video. So you can kind of look through the eyes of someone actually doing this. Well, daily looking at millions of logs and understanding threat detection. So let's get started. So how are you doing today, Amanda? I'm doing great. It's a, you know, nice blustery cold weather in northern Ohio. Same here in Detroit. We're pretty close together. So I get it here and then pretty soon you try to get a little snow as well. Oh yeah. Which sounds like a great day to talk about how to pull out logs from windows using Sysmon.  And of course I already, I kind of know you from talks you've done before. And then you landed over there at Bloomerra some time ago, which is a company we use. Definitely a fun, fun tool there and pulling windows logs is your specialty. Yeah, it's one of my favorite things. Windows by default, just doesn't give you good tooling for this. Sysmon is the glue to bring this together so we can actually get some useful information out of windows. And that's what a man is going to be presenting today. So I'm ready when you are to get started. Awesome. All right. So let's just dive in. This is one of my favorite ways to start out a presentation. And I'm sure a lot of people watching have either stories to share of this or something that you've faced firsthand and the stories around that. So let's call this company Paths Grocery Store. My team is involved after a PS exec command runs over their VPN. This triggers an alert to fire and they have servers start to be ransomware, which is definitely way later in the process than you would want to know about an attack, right? And pretty much too late in this specific case. So there's no Sysmon installed and you'll find out why that's a big deal throughout this presentation. And for some reason, their domain controllers stop sending windows altogether. They aren't sending any of their endpoint agent logs, but that wouldn't have mattered because the domain controllers didn't have the endpoint agents installed anyways.  So same user, same password. That account had zero MFA set up and full domain admin access across the environment. I think the people don't realize that the first thing you notice is ransomware, but that's by far not the first thing that happened. That is, that is the end. That's the end of, that's the end of you having a good day. Right, right. Yeah, if, if something getting ransomed is the first indication that you have, it's not good. You definitely have had them on there for a while before that. So other than the lucky fact that one of their endpoints ended up receiving that PS exec command was sending us windows logs, we would have been completely blind to all of that activity happening in that environment. And it's always so nice if anybody's had to deal with this before. Threat actors, a lot of times offer to give discounts for speedy turnaround, which is they actually have from what I've heard amazing customer service. Yeah, they build a reputation on it. It's kind of they do, which blows your mind because they're doing terrible things, right? I do believe Pat's grocery store had had to end up paying the ransom because their backups were also not working for a significant amount of time. And that could probably be a total talk all in its own. And they needed that data on the servers. Otherwise they were just going to have to shut down all of their stores, right, and build from scratch. Exactly. So they needed that server data that was ransom.   This would be a great idea security wise. And sometimes that just doesn't it doesn't work until something like this happens. And a lot of times this is also where you end up getting security budget, sadly. So as defenders, above and beyond all of the other roles that we play, you know, there's strategic thinking, process creation, research, writing, all of that kind of stuff. One of the main goals we have, no matter what vertical we're in, is defending against threats. That's, you know, a lot of times it's in our title. So there's entire conferences, companies, frameworks, podcasts, whatever you name it built around those three words. And so prior to 2014 or so, endpoint AV products were just like detections of MD5 hashes. So just plain virus virus signatures for the most part, we're doing all of the work as the amount of all of the things on the internet just started to boom off control, right? And that paled in comparison to what we see today. So now we rely on what AI, machine learning, artificial intelligence boxes or maybe just waiting for that time that you see a splash page about the server being ransomed. And as enterprise networks grow and hopefully mature, the majority of what we see is largely misconfigured or underconfigured networks and endpoint security products that just cost upwards of hundreds of thousands of dollars a year. Right? Like if you look at the budgets that security companies or security teams have to spend on these things, it just gets more and more every year. And there's this repeated behavior of just trying to throw money at a problem and hope a new Blinky Box is going to fix something. Right? I was looking for some magic solution. Right. Right. Like, I don't know. Our auditors say that if we get this WAF, everything's going to be fine. Right? So you have these Blinky Boxes that are compliance checkboxes that without the time and care and effort aren't going to do a whole lot. Anyways, yeah, I see a lot of the Blinky Checkbox Compliance stuff. And that's not where the real security comes from. We have to know what's going on in these boxes. Right. Right. And visibility is one of the anytime I get interviewed or right or whatever, one of the common questions is, you know, what do you suggest people do or learn about or whatever to prevent, you know, next year cyber attacks? Visibility, visibility asset management, top two, hands down all the time. Know what you have and know what it's doing. Yeah, exactly, exactly. So that being said, how can we make sure you're getting the biggest bang for your buck, especially when that buck is free? So Sysmon is free here to save the day. This is the Microsoft definition up there, just in case anyone wants a refresher. This was also released in 2014 as a part of the Sysinternals suite of products. And ever since then, I'm sure security admins, server admins, practitioners have been perplexed as to why it's not just installed on every endpoint by default. Yeah, Microsoft acquired Sysinternals back then. They did. Yeah. And it's just one of those, this is something you need to load on your Windows systems that was great. Those tools were so popular for years, it was the missing component. Of Microsoft for a long time. Yeah, for all admins, right? Like I remember all the time needing to use Sysmon and Process Monitor and all of those things for troubleshooting, right? Like it was just not there and you had to install it as an extra thing. So we'll cover some of the use cases around why we want this and then we'll dive into some threat hunting stuff too. Yes. So here's some fancy stats. So enterprise orgs have a lot on their plate on the cybersecurity front. There's fast adoption and move to cloud, which I'm actually just writing about now, like the move from Exchange on prem to 365 Exchange. And then the last two years, you know, just the largest amount of move from a corporate world being from home from work to home. Right. So you have this huge boom in work from home now where prior to COVID, a lot of us were in offices and that attack surface has significantly expanded, right? Yeah, all over the place. You don't have everybody in the office from nine to five behind a firewall. This has created some really interesting challenges because now they're scattered. All my users are you can't even hire someone if you don't put a hybrid in the job title somewhere. So at least part of their time is not going to be behind your firewall but behind maybe some consumer router with access to things. So monitoring what's going on in those endpoints is as critical as it's ever been. Oh, yeah. And, you know, as a former CIS admin, that always just like scared us to death, right? Because I worked at a hospital. We didn't have anybody that worked from home, but we did have like physicians that would take laptops home and stuff like that.  So going from that to like, no, everybody works from home is a huge shift. So going based on our average implementation at Blumera, so we would, we were seeing like a single digit percentage of orgs that come with us, come to us with CISmon already installed. So we had so few that now it's just included in our onboarding process, right? Like you want to get Blumera licensed like not our free one because that's just cloud stuff. But if you want to do end point stuff, yeah, you get CISmon. Because it's so helpful for detections and for incident response and it being free. You know, why not? Yeah, absolutely. Just load it. Oh, yeah. And I'm a huge proponent of using the tools that are at your disposal. So whether they're open source, free, whatever, you kind of I've always felt that the the responsibility to whatever organization that I work with to look up those kind of viable options prior to just asking for like capital expenditure, right? So why not try? Like if it doesn't do what you want it to do, then you're not really out a whole lot of money, right? Just a little bit of time is setting it up, right? So if you can reduce like time to detection, time to remediation with a price tag of zero, why not? So I'm not one to read slides, so I'm not going to read this one. I'll let everybody watching since you can pause it if you really want to. But, you know, I I've always wanted the tattoo live, laugh, log. Live, laugh, log. I love that. I've not made that jump yet. But if I were to get another one, I think this might be a little bit too wordy. But, you know, this could be a possible possibility, too. So at any point when you have an incense or if you had one or whatever and you have Sysmon installed, configured and logging, you'll be able to find your breach scope, right? I can guarantee you that time to detection, time to response, time, all those buzzwords are going to be key security metrics that people end up using in coming years. And I know a lot of people are already using it already. And it's one of the best things that you can pay attention to for program maturity. And then I'm sure a lot of people have also heard like the assume you've been breached. Yeah, just because, you know, in the beginning, getting breached was a huge thing, right? When it happened to Target or stocks plummeted, when it happened to Equifax, stocks plummeted like they did. Terrible for a couple of years. And then, you know, it all recovered. But now it's just another one of the news. Right. The stocks don't even dip anymore. You know, people are less likely to get fired anymore when it's just so prevalent to get breached that people are like, oh, just assume it. And then if you have been, what should you be doing to find evidence of that? Right. Well, and that's, as you know, is me and you both work in the infosec community with instant response teams. And they're always complaining as they should be about. We couldn't figure out what happened because they didn't have any logging tools. We know we know the results. We don't know how we don't know the backstory. We don't know what led up to it. There's an absolute lack of information. We just see the boom, the results, the disaster that we're dealing with today. But that's why getting these tools and solve the beginning is so important for that forensics information. Because if you don't know how they got there, you don't know exactly how to prevent it. Exactly. So that being said, we can go on some Sysmon use cases. First, we're going to cover some of the specific benefits you can get from Sysmon. So if you're beginning a hunt, more than likely you are inserting yourself into an event related to a Windows host that's in the process of happening. Or maybe like we do. A lot of times it's a retroactive step to find that activity that's been missed. In either of those situations, Sysmon logs give you by default an amazing amount of logs compared to anything plain windows can provide you, even one fully configured. And honestly, way better than a handful of endpoint solutions. So, yeah. So here you see a difference on the left hand side in the difference of Windows IDs and then on the right compared to Sysmon. So there's 10 total types of data that fail to appear if you don't have Sysmon. And then there's others that are possible that are extremely difficult to configure to even get a shred of information out of it. If anybody has ever tried to log DNS from a domain controller without using Sysmon, you have to like turn on debug logging. And half the time it doesn't even tell you what device requested. You know, may the DNS request or it's just horrible. And Sysmon just does it for you. It says, hey, this host reached out to this website. It's fantastic. So for threat hunting, what can we do with this kind of information? One thing that we do every now and then is using... And I think a lot of us take for granted that we do this. It's kind of just part of how our brain works sometimes. And it's using standard deviation to weed out baseline activity. So this is the normal like curve of standard deviation. And in this example graph, like 90% of the results live in one and a half standard deviation. And the rest you can call outliers. That can be really helpful in threat detection because looking at those outliers regularly gives you a good look into some of the less common activities that happen in the environment. Granted, if you take your standard deviation and you already are already are having an incident and it's been around for a while, it's just going to fall into normal activity, right? So you have to have this for a while and collect a lot of data to be able to see what is normal in your environment. Here, I just took... I know this is like a crazy graph, but I take an example of some Sysmon logs across all of our customers. And so the y-axis on the left is total destinations. So this is just like IP addresses that things have reached out to, starting at 50, going up to the upwards of 500 to 5 million range. And the x-axis shows standard deviation score. So this, we're taking all process names on devices across all of our customer data set, which is why those numbers are so large, and looking for destination IP addresses that processes are accessing, right? So some of them make complete sense. And then we calculate the standard deviations. So we're going to zoom into this, which you'll get that joke in a second. That makes sense, right? Processes zoom, it's reaching out to a ton of IP addresses. Teams, Java, Chrome, like this makes sense. These are applications that reach out to stuff all the time. Edge, go to meeting, that's Cisco, WebEx. Ring central, everyone's phone system. And then we see Notepad, which should Notepad ever reach out to IP addresses on anything, no. Good news is that was just our lab. Because if we start to see things in that standard deviation, or outside of the majority of that standard deviation, that are like this, like that's something that you're gonna wanna look into. And this becomes different with every customer, right? Like this is interesting data set because it's across to everybody. But it's nice to see that it actually captured our Notepad going out to 2 million IP addresses. Another main detection creation strategy we use is adversary annihilation.  There's atomic red team. There's a whole bunch of different tools out there that you can use. This one is also free. This is called Vector, it's B-E-C-T-R dot I-O. And you can import different frameworks into this. So I'll give you some examples of a couple of things that we started to do. And this is like a heat map of MITRE. So you can import all the things in MITRE and you can test against it whether the endpoint agent you use blocks or detects it or your SIM or your IPS IDS. Like there's so much that you can do to track all of the different levels of MITRE, right? Because there's different stages. We used to have the tech, the cyber intrusion kill chain, right? And now we have MITRE, which I think is a little bit more extensive. But there's all of these different TTPs that people use and it's good to test these. Now, can you test all of these all the time? Probably not without having several staff dedicated to that all the time. It's really hard to test all of those. But just think about, you know, like spot testing this stuff. So the amount of things that can go wrong in any given day with any kind of technology is, you know, overwhelming. But you can pick, all right, today, I wanna check to make sure I can detect on if somebody installed a new remote access tool, right? Just things that you, I always, you know, back in the beginning when I started doing this, it's something that I've kind of always used as an example. Think of the things that keep you up at night. Like, oh my gosh, I woke up in a cold sweat because I just realized that these users don't have 2FA enabled. I don't know if anybody else does that, but there's things out there, right? Well, I think it's important too, one of the things that we don't use TeamViewer within our organization. So when we see a TeamViewer install, it absolutely flags, it opens a ticket, it creates error. We don't even have to test it because, well, clients tested for us. Yes. And you're just crashing head, wait a minute. How'd they get, they're supposed to have admin privilege. So there's the first problem. How'd they get TeamViewer installed? Exactly. But yeah, going through your organization in validating, not just saying, okay, we monitor for installs of new files. We monitor for installs of remote tools. Being able to actually test and validate that finding is really important. It's a heavy lift to ask an IT team, but it's kind of an important one to make sure that those flags are still being raised. Yeah. And you can pick like 10 a quarter, right? Like you could probably do 10 tests easy in a day, especially with some of these tools. So like an example is this is importing, this is if inside vector, this is an example of some stuff that you can track, right? So this is the red team section of this one mitre technique, right? So this is an example of T1482 and it's people doing domain trust discovery. So you can run this and it shows right in there when you import it. This is actually importing the atomic red team stuff that the phase of discovery tells you what it's about and like trick bot malware uses it, but it also tells you like, these are the commands to run. So you can just go to a Windows endpoint and if you can run this without being detected, that's probably bad. Yeah, and that's the important part because if you have, we'll say someone in your accounting office, suddenly that accounting computer is running this, you're like, that seems unusual. That is not what they run in accounting. Yes, yeah. As that backs to that whole anomaly detection, this is how you have to look at things holistically to say, all right, here's the whole picture, here's this anomaly happening, QuickBooks reaches out to too many IPs and et cetera, but all of a sudden that same computer is now running this and they're looking at what is in the trust list. So this is the instantly starts an investigation flag and getting that data out. Yeah, exactly. And that discovery phase is way before the ransomware. Yep. Those are all of the things that attackers, like we just saw one not too long ago where the attacker was using who am I and T, like doing user commands via the command line, just doing NS look up via the command line. There's a lot of like discovery things that most attackers either do or just have scripted to give them more information. And Sysmon helps with that too. Yep. And this is the blue team version of that same thing. Invector and this is like where we'll store. Yeah, we detected it. And here's the logic that we used in this specific affection, right? And you can track those, you know, however you want. And this is just, you know, our tagging system and internally how we track all the, all the MITRE things, right? All right, so now that we covered kind of how you can organize that stuff, we can dive into some specific use cases. This is use case one. There's a ton of different ways that process memory can be extracted from a Windows endpoint. You can run things like meme caps locally. You can gain access to hashes, a multitude of different discrete ways. I can think of like five off top of my head. Here you can see they use com services which is a com plus service DLL that was introduced in Windows XP, yay, that you can use to extract local hashes. There's a handful of detections and a couple of these we'll cover a little bit that are called finishing moves, meaning the attacker has all your keys to the kingdom.  Or you're, you know, calling mandiant or some other IR firm to come help you. This isn't necessarily one of those game ending moves, but it does mean that someone has local access to a machine and the files in their possession that they can crack to get credentials to your devices, right?  So we consider this a priority one threat, meaning that you have to immediately act on it or you should immediately act on it. I do have stuff blurred out in this presentation because they're actually from customer events. When it's not blurred out, it's me doing it in the lab. Understand. Yeah. So here we see the exact commands and the timestamp on the device in question. So this is com services. This is the actual command that was run, right? Com services is doing a mini dump, which it lists the process ID and then where it wants to store that memory dump and then the keyword full. You see here, let's see here, there's other related findings and that kind of just goes along with, you know, there's a lot of stuff that happens along with this kind of stuff. Yeah, I'm gonna mention too, on that page, the law goes from December 7th of 2021 and these are all local tools that are found on there. I finally was happy to hear in 2023, Lullbin come part of the common vernacular cybersecurity, but yeah, I mean, cool that we got a name for it. I like Lullbin as a name, but it's not that new folks, but Thread Actors have been using the tools that are available to them for a long time. It's just been slipping under a lot of radars. So how'd they do that? Well, these tools are all in there. I mean, they're using built-in facilities of Windows to attack us essentially. This is why it doesn't flag in malware scanners. This is why monitoring what's going on from a behavioral standpoint is so critical to cybersecurity. And difficult too. I remember when PowerShell by default started coming on Windows OSes and they're like, oh, well, we have to uninstall this. Like we can't have PowerShell on every device. Like how are we, how are we ever gonna manage security if everyone has access to PowerShell? And then it didn't matter if you uninstall it or not. Like you could just reinstall it super easy and then there was all that stuff. So yeah, like lullabins and what is it? Lullabins and lullabaws, whatever they changed it, or these are all ones they have now, living off the land, right? Which means there's so much already in Windows that can be used, right? So this is, and I have a couple examples of that, yeah. Yeah, it's definitely, I left because I think in 2019 that we had our last time we had a Detroit screen size, my friend, David, presented, showing how all the live-off-the-land techniques how I'm gonna, it was the most fun these types of, how I'm gonna hone your system with everything you have. Right, right, which is easy, right? There's entire attack frameworks built around PowerShell. Yep. So I apologize in advance for all of the screenshots of Event Viewer. There's a lot of them, but there won't be too much of a quiz. So here's when we see the incredible differences in native Windows logging on the left compared to a Sysmon install. So even if you, Oh yeah. Even if you can't make out all the specific ones, you can see just like a share volume of information. Event one in Sysmon is the same kind of as 4688 in just plain Windows. And it just pales in comparison, right? And the reason that 4688 even shows up is because we have to configure it in group policy to output those results, right? You have to do command line logging and process creation to get everything that you need where you just get it in Event ID one. Here's an example of the specific configurations that you can do for Sysmon. So I'll cover it a little bit later, but one of the things that you can do is there's like the Swift on security Sysmon configuration. And then there's also Sysmon modular, which is the one that we usually use. And the Sysmon modular actually gives you MITRE techniques at the top of the rule names that you have in your configuration. And then we can correlate that to a couple different things. The second one down is like we were just talking about before the living off the land techniques. And that is a screenshot from the actual Sysmon config. And then the next one down is from atomic red team tests, yeah, yeah. So it kind of breaks them all down and it's kind of cool that you can see all of the techniques kind of match each other throughout those tools. So in summary, this is what kind of the detection looks like. You can put it in most things that you have for a SIM. So we're looking for Windows Event ID one. And then we're just looking for command to the parent command line to be that com services with a mini dump.  So nice and short and simple. And then we can move on to use case two. So this is an example, I talked about a little bit before, the finishing move, something you wanna know definitely as soon as possible. Another thing that just comes with Windows is NTDS util. If anybody's ever managed 80 databases, that's the utility that's built into Microsoft for doing that via the command line. And it's been used for years, right? You can use NTDS.dit and that's the database of all of your active directory user information. And you can just dump all of that information also. And as a threat actor, you can begin exfiltrating that and cracking all the passwords and looking and seeing what the directory forest looks like. I can tell you, I have not had to do this, but it was really interesting. The first time I gave this talk, the person, if anybody's ever heard of the, there's a Microsoft guide on network eviction process. The first time I gave this talk, I'm like, ah, by a show of hands who has, you know, had to follow this or had to start a forest over from scratch because I don't know if it is still, but that at one time are the two options you have if somebody breaks into your active directory, you have to start over from scratch and manually do everything again, or you have to follow this network eviction process. And the person that wrote it was actually in the audience. Like, oh, all right, well, that's great. And he actually told me, instead of doing network eviction anymore, he's like, I just tell people to move to the cloud. Like, all right. It's time to just throw away your on-prem. Yeah, just no more DCs. That's it. I'm like, oh, that's real interesting. It's not an easy place to unwind when people bury things and hide them in there. Yeah, there's a lot there. I mean, there's some tooling around it that exists. I believe Bloodhounds and all those tools that you can do some auditing with, but it's not easy. Even with the tooling, it's a project. It's its own project to itself after the incident. Yeah, I would not want to do that. Just, I'm glad I kind of misadmin stuff sometimes, but not always. There's a day when you're happy you're not doing that. Exactly. So here in our matched evidence, we see the full command, right? So ACI NTDS sets that NTDS as the active instance. IFM means install from media for like non-read-only stuff. And then the backups created and the two cues are just quitting the previous two commands. And it gives you a nice download of active directory information to go crack. Yeah, and with humans tendency to reuse all their passwords and the absolute amazing availability of rainbow tables, they actually don't spend as much time cracking. They're doing more of a matching scenario. Right, right. It's much faster than people think. Yeah, sadly. And then here are the differences. If you pay attention to the Windows logging versus Sysmon logging. On the left-hand side, it's just a huge list of 47.99, which is just enumerating security groups in active directory, which that's how active directory works. You do enumerate security groups all the time. A 4688 was generated and we saw that in the first use case. And then here on the right-hand side, you can see huge wall of tiny text with the amount of information you can gather when that command is run using Sysmon. So the first one is that event ID one again. It's comparable to the LSAS dump we went over in the use case one. You can see NTDS Utils, the original file name, the command that was run from the parent image of PowerShell. It even provides extended information like that a newly created process when the command was run and then the full command line. And we also see, is it in this one? Yeah, so it gives a terminal session ID. So it shows that I was connected via terminal services when I ran this as well as I use PowerShell ICE instead of just running plain PowerShell. And then here's the first place where we see event ID 10 which is a process being accessed. It's one of those 10 events that isn't included with Windows no matter what you configure. And then you see that the source image of PowerShell is using NTDS Util. And then event ID 13 is a registry value being set and that populates when that process actually executes successfully. And you can essentially, without the system, there's not the entire chain of the event. Right, yep, yeah. You get all three chain together for that instead of just enumeration and a process. So again, we see the detection screen with what you can look at as far as the detection goes. NTDS Util can be used again, legitimately across the environment but it's extremely rare. So you can still look for event ID one and then the process name of NTDS Util. And this is just one of our detection versions that has the command or parent command that has those two quick commands in it also. Because a lot of times that's pretty much all the script has. All right, then we have use case three. This is another priority one threat, another living on the land which is comm spec modifying the registry. So on the bottom there, you can see comm specs and environment variable that just opens the command line. So if you type in echo and then comm spec, all it does is open up cmd.exe. You can't really see what's going on here from the description of the command itself because it's base 64 encoded. But using that comm spec environment variable along with hidden encoded power shell commands is extremely sketchy. I've not seen this false positive yet of this type of behavior. And I say yet because there's an amazing amount of really terrible software used in production environments that like to mimic threat actor activity. Yeah, I have worked with some industrial engineering people like why are they base 64? Yes, yes. They thought they were hiding as a secret and apparently they didn't know about CyberShift. Right, right. And it's weird things too. Like I've dug into power shell encoding before and it's just been like, oh, we're updating the firmware on this scuzzy drive. I'm like, what in the world? Why would you base 64 that? I don't understand. Yeah, any time there's base 64, it's at least raise suspicion on it. Right, right. And then kind of a sidebar on that. When I was looking into some data for this, for examples for this talk, I was weeding through some of these power shell commands and notice something really weird. And normally when you see a bunch of words with upper and lower case, you just assume it's like obfuscation. Yeah. So I took that base 64, decoded it and found that it was Matt Graber's reflection method. Right, which is a tech testing tool that bypasses AMSI. So right away I was starting to worry. I'm like, oh my gosh, somebody's using Matt's like stuff in this network and they're attacking it, whatever. And it turns out there's actually a really popular tool that people use as an endpoint agent that for some reason, I don't know if they still do this, but they were using these locally on all devices as I'm assuming as some kind of, let's test our own detections. But I'm not sure. It's a legitimate endpoint security solution and it just made me freak out because I thought they were actually being attacked, which they were not, so that's good. So back to the use case is that com spec. And here we can see it's being used to run PowerShell and install on the service. Normally you see a random service name, but this talk was originally created for RSA. So I had to make the service name RSAcon. And then here we see the full command. And I've already, let's see, this is on, shoot, what event ID is this? I missed it. Let's go back. Oh, I think this is event ID 10. Okay, here we go. So this is, yeah, the sneaky way of calling command line and then you see no profile, the window's hidden. It's an encoded PowerShell command. All flags that make you raise an eyebrow at it. So you can use your window hidden encoded. Okay, there's something going on here, no profile. Yes, which are common ways to bypass PowerShell execution policy, which turns out wasn't even created as a security measure. Execution policies were created as a, let's hope these admins don't shoot themselves in the foot by running these PowerShell commands. It's kind of like having to type pseudo when you do something on Linux. So we're gonna look into event ID eight because we already looked into those other two that were there. Most true positive process injection attempts you can find in both of these Sysmon event IDs. This is create remote thread detected and another one that you don't have the ability to see it all with plain windows logs. And it lets you know here that PowerShell has injected code into DLL host. And you can see it here in the source process as PowerShell and then the target is DLL host. And then the next is 10. And this reports when a process opens another process and that's usually followed by information queries or reading and writing that address space of that target process. And during this attack in the lab there is a generous amount of both event IDs because the command was run and it attempted to inject into all of the processes. So this is just lab data. And it kind of shows that that encoded content isn't exactly the same as our actual customer event that we had in the use case. So if we dive into that, we take that encoded command at the beginning of this use case which was base 64 and I decoded that. And then I was really confused because if the top parts where I grabbed it from PowerShell on the left, the bottom is where it got decoded to but then it looks even more decoded and I tried to like double decode it, but it didn't work. Turns out that it was a base 64 GZIP file. And then after looking that on the left hand side, I'm like, I don't know what this code is. So I started to look into that decoded GZIP and it was cobalt strike. And this is one of the methods that cobalt strike uses to avoid detection. And another funny story, when you include this kind of thing in a presentation, it's best to take a screenshot of it as opposed to just copying the code because it turns out that a lot of places, it was very hard to email and share this presentation and no one could open it because all the endpoint agents were freaking out thinking that I was trying to use cobalt strike hidden in a keynote file. That's great. So you had to make sure it's an image. Yes, yeah. So definitely learning moment for me there is to always just take a screenshot. And then here's a summary. Really just look for ComSpec, right? Super easy. No one ever does that. Nothing good is happening if ComSpec is running. No, it's so weird. It's just this environment variable that you'll rarely see and is worth detecting on as a threat. So now that everybody's a little bit more familiar with event IDs and Sysmon, we're gonna tie it all together and a customer exchange compromise. I won't replay that lab because that's gonna be another million screenshots of event viewer. But this is an example of proxy logon which came out in 2021, proxylogon.com explains all of it if you really wanna know. But it's basically just a remote code execution that as long as your exchange server is open on port 443, people could attack you. So it generated tens of thousands of breaches of exchange online. So many incidents. Yeah, people's on-prem exchanges. Yeah, so not too long ago after, I mean, I guess this would have been a year ago now, we had a customer come to us and asked for help around a notification they received. There's two MSI files that are installed on the exchange server. Other commands were being run that didn't think should be running. And there were actually 11 rapid-fire findings. And so you'll notice that's the next several slides as they all happen in rapid succession. First, you see something along the same lines as that process injection we talked about in the other use case. So we basically have PowerShell being run, injecting into DLL host and acting alone with nothing else. It could be something not super terrible because we've seen legitimate software do this before, but then you get something like this, which is a discovery command, again, on its own, not terrible, but this is net group DOM domain admins, which is a command to find all domain admins in the current domain. Helpful for admins, right? Also helpful for attackers.  Right, you start to be like, all right, well, that wasn't me, uh-oh, that's not good. And then here, you have a little bit more suspect activity. Granted, this is something I've done before, right? You need to transfer files and you just decide, okay, I need to open up SMB and transfer from one to another. Not great, especially if you're doing it to a domain controller from your exchange server. Very bad, right? That's definitely somebody just copying stuff back and forth. Even the effect that they could do that means that something is not set up properly. But yeah, this is when they started to really freak out. Talked a little bit already about PowerShell encoded commands, but this is something that happens in malicious PowerShell all the time, is that net web client, so you can, you know, a couple lines of code in PowerShell call and execute strings, like it'll download that string and execute it, right? Super easy, it's seen all the time. And then finally, you see this IIS web service process spawning another child's process. So in this case, it was PowerShell and associated with that full exchange compromise. This did turn into like a full IR response investigation, but without the use of Sysmon, who knows how long it would have been to see that compromise, right? It could have been way after this. You wouldn't notice it happening, right? Oh yeah, you would have just had a ransom exchange server at that point, right? And then with that last detection of IIS, so if you have on-prem exchange, definitely be logging your IIS logs and your process logs on your IIS server. That W3WP-EXE is the EXE name of IIS, which the, what is it, Outlook Web Access uses, and it should never respond PowerShell or command line. Never, there are no circumstances for each, that is a good thing. No, there's, unless you are being, maybe being pen tested, but above and beyond that, that is a terrible thing to ever have. Even then you let them know we found you. Yeah, yeah, for sure.  There's a lot of different things you can do. You know, you can have a gap analysis and figure out where your quality of logs are now versus where you want them to be. You have critical devices that are not logging the right stuff. You have to plan for a lot of these attacks, which can be daunting because there's attacks for everything, right? But one of the first things that you can do is improve the processes of, you know, include installings this month, right? It could be a change in configuration. You can align these with company objectives, like, almost everything in security can be tied to something in your business, right? Whether it's reduced downtime, money savings, whatever you want to tie it to, you can usually align this kind of activity with bolstering like your security posture, okay? Something that we created, and it's out there for anybody to use, it's called Posham. One of the most difficult pieces we had in the beginning of Starting Boomera was installing all this stuff at scale, right? Because you can use, we've used NX Log, right? Which has its own configuration file and Sysmon, which has its own configuration file. But if you're not paying for them, you have to have like the right configuration or it doesn't start or you have to have the right logs being generated just like silently fails. So we were tired of having to do that and figure out, all right, well, this server has IIS, this one doesn't. This one's a 2016 server, this one's a 2012, whatever. This actually just does it automatically, which is super nice. It figures out what channels you actually have and just shoves it into config file. So it makes it much, much easier. You can, you don't have to have Blomera to use this. You could just plug in whatever you want, but yeah, it was much easier to just do that than manually. And then pick a configuration. A lot of times you saw the Enrichments that Sysmon Modular gave us with all those MITRE TTPs that it lists and it gives the ability to turn on and off certain levels of logging, especially Sysmon isn't made for all devices, right? One of the things we ran into is if we had a couple of customers that had really old servers with spinning disk drives, like their DCs were just some 10-year-old box on the floor, the amount of IOPS that Sysmon has to use to write all of those DC logs that are coming in kills the drive. It makes it very slow. So if you have SSDs, you're probably fine, but just know that Sysmon Modular is out there. So for the slower devices that we have, you have the ability to not overwhelm them if you really, really want to. And I always include this too. Always test your SIM. Whether you're doing it yourself, whether you have a third party doing it, having an active relationship with your MSP, MSSP, SIM vendor or whoever, there needs to be some kind of regular verification that you're detecting stuff, right? Whether you have a pen test or whatever, there's just so many things that can go wrong to make detections break, right? Service stops, a firewall rule gets implemented, something, right? There's a lot of different things that you can do to test. Like there's easy password spray stuff out there. There's just real easy commands that you can run a command line to test it. Yeah. One of my favorites, and I did this when I was doing a Blumerid demo was you could just go to who am I and you'll notice it'll flag that because this is not something commonly run. The testing doesn't have to be that hard. My favorite way to test is get some accounting firms and have them as clients. There's a secret class. They give all accountants that say, click on everything that comes through your email. I'm convinced of this. I don't know what the class is called, but there's something about accounting firms. Great, because, you know. They're quite vicious as I have named some of them. I'm like, this person will click on anything. Oh, that's terrible. Yeah, accounting for, I don't know what it is about that particular industry. I thought sales was bad, but that's very interesting. Yeah. We need some industry stats on what's- Yeah, I know, right? There'll be a panel discussion. What's the worst industry for clicking on data? Yeah, we have to ask no before. I'm sure they have stats on that. Yeah. And then last but not least, if you want to do your stuff internally, this is a great combination of tools to track what you're detecting and how you're detecting those things. And that's it. Yeah, Vector, Aramik, that's not, I'm sure there's links to everything. So it's easy for you to be able to click on everything in the description. Yeah, so this is me. Thank you so much for having me on. I love doing this stuff. And I love preaching the great word about Sismon. Well, it's fun. This is such a deep discussion on, we started with some of the theory and then some of the practical as you go through it and understanding the big picture, if you will. It's not just the commands that are run. This is like the whole top to bottom of why you need these things, why we need some of the alerting and how we're looking at it as security researchers. I think it's just, it's critical. Cause I get that a lot with the, as I'm sure you do with the younger people starting in, newer in their career. How did you see that or what made you think about it? And that's where the theory and anomaly detection really comes into play. Oh yeah, yeah. I definitely suggest reading up on if anybody ever has walkthroughs on instant response, right? So like either they've seen an attack in their lab or they've seen an attack at a customer and it's been like anonymized or whatever. They're always really interesting to read like the really technical, well, at least to me, the really technical versions of those because you can actually see, oh my gosh, like they went from this process to this process and that's how they did that. And the better articles, I think anyways, tell you what to detect on in those areas, right? I spend time reading different reports, you know, it's good, good reading. There's some great sites for that. And one of them was interesting, I posted last year, I can probably find a link to leave that down there. They had actually gotten a treasure trove of a weird different report because it came from the threat actor.  So they got to walk through what their day to day was. Of how they were doing it. It was really cool. I put it on LinkedIn a few places. I can't remember if it was Recorded Future, I think it was Recorded Future who actually published it. It was a great read because it's very, very detailed. And it just, they had logs of everything this person had done, how they made their notes on how they attacked, what was successful, what wasn't. But this is how we update our threat modeling for what you should protect against. Cause I'll see people kind of go off and really worry about this or that when they look at their threat models. I'm like, just read different reports. You'll understand it because you only have finite number of resources. People get excited about some of the physical layer stuff because don't get me wrong.   But honestly, he is probably not in most people's threat model of the, it's just from our day to day with all of our businesses. It's not physical. It's always phishing emails and some of the silly stuff. Hey, lock down. Use security cards. Don't let people tailgate. I'll agree with all those things. But if you only have so much resources putting a better door access system and probably they're getting into phishing, that's the most likely. Once you solved all the other problems then work on the other stuff. Yes, yep. Yeah, for sure. Well, thank you for this. This was a lot of fun. I love diving into these things and all that fun stuff. Links will be down below and thanks. Awesome, thank you.


So we are almost done with the time management series. We'll probably be closing things up within the next week or so, and then we'll be right on our way to Unreal Engine. It took forever, but I finally finished sculpting, modeling, and UV mapping the tactical military night character that we're going to be using in the new engine, and I'm really excited to explore and test the differences between Unity and Unreal, so I hope you guys are also looking forward to that as much as I am. In the meanwhile, since this current series is coming to a close, I've been thinking really hard about what the last three videos should be, because I want to make sure that they're important and useful for you guys, and while I'm not sure what the last two should be, I know that today I wanted to talk about one of the most important resources I think everyone has that I see a lot of people waste all the time. Your thoughts. Do not waste your thoughts. They have a lot of value because your thoughts are the fuel for your actions, both consciously and unconsciously. Every word that comes out of your mouth is just the sound of your thoughts. You cannot say something you didn't think about. Likewise, aside from your muscle memory, you cannot do things you don't think about, and this idea really resonated with me when I was watching an interview with Magnus Carlson. Now, for those of you who don't know, Magnus is basically a chess god. Now, I'm not particularly good at the game. Personally, I've always loved chess. I used to be in a chess club when I was a kid, and if you ever took a really close look at the Royals Guys logo, you can see that the king and the queen of the chess board are both integrated into the logo itself. And that's just because I love the game, it's beautiful. So I kind of just follow the chess scene with my downtime, but the point is, chess is cool, and Magnus is basically like the king of it. And when I say king, I mean, this guy remembers just about every move from every game he's ever played. He can beat you without looking at the board blindfolded against 10 people at the same time. This guy is just on another level. And on one of his dayoffs, he accepts an interview from 60 Minutes. And remember, this is his day off when he's just relaxing. And here's what happened. He had a day off. He'd been in London before, but told us he'd never seen the sites. We took him up onto this. It's a giant ferris wheel that overlooks London. And it's right on the south side of the Thames. So you can see the houses of parliament, Big Ben. We were up in the London Eye for, I guess, 45 minutes or so. He didn't look at the window. Just, it wasn't interested. Do you ever stop thinking about chess? Sometimes, but right now I was actually thinking about chess. And you were thinking about specific moves? Yeah, I was thinking about something specific in my preparation for my game tomorrow. You know, I've always wondered what the inside of a genius's mind would look like. And then it kind of just hit me that part of being a genius is that all of his spare thoughts are about chess. All the time. It doesn't matter where he is. It doesn't matter what he's doing. He could be walking through the street. So what are you doing for the rest of the day? I'm preparing for the next game. He's buying groceries? Yeah, I was thinking about something specific. On a lift with a beautiful view of London. And my preparation for my game tomorrow. Still planning his next chess move. He's not on his phone, scrolling through Twitter. He's not sitting idly thinking about what to eat. He's constantly thinking and planning chess. Usually, how to prepare for his games tomorrow. So I actually believe that one of the most important differences between a genius and a normal person isn't necessarily how their brain is structured. It's how they organize and focus their thoughts. Like, of course, that's not everything. I'm not saying everyone could be a genius. But a part of it is definitely a genius probably spends 99% of their spare thoughts thinking about the thing they're a genius at. While a normal person probably only spends 3% of their thoughts focused on any one thing at a time. I mean, as a normal person, I can barely focus 90% of my thoughts on a single thing for more than 30 minutes. But really ask yourself, where are you spending most of your thoughts? Because how you spend your thoughts determines what you're going to get good at. If you spend your thoughts scrolling through Twitter and Instagram, you're going to become really proficient at scrolling through Twitter and Instagram. But if you spend most of your thoughts exploring game design, you're going to be more proficient at game design. Because obviously, the prerequisite to doing things is thinking about doing things. So to properly explain this, we're going to talk about something I like to call downtime, which is basically the dead time that happens when you're really not in a position to be productive the way you would like. You're not at your desk. You do not have access to your computer. You couldn't really work hard even if you wanted to. Usually these types of moments are when you are waiting in line at a restaurant or something like waiting for the next customer to arrive. Now most people I've noticed in these situations default to just opening their phone and scrolling through Twitter and TikTok until the situation has passed. But I highly recommend taking a page out of Magnus' book. Utilize this time to organize your thoughts. Plan out your attack for when you enter the zone at home. For example, I used to work on the sales floor at the mall and at Best Buy, selling whatever my boss needed me to. Now I wasn't particularly passionate about my job. I much preferred spending my spare time learning how to program games in Unity, but I really needed to pay rent. So I was stuck at this job. There was no way around it. This was just a sunk cost every day for eight hours. My dream was not being built. But then I realized there were certain points in my job where no customers would come and I'm just waiting. Everything is stacked perfectly. I've done my job to the best of my ability and I'm literally sitting there with nothing to do, which for me felt like a big waste. So I started to use this time to plan the rest of my day or to organize my thoughts and think about the best way to fix that glitch that kept popping up. I started to think about what are some potential solutions to these glitches? What are some things I haven't tried yet? What parts of the code might be giving me the most problem? What are features in the game that still need to be created? What characters do I still need to make? What animations are missing? How many frames should those animations be? I started filling up my downtime planning my next move so that by the time my shift was over, when I grabbed my coffee and arrived back at my computer, I would be in the zone and immediately start executing everything I had planned for the last eight hours. And the performance difference was crazy. Like the first day I tried this, I got so much done when I got home, I was baffled at how smooth my workflow was getting because all I was doing was things I rehearsed in my mind. And simply rehearsing what you need to do in your head before you do it allows you to hit the ground running because you're not trying to think about it for the first time. And that makes sense, right? Like, obviously, the less time you spend thinking about something, the less good you're going to be at it. Which means, conversely, the more time you spend thinking about something, the better you're going to be at it when you finally turn those thoughts into action. So, don't underestimate the power of filling your downtime with planning your next move. Just because you're stuck at your job for eight hours a day doesn't mean you can't still use that time as fuel to hit the ground running once you get home. So, I hope that helps. And as always, hope you have a fantastic day and I'll see you around.


On Sunday, October 2nd at 3 p.m., the historic Snively Chapel will be holding its annual celebration. The church will be open to the public and welcomes any and all community members to join for music and fellowship. So we only have two annual public events, and this is one of them in October, usually the first week in October. And then we will have a Christmas celebration, and that will always be the first Sunday in December at 5 o'clock p.m. Our goal in these annual events that we invite the public and our community is to allow them to be part of the historic treasure that we have there in county. I mean, you know, it's not just faith-based, it's a community treasure, this building that, you know, it's been there for so long. So, yes, every year we want to try to do this annual homecoming and invite our public to come and be a part of this historic treasure and just, you know, it's a wonderful opportunity to share together with the public. My name is Aaron Crump, Chief Medical Officer at Pikeville Medical Center. The Heart and Vascular Institute, it's really an incredible program. Made incredible because of the providers, because of the technology, and because of the staff. We've truly become the leading provider of Heart and Vascular Services in eastern Kentucky, and in fact, what we do compares to anywhere in the nation. Pikeville Medical Center, when it comes to your heart, place your care in our hands. With newly finished renovations, the church and its members can be proudly recognized as one of the oldest in Pike County. The community is willing to do anything to keep the church up and running for as long as possible. We're thankful that we have this historic treasure still with us. St. Amunitum Methodist Church has been the ones that have been the caretakers of the church for a long time, and they continue. And we take that seriously, the responsibility of keeping the treasure for years to come for the community. Reporting from Mountain Top News, I'm Nick Collum.


I cannot believe I am auditioning at the House of Mouse for different roles for the live-action Little Mermaid. This is such an exciting time for me. The obvious choice, of course, is for me to play Prince Eric. I already have the sea legs. I'm already very proficient on a boat. My uncle has one. I've been on it several times. I mean, I wasn't actually on the boat. I was on the dock. They won't let me come on the boat.   Looks kind of attractive, but it doesn't really matter at the end of the day. He's just an NPC. Okay, perfect, yeah. Hello, Ariel. It's me, Prince Eric, too much. Hello, Ariel. It's me, Prince Eric. Is there something out in the ocean?


I'm Dan Jigalski. I'm a maintainer of the OpenTelemetry Collector and a principal software developer at ObserveIQ. I've been developing observability software for over 10 years and contributing to the OpenTelemetry Collector for the last three. Most recently, I've been working to enhance the collector's ability to manage and process telemetry. And that's what I'll talk about today. First, I'll talk about the context of these enhancements, specifically why the OpenTelemetry Collector is an ideal place to add more sophisticated telemetry processing capabilities. I'll describe the collector's pipeline architecture and make a distinction between managing telemetry and processing it. I'll highlight some limitations of the architecture and finally, I'll introduce you to a new feature set called Connectors, which resolve these limitations. So traditionally, there are three types of telemetry data, metrics, traces and logs. And until recently, observability tools were designed as end-to-end solutions with one data type in mind. So to achieve a high degree of observability, you would have to deploy several independent tool chains. And it also meant that if you wanted to make a change to any one component of that tool chain, you might have to replace the entire tool chain. So OpenTelemetry is improving upon this by supporting multiple data types from the start, but it also limits the scope of the problems we're trying to solve. We're focused on processing and generating telemetry. We're leaving the other problems to other tools. A typical OpenTelemetry solution looks something like this where if your application is on the left, you bring OpenTelemetry's instrumentation libraries into your code base where they can capture and emit telemetry. Often that data will flow through the OpenTelemetry collector, which can process it and forward it to whichever storage and analysis backends you like. And the collector can also generate its own telemetry. It can read logs from files, scrape metrics from APIs. It's also very highly interoperable. So if you've already established data streams using other tools, you can usually redirect those through the collector. So between the collector, the instrumentation libraries and compatibility with other ecosystems, the collector has, we have many ways to generate useful telemetry, but it's also, of course, very important that we have robust processing capabilities. A current trend in the observability industry is to shift left, which is with the idea being that we should move processing capabilities from backends into the collector or from the collector into instrumentation libraries. And while it's generally true that you can reduce costs this way, the collector has some unique advantages because of its role in the ecosystem. So in some cases, we can only shift left so far. So first of all, the collector is one of the only components that can process telemetry regardless of the source. If it comes from instrumentation libraries, the collector itself, other tools, it can process all of that data. So any capabilities we add to the collector are more broadly useful than if we were to add those capabilities elsewhere. Additionally, the collector has the opportunity to correlate telemetry that comes from multiple sources. So of course, the collector was designed with processing in mind, and it has had many useful capabilities for quite some time. We can filter, annotate, aggregate, redact, transform, batch. But I think what the collector has lacked is a flexible system for managing telemetry. And when I say managing telemetry, I mean something different than processing. I mean getting the right data to the right place, both internally in the collector and externally. Talking about things like merging data streams, replicating data streams, and routing data streams. And that's what the connectors feature set is all about. It provides new ways to manage telemetry within the collector, and it does so in a way that's backwards compatible with the existing architecture. As a bonus, connectors also give us the ability to work with multiple data types in the same place. And the end result I think is that we now have a generalized framework for processing telemetry within the collector. Okay, let's get into the details here. So the collector is built around a notion of data pipelines where each data pipeline is made up of individual components, and each component will essentially do one thing and then emit data. We have three classes of components, receivers, our inputs into the pipeline, processors will modify the data in some way, and exporters are the outputs from the pipeline. The basic requirements for the pipeline are pretty simple. We need at least one receiver, at least one exporter, and exactly one data type. The receiver and exporter requirements are probably pretty intuitive. We need to have an input and an output, but the data type requires a little more explanation. The collector, as I said, supports multiple data types, but a given component may not. It may only support one or two. For example, we have a syslog receiver that only supports logs, or we have a Jaeger exporter that only supports traces. So by framing a pipeline with a data type, we're adding a lot of clarity about what is expected of the components in the pipeline. And this, you can kind of look at this in two ways. We can, first of all, the components in the pipeline must be able to handle that data type, but secondly, we're also giving the user the ability to ask more precisely for the capabilities that they want. So if I put a OTLP receiver in a traces pipeline, I'm only asking it to receive traces. It won't receive metrics or logs unless I ask it to by placing it also in a metrics or logs pipeline. I'm gonna be showing you a lot of pipeline diagrams, like the one on the right. So I think it's important to understand the configuration that goes along with these diagrams. So I'll quickly run through the configuration of the collector and just highlight some of the relevant parts here. So first of all, we have a section for configuring each component class, receivers, processors, and exporters. And then within that class, you can configure multiple components. Each component, of course, has its own parameters, whatever is appropriate for that component. But importantly for this talk, component IDs, these are a unique way to refer to this configuration. And it's a very simple format. It's just the type of component and then optionally a name. Down here, we have a pipeline configuration. Pipelines also have a unique ID. It's the data type and then optionally a name. And then we specify the receivers, processors, and exporters that we'd like to use in the pipeline. And this implies the structure of the pipeline. And this is enough for us to understand the data flow in most cases. And we refer to the component IDs that we specified above. And this is how we specify the receivers, processors, and exporters. So this is enough for the structure, so that's what I'm gonna show you for the most part. All right, let's talk more precisely about the meaning of the structure. In particular, I wanna highlight the difference between pipelines and data streams. I think if you're familiar with the collector, you may not be making this distinction, or maybe you are, but when we think of a pipeline with multiple receivers, we should think of it as receiving multiple independent data streams. Those data streams will then be merged together into one data stream, and then that data stream will run through the processors. If we have multiple exporters, that data stream will then be replicated. So I'm talking about merging and replicating data streams here. I mentioned earlier, these are the kinds of things that we need to be able to do in order to manage telemetry. And so the collector can do some of this already, but the problem is that it's slightly constrained by the pipeline's assumptions. But we can look for other ways. There actually are a couple others. We can share components, we can share a receiver, and when we do this, the data that it emits will be replicated so that each pipeline will have its own data stream. And we can also share an exporter, so this will merge the data streams before it goes into the exporter. It's also possible to share components between pipelines of different data types, assuming that the components involved can support both data types. But for example, the OTLP receiver, OTLP is short for open telemetry protocol. So if you send metrics to this receiver, it will route them onto the metrics pipeline. And if you send logs, it will route them onto the logs pipeline. So we even have some rudimentary routing capabilities here. Okay, so at this point, I've already explained to you all the rules that govern how pipelines work. These are the expectations that people have based on years of using the open telemetry collector. So we need to try to respect these as we look to resolve any limitations with the system. So I'm gonna show you some scenarios that highlight the limitations and talk about how we can address them with the connectors framework. But keep in mind that all of these things will still apply. Okay, so let's look at a scenario here. We have an expensive analytics tool, and we have a lot of log data. We can't afford to send all of the log data to the tool. So we're doing the obvious thing and just filtering a lot of it out. But let's say we have another requirement where we must preserve all of the data, some regulatory requirement, for example. So we can't afford to send it all, but we also can't filter it. So we can use the mechanics that we've already discussed, share a receiver. This will replicate the data stream. And now we can filter one of the data streams, send that to the expensive tool. And the other stream, we can do whatever we need to, but ultimately it will go to some kind of cheaper cold storage. So this so far is so good, but let's add another requirement. Let's say that we have a policy where we need to redact PII immediately as soon as we touch the data. So we're adjusting one data stream, we wanna redact the PII, then we want to replicate and then process some more. The problem is we can't do this with the existing pipeline structure. Or at least we can't do this within a single pipeline. And the reason again is because we can only replicate immediately after receivers or immediately before exporters. So we can kind of look at this as partial pipelines, but none of these are valid. The first one doesn't have an exporter, the other two don't have a receiver. So as a workaround, we can use the OTLP exporter and receiver in combination to forward the data. So the first pipeline has an OTLP exporter, and the second one has a shared, the second two have a shared receiver so that will replicate the data like we wanted. But this is a networking protocol, doesn't seem ideal, right? We've, there's a lot of configuration involved just to get them to communicate correctly with each other. And just intuitively, this can't be the most efficient way to do this. So what we want is something that can fulfill the requirements of being an exporter in the first pipeline and simultaneously fulfill the requirements of being a receiver in the other pipeline. And all we really want it to do is to just pass the data along. So in the simplest case, this is a connector. It's an exporter and a receiver at the same time in different pipelines. And this is specifically the forward connector and it's just called the forward connector because we are just passing data along. So to configure this, first of all, we have a new section in the configuration for configuring connectors. And this is just like the other classes of components. And then when we use the connector in the pipelines, we use it in place of the exporter or receiver. Here's another example of using the forward connector. I just showed you how we could use it to replicate data streams. Let's merge data streams with this example. So we're receiving two different data streams. We're labeling them because we can tell them apart from then on. Then we'll run them through a forward connector, do some more processing, export. But the idea here is that it's a shared exporter for the first two pipelines. And so when we have a shared exporter, we merge data streams together. And then it's a receiver and the other pipeline. So this is the first capability that connectors give us. We can sequence pipelines. Previously, pretty much all pipelines were in parallel. You do all your receiving, you could do some merging and replicating. Then you could do all of your processing, maybe some more merging and replicating than all of your exporting. But now we can interleave the merging and replicating operations with processing operations. We can do this arbitrarily. Okay, back to the first scenario. Let's continue adding requirements. Let's recall we were ingesting data that we can think of as both high and low value and whatever the high value data is passes through the filter on the upper pipeline. But the lower pipeline is sending all data both high and low value. And let's just say that we want to really optimize this and we recognize that to meet our regulatory requirement, we can actually filter out the high value data and only send the low value data to cold storage. So one thing we could do is just add a filter that's basically the opposite of the filter in the other pipeline. But we can do better than that. We can route the data with a connector. And this works probably just as you'd expect. It's a exporter in the first pipeline. And then it plays the role of a receiver in the other two, but it doesn't always emit data to both pipelines. It will do so conditionally. You can have a pretty standard routing table. We have some criteria to evaluate. And then we list the pipeline or pipelines to which we would like to emit the data. Another way to think about that is that we're telling the connector in which pipelines it should act as a receiver when that criteria is met. So this mechanism of referring to pipelines and then conditionally emitting data to them is built into the connector's framework. So this means that we can have multiple types of connectors that behave somewhat differently, but use that mechanism. Here's another example, the failover connector. This will, by default, route data to whichever pipeline is the highest priority. But if there's an error downstream, let's say in the exporter, then that error will propagate back up the pipeline to the connector, and the connector can react by rerouting the data to another pipeline. So conditional data flow is another capability of connectors. And just between these first two, I think we've basically solved the problem of needing to be able to manage data streams. So, but we can keep going. Connectors can actually do more than this. So let's think of, we're sending all this data to cold storage. We're not really looking at it. We don't know anything about it. We can pull it up if we need to, but on our day-to-day basis, we're not doing that. So what could we do to get more value out of that data? We can summarize it in some way, and we can do that by generating telemetry that describes it. And the telemetry we can generate could be of any data type, but let's just say we're going to count the logs and counts our naturally metrics. So we're gonna generate a metric data stream that describes the log data stream. So we can add a, we have a count connector. And what I've done is I've added this to this pipeline down here as a second exporter. So the data's replicated. One replica goes to cold storage. The other goes to the count connector. The connector will then count the data and generate metrics that describe it. It's acting as a receiver in a new metrics pipeline as well, though. And the exporter there is the same exporter as before, I'm just assuming that we can support both metrics and logs. So why does this work? We have the count connector acting as an exporter in a logs pipeline and a receiver in a metrics pipeline. So the way to think about this is what does it really mean to be an exporter, a logs exporter? It means that you can consume logs and basically that you're the last component in the pipeline. So it definitely does that. And as a metrics receiver, it just needs to be the first component in the pipeline and emit metrics and does that as well. So let's talk about how data types is, where data type support is defined for connectors because for other components it's very simple, but with connectors we have to consider the dual role that it plays. So we have pairs of data types and more specifically we have ordered pairs of data types. The forward connector can forward logs to a logs pipeline, metrics to metrics pipeline and so on, but the count connector can take any data type and generate metrics. But you wouldn't expect that if we gave it, if we were counting metrics that it would generate logs, right? Like this is not a reversible operation. So each connector will pick and choose the ordered pairs of data types that it will support and implement the appropriate functionality. And then when you use it in a configuration, it's implicit what functionality you're asking for, just like if you put an OTLP receiver in a traces pipeline, you're only asking for the traces functionality. If you route logs to the count connector and put it in a metrics pipeline, then you're clearly just asking it to count logs. So we can generate data streams. And I think this to some extent blurs the line between processing and managing data streams, but there is an important aspect of the management here, which is that we're generating a new data stream. And I think if you consider what it looks like to count metrics or ingesting metrics and emitting metrics, you could just mix those together and put the new data into the old data stream. But this is a new data stream. And some users may not want to do that. And the nice thing is with the connectors framework, they have full control over this. They can put the connector in whichever pipeline they want to receive the data. And if they wanted to preserve the original data stream, they can, you can replicate it. And if you wanna mix them back together, you can merge those data streams. So you have whatever options there that you need. Now I wanna go through the first three scenarios quickly and create a visualization of the data that we've sent to both the expensive tool and the cold storage. So let's say that these squares are resources. And by resources, I just mean things that are emitting telemetry or things that the telemetry describes. So I'll use the space above to represent the data we sent to the expensive tool and the space below to represent the data that went to cold storage. So we, we sent, first of all, we sent all of our data to the expensive tool, but we couldn't afford that. So we filtered much of it out. Then we replicated it and preserved it in cold storage. And then we optimized a little bit. And then finally, we characterized it by generating some metrics and sending those to the expensive tool. So that's where we're at. Let's, let's assume we've done similar things for metrics and traces. Maybe the workflow is a little different, but basically we're trying to separate high value of data from low value data. So we have something like this and this is a pretty good place to be, but sorry, this is what we have. So this is a pretty good place to be. We have some of the telemetry from every resource. And this is a pretty standard approach to reducing data, right? We apply rules across the board that will remove what we perceive to be low value data from the data stream. Now, the problem is that we're only really looking at the low value data when we're, when we're developing the configuration and deciding what those rules are. So if we make a mistake, you know, it's very easy that there would be some kinds of, even if they're minor, but some kinds of problems indicated in the low value data that we've sent to cold storage. And we're just not going to see these even if we're characterizing that data in various ways. So one thing we can do is to sample data from there and sort of promote it to the high value data. So we'll find a little bit of the data that can give us some more insight into what's going on. So if we just think about this from the log's perspective, what we could do is before the data goes to cold storage, we'll replicate the data stream and we'll send it into a sampling processor. This can apply whatever criteria we want. Maybe it's random. That would probably not be very valuable though. So let's say we run to sample resources. So the logic would be roughly when data comes in, we look at the metadata that describes the resource that the telemetry is associated with. And then if we haven't seen that resource before, we make a decision about whether or not to include it in our sample set. If we do include it, then all logs associated with that resource will pass through this processor, otherwise we'll drop them. So that can give us something like this where we have all the logs for some of the resources. Now if we do the same thing for metrics and traces, we might end up with something like this where we have all of the traces for one resource, all the logs for another, and if we're lucky there might be some overlap. And it's possible that we could use deterministic criteria to decide what the resource would be, but that limits the types of criteria we can apply. What we really want is a framework that allows us to apply any criteria we want so that we can develop connectors that behave in the ways that we need. So let's zoom in on the tail end of the workflow we had there where we're routing the low value data to cold storage. Let's add in the metrics and traces. So we've got these three low value data streams going to cold storage. Now let's replicate all three of those streams and also send them to a sampling connector. And this can apply the same exact logic as I just described with the sampling processor, but the difference is that when it makes a decision about a resource, it can flow all of the telemetry from that resource through. And it also doesn't matter whether the first thing we see from a resource is a log or a metric or a trace, we can make that decision at the first possible opportunity. So we end up with a much cleaner sample set here where we have all of the data from some of the resources in addition to having some of the data from all of the resources. So we have a nice cross section now and we've done this without adding a huge amount of data to the expensive tool. If these were Kubernetes pods, for example, we may only need to sample 1% or less and we can still get a much better idea of what's going on. So correlated data processing is another capability of connectors. And if it's not clear what's going on here, the connector would be acting as a logs exporter and a logs receiver together and then a metrics exporter and a metrics receiver together, et cetera. So it's kind of like the forward connector but where data is just flowing through but it's all flowing through one single component and we can't do this with any other with processors. We can only do this with connectors. So this gives us that opportunity to reason about multiple data types in one place and make decisions that impact all of the data together. So just to review the mechanics that I showed you earlier, all of our pipelines still require a receiver and an exporter and a data type but when we say receiver, we can think receiver or connector. And when we say exporter, we can say exporter or connector. One thing that has changed is we can add a rule here that says connectors must be shared and this is only if you're using connectors and this is just in the sense that when you use a connector, it must be an exporter and a receiver. And the merging and replication points are the same. We've just taken advantage of where they exist in pipelines and the connectors because they are receivers and exporters exist adjacent to those points. So data going into connectors or out of connectors can take advantage of these. So these are the new capabilities that connectors offer. We still have work to do. Some of the specific types of connectors that I showed you are available today but others are still in development. However, the connectors framework is in place and I'm happy to announce that as of this week it is considered stable. So at a high level, I think the important result here is that the collector is now a much more capable tool for processing telemetry because it now has a generalized system for managing telemetry. Okay, that's it for me. I hope this has been helpful. Thank you for coming. If you have any questions, please reach out or come find me afterwards. And if you'd like to review the session, please scan the QR code.


Valuation of privately held companies. Why would it be important for an entrepreneur to know how to calculate the valuation for a company? And where are valuations applied? Let's start by looking at the context in which valuations of entrepreneur companies are used. Most people know valuation from this context. Pretty good deal. The entrepreneur is looking for $800,000 to expand his business. But what are these numbers based on? Why $400,000 equity? Why $400,000 loan? And where does the 10% come from? And that's the question of valuations. So most people know valuation or hear about the valuation the first time by watching this Sark tank program, where entrepreneurs pitch their products or their ideas to a panel of business angels who have made their money in business and now are willing to help other entrepreneurs or to invest their money in these smaller companies. So where does this $400,000 come from? And in what other context we would use valuations? Valuations are generally used in two different contexts in entrepreneur companies or small companies. First, there is equity based investment. So we look for more money into the company. And to do so we issue more stock, new stock and the purchase of that stock, the purchase price goes to the company. Business angels, venture capital, crowdfunding are some of the kinds of equity based investments that require valuations that entrepreneur companies use. Valuations can also be used when the company is divested. The divestment can be in the form of harvesting where a company is sold for profit. For example, the entrepreneur decides to sell their company to a larger business in which case we call it a trade sale or do an initial public offering where the initial entrepreneur sell some of their stock to the stock market and then cash in on the venture. Another kind of scenario of valuation might be required is liquidation. When you decide that the company has been failed, no one wants to buy the company, then you simply take the company apart and sell those assets that can be sold, take the money and then the company ceases to exist. In that case understanding valuation is useful as well, but how we calculate the actual valuation is slightly different. Quite often we talk about the investment more because that is more relevant for most companies. Having a lucrative exit is not as common as seeking funding and to seek funding you need to understand what is the value of your company. Why do companies need funding? We need to understand that a small company does not become a large company overnight but they need to grow and to grow you need money. If you want to expand buy a new factory might cost tens or hundreds of millions of euros, that money needs to come from somewhere. If all the money that comes into the company is simply revenues then growth will be very slow. So in practice companies seek external investment to grow and why an entrepreneur would like to do that? It's simply because originally you have all of 100% of a very small pie and often times it's better to actually have a smaller slice of this a lot bigger pie. So which pie? Which one is better owning 1% of Google or owning 100% of the Surgeoning company that you started last week? Of course owning 1% of a large company. On the way from a small company to a large public listed company the entrepreneur typically goes through different sources of financing. Most of the time you start with your own money or a co-founder you might get some money from friends and family that gets you started. Then you might get an angel investor.  Once you are already growing then venture capitalists become relevant for future growth. So these companies invest in the ballpark of hundreds of thousands of euros to millions to tens of millions and you can have multiple rounds of venture capital funding at different points of growth. Finally companies that are highly successful file for initial public offering in which case general public can buy their stock and they become stock listed large corporation with diversified owner base. At every point of this chain whenever there is equity being issued or stock being traded someone needs to calculate value for that stock and that is where valuation comes in.