wnloader will then reach out to the command and control server Which will issue new commands to the downloader which will then execute those commands on the infected endpoint Now as in the case of the previous downloader, we see this new variant also is very short and very simple, you know Basically all there is the core of the functionality in this downloader is to Take commands from the command and control server and just execute them non-stop on the infected endpoint. That's it So basically the that they're using very small and very compact implants to establish an initial foothold into the networks Now, uh, I spoke about the sec the twofold approach, right? So the second approach that the attackers took during the course of this campaign was to use malicious pdf That would be opened up by the victim and the pdf would have language that said We cannot display the the the content of the document to you However, the correct version of the document is available at this location Please click on this location and open up this document And that's exactly what happens during the infection chain The pdf reaches out to a remote location that downloads a mal doc The mal doc consists of malicious vba macros that will in turn drop a vbs based instrumenter and a power shell based downloader And then the infection chain is pretty much the same. You know the instrumenter will Execute the power shell based downloader which will download commands from the remote locations and execute them on the infected endpoint Now what's interesting here is The mal docs disguise themselves as reports or forms belonging to different ministries in the turkish government And I've got a few examples here. You know, we saw reports that you know mal docs that masqueraded as reports from the health ministry the interior ministry or from turkey So it's it's kind of interesting that you know, they were trying to this gives us an indication that they were actually trying to infect users who have relations with these specific ministries now Another interesting point to note here is that the attackers started using canary tokens instead of homegrown tokens in their mal docs So for those of us that don't know Canary tokens canary tokens.com is basically a free service that allows you to register a url That you can then put in your artifacts and when those artifacts are run You know the artifact the executable or the mal doc will reach out to That specific url on canary tokens.com and register a successful infection so Now we have I tried to the attackers now switched from the use of homegrown infrastructure for tracking infection tokens You remember I spoke about homemade tokens. They moved from homemade tokens to Canary tokens and we believe that this is an attempt at legitimizing their infection tracking Muddy water, you know a lot of them most of them use power shell based downloaders in one form or the other To establish an initial foothold on the infected endpoints uh However, uh, there was a new campaign in december 2021 and that you know made its way well into 2022 That targeted a multitude of countries in the arabian peninsula What basically happened during the course of this campaign was uh, this is the infection chain You know a phishing email would arrive in the victims inbox with the mal doc Uh, the malicious document would be opened. It consisted of a malicious vba macro That would instrument the next stages of the infection chain However, what's different here and what's unique here is that the attackers now moved from using power shell based scripts To using wsf based scripts Wsf is the windows scripting file. Um, it has the ability to Have multiple Scripts if I may say or multiple scriptlets inside of the file from different languages And that's what we saw during the course of this campaign. We saw The wsf based rat, which we're calling slourat Being used to deploy additional malicious payloads on the infected endpoint And we saw the reuse of the ligolo reverse tunneling utility that we saw earlier in march of 2021 Being used again in this specific campaign as well So, um, let me give you a brief overview of what slourat is We we we named the terms slourat, but it is also known as canopy This rat was also disclosed by the cyber security and infrastructure security agency of the united states The cisa agency This is a wsf based rat. So it contains different Snippets of code consisting of vb script and javascript and the execution jumps from one snippet to the other In order to carry out the malicious functionality that resides inside of this rat um If you take a look at this specific rat, you know, if you just open it up in a text editor You will see that there's a lot of obfuscation here But if you peel through the obfuscation layers at the end of the day, it's a very simple rat It has the ability to execute arbitrary commands on the infected endpoints, you know The server will issue a command the rat will Execute that command on the infected end point. It then Stores the output of the command in a text file that text file is Subsequently read by the rat and then x will created out to the command and control server The the rat also had the capability to deploy additional malware payloads and we saw the use of you know As I said the ligolo reverse tunneling utility being used here as well By infections by infected endpoints that were running slough rat Okay, so at this point in time, uh We've been through all the different muddy water campaigns But what I'd like to highlight here is that if you look very closely at these slides You will see that there there are there are some very common ddbs, you know the use of executables across different campaigns, you know from armenia to pakistan to turkey the use of Remote control utilities from march 2021 that were reused again in december 2021 and well into january at the beginning of this year All right, so let's talk about canary tokens. So, um canary tokens The the usage of canary tokens for tracking infection chains is very novel Uh to this specific apt We haven't seen this before the first time we discovered this was when uh, we were taking a look at the campaigns You know sometime in april 2021 um And the more we looked at these campaigns the more we realized that The attackers were slowly refining this technique So in april of last year in the campaign targeting pakistani entities We saw that the attacker started using homemade infection tokens, right? This was the experimental phase for the for the for the specific technique, you know the attack The attackers were testing the waters. They were trying to make sure You know, they were trying to understand the utility of this technique Um, they used this technique up until august of 2021 Perhaps around this point in time the attackers realized that using homegrown infrastructure and homemade tokens was a little bit too noisy You know a random document, uh, you know suspicious document reaching out to a random ip on your network is, you know bound to catch somebody's attention at some point in time Uh, which is why beginning with the campaign targeting turkey in november 2021 We saw the attackers try to legitimize their infection token tracking systems They started using a more professional implementation and they started adopting canary tokens.com in order to keep track of their infections, right? now, um Canary tokens is a very interesting concept and uh during the course of our research We try to come up with a hypothesis of why the attackers were using, uh, you know, homemade infection tracking tokens and canary tokens specifically um So canary tokens can be used for a number of purposes and these are our we have four hypotheses The the most obvious one is, you know, you want to keep track of successful infections Uh, on the other hand, we believe that canary tokens can also be used for anti analysis So, you know, basically you can have an instance where, um, you send a request to the token server first and only if the token request is received will the server issue a payload to the specific, uh, infection So this is a way to thwart, uh isolated analysis of specific components of an infection chain Um, I believe that canary tokens can also be used as a form of uh timing checks for anti analysis You know, you check the duration between a token request and the request for the payload And if it's too small, then you you know, the infection is probably running on a sandbox or some kind of an automated system Uh, canary tokens can also be used to find protections. Um, you know, if a token server keeps receiving requests from an infected endpoint But there are no requests for the payload from the infected endpoint Then that means that, uh, you know, the victim has some sort of blocking and protection mechanism in place And the adversaries can then modify their strategy and change their, uh, techniques and tools In order to achieve a greater degree of success against that specific victim All right, so I know this is a lot and uh, I know I rushed through a lot of the The content because there was a lot of content But I promise that I'm just going to present two more slides to you and I'm just going to talk about the conclusions here So, um, in case there weren't enough Timeline slides for you. I have yet another one This time we've tried to do something different. Uh, on the left hand side We have a timeline of all the campaigns that we've discussed today And we've also presented and color coded all the salient TTPs, uh, along with the different campaigns, you know in the bullet points But on the right hand side, we this is our attempt at showing the different, uh Commonalities between all the different campaigns that water has conducted over the course of the last year You know, you see the use of honey tokens across different campaigns We see that the honey tokens were then evolved into an into a completely new and uh Distinct campaign that was targeting turkey We see that, uh, you know, the similar payload similar power shell based downloaders and file names and extensions and load bins were used across multiple distinct campaigns And uh, this indicates that the adversary is reusing their TTPs You know, they're reusing their TTPs and they're borrowing from one another based on the reliability of and and you know Utility of a specific technique that they've used in the past All right. So in conclusion Uh, we believe that muddy water is a super group. We believe that it is Uh, an umbrella organization that consists of separate teams that are targeting different geographies All of these teams borrow TTPs from one another Um, and we've seen that we've seen evidence of the reviews of their TTPs from one campaign to the other in distinct and different geographies However, that does not mean that the group is not that the that the individual groups are not innovating on their own as well We see that the groups are innovating on their own. They they develop their own TTPs They develop their own suit of tools. They develop their own tactics They test them out. They refine them. They make them more reliable And uh, once the utility of these techniques has been proven They will put them in a common pool of uh tactics and tools And these tools can then be utilized by another sub team in a completely different campaign targeting a completely different geography now, uh Although there's different tools and techniques being used and there's some amount of uh overlap between the TTPs as well We strongly believe that all of these teams under the muddy water umbrella Serve a common set of goals, you know They serve the interests of a common nation state, which is Iran in this case And their primary focus again is to conduct espionage to establish and maintain long-term access into their victims networks And to also do some amount of intellectual property theft as well And that brings us to the end of the presentation Uh, I think we have we have a separate session for answering questions as well. So we'll we'll take that up. I believe that's a 310 Today Welcome back everyone So here's the time for the qna and panel session for the mower Uh, the mower block of the north sec conference. Uh, first, please welcome suhara She's a senior security researcher at crowd strike She will be doing a workshop on static malware analysis Starting at four o'clock today. I think just a few minutes after the the panel And it will be continued tomorrow. So it's a two two day workshop probably super interesting And we have our presenters Everyone is here so I will be asking question from the audience. It's not too late if you do have questions.  So if If you still have questions or follow-up questions, feel free to send them to slido slido and I will be able to ask them First of all, thank you all for your very good presentations And I have a I have a few very good questions from the the audience. So I'm going to start with The ones about air gap network, this is the subject that I think We got the most question about So I I have a few questions where people were asking about, you know, all the Proof of concepts that that was presented at various Places where, you know, you can extract data with with light or with LEDs or with sound or with any kind of Other methods, so you're presented about usp. Do they do you think that they are Really a proof of concept or do you think that they really apply to the in the real world? So great question because One of the reasons also behind our research was to see how much of these techniques were in the wild and the answer is really like Not as so they're not in the wild As far as public knowledge is concerned. So it's not impossible that Attacks happen and they were either never detected or never reported on publicly That being said Those techniques don't execute out out of thin air. They there still needs to be a payload that will implement Whatever like using speakers and microphones or things like that So they're still malicious code that will implement those techniques and that code can be detected. So Um It's not it's those are cool techniques for sure. Um, but Before if if you manage an airgate network before you implement a Faraday cage around everything Make sure you tackle the usb stick problem first properly And that you manage software updates and then you can focus on on that kind of stuff And do you think that the the lack of of Us seeing it seeing it in the wild is due to the fact that we We don't have the telemetry. We don't we couldn't actually collect them It's a it's a factor But then again what what we detect in terms of like the malware that we know that uses usb We don't necessarily detect the use of usb as a covert mechanism itself We'll detect the files like the malware that will write to usb to usb drives or that will gain persistence on the system or that will I iterate on the file system and find files to to to to copy So the the physical communication mechanism that jumps the air gap is just one tiny part of the whole attack And even though it's yeah, of course, it's hard to detect like lights flashing at a certain speed It's hard to detect But there's a plenty of other things that that can be detected So I'm I think that it's not necessarily because of lack of being able to detect that that we don't see it in the wild I have a follow-up question, but I think everyone can can answer. I will give their opinion Is about how, you know, it's always more sexy to talk about the the how we break stuff, you know, the How we we're actually able to compromise something or talk about potential attacks But It seems that we get more and less media attention when it comes to new ways Around protecting against those those attacks. Do you think as an industry we should focus more on? The novel defense techniques rather than always like spending so much effort into Like red team or, you know trying to break the The security of the different systems, so I've got an opinion on that Let's hear you first Of course, it makes better headlines saying this is how I broke that instead of this is how I defended against the hypothetical attack But the problem when you show how to break something without showing how to defend against it Like who is who will benefit from that information? Who benefits from a technique to break something that cannot be defended yet? So I think it's the responsible thing is at least to give some ideas sometimes it cannot be fixed easily But at least give some some ideas of how to mitigate how to detect Otherwise, you're just giving hints for attackers to to break more stuff Hello, okay, so I agree with that and I'd also like to add that it's a question of risk assessment as well So, you know attackers and I mean even including ourselves that are not attackers. We're still kind of lazy, right? So you will try to find that low hanging fruit and that's kind of what you're going to start by going after And so yes, all these flashing new techniques are very great and stuff I don't think they have to be overlooked. I think they're worth talking about But we have to not forget that things like defense and death and Making sure that you have all your basics covered are also really worth it as well so I've had a question for you regarding your your presentation leon so Do you think it would be possible to further help the decompilation To just to see an equivalent of the javascript that was actually Used as source for the for producing the v8 snapshots Yes, certainly, and that's a great question. So what we saw was a Okay result and we were able to work with it, but it was not ideal And I do believe that we could make that a lot better Um, especially regarding like all these properties. So like when we're you know loading a library and it you know You do like dot something dot something dot something it creates like tens and lines of codes And it's like really hard to read. So I do believe that the next steps for that specific tool will be to improve the result Of the decompilation and I do think that it's possible. Yeah Thank you um I have a question for ashir so Regarding the canary tokens so What do you think? So when people make defensive tools Uh When they provide some kind of Services that can be used as canary token. Is there any Mitigations that they can use to avoid misuse by bad actors Yeah, so canary tokens is one of the tools and services that fall into the career You know, they can be used by legitimate parties such as red teamers But they can also be used by bad adversaries to serve their purposes as well I think it's very important for Service providers of these services to proactively figure out if their platform is being misused And also it is the responsibility of the community, you know Researchers like us so that we can provide the right kind of threat intelligence that prompts people and organizations to block Certain events that are being used or certain artifacts that are actually misusing them So you have to have a proactive model where they can do takedowns and they can proactively go out and find Um misuse of their platforms This is for for the service providers And then you have to couple that with the right kind of threat intelligence So that other organizations can also protect themselves, you know until of when there are a service provider takes action And I have another question. Do you think it's also a problem because Those canary tokens generate logs on someone's server And and do you think that can be used to further find identify victims and perhaps help remediation correct So using third party services like canary tokens, you know that that would leave logs which can be recovered and tracked sometimes even by the service providers it's it's You know, it's They can they can be used in any form. The problem here is that Canary tokens are not usually You know something like canary tokens is not usually by Blocked is a part of block lists So it becomes a problem for various remediation teams to figure out whether this is actually red team activity Or this is an actual adversary in your networks. That's actually doing some activity of this form All right, so thank you. We have some more general Questions as well that everyone can can answer So there was a question about if we are seeing If there's an increase in trends in malware trying to steal or generate cryptocurrencies On non-traditional targets This is something we've you've seen or So I'm not the the expert that at he said in in that field What's super common is Is iot buttonets deploying miners there for Some reason because those are not really powerful machines What's what's more What makes the most money I guess is stealing rather than mining crypto And that that's that's going on quite quite intensely right now Yeah mining requires I think A huge buttonet rather than stealing where you just have a single target that you must compromise So malware research, you know We've been doing Well, the the the field of malware research has been has been there for about 30 years now. It's quite Quite long recently we've seen new kind of A job title, but you know helping in in in in this this research called threat researchers where there's People with no or a little knowledge about reverse engineering Trying to identify malware campaigns and so on Do you think that there's A risk that Attacks get wrongly attributed or that there's missing technical information When So that if do you think that there's a trend that we are perhaps not giving the the the public the the the Factual or scientific output For sure, I think the first part is that attribution is hard So there's many many threats and some of them use common TTPs right tactics procedures And it is hard sometimes to distinguish them between themselves But I do think it's kind of like a best effort and because there's so much industry collaboration in the field I find it makes it so much easier for researchers And so it ultimately helps the public have The best information It's not perfect, but it's a best effort. I find I think The truth is in the code So whenever malware is involved, you can't fully understand what's going on without reverse engineering thoroughly the the samples We actually had a case last week. I think where Some some people put up a blog post saying they found similarity between this industrial 2 and another malware family And they used the code similarity analysis techniques And indeed there were some parts of the code in industrial 2 that was very Pretty much identical to parts of the code in that other malware family, which I forgot the name of So they were like, oh, why why is this going on? Is there a connection with with with that? The problem is that one of our reverse engineers spotted that the code similarity was in like standard libraries That are probably present in 75 percent of C code So yes, there was similarity, but the person doing the analysis didn't understand what they were looking at So that led to some very very wrong attribution And we actually called it out. I think nicely, but And they updated the blog post saying further research showed that so it's not a real connection But so it's important to pay attention to those to reversing because As I said, the truth is in the code I believe Hi there, okay I guess I want to add to that to lian and alex that would make valid points about code and You know, that's where a lot of the hooding is But also I've seen code that would trick analysts into thinking That oh this attribution I see some interesting strings like it may look like it comes from a particular group and And and this is where I think the original question was, you know, there are individuals out there that do data analysis they try to look at this They try to make a I guess with whatever existing data they're looking at to make attribution They are also those individuals also are important in a controversially speaking manner But I think the key is when you have reversers and analysts work together And that's also kind of where I come from in my field of work And I I have to always I create a lot of Right up on what I've reversed, but also I work with analysts to see What they have on the end and it's a lot of discussion from there on and I think it's it's like that It's it's a hard thing attribution, but it's a never-ending process Thank you I have another question So we we've seen we've seen it during the the invasion of Ukraine But also in in previous conflict, but I think it's it's been I think it's never been as clear during the past few months in Ukraine that malware Is used in in the context of war Do you think that malware research as a Field as an impact outside of the technical sphere So we talk about about we talk a lot about our analysis But do you think that our work actually As repercussion outside of the technical crowd?  You're going to expose assets used by by the attackers Some attackers will immediately like pivot to something totally different Some attackers will keep using the same c2s the same malware just tweak it a little bit In in those cases, did did the publication really have an impact? Maybe not or it helped people defend better, I guess So it's a it's a trick question.  No, okay, I'd like to hear a sheer stake on that one So I believe that the the intelligence that we generate Leads to protection mechanisms, you know existing enhancement of existing mechanisms as well as creating new mechanisms And when we stop an attack at the right time, whether it's for Critical services, you know utility or for a hospital We don't know we don't really at in certain instances We don't really know how much of the attack we actually stopped and you know how much of the damage we actually prevented And you know, that's a conundrum because you know, if you want to stop an attack as soon as possible But if we do that, you know, sometimes we don't know the scope of the entire operation And so it's very important The work that we do even if we publish disclosures where the attackers are not forced to change their TTPs I'm very sure that in one form or the other we are protecting some of our customers or the you know, the general public as a whole We have two two interesting questions from the audience The first one is regarding the trust of the security vendors. So the security products to install How do you address the issue of trusting that vendor and the possibility that it may be compromised or Oversight by their respective government That's a political question to be honest Not really a technical question I'll I'll just say that honestly, this is out of my field of qualifications and I'm not qualified for Sorry captain kangaroo Um The other one was Yes, the other one was Whether um, we've seen an increase in in mobile malware And versus desktop malware or versus what desktop malware regular Stuff we see I've got another one the the He said it's it's for the the Noobs, but I don't think it's it's it's I think it's a good question is is about how attribution works. How how do we decide to publish an assessment on the the that this attack was performed by a particular group um, and if we have thresholds for Thinking we have enough evidence or clues that It's uh, it's them It's a difficult question actually I can try Um, so I guess it's how do you assess how do you? Uh, it's a hard one and every time I have to write an intel report on a piece of malware I am constantly reviewed and critiqued on my assessment Uh, but a lot of the criteria is it's I guess it's easier for those as reverse engineers because you Look at code and that's a bit more factual And I can especially if you see a particular technique doing something and it then it can then it's A high probability a high confidence that that proves that thing But you cannot but when it comes to answering questions like do you know or can you foresee?  That's difficult and that's not an assessment that I Like to write and I try to avoid that so It depends on the individual so when I do an assessment, it's very technical And that's how I like to keep it I agree with that and I'd also like to add that sometimes we do let get lucky um, so the Opsec of the threat actors is not always on point and it happens where we do get lucky and we kind of Gain extra insights into who these people's are who these people are or um, how are they Doing things and why and it's purely out of luck. So this does happen And there's no one recipe to attribute an attack to to a threat actor if there if there's one Well, we're not aware of it And there's it's it's about comparing different indicators. Some are really strong Or stronger some are weaker, but they still matter There are technical indicators for example the reuse of of an ip address that is not like a shared hosting provider, for example That's a fairly good indicator. It's not foolproof, but it's a good indicator The the malware itself have we seen that malware before is that that malware was attributed previously? by the industry as being operated by a single entity that is labeled as You know apt something And then you've got the targeting you've got other software ttps Use of like spearfishing or whatever like that are all put together You can make an assessment that everything points to a campaign that was launched by this this known actor But again, it's it's an assessment. It's not a it's rarely a 100 determination Do you want to add anything this year? So I agree with everyone on the panel Attribution is hard and it tends to be very subjective at times It depends on the amount of information you have, you know the output of your reversing exercises as when you're as your Institutional knowledge and looking at open source intelligence and code similarities and stuff like that What I also want to highlight here is that It's okay if you can't attribute stuff It's it's that's that's not a problem You know we can convey doubt in an effective manner and that's completely okay and justified You know, I'd rather convey doubt more effectively rather than come up with the wrong attribution any day of the week All right. Thank you very much And thank thanks to everyone for your presentation. Good luck sue for your workshop And this will conclude the q&a and panel for for the malware section Uh, I think we'll be back. I think it is a 15 minute break if I'm not mistaken or perhaps even a little more a little bit more But we will be back with the next section after Thanks everyone for for being here. Uh, thanks for being online for those who are listening live streaming And I'll push you a good, uh, rest of north sec. Thank you everyone so Welcome everybody. So this is the detection engineering slash blue team block aka the best block Um quick reminder. This is going to be three talks of 30 minutes No questions after the talk, but we have a q&a of 30 minutes after you can ask your questions on slido so The QR code is going to be get shown on the the screen It's on the discord as well. Make sure to ask your questions So the first talk is by kaskian kill kelly He's a senior consultant with crowd strike canadian services team kaskian's career highlights includes a variety Of roles over 20 years that usually end in the word security or the word consultant He has worked with different companies across north america and europe security conferences You've probably never heard of So kaskian the stage is yours. Thank you. Thanks I like that variety thing. It's uh, kind of what i'm going to be talking about today So my name is kaspian. You've all heard that. Uh, i'm going to talk about 10 things I wish I knew before my first incident I'm not the only one, uh, who wrote this talk. Uh, so my manager at crowd strike shelly geesprick who can't be here today Would have been doing this talk And I think i'm looking forward to seeing her version of it because it's probably going to be pretty cool We're going to go over a story or two. Uh, and then we're going to get into my top 10 list and I hope you'll enjoy it I'm just going to Proface this I don't know by saying Nothing i'm talking about in here is not public information. So it's already stuff that's gone out I don't speak on behalf of my employers while i'm here. I'm speaking from personal experience I've been with crowd strike for a while. I've been doing this for 20 years though. So That's kind of the whole thing. So let's start with my first incident. Um, I was pretty young It was a long time ago ransomware wasn't a thing yet and I was kind of a junior incident responder really like getting into my stuff getting very excited about things And we had a malware case kind of like what we've been talking about for the last couple of hours Except much less advanced than anything. I've been talking about for the last couple hours Um, we hit the big red panic button a bunch of people go running into this executive's office We're freaking out. We're tearing apart computers.  Oh my god. It might be a state motivated threat actor They might be attacking us Turns out it was actually just an email campaign um, and the only reason We were freaking out was because the email campaign deployed something on this one executive's desktop And the poor executive went. Yeah, let me just click on that and open it That ended badly. It ended in a huge waste of resources It ended us scrambling pretty much all the jets And the search dogs and the fire department For nothing it also kind of did some bad stuff for our security team because we Got looked at as that the people who freak out over nothing, which is not a great situation so There's some commonalities here. You're seeing them up on the screen, of course But one of the things that I like kind of pointing out is that There are usually three things you need for doing Incident response. Well, they're up there. I'll let you read them I'm going to talk about a second story where and this was a little bit later in my career Um at a hospital, uh, we had something worse happen. This time it was real So ransomware hits a hospital This happens. Unfortunately. It's a it's a thing this time It's not so bad that it's like making the hospital not work, but it's bad enough that it's messing everything up for us And we hit the big red button and everybody panics and everybody runs and thankfully We had the ability to call out for help. So we got some extra people in but it took us a little while to get to that point Um, we also didn't have any way of talking to any one of the various divisions of the it team or anything else So, you know, it took us a while to get to the point where we were recovering things So by the time we finished the recovery, you know about 16 million years later, uh, actually it was close to six months The threat comes back and wax us again Yeah, that was great um I think the main thing here basically is plans communication and visibility obviously the visibility part is actually My number 10 on this list. It's the first stop Can you even detect threat actor activity? Uh, and a lot of people are going to be like, yeah, cool We've got tooling. We've got automation. We've got all this cool stuff. And that's great It's very important. I work for a company that sells automation and tooling I worked for a company prior to this that also did that. I've spent a lot of my life looking at other people's work around it and honestly Everybody's doing a good job in this space I'll obviously say that, you know, my employers are better than everyone else because I work there But I've also actually gotten to work with their stuff a lot But The main thing here for me is actually people Um, you can have all the shiny blinky lights in the world You can have all the machines that go ping it really doesn't matter if you don't have people who are capable of handling those things So, you know, there are two questions on this slide Obviously, uh, and these were ones that I would keep asking every time I change jobs every time I got a new role as an incident responder What what's the security stack look like and it's gone from Norton antivirus, which should give you an idea of how long I've been doing this To advanced endpoint detection and threat resolution blah blah blah intelligence sprinkle some blockchain on there Whatever you want Um, and and the thing with this is that That stuff's great But sometimes your solutions bail or twine and duct tape or just grab so Knowing that you can detect stuff is great Knowing what happened is even better and a lot harder and and the three things here on this slide These are kind of, uh, you know an ideal situation I've never worked with a team either in consulting or in you know in practice when I was doing it where We have all three of these lined up perfectly Usually what we've got is we've got a situation where we've got some good investigators, but The logs aren't being stored anywhere or the logs are only being preserved for a week because they're too many of them or On and on and on In a lot of these situations, you know your your outcome from your your actual incident response Is probably not going to be as good as you want it to it's not going to be one of those things where Hey, you know, this is great We resolved everything but let me talk about one where it did work for me And this was a little while ago It was another minor ransomware case and this time basically I walked in the door I think I was working alone at this point and I said, you know Where your logs and the IT person comes back and he says, oh, they're right over there All 40 gigs of them go nuts So several hours later, uh, I had started to be able to reconstruct how the threat actor got in what they did Everything else. This is you know, not not a lot of systems. We're talking about it's a fairly limited scope um, and what was kind of nice about it was We actually had a resolution within a week, you know, it wasn't an incident that just dragged on and on and on We actually knew what we could do So that's a good situation um, I've been in a lot of bad situations and usually it's because Logs are missing or they're not stored long enough or people aren't trained on their equipment I think I'm repeating myself, but I think you get the point Um, I've also worked with a lot of grep in the past. That's why this is up here This is actually me, uh, 25 years ago when I got my start in incident response So the next question we've gone with Can we see the ta? Can we investigate? This is a bigger piece for me. What can I contain? um, I love this picture and that's why I put it up here, but I There are a couple of main things about containment. Uh containment is should be your first move on an incident but in a lot of cases your first move is Once you get past this part go to containment Look at what you can contain and you should know what you can contain, but a lot of people don't because you know, um There's more than just the incident response team working in it so you're any going to end up in situations where Your containment isn't working because you're missing pieces. There's missing communication That kind of goes back up to my top slide, right? I've worked with teams that really know their stuff and they've got good processes and there's been tooling and investigation And this is also doing the consulting side of things Um, but for me the containment part is usually where people trip up. We think we've succeeded. We're not sure Can we call you and get some more information? And I've been the person asking those questions And for me, it's kind of perpetually a case of like How sure are we? Uh, how far can we see? So let's go back to slides, you know, who who's actually able to help us here? Which is another piece so I think with ransomware, it's actually really simple to talk about containment. We're basically saying, okay Stop the spread cool Great, we've done that. Uh, what about the threat actor and the c2 networks and the rats and the data? Well, you can't prove the data was exfiltrated unless you can see what's going on on the network And if you can't see what's going on on the network, let me back up a slide. No, i'm kidding So the next piece after the containment side and actually in some cases at the very beginning of the containment side is Who do I call for help? I where's your team of avalanche search docs coming from?  Across the room to the it engineering department and say, okay, can you like, you know close all this stuff up? I've got a really interesting story about that one actually. Uh I was working at a place where we had a really really large network and What basically happened one friday, of course at around five o'clock when I was on my way out to a conference Was the entire network went offline not just Not just the internal one, but our connection out and literally everything else. We basically DDoSed herself from the inside of the network And network engineering lead comes over to me and says you're never going to guess what just happened And i'm like, well, I think I am because you're going to tell me and i'm going to bet it was a worm. Yeah, it was So the problem with this was we had no way of reaching the side of the Large campus that I was working on to talk to the people who had the worm to tell them to stop For three hours. So our entire network was offline for three hours because we didn't have a phone number The other piece of this and this kind of turned into something later because the reason this DDoS happened was We had a piece of technology that had really really really really really poor internal security um I don't know if any of you have ever run across a username of developer and a password of You can guess So This is where the functions that end in r come in. Um, in this case hr wasn't involved pr thankfully wasn't involved, but lawyer was involved And then there was a third one that we didn't have to call up our fourth one. Sorry external responder um Knowing when to escalate is kind of important because getting those people those r's on board Is sometimes going to require a call at three o'clock in the morning and sometimes it's going to require that they actually Get over there in time. Uh when I've worked for on-site incident response teams, which I've done a little bit We usually try and have a 24-hour readiness time if you're in the middle of a ransomware case. That's That's a lot of time, you know So be prepared to get extra help know who to call for extra help And know who to escalate and if we you know go back to my original sort of not exactly an incident incident The escalation went to me, but I was the only incident responder there I was the junior incident responder and that's all I did Um, and it was coming from an executive who probably should have been the person who actually had the power to do that But didn't really know how that worked and I think this is probably one key point to take away from this is You need sponsorship from somebody further up in the organization Even if you're at the top of the organization, you should probably have that backing So the next piece in my top 10 list is criticality This is one of my favorites because I do a lot of strategic practice work as well as incident response Uh, and I help run red team blue teams and you know do tabletop exercises with people So we spend a lot of time Doing simulations and sitting folks in a room not as many folks as are here right now but sometimes we get quite a lot and We usually will have a question of like, okay, so where are your critical systems? And my favorite thing in the world and by favorite I mean, I absolutely hate it and it's kind of terrible is the number of blank faces I see when I ask that because there's always a Oh yeah, I mean, I guess I guess the point of sale machines are critical or Maybe it's that that electronic health records that that's probably important or or maybe our sql data and then it balloons And this is the thing with what critical is Uh It's obviously get a very poor organization. You know if you're if you're in You know if you're running a store for example, you're doing sales It's going to be a very different thing from if you're a hospital or a factory that makes widgets It's also going to vary per team.  This is my favorite sort of like dig down when we're doing uh tabletop exercises The worst thing in the world in an incident is to not know who owns the system you're working with I actually had a case years ago where We uh, we had a it was another malware case. So it gives you an idea how long it was because you know Malware now is solved by everything, but it was a malware on a windows xp machine Malware on a windows xp machine and a part of one of the buildings I worked in that no one knew even existed until the malware pinged Our antivirus and went hi. I'm here Calling out to russia Come get me And we're like, okay. Where is this? Where is the system? Turns out it was in the basement and it was attached to our hVAC system And when we got down there We start tearing it apart. We're like, okay. First of all, no one knew this was here. Who's is it? You know, can you tell me what it does? It took us two days to figure out it was connected to the hVAC system That was another one that kind of turned into an emergency for us because it was around the time of stux net And there were questions being asked and thankfully again, it didn't blow up too big we didn't have to do the whole you know panic arms waving thing, but We did have to spend a lot of time doing a very long and deep investigation after that just to make sure everything else was okay because We also discovered that the hVAC system was controlled by a whole bunch of plcs and workstations and servers That we hadn't seen yet and weren't on our radar Is this critical? I don't know is an hVAC system critical for a building. I think so Um, I ran a simulation a little while ago with one of our hospital clients that I kind of love because we actually set it up So that we just turned off refrigeration for them Turn off refrigeration in the middle of a covid vaccine campaign You have a huge problem on your hands. So again going back to knowing what's critical and who owns it That's going to extend outside the it team sometimes The next piece is how do we customer and I I like saying it this way because it's also how do I even? Or how do I cope? um This isn't a self-help seminar So i'm not going to get into the individual coping skills that you have to have as an incident responder Which i'm sure all of you have probably struggled with if you're an incident responder And probably if you do security at all you've had to struggle with this This part's about the organization You can see what's on the slide obviously But let me talk about something that is in the picture on the slide. This is a picture of woodstock This is the nice picture of what was up the field from what was down the field at the end of woodstock And all the way through woodstock They had to hire helicopters get musicians in and out because they were so narrowly focused on the idea Let's do this show, but they sort of forgot to install porta potties and people got sick And they couldn't get ambulances in Woodstock was actually worse than the fire festival in some senses just in terms of the emergency preparedness and kind of dealing with All of this stuff so I you're probably asking yourself caspian. What's your point? Which is a really funny thing to ask yourself Because your name is not caspian, but my point is really simple Be prepared for denials of service be prepared for things shutting down be prepared without ban communications Going back to again my first non-incident story and actually the second one as well that out of ban communication piece was huge If we had had it it would have been very simple if we thought all our networks were compromised Cool. Let's switch to signal and talk or something else Right now though, you know We're kind of having a hard time here because we can't talk. We can't speak. We can't see we don't know what's going on so This brings me to the next piece. We also don't know who does what I kind of talked about this in the criticality piece Not knowing who owns what's going to slow you down I mentioned that with the story about the windows XP box in the basement Not knowing who does what's going to hurt you even more if I go in in an incident and turn off a critical system Or I just turn off a system or I trip over a power supply that knocks out the billing system for somebody We need to know Who's going to call the people at the end of the billing system? We need to know who's going to turn the power supply back on I need to know all this other stuff Incidents don't occur our current a vacuum so as much as an incident responder my focus is going to be put out the fire You know stop the bleeding fix all the stuff There's all this other stuff going on where we're going to need to actually Call these people in and get them to work. So, you know the 4r as I mentioned before Your your lawyers your hr your pr those folks I a lot of good Prepared organizations I work with and I'm saying good and prepared sort of separately because there are a lot of Good organizations that are unprepared The ones that are prepared usually actually have a sense of who does what in this case, you know We do check who the pseudo reports go to That's actually kind of important knowing about that Putting that down somewhere in a way that it's accessible for your responders for me if I'm your responder It's going to make a huge difference Because if I don't know then who do I talk to? So the next piece Speaking of talking If I can actually get it to advance There we go Next piece is connections you'd think I would have put this at the top, but I'm kind of going in reverse order I don't know if anyone's kind of noticed that Um connections are really important for a couple of reasons The first piece on this slide is actually the one that you probably should be thinking of before the incident Have you scanned externally? Do you know how everyone's getting in? Do you have a way of getting in to that data center or anywhere else remotely? And if you don't Are you prepared to drive for two hours to Laval? Or Toronto? That's going to take longer than two hours if you're driving from here Access is a really big deal Loss of access especially when everything goes out also a really big deal Um, I don't have a good story about this because I can't talk about some of the stuff that I've seen except to Give a vague indication that that two hour drive was something that one of my clients did experience a few times And they had to basically say Yeah, we need to figure out a way to get in Well, and I'm I'm merging a whole bunch of people in this case So they build this bastion host and it's really nice, but it's getting scanned constantly And they haven't put any kind of anything on it to prevent it from actually being used for remote desktop access One of my favorite threat threat actors really loves using remote desktop to just sling stuff all over the place Guess what happened It was a lot of fun to solve that one Uh, we had this really cool new tool that had just come out that we could actually map the remote desktop sessions out from So we could see exactly what they were doing I sort of chuckled when we saw this person just basically download metasploit from rapid seven's site and start using that Uh, this piece, I'm sure you've all had to deal with at some point. Um, I have extensively This is actually where and this is one of my favorite parts as well. This this is criticality They go hand in hand. They're like They're like drinking too much in a hangover, you know, they're they're just going to be there all the time So First of all Please please save me next time and test your backups I've been in so many situations myself and with other people where Those backups weren't properly tested or the restore process wasn't properly tested. That's an even bigger one I mean if you're restoring over a t1 line and you've got terabytes of data That's not going to work out. Well if that a s400 That you don't have a backup for that drives all of your production systems goes down You may want to have a backup system for it.  We're you know, we may find you to cryptor. We may find a key. That's cool But man, if you can restore from backup quickly enough, you're going to be fine The rest of it's going to be an investigation So this brings me to number 10 On my list I've been hinting at it all day Do you have a plan? Uh, I work with lots of people who don't they didn't write it down their plans A nice iso document that has a phone number in it. Um, and that's it So there are good plans and bad plans the plan you wrote for the auditors is a bad plan The good plan is not the end of my slide deck It's all these little boxes here and let me let me let me walk you through this just in case you got tired of being rick rolled Um The never gonna is your actual master incident response plan That's the thing that has all the phone numbers that you have tearaways from you give them to people and say, okay You go it engineering guy help me out here The next piece playbooks Playbooks are kind of useful when you're doing this all the time And you're kind of panicking your way through something because you've actually got something written down that you can go back to And say, okay, we've done this before Let's see how it works or we've planned this out and tested it Did I mention testing testing is really important? Um, and then the last piece the awesome piece is the fact that those playbooks and the plan and everything else goes Right back into a continuous improvement process that you can use To build better it and build better incident responders So that's it. Um, I'm just going to do this one more time. I've never rick rolled a room full of people before Thank you all very much I hope you hope you enjoyed that and obviously questions will be later. So I'll be back for that All right, it's time for the next talk Which is about javascript javascript Obfuscation classification with machine learning Those are scary words by Yuri arbitman As a data scientist in imperva Yuri develops machine learning solutions for various cyber security projects He's fascinated by the wonders that data science and machine learnings bring to the world and the wealth of open source frameworks Enabling us to build systems today at a scale and he's unthinkable just several years ago In the last 20 plus years Yuri has been working in the high tech industry in israel Including for several great companies in engineering management and research positions Yuri holds a master's degree in computer science from the weaselman institute in israel Round of applause for Yuri Thanks. Thanks, Amelia. Hi everyone. I'm excited to see all of you face to face And those watching the live stream. Thanks for joining My name is Yuri. I'm a data scientist in a threat research group in imperva Imperva is a cyber security company a leading cyber security company That's been around for about 20 years focusing on data and application security We provide our customers the wealth of solutions like well application firewall DDoS protection client side protection account takeover prevention and more Today, I'm going to tell you about one of our recent projects that combines research in security and machine learning And this project is about classifying pieces of code into obfuscated and clear text ones So let's start So the motivation is clearly to help our customers to prevent attacks And one of ways to do so is to concentrate on a client side perspective so A typical website has many many resources Most of them are totally harmless But there could be several malicious ones if attackers managed to inject them into the website And the question is how how we find them and how we distinguish between those So since a website has many many resources This is something that clearly cannot be done done manually And more over even semi semi automatically is not good enough We need a totally automatic automated process to do that So let's take a simple example of of of resources and and and see how how it looks So here We have a piece of code in javascript Which is actually a keylogger So you can see i'm not sure if you can see my laser But anyway here above you can see the command and control endpoint, which is example.com You can see that we are listening to key down events And We're sending the data to the command control Back every every 10 seconds And now let's take a look at another piece of code So it looks really similar. You can see that the command control endpoint here is missing And you can see that we listen into the same event And we just log in it Locally once every every hour So I Just highlighted the difference here for you And as you can see For this very very simple example almost trivial example the differences are very subtle So it requires really TDs work To to even spot them and attention to the tiniest of details As I said in imperva we are interested in this problem and and an example is our client side protection product Which enables our customers to look on the resources on the website And and see which which ones are potentially harmful and we provide Several scores and several data for those resources so the customers can really understand what's going on And and as you saw in this really trivial example This this problem is really really difficult for humans And the question is whether it is solvable at all for machines and and by machines. I mean of course in machine learning sense So Just one more word about this so before this project The way we tackled it was to use the best data we have in a company and to calculate reputation For ips and domains and combine all this data in rule-based method In rule-based techniques and and that's how we we provided this this course So now back to our story when we see You know a difficult problem What do we do? Well, we search for an easier one and and this is of course for dramatization purposes only for this presentation So an easier problem with which i'll just explain in a second why it is at all related is obfuscation so for those of you who never heard of it before obfuscation is Is an interesting thing The theoretical foundations of it go deep in computer science and computational complexity But if you are down to earth a bit is just a family of algorithms of techniques that make transformation on your code They take your code and then transform it so it preserves the functionality and hopefully the right time But it really hides the inner structure in a way that people find it very difficult to understand And it makes it almost unintelligible To to humans to to really understand what's going on. So let's let's take an example So this is like the most trivial piece of of code in javascript. You could think of it's just You know printing hello world to the console log And when I take this piece of code and obfuscated using one of the readily available tools in the internet We end up with with all this and actually even All this too So you may say well, this is ridiculous. This is probably some contrived example But but but you are wrong And and the reason is that I I just used a tool with the lowest obfuscation possible In this specific tool, which is obfuscator IO and if I would use This tool with the highest level of obfuscation We would end up with many many pages that I would have to show you here So this is this is a real world example and this is really how obfuscation work Just for for this trivial piece of code imagine what happens when your code really does something interesting So And of course the question is how is it all this is related to maliciousness because remember we started with maliciousness So the answer is that it depends on the language so The the thing with With obfuscation on our client side for example in javascript Is that sometimes it can be used for legitimate purposes for example for preserving intellectual property If you want to to hide your code because you invested a lot of time developing it and there is nothing malicious there And there are a few more a few additional reasons but On the client on the server side These two two problems are really closely related because usually you don't have to protect your code from hackers On the server side if you don't you know move it anywhere and do not give it to anyone And still if we look on these two domains together client side and server side Still obfuscation is a very very interesting signal that usually helps us to determine maliciousness of the of this piece of code Okay, so So what we want to do from this point on is to classify every javascript document into clear text or obfuscated Um, and the question is is it easy for humans? So let's look on another example Um, I hope you can see I hope it's not too small. I tried to make it as large as possible Take a few moments to try to understand what what it does if you if you will And then of course we have another piece Um Look pretty similar aren't they? Okay, so without, you know, torturing you too much This is a a part of Of a malicious script that I took Um, and and this is indeed obfuscated And and this part the first one Is is actually a clear text. This is a part of a script from from youtube that is Used to speed up The if you embed a piece of youtube in your on your website So this is the code that just just a small part of this code that uses to speed it up and and this is probably Calculate some hash function or things like that So, um, as you saw it's not really, uh, it's not really easy to to distinguish between the two cases for humans And actually it turns out that it might take a lot of time for seasoned security researcher and the reason is That in order to Decide if the code is obfuscated or clear text what what this researcher does usually is Really trying to understand what the code does. It's not just trying to see I mean It's not written anywhere, you know, it's obfuscated or not We need to understand what the code really does and sometimes it takes time So, uh, it's clearly something that is unscalable in in any in any way um So the question is uh, is it Easy for machines and of course we didn't jump into the machine learning solution right away We we try the working with heuristics and it turns out that heuristics are not good enough simply because They are too specific and we will talk about the various obfuscators in a few moments and you might then understand why um So The mission from this point on is to build a machine learning classifier for this problem So i'll tell you about several approaches from the literature we could use Um, so the first approach is a classical one There is a paper by uh telenbach and um and several additional researchers from 2016 Where they propose to do a classical feature engineering meaning to extract Things like average length of line frequency of specific words specific characters and then Build a decision tree. So I guess you've heard about these kind of trees And here is a simple example So This this tree is checking whether the average length of line is larger than some threshold And if it is it then checks the frequency of certain character And if and if it exceeds the threshold, for example, it says that the script is obfuscated Of course, this is really simple example, but just to get you understand the the principle um another approach was suggested by um skolka and his friends in 2019 they suggested to build a deep learning classifier based on a convolutional neural networks and abstract syntax trees so convolutional neural network just in a sentence is Is a method used a lot in vision But today it also used in additional fields in in deep learning and the abstract syntax trees is you can Easily download a tool that takes your script the java script in this in our case And the component builds out of it the syntax tree for For the script where you can see the structure and you can extract many interesting features out of this tree And then these features are fed into the network of these guys And the last possible approach is natural languages processing So in the recent years we we see a really tremendous advances in this in this field. We we see Amazing models that understand human language and enable us to to solve a multitude of tasks for example question answering and and Text classification and many more so So in our case what we can do with it. We can take a model burt is just an example To use the weights That the model was trained on and then we train it on downstream tasks may and namely our task We just feed it documents Which are java scripts classified into clear text or obfuscated ahead of time And then the model learns it and adjust the weights That it had learned previously Um So our first approach Was uh inspired by the by the first work. We started simple We wanted to make to build a simple model and we want to benefit from decision trees that enable us to Gain explainability it can uh when we use a decision tree It's very easy to see really which features are affecting the model and how Um, so we took something about 40 features And we trained on a single obfuscator And it turns out that this model didn't generalize um, so What went wrong? In order to understand that we have to dive a bit deeper into the various obfuscators for javascript So here I prepared a list of the most used obfuscators for javascript Sorted by the popularity metric of github, which is the number of forks as you can see Uglify js and obfuscator i are the most used ones and there are additional ones Here um, so What methods do these obfuscators employ? Let's see a few examples So the first thing people immediately think of when they think of obfuscation Is renaming of variables and functions and indeed if you think of your code if you rename it a bit sometimes it makes it clearly unintelligible Uh, even before it might be not really Easy to understand so in this simple example, you can see we just rename a few things and it's already looks a bit more intimidating Additional technique is modification of functions Uh function calls the function arguments and the return values So in this simple example, I just took a function that just squares the number And I show you a snippet of the code. It's not the whole code that was uh result That was produced up after obfuscating it But you can clearly see that The function gets many parameters and returns many things and it's really complicated things a lot Additional key example is a modification of strings By using encoding encryption and string generators So in this example, we have a simple variable and uh, if we want to to find the strings that were used inside of it Like it's you know, the honda accord car You can see that the strings are splitted and then several Arithmetic operations are used and if you look on the resulting code without having seen the Clear text one ahead of time. It's really takes time to understand what's what happens there And and and the last example in this context is the manipulation of constants So if we have a simple constant, we might end up with some expression Which we need to calculate to to understand what the constant is and there are additional methods like changing the base of integers And injecting dead or redundant code, which of course, uh, you know, it's clear what it does complicated things a lot So what are the differences between those those obfuscators and in general between the obfuscators So the differences are as you might imagine specifically naming methods The the the encoding encryption functions that that I used and some more so more over Uh Most of these obfuscators can receive a lot of parameters to tweak them and the resulting The resulting Docus the resulting code Looks much looks very different if you run a single obfuscator in mode one versus mode two. So Just an example for for javascript obfuscator. It has 40 parameters and the aglify js has 30 parameters. So Uh, as you might imagine When we compare between different obfuscators that clearly the the output distribution looks different and in in If we think about distributions so We get different Everything that you that you like average length of line length award Function size a proportion of encoded characters And in general this is just an example if we want to Measure the normalized backslash count Which is one of the features that was proposed in the paper that I mentioned before Or average length of line you can see that the differences between the obfuscators can be really big And then this is log scale by the way So before uh telling you about the approach that really worked for us I just say a few words about the data we train the model on Uh, so we use the public data set Of about 150 000 javascripts all of them are clear text And we applied four different obfuscators of them and and four because we wanted to uh test the model on additional three That the that the model has previously not seen and we um We got perfectly balanced data set of about 100 000 clear tech javascript and 100 000 obfuscated javascript Equally divided between the Between the obfuscators so um our approach combined Approaches number one and three that I mentioned And and and the idea is as follows we tokenize the input into words And we calculate the most common words in the clear text javascript So we take all the clear text javascripts We extract the common words out of it and look on several hundreds of them And then for every input we measure the difference between this input and this uh, and this calculated distribution So let me give you a small example So assume that the top three words in javascript are function document and input and assume that Like function appears about once in every 20 words Document once in every like 33 words and input like once in every 100 words So if we have following javascripts and and if it's too small, I'm sorry, but you don't have to see the exact details I will just read them out for you um So here we have 71 words And uh, if we look on the word function, it appears twice Meaning it appears in three percent of the cases the word document appears once and the word input appears And never appears So if we calculate the difference between the clearest occurrence We calculate it as I showed you in the previous slide and the actual occurrence. We can see the differences in red So, um What do we do with these differences? So we we feed them into a boosted decision tree and this is very similar to the example that I gave you before Uh, and then we just look on the differences between Uh, the specific word how how many times it appears and then we can know whether the Text is obfuscated or clear text. So I hope this this this is clear This is just an example of decision tree with the different features. Every feature corresponds to a top word We extracted from the clear text javascripts So it turned out that the performance of our model is is pretty good We were mainly interested in this case in false negatives and false positives You can see that both of them are less than one percent About that this was the Product requirement and specifically, uh, we wanted So what's false negative in this case false negative means that we are classifying some document as Clear text when in reality it is obfuscated and false positive means we calculate we classify document as obfuscated We're in reality. It's clear text. So, uh, we were interested in in a case where false negative rate Is smaller than the false positive rate and the meaning is Since remember, uh, as you as you recall as I said, it's related to maliciousness. So we do not want to flag Cases where uh, people would look and say oh in reality it's not obfuscated So we don't want to um, you know to increase the number of these cases Um And the next question we asked whether Our approach is generalizable to additional languages and it turns out that that the answer is yes So we looked on python and php. For example, as I mentioned before they are closer to So the case for these languages since our server side is that obfuscation and maliciousness are closely related to one another and um And this is why in this in this case is it's even more interesting And and we got really good results. Of course, we trained these models on on the specific data set that were chosen for these languages And so let me show you just a very very short demo. This is the qr code you can scan So we set up a website That you can use it's publicly available Uh with some version of our model And you can just play with it and see And see what's uh, what's what's what's going on? So if we take This script as example, you can see it's uh, non obfuscated Okay, so we can just take it We can feed it into our model. This is by the way A live demo so You can see that the model says that this is a clear text with probably probability 0.95 Meaning the model is pretty certain And here you can see a sharp values Uh, I don't have time to explain what what they really are They are just the significance of features that affected the decision So in this case you can see uh an example of one gram model Which is a little bit different of what I described before it's one of of our models that we Built during the development of this project and If we take another example just just a quick one So I took I took A piece of code And obfuscated it so you can see that I'm not sure you can see unfortunately the window is a bit small here, but You have to trust me on that So if we feed the code here Then um, it says that it's obfuscated. Of course the probability is rounded And you can also see the various features and how they affected the decision. So I I cannot promise that this This website will will remain in this in this exact form But if people are, uh, you know interested and and will use it we might enhance it and we might Continue maintaining it So What are uh, what what have we learned in this in in this? project So the first thing is uh about the relationship between maliciousness and obfuscation in different languages As I mentioned, it's not a one-to-one relation But usually obfuscation is a strong signal for maliciousness and It's really cool because I think that solving the obfuscation Classification problem is much easier than solving the the maliciousness one in terms of machine learning, of course So we saw that Classifying code we saw many problems that are difficult for humans Uh, but even the even the problem which which mine which might seem simple of classifying the code into obfuscated or clear text is not is not so easy and And as we saw a building a machine learning model that solves it is relatively not very sophisticated and not too hard And another thing is about the obfuscator. So we had Several choices here building a machine learning model per obfuscator is not scalable And looking on the internals of each obfuscator is not scalable. So we use this You can call it a trick or approach of looking on the clear text Extracting the top words out of them, which is inspired by natural languages processing Uh approach and um and and this this is really enabled us to solve this problem And another nice thing is that this framework seems general So we didn't test it on all languages, but I think python php and javascript are like the more the most interesting Interesting one in terms of obfuscation And there is no reason to believe that this wouldn't work for additional languages Of course if you train The model on a large enough data set that is properly balanced So thank you so much. I uh, I'm looking forward to your questions. Hope you enjoy the talk So it's time for the last talk of the day by Maya I forget how to do I just go to pronounce your name kacherosky And erichang So maya is a product manager at tail scale providing secure networking for the long tail She was previously at github in software supply chain security And before that at google working on the container security and encryption key management Prior to google maya worked at mckinsey and studied mathematics at megal university Eric is an engineer at google's enterprise infrastructure protection team where he builds system to scale internal security processes He has previously worked on linux fleet security at google upstream communities cubanities and cloud authentication systems Thank you. Is your mic on? My mic is not here me. We can great. We're here to talk about the road to beyondcorp and how to get to beyondcorp So as we just introduced i'm maya. I am a product manager at tail scale Tail scale is a wire guard based mesh vpn. And yes, I do have stickers lots of stickers. Lots of stickers. Yeah And i'm i'm erich i'm a security engineer on the enterprise infrastructure protection team at google We deal with google's corporate security, which Relatively relevant to what we're about to talk about today. Yeah, exactly So what are we going to talk about today? First we'll cover what we mean when we say both the words beyondcorp and zero trust and why those are important and particularly poignant right now Then we'll dig into what beyondcorp is to better understand Its components and how and what what is needed for that to function We'll talk about what companies have typically done or tried to try to do to adopt controls like beyondcorp And what steps you can take towards a zero trust architecture And we'll wrap up with going over the long tail of problems We're going to hit by trying to make something truly zero trust. So to jump in Zero trust zero trust is a very trendy phrase these days But in addition to coming from you know, lots of vendors and annals firms and yes conference talks It really does seem to like the concept of zero trust has become particularly trendy and mainstream like very recently And that's probably due to what just happened south of the border In late january 2022 the u.s. Government published a memorandum on zero trust principles And about moving the u.s. Government's institutions and networks towards zero trust And this was kind of surprising like although maybe not that surprising because after this was after in may 2021 The u.s. Government also published the executive order 14 028 for improving the nation's cyber security and this is a follow-on to that So as a result of this memorandum By now u.s. Federal agencies have adopted their zero trust plans Into their cyber plans and they have to Implement what they've decided they're going to do by the end of 2024 So I guess I guess this is just how we do security trends now So how are zero trust and beyond corp related? Let's start with traditional networks in a traditional network You're in the traditional network architecture you you rely on network perimeter to delineate between trusted and untrusted users such as trusted employees inside of firewall versus potentially Untrusted employees outside of or untrusted actors outside of it By moving to a zero trust architecture the location of an individual specifically what network they're on is no longer solely What determines whether that individual is trusted? But other context is use to determine whether or not they can access a given application So with the zero trust architecture, there's no longer such a thing as a privileged physical corporate network One thing that zero trust doesn't mean though and people get confused about this is that vpns are bad And I have to say that despite you know the fact that I that I work at a vpn company There's nothing inherently wrong with vpn But there is something wrong with assuming that because someone's on the vpn they're trusted or you know that they're on the corporate network They're trusted So if it's more like you know vpns that have no application level, you know access controls are bad sure Beyond corp Introduced in a paper in 2014 is google's specific implementation From which the broader zero trust principles emerge and zero trust architecture emerges yeah And so beyond corp is fundamentally an access question This could be an access to something like your internal corporate wiki And the way it determines if you the user making that request have access Is it considers two primary principles one is pretty classic right? It's who you are You know, what are your credentials? Can you can you get into this wiki and then more novelly it also considers what? Devices you're coming from so a really good example might be a personal versus a corporate device right? You might be able to have your credentials on your personal device But that doesn't mean we necessarily still want you to be able to access the corporate wiki from that device Um a really important thing that's just not even in this slide is any discussion of networking You know you're talking about vpns a little bit before and that's kind of dominated the conversation And I think that networking is interesting and your network topology kind of falls out of trying to get these core things But ultimately what we're done by talking about today is access users and devices and trying to harden those as much as possible So first, you know, I said this before users were all kind of familiar with that right you are the user You've done an oauth dance. You have some credentials and and so on and so forth So it's pretty much what you'd expect out of any normal application level access question A few things we will be talking about today is how to harden that so you know How do we actually get strong user first identity? And then what are the properties of a strong credential because this is fundamental to the access question Having a strong credential results in stronger access or access decisions Um, and then the next one is we're going to talk about devices. So Um, we've discussed stuff like you know, is this a personal but it's a corporate device But once you start thinking in that direction, there's a whole kind of different set of questions that you might be interested in asking Right, what is the patch level of that device is a really good example And then how do you store this information? What kind of inventory do you have and then once you have all of this How do we again in the same way that we're going to think about users credentials? How do we think about device credentials? How do we harden the identity of those device credentials and the storage of them and so on and so forth And then finally access is the the terrible nitty gritty of given These two pairs of credentials. How do we actually make an access decision? This looks totally different depending on what protocol you're actually using So as an example if you're going to go visit the corporate wiki That's going to look like a very different access decision from a technical perspective of if you're going to ssh to a prodbox But we actually really do still care about that combination no matter what protocol you're using and then again Because we now magically Have all of this device information. What kind of device information can be used to make an access decision? So if you're trying to get to beyondcorp or a zero trust architecture How can you actually go about doing it and how do you know how far along you are? So eric and i put together a roadmap or a maturity model where we want to call it For how you get to zero trust as you go higher from left to right You your organization has more capabilities in terms of how it secures its users devices and access applications on the network Basically the more that you harden the more you're doing zero trust So your first step on the road to beyondcorp is level one is having an inventory of your users and your devices At level two you have a way to measure most of your security controls and enforce some of them You're able to segment access to specific applications At level three we have what's typically called zero trust in the market today It's tiering users and devices based on measurements and enforcing access based on these measurements So like eric said a device that's patched might be treated differently from a device that's not patched And lastly level four is what people think they're being sold It's being able to have dynamically changing risk based access to applications This is incredibly hard and something that's not really available in the market today because there are a lot of edge cases It's what we should be aspiring to as an industry, but we're all still working on the details So we're going to go through each of these level by level So level one as i mentioned it's about having an inventory of your users and devices To inventory users who are accessing applications in your environment The easiest thing to do is to use a single sign-on identity provider sso It's also a great simple security improvement for your organization This is typically tied to your hr information system So when a new employee joins changes teams or leaves then their identity and so their access to business applications can easily be updated Even if you have an sso you might not be using it consistently Unfortunately, a lot of sass tools charge you extra for the ability to use sso with their application known as the sso tax Or sometimes charge you extra just for the ability to enforce sso for their application And so for applications where you're not using sso you should ask employees to say use a password manager But that's also not something that you can enforce At this level you probably also treat employees and Contractors identities the same way although you might not give contractors access to all of the same applications In terms of devices at level one you have a device inventory and some way to manage devices This can actually be pretty complicated depending on if you have a you know a mix of both company and employee owned or b yod devices You probably have a wide mix of operating systems. Especially if you support b yod devices. I think one of my colleagues runs his own os You might also support mobile devices. So that makes your makes your life a little bit harder So although your corporate policy might be to give employees, you know company owned MacBooks If if you let them be yod their own mobile devices, you've already added a ton of complexity to what you need to manage In terms of keeping track of company owned devices You ideally have a device inventory of who you bought what device for Though this is not necessarily more complex than a spreadsheet And before you let a user connect to a network Then you might have some manual verification that that's actually the expected device And then at level one in terms of access It's that flat sort of traditional network where if you're on the network you're trusted like a traditional vpn You know and ideally can control what users and devices are on the network given that you have these inventories Even if it's manual, but you're not segmenting users or devices any further If you're only dealing with corporate devices and only self-hosted applications Then this like having a vpn plus an sso is a pretty good initial set of tools to limit access to your internal applications And now we're going to start talking about not just knowing and enumerating but actually Measuring and starting to ask questions from I don't want to say our users, but you know our devices at the very least So this is the slide where I just tell you to use security keys There was a conversation this morning Hand wave, uh, this is a conversation this morning about passwords and all of the You know terribleness and I think the password list phrase got thrown out or something like that Security keys are effectively unfishable from a browser context Yeah, you can go in the technical details of like everything but in terms of producing a good Security control that ensures that the initial authentication of your users is strong There is nothing that compares to just using web off end based solutions. So please I think that's the whole slide Right just use security's oh wait. No. Oh, yeah, and so then we also have things like user properties, right? So um, this is more classic, right you want some sort of Grouping system like active directory you want a way to say The properties of a particular user are they you know an engineer? Are they an sre so on and so forth because we're going to want to use that access information or for access decisions later Um, and then a level two for devices We're going to start thinking about management of devices now You can initially start talking about things like mdm's mdm solutions for mobile devices But it doesn't actually need to be that complicated I know plenty of people and plenty of organizations where it's like Yeah, we just have a script that we run and it checks in occasionally or maybe they use something like puppet The ability to just say Ask your device what its patch level is or say later on you might want to force it to update Having those basic capabilities Is fundamental because if you don't you just can't advance past this particular point and then yeah, uh, there's someone else on our panel today who is going to I think of a os query Sort of training os query is a great example of a tool that is open source that you can use That you can install an agent on your devices and then you can start asking questions of your devices You also want to start thinking about less of like here's this key that I You know gave you that maybe a shared between a bunch of things that lets you on the vpn and start thinking about per device credentials That you can actually associate with the device This is also where the spreadsheet isn't going to cut it right like you do need at this point To not think of your inventory as something you only update when you on board or off board a device But as something that is continuously living because you're going to be profiling and gathering information from all of your devices And then uh level two is of access management Is now we have to start thinking about not just having one single policy for all of our applications But we might want to start doing this at a per application level and there are plenty of Open source sort of solutions that do something like this You know some nginx plugins go quite a long way. There's also things like the oauth proxy, but the Level seven http proxy is a really good way of achieving this because you can't always make the assumption that all your apps are Going to perfectly understand, you know your format for your users credentials or your format for your device credentials And then you also have to start thinking about what are your strategies for non browser traffic? So this again might be ssh or command line tools while we generally think of Beyond corp as a solution for like your browser based traffic The being able to ssh or a prod box is also something you definitely want to have a strong authorization decision for And you can't just ignore these sort of access requests because they're not you know your classic http browser traffic And then this is where we also are going to start consuming device credentials So mtls is a really good place to start here, right give your devices a client certificate You can there are a lot of solutions for like android phones or that kind of thing that I allow you to Through a enterprise enrollment install client certificates that you use for your corporation And now we're going to finally get to level three, which is sort of where we see as like zero trust today or whatever So a very very common pattern Is you have a user and they have a security key and they have a really strong initial Authentication and then they exchanged it for some Token that is valid till the end of time that just goes on disk or gets thrown at some third party service, right? So iMap is a great example, right? You have this strong initial authentication and then this device just have a credential that can read the email forever. So The the really important part of user authentication is not just to think about the initial authentication But thinking about what are all of the other credentials that a user might have right your ssh keys Your access tokens so on and so forth and thinking about Time-bounding those right if you have some initial login and you give somebody a cookie that lasts for 24 hours That's great. But what other credentials have they also been able to grab from your institution? API access keys another great example where these things just live forever What we found very useful Is to take weak credentials maybe Barre tokens are problematic because you can't really hide the secret key, right? There's no secret key you throw it at someone you you check it into github and it you know gets on the open internet We found very useful is to take weak credentials and bind them to strong ones So for example if you have a device credential you might when you issue someone an api token Put the hash of that device credential in that api token And now when you're making access decisions you can sort of force a binding between a weak credential and a strong one And speaking of strong device credentials First off we're going to talk about active management. So this is where we're on just measuring We actually want to start enforcing things. So force patching great example Force reboots are another great example where you might patch but until you reboot there's nothing that's available This is also where you can start thinking about what are the other security specific configuration on your devices, right? And do you want to force these clients to do this? Device-based firewalls another great example This is also where we can start thinking more Ambitiously about how we are provisioning credentials to devices. So basically all modern devices Right my laptop my phone so on and so forth has some sort of hardware identity that you know magic pixie bucks Magic pixie pixie dust. Yeah, it doesn't change Um, so this device has a tpm in it My phone has a strong box is an android phone. So it has a strong box as a device hardware based Identity for the device iPhones have the secure enclave. These are great because you can generally challenge them for the identity and then issue Corporate credentials based off of them And that's great because you're no longer like physically and manually going and doing this But you might actually be able to automate some of this Also, all of these hardware based identity mechanisms generally have the ability to store private keys So now you're taking your mtls certificate that just had a Private key that was on disk and you are able to back it by one of these Hardware based solutions and now you have real strong confidence About when that device comes in to my network that it is that device and the request is coming from that device And finally, uh, we're going to talk about access and there's a general term that google has published called the tiered access And the the sort of summary I want to give is this we've talked about personal versus corporate But anything beyond that we're not going to start talking about device tiers, right? A personal device is interesting in a corporate device. But what about the patch level of that corporate device? What about other security relevant? Configuration if it hasn't, you know taken up these particular things we've prescribed to it Do we want to continue to allow it access? So when we talk about tiered access, we generally talk about devices losing trust So this is a device that is a corporate device that should have access But doesn't because it hasn't done something we want of it. For example, it hasn't patched recently And critically the most important tier is the tier that your devices get placed in when they lose access Because if they just suddenly lose access And your user has no way of actually Regaining that back. That's an extremely bad user experience and you're going to get some angry messages So level four So what we're going to describe at level four is what people think they're getting right? So they think that something that implementing a zero trust architecture means that you have dynamic risk-based access to all of your applications for all of your users on all of your devices and That's not real That's not possible today There's a long tail of things that are hard to get right or that are not yet solved for like a normal company To have a zero trust architecture Specifically, there's a couple things of the long tail that are hard SaaS applications truly risk-based access device state And just random network devices. So let's dig in So this is a friend of ours Matthew Garrett who Left google to go, you know work at a startup and realized Oh my god, all of the services my customers or my clients use are now hosted web services and starts to think How do I do zero trust there and the reality is that this is really hard when you can't put a service behind the proxy And I think uh based off the second tweet Matthew's solution, which I think is a good You know if unsatisfying one is that oftentimes your sso that you know corporate sso that you pay texture to get Now becomes the access point. So when you go and log in through this and get bounced to your sso provider That's run by your corporation. It now needs to make intelligent decisions about when I issue you that, you know, sammel I forget the assertion or the oidc token You know, can it is does this device have a client certificate? Does it have is it correctly patched? And do I want to now allow this to access the third party sassap for some, you know given amount of time Yeah, and in one workaround that I've heard and I really don't like it But it's a workaround is to is to have them Pure have the sassap's peer to network so that ingress traffic to the app is only coming from allow listed IP addresses that are presumably your corporate network So an employee who's trying to access, you know workday or senate fits Whatever happens to be needs to pass their traffic through your corporate network out the other end to that to that third application Some security teams like this because they want to inspect the traffic or enforce certain things on that traffic But it is kind of shitty for the user, right? It's like much slower You have yeah, anyways, so in addition to being slow, there's a couple of security downsides that are not great This is completely a step backwards in time, right? Like you're going back to having a flat network and you're saying that anyone who's on the network can access these sass applications Which is not what we're trying to do Um, it requires the sassap to have this functionality Not every sass application lets you have IP allow listing to a set of IPs And you probably also want a way to correlate audit logs from the sass app to your audit logs To know who's actually accessing the application what they're doing And you also don't necessarily have fixed ip's right your network changes You migrate data centers or more likely you're using a cloud provider And then you're allow listing like all of google's public ip's or all of amazon's public ip's and that's not really Closed anymore to anything that's like you've allowed listed like a lot of the internet to talk to your to your internal applications So relying on ip trust isn't the best way to deal with sass apps either And I think the temptation with tiers is to have a lot of them People want to do this they want to say you know more than just you have you know privileged access plus some middle ground plus no access And this slide is sort of one of my friends who used to work in mozilla was working on some of the ways in which They were going to allow they allow Individual applications to prescribe what the what the requirements are for their own service So there's two main things that you hit one is that every service now has to be aware of like what tier level it accepts This gets really messy when you think about things that also might have different levels of tier a bug system Is a great example where some of these might be extra restricted Additionally the more you measure particularly from operating systems the more you realize that everything is broken and filled with like held together with duct tape You know the version number changed and suddenly all these people lost their access these type of things happen regularly because Ultimately osprey is cool But like the operating system vendors should be providing us that information in a way that isn't going to break because somebody tweaked, you know a string And then I put this in basically all of my slides as well This is the nsa Taking over a sister router and doing god knows what to it State is great until your device starts lying to you right. Hey, I'm patched and I'm cool and you can let me in The reality is that there are device state attestation mechanisms. We talked about stuff like tpm's Um, they work really well for closed ecosystems, right? iPhones can are pretty good at detecting if they're broken or not For when you get into more open ecosystems like microsoft where they have to support all of these random devices I've looked at tpm event logs and I have no idea like Is this the thing hacked or is it not because to know that information you'd have to work with one or two or three vendors just to get the right hashes as an example And the last challenge we're going to talk about is random network devices Some devices like thinking of you printers are just a pain to deal with and manage on your network the same way I'm not talking again. I'm still not talking about production networks still talking about the corporate network It has a lot of devices Printers don't have secure boot printers, you know can't be managed by mdms The easiest way for you to set up your printer is with a fixed ip address and then scan your land to find your printer And then use it But I can't you know, it's still a device on the network and I still want to be able to verify and manage access to it in the same way But I can't this comes up More in brownfield than in greenfield deployments because you know, maybe if you're at a startup you can just If you're trying to adopt a zero trust model, you can just not have printers But existing legacy networks will have a harder time With dealing with these random other devices So then kind of waving a magic wand and looking at you know ignoring those challenges We just talked about if you could if you could have level 4, what would that be? It would there would be a decent way to manage users and user access to sass applications and correlate logs for when users access those applications Probably there's only two okay partial solutions today One is to host everything yourself Which is not super realistic But you do see come becoming more and more common with like larger infrastructure tools that offer on-prem solutions Or the second one and I'll add on to To Matthew's list is you can be Google or you can be Microsoft and you can use your single sign-on to access everything Fundamentally as an industry we don't have a way to deal with this and we need a better way to deal with portable credentials for third-party sass apps Ideally also including device credentials as Eric has pointed out In terms of devices You would need to include all the devices that are on your corporate network because they're all a point of entry to your network Usually the easiest way to address this is actually To move these devices off of the corporate network rather than do whatever they need to do and meet the requirements to Remain on the corporate network And as Eric explained, you know having hardware based device out of station from a root of trust that you can actually trust That's still a very hard problem And lastly with all that info making real-time decisions that aren't just rule based But that change based on what you know about the user device Quickly enough that the user doesn't notice or get frustrated and so that's it right? But as you've learned that's that's not exactly easy So to recap what we talked about today We proposed different capabilities On your roadmap to adopting a zero trust architecture that looks a little bit like google's beyond corp At level one you are treating the vpn as an access control point for the applications on your network You can enumerate users and devices You're using an sso to inventory users and you have a manual way to list devices These are table stakes it and security capabilities At level two you have per service authorization For example by using a proxy you can measure and enforce you can measure most and enforce some security controls You have a vpn for your network an mdm to track and manage your devices and your users use sso and mfa ideally security keys This is where most security focused enterprises are today At level three you're all in on what the market calls zero trust architecture And you've maybe even bought a solution that builds itself that way What you can do is enforce tiered access to applications based on device characteristics And level four is what many people will aspire to but no one has yet to fully achieve You want to be able to dynamically enforce risk-based access to applications And there's a long list of of user device and access issues such as sass apps that are very hard to get right today I think um The wrong takeaway from this talk is to look at this list and say I'm going to do a check mark and get to level x right Beyond corp is never really done. It's more the friends you make along the way right like we're friends You want to be focusing on these core pillars and it's that's not a vpn It's users devices and the way you combine those two pieces of information to determine their access and That in and of itself brings a huge wealth of security benefits Regardless of what level you're at or if you ever claim that you've accomplished beyond corp And that's it. Thanks so much for joining us and drop questions in the slider Thanks, so those were three great talks in about 10 minutes. We're gonna set up the q&a You can still ask questions on slido and uh, we have many questions. It's going to be a really interesting discussion Stay tuned so Welcome to the q&a. We have 30 minutes of Of questions and we have a lot of them So I go into too much of an introduction, but I'll still introduce a guillaume Who was doing the workshop on the fleet and os query So guillaume is the head of security at fleet device management the company behind the open source fleet management platform for managing and using os query While he prefers working in startups He's been working in security forever in organizations of all types and prefers working at a bright side of things And things that work instead of repeating 30 year old best practices that never have All right, so the first question the most upvoted is uh, I think mostly for caspian What is the impact on mental health of cumulating incident response and what can be done to better handle it That's uh, it's a great. That's a great question. Uh, so It kind of depends on what the level of incident response people are working at And how often they're dealing with it But in a lot of cases your first incident is going to be stressful If you're dealing with a lot of incidents It's going to be stressful And sometimes you're going to see things that you absolutely do not want to see hear things You don't want to hear and experiencing things You know that you won't want to be there for so I think the mental health impacts can be kind of high and it's important both for employers And people running secure teams and managers to understand that the folks that they're putting on a line to do this actually do You know experience that and to have that awareness and have that support in place when it's possible All right, thanks Any tips for capturing momentum after an incident to guess to get better practices in place so glad so like In post incident. Oh my god, are these all for me? I feel really special right now Yeah, a couple one Post incident basically take what you've learned from the incident It's going to make a huge difference if you can say, okay, you know, let's not waste this incident Let's learn something and let's build from there Um, and in terms of capturing momentum from the teams like it also kind of goes back to a mental health thing Right, you you basically got knocked down. You have to get up again. Oh wait, that wasn't yeah I was going to go on with this song, but uh It's not a great situation and a lot of times, you know, you can use the incident as a rallying point for people But the most I think the most important point is to not point fingers, you know To not turn this into a situation. We're like, yeah, so-and-so is responsible for this So they should you know, they should get fired or demoted or whatever. That's not something you should ever do in these situations Yeah, I think we this the blue teams in general they We have a reputation of pointing fingers that maybe we shouldn't be doing or maybe we don't do it, but I think people external to the the security people they they see security as Pointing fingers and it's bad for the the business or the industry as a whole Yeah, and I think so I've worked with Guillaume in the past in a couple of In a really fun role at a former employer and I've heard a lot of his stories We're kind of like losing it and getting angry but being positive and I think that's actually kind of important in these situations Like be angry at the incident not at the individual I think to add on to some of what you were saying about, you know, not blaming a particular person or etc Like google has a culture on sre's of blameless postmortems after an incident happens And I think as much as possible that that also applies to security like something bad happens Document what you should have done instead There's something that's critical that you absolutely have to go fix And it's not someone's fault that I don't know the alert system failed or whatever, right? There's nothing more There's nothing that'll get a proof of concept built faster than some terrible all hands on deck like security Something happening, right? I wouldn't point to specific things but over the past few years like you could imagine Those moments where it's like everybody stop what you're doing. We're working on this one thing Like those are often times where you end up getting to proof of concept the things that you wanted to all along and You know, suddenly it's running on every machine and you're like, okay, maybe we could do more of that And that's a much easier place when you've kind of proven that out out of Panic more than anything and then going from the the proof of concept to actually Implementing correctly is another topic, but it's really complicated as well I have a question for either Eric or Maya So you talked about the the security security keys and web and so It's a fairly new technology, but the technology is there. It works And I think the the adoption is quite slow and My question is I think I think people A lot of people even security people are not aware of the the benefits So the biggest one being it's basically un fishable. So how do we Speed up this process of getting people to know It exists. It's very good. You should you should get through that at some point I would say that I have a much easier job because I work in a corporate environment where you can sort of just say like Hey, we're going to use these like, you know We start enumerating all the places where we don't use them. We have exception processes for that But if you look at like what the phyto alliance is trying to do They're trying to get the whole world to get onto that and you've seen things like their white papers of trying to have Even like soft phyto tokens that you can move around between devices and back up to your like cloud, right? So I think that these are kind of two different topics, right? Like a corporate entity is going to have all of their weird little things that you're going to have to Know that okay, this doesn't use web off end or it wasn't used to security key To get the entire world to use security is key is something I'm dramatically under prepared to That's to speak about. Yeah, I'll add a couple of things. Um, one is that your sso Probably has a control that says something like require security keys so yeah, you just like prep your your Employees to be ready for that and send them all security keys and all that kind of stuff explained that you can put Multiple credentials on the same key like that they don't that they can have lots of keys. They all work like all that kind of stuff So you need to spend the money up front on those keys and flip that switch But that's absolutely something an enterprise can do The other thing I will say is I think some of it's going to be forced on us in a good way So github announced a couple months ago that they're going to force all developers to have 2fa not security keys But just 2fa baby steps. Uh, I think in 2024 or 2023. I got an email from them. It's time It's like really really time like I'm I'm shocked this hasn't happened before but a lot more people are going to just start using second factors and security keys and Adoption will continue And about a github, I think I think they said that uh, I think 15 percent of the users have 2fa which I thought was really low because their developers They they're closer to security than most people and they should be aware of the benefits of it, but I don't know the 15 percent but the last public number I saw was 11 percent of projects that have over 100 developers um have developers who use security keys Or sorry use 2fa because I don't think github tracks which ones which that you're from that they publish. Yeah All right question for urie. Uh, can you successfully distinguish obfuscated javascripts? uh from minified ones because I think uh Minified javascript is pretty common in In the web as a whole because it reduces payload size and there are lots of benefits. So how does your uh, Your model Do with that? Yeah, uh, it's a great question. So let me first explain a few words for those who are not sure what are the differences So unification Means that you do not intend to hide the code, but rather uh compact it So uh, usually the purpose of minification to make the code run faster And and then it just changes the names of variables mostly and like compresses the code to be on a single line So as you one of the examples I showed was indeed minified. So In our case, first of all, uh, the minification of itself We didn't see um an interesting cause to distinguish between the cases. So From our point of view minification is just clear text And and this is why We did so so I did try To create a model that do a multiclass classification like three classes minified clear text and obfuscated And I didn't see that these model performs better But but again the point of view was to distinguish ultimately between obfuscated and clear text And this is why um So so in this case it didn't work better than that But in general to sum up, uh, it's possible to um to uh, you know spot minification by machine learning very similarly to what I described Before in my uh in my presentation And if if you really want to then just you know, just use a minifier Create the data the data set as you like and just run a similar model and I guess it will work Thanks um question for I guess anybody that Work in incident response or in a blue team given an incident Have you ever considered letting the threat actor operate in the hopes of capturing more tools and tradecraft? Is there someone who has a story to share about that? Can you talk about the story? um I don't have a lot of stories to share about that that I can talk about but I That is definitely a thing people do. Yes. Uh I mean I currently work at a place that mostly handles ransomware incidents. So tools and tradecraft We have a really big intel team for that. We've got you know, we're basically that's what we do But yes, I have seen that done The game that you're playing with that is a dangerous one because you are basically saying let's let this roll out further and you're doing that hopefully with legal oversight because There are a lot of questions that are going to be asked after the fact if you basically let Any threat actor wander around in your network and start messing with things? Yeah, and it's a big uh, it's a big gamble because Uh, I guess one uh, one of the good reasons you would want to do that is if you If you suspect the threat actor has many footholds you want to identify them before Before notifying the threat actor that you're onto them. So it's uh, it's a really tricky questions And I don't think you can have a general answer it really going to depend incident on incident No, I I mean really this comes down to a visibility piece again And this is one of the things that I think a lot of blue teams struggle with is kind of being able to see As much as they can on the network and and you know There's also the controls which I know you guys were talking about because I got to see that So there are elements of this where We're only just now getting to the point where we have the technology to maybe see this as well as we want to But there's a lot of resistance to putting that in in places All right a question for Guillaume At what size of company does something like oscar query and fleet I guess Start to make sense I would say if you're um, if you're remote and you have computers that are owned by the company as opposed to byod Like 50 people 30 people 50 people it probably starts to make sense. I think as soon as You start having enough systems that You're you're wondering what's happening on them, right? So if you if you're like a dentist's office and you've got three computers and it probably doesn't make sense But you don't need to have A thousand Definitely under a hundred make sense And is there would there be an upper bound like if we have 60,000 machines is it is it too much? Should we go another solution? No, I mean the thing with uh with oscar is There's like 300 different types 300 different tables with data that you can get from Some of them can be extremely Generate a lot of data for example There's a table that will tell you about every single connection that's established If you run that on a linux server that's a load balancer and and like you're you're on the internet And you get a billion connections every like it's probably not the right tool Um, but I've seen a lot of deployments with you know 100 200 300 000 servers And it scales really well. It's more about Where do you put the data after? But it lets you be very um precise about what you're collecting. You don't have to Grab everything and then decide what you're going to do with the data later, right? You can decide Okay, we have a new use case. We need this table. We're going to collect that every 15 minutes every hour every week And then start using it. So it's not the same type of You know, let's grab everything and figure out what we want to do with it later, which can be very expensive I guess, uh, you know, whoever sells a License based on how much you can index where sells hard drives loves that model But we don't think that's the right model in most cases, especially if it's a big environment All right, thanks Maya or eric, I guess What level of logging do you think should be in place for each level of? Of zero trust. So I guess this references risk-based Access I mean I don't know exactly where but it but like I think the a lot of what we were trying to explain in the talk is that This is so complicated And like logging when you talk about logging is not one thing right like device logging endpoint, you know, um With like stuff like or you know, and you have stuff like os query, which is kind of logging but kind of not Visibly these hard. Yeah. Yeah, and um Yeah, so I don't know if I have a great answer of like what you should log. I mean Experience wise. There's also this kind of weird Interaction between like you need different kinds of logs now, right? Because you're not just gonna have All of your things on the corporate infrastructure and you can do that right where you can You know say, oh, yeah, here's all the ip's that came in and out So I would say that logging is just like another one of these wonderful challenges that you'll experience Um, and it's it's not really a good answer. I mean log everything. I think it's the answer I mean, is it uh question is there an answer to response where you you had too many logs like Yes, yeah, absolutely. And and the thing is like I will always say log everything like it is We have too many logs sometimes that takes us a little bit of time There are really good data engineering tools. They're really good databases for this stuff So if you've got the right tooling in place, it's not a problem Uh, I have been in places where you know, I keep making this joke about grep like it I've done it.  Uh, and it takes a long time, but I Too many logs operationally as a problem too many logs in terms of an investigation never is there's there's there's no such thing Is too much evidence Grip is the best tool ever I also say to like Logging like on traffic on the network Is maybe a little less interesting than like I have an application proxy and it knows exactly who is connecting and what device they're coming from and So, you know, I don't think it's a matter of like more logging or less logging It's like sometimes you just get better quality logs out of these kind of solutions too I will add on to that I feel like a lot of people have built these like proxies or load balancer proxies Whatever they happen to be and they're like wow now that all my traffic goes through here I'm gonna do all kinds of other stuff to it too and you're like that's not that wasn't the point Like you can solve that problem in other ways. It's harder, but that's not you you solved a problem in the same way because you already had someone to put the logging um I mean if what you what you really need at the end of the day is going to be some sort of audit log of like this user access This application at this time and did these things And there's lots of ways to do that. Uh, it happens to be easy with a proxy But there's lots of ways to do that For Yuri, have you found that a lot of non malicious JavaScript is obfuscated to for uh, interdictorial property reasons? so Do you do you have like numbers or something a You you said it's a valid use case, but how how widespread is it? Do you know? You mean the um the intellectual property thing Um, so I think in one of the papers the dub dub dub 19 They really did a deep analysis of of uh Of the distributions of javascripts in terms of what happened there Um For our data in imperva and as I mentioned it wasn't really feasible to uh tag tag it manually so we use some Some heuristics just just to test the model But it's not it's not something that we can you know accomplish in any way if we have just an example like Almost half half of 100,000 scripts as a test set I have no way to take security research I mean we have many But I can't really ask them to tag to tag it and to say which one are by chance, you know intellectual property, so obfuscation type Yeah, you would have to reverse them and then yeah steal their intellectual property or something like that um About machine learning in security we have a comment here It seems that everyone wants to use machine learning for stuff like id system security Is it worth it or can can it be just a waste instead of boring things like good logs? So i'm going to rephrase it. Where do you think? Machine learning fits into security As a whole as an industry as a as a part of doing business Yeah, it's it's a good question. So so Generally if you ask data scientist, what is machine learning good for something he will tell that most probably yeah, it is it's very good So so I think there are many applications for machine learning today as as you know The algorithms become more more sophisticated and more more Libraries you can just download and easily use and and you don't really need to understand the inner working of the models So but but indeed there are cases where You don't you you cannot use machine learning. Let's let's maybe talk about this negative a negative example So when when can't we use so there are some cases that we don't have enough data and there are Cases where the data is too dirty. So you need to make You know It always starts with the data when we look on the data We can see if it has to be cleaned if we have enough of it if we have a very unbalanced data set So these are maybe like the corner cases where If you just take an out-of-the-box algorithm and apply it on on on such data You may be come up with something strange So there is no you cannot really replace This pipeline of of gathering the data and and a good amount of it If if you want to solve like anomaly detection and you have 10 examples Then you know you don't have to use machine learning. So that's I think my take on this as a rule of thumb I'll add on to that because I think the we talked a little bit like risk-based access And I think people like oh, yeah, you can just decide whether or not It's one kind of accesses application and just use spring to some ml on top and like It's exactly what you're saying. They're like anomaly detection is a great example That's actually kind of bad for machine learning Right like if if machine learning is learning is good at a couple things One of the things that's really good at is sorting things into two categories a and b That works well if A and b both have lots of data If you only have 10 examples of attrition attempts to your system Like you just can't train a model on that, right? It's not possible And so something like risk-based access you don't want to put that in front of a user Who's going to like get even more upset if they can't access their email or do their job or whatever it happens to be like Sometimes you you can do some of that stuff after the fact to like look at logs But even then like you don't have enough data to train a model if there's just nothing to go on Yeah, I think this thinking comes from The fact that a lot of security people have very little experience in machine learning and they Either go to the side of ml can't do anything or ml can't solve everything and it's uh We need to talk more to data scientists about about this stuff. So So so so we can know that ml can do this but it can't do that And just trying to maintain Even getting your signals to fire in a big system like I talked about the the quality of you know stuff you get from an operating system even just ensuring you're like like I do a lot of development where you have like nice hermetic tests and you can say like You know does this run if I run it against another system like a big Corporate infrastructure, you're not really going to have those intense very option and you know signals rot pretty quickly So even just getting signals to fire. That's like before you even can take that