Thanks, everyone. Okay, so I'm here. We have a big agenda. So let's get right into it So I'll talk about the recent evolution of server-side malware. So What it used to be and now what it kind of became I will attack one particular campaign that we analyzed During the past two years, which is operation windigo, which we released a big report about We'll touch on the subject of are the malware operators using DevOps principle and we'll show some examples of why we think so and Then we'll dive into forensic and incidents incident response and you know help you deal with the threats Who am I I'm Olivier Bilodeau. I work as a malware researcher at ESET Slavak company Eastern Europe company. I'm I've been info security lecturer in Montreal at University I also did some open source development in pearl writing for Inverse the packet fans network access control program and I did in the past also Linux Unix type system in and for large Canadian ISP I work directly on the operation when they go report. I'm going to describe today and I did some reverse engineering on it We I built some honeypots for it did the written in the report did Spam analysis and written a fake spam bot in pearl for it. So I'm pretty familiar with that topic here But first so let's get everyone moving after lunch. So everyone I would like you to get up It's real get up, please It won't be long So, okay. I see people still sitting down. I come on come on up So sit down if you never touched the Linux system before That's that's what I expected, right? Sit down if you never administered a Linux system before so as a Like, you know doing maintenance on it and stuff. Okay, sit down if you never work professionally as a Linux system administrator All right Sit down if you never done incident response or forensic on a Linux system Wow, that's good great audience. I'll have fun today Sit down if you aren't a distro developer so an active distributor or working in the vendor Nice, okay So enough with the silly commands. Let's dive into the topic So before what we used to see was old-school the face man This is one of the fancy one with a sound in it But it was hackers that were motivated by visibility showing up this their skills And also finding low hanging fruits websites that were easy to compromise some of them may be harder You can find a big list of most the face men still in on the zone dash H.org website which hosts them so this was really Like an animated gift background music type of you know kind of kid looking stuff if you want There's also old-school damage So before what you saw that a form hour or stuff was Motivated by to do damage because of hatred or personal issues with the company or maybe past employees we saw that too and The arm dash RF could be hidden, you know really well carefully So it's a declination of if you want this this thing what we've seen as an evolution is that Now it's what we call crime where because it's motivated might buy money and it it makes decisions like You know, we won't spend money or time developing this this against this target because it's not a good return on investment if you want so The this means that the threat that are targeting the servers are really specific and it leveraged the servers property Which are that it's always up Good uptime. It's almost always reachable. So the threats we see for Linux aren't Targeting desktop users mostly Servers have good bandwidth. They have good IP reputation and the IP reputation is maintained by the the system administrators So if the server ever gets blacklisted someone else then the criminal will take care of Bringing back the good IP reputation, you know, this thing trying to clean it clean the server and everything And also it may contain sensitive information which is interesting interested the Which is interesting to them to steal So why people should care about crime where on Linux server? It's not it doesn't affect directly the servers now anymore since it tries to leverage this these characteristics it's You should care about it because you will eventually get blacklisted and anyone visiting your website will get the Google, you know Like this site served malware in the past. So there are several blacklists in in effect There is an anti-virus company blacklists. So where where visitors get a warning. There is the spam house XBL, which is the server cannot send email anymore Which is very popular list and this is one of the we've get the most interest of people infected through being blocked by spam house And there's a Google safe browsing API, which is an effect in Chrome and Firefox Microsoft has something similar. So it's bad for your traffic and it's bad for reputation also So let's look at a Recent history of the the pieces of crime where we were able to find on the on the web the basic stuff old-school stuff used to be HT access redirection, so people were doing mod rewrite rules inside an HT access file to do to Send traffic to an external site, which would host JavaScript and or you know show ads and stuff like that The infectious vector is usually a vulnerable Vulnerable web application, you know with easy remote command execution and or credential stealing and brute force and And even if you had only user level access you can still drop the HT access file So this would happen an example of that is something as simple as that this what it does is if The the client the Person browsing on your website is coming from Google ask yahoo badoo YouTube anything like that You will get redirected otherwise you are not redirected Which is kind of interesting because most system administrators when they test they will type in directly the Website and will not be redirected through Google, which is a kind of a easy step for them to avoid the being detected Then what we saw and one of my colleagues Sebastian duquette blogged about this campaign Which we call the home campaign is dark leash So it's an Apache module that was redirecting web traffic also The advantage of using a module is that it could avoid doing the logging Why is F is if you use the HT access capability you are You are in the logs you are traces so you could know when you got infected and stuff like that But this requires privileges to be able to install this the Apache module of course and it was usually compiled directly on the servers It was we found that it was sold on Russian forums for $1,000 so it's kind of a malware as a service If you want and so since it was sold it it was used by different people for the different purposes as soon as We blogged about it. It's when they started selling it. So instead of I guess they were afraid of getting caught So they said oh, let's have it everywhere on the internet. So they will have a hard time tracing back to the original authors This is the ad that was seen in in forums online. So it has like features and stuff So this is how you know malware is sold right now and then Yeah, there is some more fancy malware which is Phalanx and phalanx to such a type. It's a rootkit. So it's inside the kernel Very effective and clever It hooks syscalls by injecting code inside the kernel But since the kernel moves quickly and this is not, you know API or you know, ABI compatible It it will break Sometimes and I think this is why we've seen people going away from the rootkit model instead of using now the the things I will Explain later on so now they're more on the user land stack because it's it's a lot easier to have your your malware work against Several distributions several version of Linux and not be Binding binding inside the kernel like this these guys were doing so one interesting thing of this rootkit Also is that it did all its job before it finishes loading and then it returned an error So the module would in effect never be loaded if you do use ls mod or stuff like that Because it was it would fail to load but it would have still hooked all the syscalls it needed to hook So it kind of successfully Infected the machine without being still loaded So now I'll describe operation when they go So what is specific about this threat is that all of the infrastructure? So the server side things that they are doing is all Operated on compromised servers and there are many layers of servers used we will see a diagram later on about it But the the angle of the all this all this thing is still sending doing traffic redirection and sending spam So it's it's pretty you know Not not you know hardcore erasing servers and you know DDoS and and botnet thing It's really about making money. This is why we dub this again crimeware So it's several pieces of malware that I will describe shortly But first who worked on this operation. So they are these are the people who worked together with us You have a cert bond, which is the German cert government cert. There's the CERN We know these guys from the hydrant collider, but they also have a you know good IT people there and so They they did help us greatly inside during this analysis and there is a SNCC Which is the Swedish national infrastructure for computing which helped us also during this investigation and We also collaborated to have access to servers with law enforcement and there is an ongoing case right now with them So Linux eberi is the the basic if you want of all this operation, which makes everything else possible It's an open SSH backdoor So you are they are really at an interesting place where they can get easily a shell and when they can interact with the Network also with the tunnels It it's avoiding lugging when it's doing its malicious payload So it leaves no trace and it steals all passwords and keys So everything that is connecting from an infected server will get stolen Including encrypted and decrypted version of the keys and anything connecting to the infected server will also be stolen And no matter if it's a successful password or not it will steal it and steal the state of it So did it work or didn't it work? The name was given by Steen R. Gunderson in November and 2011 We used to call it SSH door before but we liked the eberi name and a lot of people were already using it So when we found out we switched to it The way it did it works in the past it used to replace the binary directly so SSH SSH D and SSH add for you hooking into SSH agent We're all replaced, but then it decided It was easy to do an MD5 ash of the SSH D binary and compare it with a clean system So they decided to do everything in an external library, which is not an open SSS open SSH specific Which is a general purpose library, but that would detect that when the code path was from Open SSH or SSH D then it would do its its hooking So this is something very common on the windows side of the world and on Windows malware But on Linux malware, it's not something we saw very often Which was interesting to us to see and now since the LibQ till size of the library increased by 23k of code It was a easy way to tell if it was infected or not So when we started telling people to look at the file size of their LibQ till's the guys what they did and instead is now They put in everything in a new library, which is LibNS2 in this case I think right now it's LibNS3 that they call it and then Still modify LibQ till to load this additional library So the indicator of looking for the size of the library is not good anymore and they've been you know Going back to the hard-coded binaries and doing live the LibQ till trick and or LibNS2s And they move around these these different techniques now So how the shared library works It has a construction for a function which is executed when loading It detects that it's an executable that it wants to hook that it will it will do it hooks the important functions that it want to do and It used the L open to detect the the address space of the main executable to get the address And then it will patch the code inside the main executable to redirect the functions to the new code That is in LibQ till's so on the if you we look at the assembly for the the way the hooking works here You you can see clearly that it makes the segment read writeable and then put it back read only and then You know move the the address inside. So what does it look like when it's when you analyze a binary? Well, this is a key parse function that is clean So you see that it's calling key parse private PEM But after it's hooked and by the way you you need to you know get it out of memory If you want to see something like that because on the disk, of course, it will be it will be the good code path But when it once it's hooked by the code we saw two slides earlier It will look like this so a pointer to an address, which is now where the function inside the LibQ till is loaded How is the the credential informations? Exfiltrated from the server. It's using DNS So it was and DNS packet with username target IP address and port and it uses RSA to Encrypted so you you you cannot see it I'm not even sure now is it if it's encrypted or signed but anyway, we cannot temper with the thing It's well done and since the keys are too large to fit inside DNS the the Keys are stored in shared memory in the on the server and you can fetch it later using the xcat command So the the back door supports new commands Which are not in the SSH protocol and you use one of these commands to extract the the keys One interesting thing about the DNS packets going out is that if you have a TCP dump running With an interface in promiscuous mode on the server. It will stop sending lead credentials. So it's so it's kind of self aware it will it will You know try to trick system administrators if you want The the way you interact with the back door is using a password Which is added to the SSH client string Which is something that you can do easily you don't break any RFC if you do So this is kind of a hard-coded Password it used to be Directly compiled into the servers code so you could easily Find it and then trigger the back door on your own system if you want, but now it's hashed So you would need to break the ash First and then use it or sniff traffic because this is not in the the encryption layer yet The the commands that are supported by the back door is x ver which is a prints the version installed They do versioning which is kind of nice when we track them. We have like nice Like version this version does that this version does that You can print stolen credentials X bind is interesting because you you kind of they kind of added a feature that they needed for the SSH tunnels So we'll see later on why they use SSH tunnels But the X bind allow them to choose the IP address that the tunnel will be out from so this for them I I don't think open SSH supports that or if it does it's recent and so they needed that in ancient version too because they infect Puzzic systems, you know, they see a lot of different open SSH and so they added this feature and X password is Setting a new key for the future back door use so changing the default key that is compiled in and if you You put in no command at all you get a shell. So they they don't need to authenticate They like trigger it with the version string. I mentioned in this slide The other piece of malware which is so when they open SSH they have access to servers. They are root and and Then they do their thing for web redirection, which they use the seed or component for Seed ork is an htpd engine x or lighted htpd back door It's maintaining so they they have patches that they maintain to support all of these These binary these servers it will redirect htp requests from legitimate websites to exploit pack or affiliate ads so casino adult Dating type of things it will use a Shell memory so shm for state and configuration. So nothing is in the binary and nothing is on disk So you need to get a memory a memory dump of the the share memory if you want to know What was the condition to put to do the redirections? It's encrypted with a static XOR key, which is unique per infection So this component is installed on a small subset of the every infected server It's not everyone and they will replace the binary on the server. So if you do MD5 and stuff you will see it's not the same as the original one and the They put a lot of effort into making the patch really cross server So they they have less work to do and we've noticed since that we published our report What happened a lot is that the install seed ork and now instead of using any sites that they could use before They mostly target porn sites and we were like, hmm.   Hey, you try to infect me, you know or something like that So we figured this is a matter of for them to stay stealthy Talking about stealthy, this is the all the conditions So the these are all ifs in yellow if you want and then in red is the redirection We perform the redirection and do the JavaScript injection So it's these are all conditions that it will not redirect a user for so if for example It's seen your IP in the last 24 hours. It will not redirect you again if in the URL of The websites there is strings like cpanel secure bill admin, etc It will not redirect you to the exploit pack if The there is a specific accept language string So at some point some of the researcher in the community was saying you guys are full of shit This is not this is it's not real that this malware works like that because I cannot reproduce your your findings And then what we realized is he was using a browser that was set to Japanese and Japanese user will never be infected by this trip So that's why he wasn't able to reproduce what we we were seeing interestingly like Ukrainian Russia Japanese or languages that will never get infected so we don't really know why but it kind of We kind of think maybe it's because it's written by guys from this country We'll see So if we do a proper timeline we first encounter encountered C dork before ebri and How did we link the two together? Well first is there was a lot of a correlation in the IPs of the infection So we realized most people infected with C dork also are infected with ebri but another thing that we notice when we look closely at the The code is this is the encryption function So it's really cheap XOR with multi byte key But we realized that the constant for the XOR are the same so this is in effect the same thing It's just the compiler produced it in in the other way, but you can see the The same constant are used for the encryption and this this but those two indicators Let us to really have a strong opinion that it's all operated by the same people So What did we do from there when we realized there were a lot of servers? An interesting pattern, you know and in the correlation of these binaries is We decided to reverse engineer the domain generation algorithm and to buy a domain to be able to get the credentials That were sent through DNS and when we did that we would witness immediately 7,000 infected servers and this is where we said okay. We know it's big We we need to to do something about it and When we started also cleaning people or telling them how to remove seed or component a lot of time They were getting reinfected and so this is why we This led us to think that this is really all about stolen credentials that there are no exploits Used to to seed if you want this botnet it just really they're stealing exploits using SSH And and this is how it's been growing since the beginning The last component of the the operation which is on the server side if you want is pearl Calfbot, which is a pearl spamming demon it Deletes itself when it starts so it resides only in memory So you need to to core dump it if you want to to get the content But since it's pearl it's really complicated to reverse engineer Inside the the perimeter and it will hide itself as cron D. So, you know a simple dollar zero equals a string cron D And and it will run like that What's interesting about calfbot is that it will validate that the spam is successful So every time it's get it gets a job from the command and control server It will Get a test job first and the test job has gmail and yahoo and hotmail address to reach and it will It will send to these fake if you want so these are operator owned Address and it will send a test message with a specific Integer in it and it will never get a send job until the these this test email was successfully sent So when we implemented a fake bot we needed to re-implement this same the same check So we did actually send emails, but only the test emails and the the real emails were not sent So this makes it effective as soon as they get lack listed or the IP losers reputation Then they knew that it was an effective already and this is why it's been Efficient we noticed it sending 35 million messages a day spam messages And they are mostly again adult or a casino or Dating type a lot of dating lately So this is the we called it internally posix calf butt as a joke, but the guy from our virus lab They didn't really get it but anyway What we saw is a several OS that we're targeting and interestingly Two of them were Windows you running the malware on their sigwin. So which is interesting OS 10 a lot of bsd's and in the unspecified there were lots of things like arm machines and stuff And the way we were able to get this information is because we at some point at some moment We were having access to the command and control server But not the ultimate command and control but one in the middle which was doing running an engine x reverse proxy And we replaced the engine x reverse proxy with one where we disabled the encryption We had someone from law enforcement all typing in those command And so we were able to put in instead of using fmr all the fielman and all the strong, you know cipher suite We dropped it to AES 128 and we were able to steal the key because it was still there in this Intermediate server and so we dumped the cap then of the of this traffic And we were able with the private key to decrypt it and this is where we saw the user agent strings because it was using W get to do all the downloads and so with all the user is agent strength We were able to build this graph. This was pretty fun to do breaking SSL encryption All legally with the help of law enforcement. I don't know if I will achieve something as fun as that Once again, I hope So beyond the mower components. How is this operated? Well, we have a big graph that is really heavy and we won't go to too much detail But what is important to know is that the victims or the users are in top Left hand and the operators are bottom in the middle all of these are servers compromised running Iberi which are doing one of the various network evasion techniques that they do I will cover them later. So they have four different techniques. The users are gaining infected through C dark infected website, which is the web the web threat I mentioned and they are getting spam and When the exploit the server decides to run an exploit kit on a victim It will install the glup table malware, which is a Windows malware. So it will never target Linux desktop users it's all Only targeting Windows users and glup table is another generic proxy component That is again used to send spam. So all of this is really only spam and adults Redirection and so we were never able to reach to the bad guy at the bottom We always had access only to intermediate machine Which means that if you work for OVH or for, you know, people like that Please come and talk to us I would be really interested in having good contacts with people from these companies Because a lot of servers are sitting at providers like that That's not also in Germany So I realized that I'm taking I'm taking my time to talk I thought I had 45 minutes though, but I guess it's questions at the end So I'll go a bit quicker with the thing. So as I mentioned earlier This is all about stolen credentials and this is the way it steals them but already covered them How come it has so many root credentials? Well, this is a graph of the the stolen credential we analyzed and you can realize that 40% of the stolen credentials are root credentials. So so this is what allows them to spread more The money trail already mentioned so the exploit keep Windows malware and then spam and Adult affiliate program. So this is where they do their money and the impact of this thing is we notice more than 25,000 compromised server. We saw half a million browser redirection per day with 20% going to exploit packs 35 million spam sent daily and kernel.org was infected at some point and this is when they did their big You know takedown and removing shells of people. We had some good contacts confirming us that it was Iberre infected So why is this thing advanced because it's stealthy and because it's effective For us, it was really the the two things that made it interesting and worth Going after so the DevOps why? Are they DevOps? Well, they have an interesting Monitoring and deployment script and reconnaissance script and they use SSH in an interesting way What we've seen in the past is they were W getting code and so you could with a packet capture Get extract the code from the packet capture But now they stream the code using SSH redirection and an interpreter Which means that we not we never have access to the code that they run unless You build a man in the middle for SSH, which is what we did we use the one from from sign this org and we put up a very Full interaction on you put to be able to steal the scripts So the scripts are written in pearl. They always report with standard out errors and status So they are not obfuscated, but it's still pearl scripts What they do with the type of code that they run is they eliminate evidence So what here what this does is it will eliminate any log line that has a user name that it used to connect to the system Failed password or its IP address in this it will avoid all large files So this is what the find is doing so any log file larger than a hundred megabyte will not be Clean and it it's fetching all the files to look for into the ATC scon Configuration and also everything under of our luck The reconnaissance scripts are really interesting. They do they check extensively for anything LDP reload related So most of the low interaction or higher interaction honeypots are built using LDP reload tricks And so this will check this to avoid being if you want man in the middle or Investigated it will check for various restrictive SSH configuration And it will not deploy in these environments, and it will check for BSD jails C panel BR and men at Joe's several different things like other D if it's activated the One is such example of a generic SSH honeypot if it will look in the strings for its HD If certain strings are in there like for a user local LibExec, which is a jail the SSH thing And it will bail out if it sees stuff like that the reconnaissance also detect available tools, so if it's able to compile or if there is package management and Present it will it will look for the headers if it can compile open SSH already without having to download them And it will check if it's already installed of course the deployment script so the the pass tar ball with all the code that they need or Eat them the pre-compiled of the malware and they used a pearls data special literal to do that So this is an example The but the binary data usually is really binary. This is this was encoded by a vim. So normally After the data special literal you will get, you know, raw binary So they use this to pass the file, which means that again an external packet capture will not give you the malware You need to manage the middle the thing The the deployment script will also alter package management manifest So it will install the new ash So if you do a rpm check or a deb some you will not get any warnings because it modified So this is the code for the Debian version It will go and remove the the hash of the file inside the var lib the package info And it will put the hash of the malware inside. So you will not find It will it will seems like the system was not compromised There they use also an LD preload trick to be able to install an rpm in the past so that if you look at the rpm Information the only thing that is different is the the key ID of the GPG because yes, of course They sign their malware with GPG But they don't care about Key problems because they are root and they install the key or they use an installation mechanism that says don't check the key But nonetheless the thing is still signed The daily monitoring script is written in bash and it will gather, you know, user names and SSH keys and stuff like that. So if new users come on the server You they will be able to still know and fetch their these These user names and these and those keys other findings it will modify the SLNX policy and instead of Installing or disabling SLNX like some of the more simplest malware do This thing installs a policy that is tailored for what it does. So it it's really it understood SLNX More than most system administrators, you know Which was interesting. Yeah props to them for taking the time to understand the thing So it does various style of installation and it looks for over 40 back doors and root Kip to be able to put them away and be the one on the server and some of these were are not publicly documented so we looked for some of this of these strings and Google yells no result. So it has access to more malware than us have access to So we would like to get samples from them, but they're not cooperating really So, yeah, I will skip the recap slide because I'm gonna I have five minute left So forensic and incident response forensics is evidence gathering process analysis network analysis that I will cover So you need to be careful because you're running if you do it in band So inside the infected system you're running at the same privilege level that then the bad guys So it's an arms race. They like I'm telling you how to use net stat But in a future version they could provide a modify net stat that will you know remove the results that I'm telling you to do So the better is always to be out of band and to do memory forensic and this forensic, but it's a lot more a tedious process, but I'll still cover it later So yeah, it's better to aim for out of band forensics so how to spy on a User with the same privileges when it does all those tricks Well, we found out that audit B is what worked best for us I'll skip this because Audit D is very nice and it really helped us a lot But in the end we needed to build a man in the middle to be able to have access to all of this What what can you use to do process analysis? You can use g-core to dump the process and then use strings or gdb or either pro to do proper reverse engineering and Yeah, so this maybe is more advanced if you want, but there was a reverse engineering tutorial yesterday You could use the tools that were Right radar to toolkit to do your reverse engineering also Something that I talked that the lot of the mower was deleted I learned that you can bring back something that is running out of proc So if you delete a binary You can still copy it from proc even though it says that it has been deleted Which has been really helpful helpful for us to do the man in the middle. I talked about earlier and More tricks is that what you can do is you can check what is the target of the exe that is running So even if it's it's hiding itself as cron if you look inside proc You'll be able to see if it's really user bin cron or user bin pearl which is hiding is cron Something that we I tried to tell people to do is always get everything you can from proc PID before killing a process or six stopping it because in some cases We saw script that are encrypted and that the encryption key is in an environment variable So if you still have the malware, but you killed it You don't have the encryption key to analyze it because first thing it will do is decrypt itself and then do its malicious payload To do the process analysis, you can use LSOF net stat IP CS and stuff like that to S trace to trace system call L trace to trace library calls Share memory analysis is done with SSM cat if you want to output it into a file I'll skip that to do a pearl reverse engineering You can use pearl TD to pretify the pearl which is the first thing you could do And then I use vi and I rename variable, you know using the by the quick key bindings And if the script is heavily packed like some of them are only brackets thingies You could use BD parts which will do first step of like The extract that from the p code instead of looking at the code. So you you get more readable pearl But it's still pearl at the end So the various network evasion I wanted to mention but unfortunately I will run out of time our SSH tunnels engine x reverse proxy IP and IP tunnels and tree proxy The SSH tunnels are used for spam You you can find it pretty easily looking in the process Looking for process of SSHD the reverse engine engine x proxy are used for the command and control servers and the exploit pack This is an example of the config they run They use a lot the upstream server feature to have load balancing and you know resilience if you ever take down one of the server How do you find them? It's simple. You look for sockets or and or process the IP and IP tunnels are used to do to hide the they do credential brute-forcing and general network scanning and They they use several layers so the IP tunnels are handled in the kernel So there are harder to to find but if you do IP tunnel show you can find them Or IP route show you will see a tunnel interfaces This this was very interesting for us and we did some TTL analysis and really more You know again offensive stuff against them which was interesting They also use IP tables to do port redirection So you need to look also to think about auditing your IP table rules if you want to find it so just audit the thing tree proxy is a cross-platform its protocol proxy server that is not malware and sometimes it's renamed so you need to look for it Again, you know you it's a process so you will find it in the process list So out of ban forensic I'll go really quickly but wire shark SSH men in the middle with the tool we mentioned here This capture dumping memory in ban with lime Out of ban you can use virtualization snapshots if you have the if you run your server in a virtual machine Memory analysis you can use volatility We release indicators of compromise But as I said earlier every time we published about something they adapted to it So what we prefer to do now is? Tell people to contact us and if you provide us with a proof of that you're not you know a malware operator Then we will send you the latest indicators of compromise We've got and so an example of a reaction is that after we release our report the updated there They're malware to avoid the the thing and the added in the strings of the malware. Good job He said and thanks for either Pro was leaked from our company a long time ago. It was stolen from an analysis machine So they they kind of joked at us a little bit But that means they read our report, which is good So incident response quickly. I just don't be in denial reinstall everything from scratch and Be really careful to not reuse credentials that you use because you will get reinfected and we've seen it a lot Ideally consider implementing a password policy to prevent password from being reused But again, this might be more complex and this something we don't see often in servers Another way to completely prevent a threat like that is to use two factor at investigation So anything like Google investigator would prevent the thing because it relies only on stolen credentials And so see if you have to factor You you it won't work One thing I like to do when I have some time is like every who uses two factor for their Gmail and Hotmail or Twitter and a lot of people raise in and then who uses it for servers and then no one Yeah, nice. We have like five people. Good job guys Yeah, so recap and then Lugs I think we should ask system developers to make logs harder to temp temper to have some live CD to verify package integrity instead of doing it from the system and Things that should make Linux system more resistant to attacks, but Unfortunately, I think this will take time, but system D with journal D is doing forwards Ceiling which should be a technique you can use to detect that your logs were tampered with So yeah, the reason we gave we talk at conferences is to spread the word about these type of threats And so that people like you who get sometimes asked about Linux malware or and or have to react in some cases know what to do so I hope that this was helpful and I you will be able to refer to the slides to you know grab the list of command that I went quickly through but Let's work together to make the ecosystem more resistant to attacks like that And if you find anything suspicious, we are always interested in looking at malicious code And we often can quickly assess if it's malicious or not And then it takes a longer time to analyze if it's something completely new and very Obfuscated impact. So thank you very much So we have about five minutes for a few questions See so I'll start We'll go one two You talked about the tools like IP tables IP route LS etc Don't most of those toolkits replace those anyway, and so you can't trust them. What do you do about that problem? So the in this specific is my mic still on In this specific case, we know that they weren't modifying these Bineries because we were able to notice the way they were infecting the computer So in these cases we were relying on them In in the cases where you know, you you don't know what you are Investigating then of course it's always better to go completely out of ban and you know Look from a disk image perspective what the bank binaries are and once you confirm that they are clean or not then you know Do trust them or not trust them, but it's a very a very hard problem, of course You mentioned that it would actually check if it was already installed Before it would like a system. I mean it was possible to like trick it into thinking you were already compromised to prevent it Installing itself. Yeah. Yeah, someone could do that definitely, but again, you know at this point It's already running code on your machine. So you would prefer to close that door, right? But so yeah, it's using the it's using the shell that it gets from the back door so in effect like if If you're running the back door you are kind of infected so I guess I would need to look what specific test it does to detect if it's installed and then if we could easily emulate it But it's probably something like Looking at the binary or the presence of the file and the file size, but yeah, I have this script I might look at it later Could you put a box between your server and rest the internet which only did check the traffic going through for this stuff Yeah, so when we released our report we released snort rules, which is unusual for an antivirus company To be able to do just that like being Alerted when you are infected So we did that Okay, so this will be a second last question What would you say is the Three or four things that this admin should be doing to protect our servers from these sorts of attacks Not just this one, but in general So the Linux threats are really mostly about bad practices if you your users passwords are good So like Cracklib for me would be one of the first thing I would do if I was a shared hoster You know who I would have users having dozens of you know cat websites and stuff like that So, yeah making sure credentials are really better even disabling passwords Entirely no root login, please seriously like disable this thing remote root login Well, okay, so at the this thing steals SSH keys So they are aware of this possibility But of course to get it stolen in the first place It's harder and what I will advise people is to really be good with SSH keys and never forward them to server Use SSH agent if you need to You know do the hop from servers to servers But yeah, okay, so SSH key has ruled then is a matter of you know personal choice And what is good if you use a sage key for route is that or for non route? I should say is that after you get inside the server You still need to pseudo and if you login with a key and you use a password for pseudo Then you kind of have two different credentials that they need to steal instead of just one so You know it's all about risk analysis, but the people that are infected are really lousy sloppy and men's So far, you know and everyone in here. I'm pretty sure would maybe know someone infected But wouldn't be directly affected unless they haven't updated their box and their passwords since a long time Hi quick question about SC Linux. You said that Some of the malware shipping their own SC Linux policies Wouldn't that be a dead giveaway because you'd have to publish the policies in the kernel or have actually Done something to the kernel SC Linux modules to actually hide the policies as well No, it's it's right there for you to see but it's altering SSHD's policy already Is it SSHD or HTPD? I don't remember but it's altering, you know, one of the pieces already existing policies not creating a new one and so you could like Wander and the policy it adds is for this binary like a star slash star or something like that So it's pretty obvious of course, but you know, it was able to hide itself in IP tables it was able to hide itself and stuff like that So of course anyone is using like tripwire or stuff that would detect configuration changes would be notified But I guess a lot of the sloppy and men's are not auditing their SC Linux configuration Often enough. Yeah, so I say Linux is still safe. It's just yes. Yeah. Yeah. Yeah policies. Okay. Thanks Thank you very much for your presentation today, Olivia. Thanks and We also have a small gift just from Linux conference to say thank you today. Thank you