Do you know grab the other box? Yeah. Okay, so I'm always getting asked How do I ship my paintings? And so today? I'm gonna kind of show you that maybe a quick video on how I ship my paintings I shipped my paintings as you can see woke up with a bunch of snow Which I did not know what's gonna happen. So thank you, Missouri. Appreciate you today We have about 50 50 to 60 paintings going out. And so yeah, let's get it Normally we'd put these bins in the trunk, but I think I'm just gonna stack them Just one by one in there instead of putting the bins. So this is uh, this is what we do Thanks, Sarah You can put it down. Thank you now You put these pieces right on top Should be good. All right. There it is Thank you. Why are you putting up? Why is it over your head? Bye. See you later Also, I just want to say thank you for everybody who has supported My channel my journey if you've bought a painting. Thank you if you subscribe or follow Or just just been around. Thank you so much It's been honestly a huge blessing and we just can't wait to see what? 2021 brings quite honestly It's so freaking amazing to even think about that. This was possible Like this is my full-time job creating videos and Doing spin art for people all over the world some of my dreams coming true because of the hard work And I'm just I'm excited. So it's no one I should focus on the road because I'm gonna go on the highway Yeah Here I'm just backing up now spot Don't want to hit the car there All the way in around Taking up two spots. Whoops We'll be quick. We'll be quick Then park. That's all of them and they ship them out. We're good to go We're all done Now we head out. There it is just a brief inside look as to how I ship all my paintings We'll be as like so let me know what else you guys want to see. Peace


Alright Aloha and welcome to my talk. Uh today we're going to be talking about harnessing weapons of Mac destruction. My name is Patrick. I am a security researcher at Jamf. Uh previously I was a co-founder at Digital Security and I'm also the creator of the Mac security website and tool suite Objective C. So today we're going to be talking about repurposing other people's malware for our own surreptitious purposes. We'll start by showing why this is rather an excellent idea and then show exactly how to repurpose various Mac malware samples. Of course we want our repurposed samples to remain undetected both by Apple's built in malware mitigations and third party AV products so we'll address that as well. Finally we'll end by covering various generic ma- methods to detect these repurposed uh creations because really our ultimate goal is to ensure that Mac users remain protected. So let's dive in. So here we have a diagrammatic overview that conceptually explains the repurposing process. So imagine that spy A has developed some neat uh sophisticated uh fully featured malware and has deployed it to various targets, right? Infected various systems around the world. Spy B somehow captures this malware. Uh perhaps either in transit or pulls it off an infected system. Spy B then will take this malware sample to her lab and then repurpose it. So what do we mean when we talk about repurposing? Well we'll get into this more shortly but basically we're talking about reconfiguring a piece of existing malware so that you can use it for your own surreptitious purposes. So for example we can see on the slide once spy B has repurposed the malware she can now redeploy that against new targets and instead of beaconing back to spy A's infrastructure for tasking it'll talk to her new infrastructure instead. Now you might be thinking okay this seems like a reasonably neat idea but why? And we'll talk about two reasons that hopefully show that we should really be asking why not. So first there are incredibly well funded uh incredibly well resourced uh very motivated hackers, APT groups and three lever three letter government agencies that are creating some incredible malware. Uh this malware is fully featured and has the benefit of being fully tested a lot of times already even tested in the wild. Now I'm rather lazy I would rather be sleeping on the beach or surfing or doing yoga so I figured why not let these groups create this incredible malware and then we can just repurpose it for our own missions. The idea is to let those with more time resources and money do all the hard work. Secondly if our repurposed creations that we redeploy get captured it's very likely that they will be misattributed back to the original authors not to us. This is great. Now you might be thinking okay Patrick if this is such an amazing idea why aren't people already doing this? Well the answer is they are. Now I can't comment specifically on this but leak slide uh slides appear to indicate that the NSA has had this capability for a while. Also recent reporting from the New York Times claims that the Chinese are doing this as well. Now to these well resourced agencies the benefit of using repurposed malware is twofold. First it allows them to deploy other countries malware into risky environments. For example perhaps a target where another agency, another country, another APHT group already exists. So imagine you're in the NSA you hack a high value target and you find out someone else is already there. Obviously you don't want to put your primary implants on that system. So if you have a repurposed implant that you can use might as well put that down. Also we already talked about attribution. These agencies don't like people to know what they're up to. So if they can use repurposed malware perhaps this muddles the attribution picture. Now of course there's going to be challenges to repurposing malware. If there weren't we wouldn't be talking about it today. But we'll see these challenges are not insurmountable. So first generally speaking if we capture a piece of malware to repurpose we're not going to have the source code for the malware. We're just going to have a binary image. So we'll have to reverse engineer the malware at the binary level. This is kind of the analysis stage I like to call it. And what we're doing is we're trying to figure out and gain an understanding of what the malware does. Answering questions such as how does it persist? How does it install? And what its capabilities are. And of course what command and control servers it talks to. Then with such an understanding we have to figure out exactly how to patch the malware. Again usually we don't have source code so this involves patching at the binary level. Now if the malware talks to a remote command and control server which most do we will have to figure out how the malware determines where to get the address of that server. And then also understand the protocol so we can create our own command and control server so when we redeploy the repurposed malware we can communicate with our new infected systems. And then finally since we are repurposing generally known malware it's likely that Apple or AV products already have detection signatures for this. So we have to talk and we have to figure out how to get around that because obviously it would be rather unfortunate if you deployed some repurposed malware only to have it detected. Now I want to reiterate that it's very important to take these challenges seriously and get them right because if not relatively bad things can happen. Here for example we've attempted to comprehensively repurpose a piece of Mac malware but it turns out that we forgot to modify or patch out the addresses of the backup command and control servers. This means if we deploy what we believe to be a fully repurposed piece of malware at some point it may beacon back to the original hacker's infrastructure given him or her access to our infected targets. So obviously this is something we want to be very aware of and make sure when we are repurposing these samples we're doing it comprehensively. So now let's talk about exactly how to repurpose Mac malware with the goal of making their malware ours. So the first step is to select a malware specimen and that is the sample you want to repurpose. This is usually based on two criteria. First you decide what you need the malware to do. Are you looking for an interactive backdoor, perhaps a piece of rent somewhere, cryptocurrency miner, take your pick. And then maybe attribution is important to you. Do you want your repurposed sample to originally belong to a three letter agency, a Russian hacking group, so again if it gets caught it will be likely misattributed to them. So on my website ObjectiveC.com I have a very comprehensive collection of Mac malware. So that's a great place to start. And all the samples we're talking about today that will repurpose the original specimens are available for download. Now once you've selected your malware sample it's time to analyze it. In terms of purpose or as we mentioned the main goal is to figure out things like if it has any remote access, right? If it's talking to a remote command and control server we want to know what that server is so that in the malware we can modify it to instead talk to us. If the malware does talk to a remote command and control server we also need to understand the protocol so that we can task it when it connects back to us. And of course you want to fully understand the malware's capability both locally, how does it persist, how does it interact with the system and then also remotely. What commands can we remotely task the malware with? So once you have a comprehensive understanding of the malware it's time to actually reconfigure or repurpose it. Generally this means we'll have to patch the malware at the binary level. So here for example as we can see on the slide we've located the addresses of the malware's remote command and control servers directly embedded in the malware's binary. We can simply modify these bytes for example in a hex editor to change that address to point to our own control. This means now when we take this malware and redeploy it the malware will be can connect back to our infrastructure and be ready for tasking. Now as I mentioned if the malware is a back door or an implant and expects to talk to a command and control server well if we're repurposing it we're going to have to write our own custom command and control server as well. This will obviously ensure that when the malware checks in and connects back to us we can give it tasking and fully exploit its remote capabilities. Now such a custom command and control server really doesn't have to be anything incredibly fancy. We can whip up a simple python script but if you're planning on deploying these repurposed malware samples to many targets you might want something more comprehensively. In a previous demo we talked about creating one such custom command and control server with the goal of analyzing a sample. Now obviously each specimen you repurpose is going to be different in the sense that it likely has a different protocol and thus we'll need a custom command and control server for that specimen but that paper is kind of a good place to start. Alright so now let's walk through some actual examples repurposing some Mac malware and we're going to start with a sample called Fruit Fly. Fruit Fly is a rather unique piece of Mac malware. It was originally developed over 15 years ago remaining in the wild and only very recently became detected. It's a fully featured back door which supports a myriad of remotely taskable capabilities which really makes it the perfect candidate for us to repurpose. As we'll see we can very trivially repurpose this malware and then that gives us access to a proven fully featured back door that we can utilize for our own subversive purposes. Fruit Fly is actually written as a pearl script though it's highly obfuscated. If we take a peek at the back door we can see logic that processes command line arguments that is to say the script will check if it's been invoked with certain command line arguments, parameters or flags. Turns out it's looking for a port or a port and IP combo of the command and control server to connect to. So this means if we specify our own via the command line we don't actually have to modify the pearl script or the malware binary itself. So what we can do is we can persist this this malware Fruit Fly as a launch agent and as we can see on the slide we can pass in our address of our command and control server. So this means we can trivially repurpose this back door to connect to our custom command and control server right because we can specify in the command line we can now push that out, give the address of our own server and the malware when it runs will connect directly back to us instead of the hackers original servers. But to deploy this we need an installer. Good news previously I wrote one that's available on Payspin as we can see it's just a few lines of code which persistently installs the malware in a manner that matches how it was and covered in the wild. Finally as I had previously written a custom command and control server to analyze the sample we can just review this. So we're gonna do a live demo now. Live demos are always risky and actually we're gonna run this on my laptop. This is my own personal developer laptop.  So let's see this in action. So the first thing we're gonna do is we're gonna run the command and control server. This is listening for connections and then we're going to execute the malware specifically via the installer passing in the address of our command and control server. When we run this we can see that the malware now has connected to our command and control server on the right. And we see there's a myriad of capabilities that we can now task this malware. Again this is running on my own personal box. So for example if we execute command number thirteen this will tell us where the malware is installed. We can see it's running out of users Patrick FPS odd. We can also do things like take a screenshot that is command number two. Execute that. We give it a parameter of zero saying we want a full screen capture. And this will tell the malware which has been deployed to my personal system now to please take a screenshot and send it back to the command and control server. So we can kill the server and we can open the screenshot and we can see yeah it took a tire screenshot of the desktop. Okay so that's fruit fly. Now let's talk about repurposing a cryptocurrency miner because maybe you want to make a lot of money mining cryptocurrency on other people's computers but you don't have the skills time or resources to write your own. So our target is called creative update. This cryptocurrency miner was spread via trojanized applications and then spread via the popular third party application website macupdate.com. If we mount the malicious disk image that contains one of these trojanized applications and reverse engineer the applications main binary we can see it invoking a method called execute script with privs. And if we look at what this method does in a decompiler or disassembler we can see it simply executes a file named script out of the application's resource directory. So if we take a peek at this script we can see it first launches the real pristine copy of the application that it had trojanized for example firefox. And this is so the user who has downloaded this infected trojanized application doesn't think anything is amiss. We can then see in the script it beacons out or connects out to a download server to download a zip archive named mdworker.zip. And then extracts this archive and persistently install something named mdworker. If we look at the launch agent P list that persists the malware we can see the path to the persistent binary which is that mdworker binary we just discussed. Also we can see some command line arguments that are passed into this binary. If we execute this mdworker binary it turns out it's a legitimate cryptocurrency miner belonging to a company named miner gate. The command line arguments therefore specify the miner account that is where to send the mining results and the type of cryptocurrency to mine. So similar to fruit fly since the relevant parameters are passed via the command line as say opposed to directly embedded in the malware's binary it's trivial to repurpose this sample. So first we simply modify the property list the P list to specify our own miner account and then zip up the malware's components. Now instead of having the malware download that zip from a download server we just add the archive to the applications bundle. This is far more simpler and self contained. It means we don't actually need to set up a remote download server to host the second piece. Next we tweak the script so instead it utilizes the zip archive we just added. Finally we repackage everything up and we're ready to go. So we're going to do live demo number two. Finder window. First I want to show you the repurposed malware it's kind of application bundle. We can see this is an app and within the app there is this script file. There's the original Firefox. This is the one that signed Mozilla. If we pop over here though we'll see that this is what the user downloads and expects to run. So what we can do is we're going to run this. This has been reconfigured. Again I'm running this on my own personal system. Figured while I'm up on stage I might as well be mining some cryptocurrency. So we're going to run this. Firefox pops up. Everything looks great. The user is stoked. They've got this great browser. Now if we pop back to a terminal window. Behind the screens I make this a little bigger. We can see that the malware has persistently installed itself and again if we look at the program arguments we can see that it's going to execute the miner named MD worker but we have repurposed it so it's instead is going to send all the results of the cryptocurrency mining operations to me. Sweet. Awesome. Alright well since we're talking about making money right we all love to do that. Perhaps you want to deploy some ransomware without spending time writing your own. So now we'll walk through exactly how to repurpose KeyRanger. KeyRanger was the first fully functional in the wild ransomware targeting macOS. Hackers infected the popular bit torrent client transmission on the legitimate applications website. This means that if users went to the transmission website to download the bit torrent client inadvertently they would get the malicious infected trojanized version and become infected. Looking at the malware's binary code we can see it executing something named general.rtf out of the application bundle. Though it appears to be a document if we examine that file a little closer it turns out it's not a document it's really a mock-o binary. So what this mock-o binary does is first sleeps for a few days and then it connects out to the attacker's command and control server expecting a response consisting of a public RSA encryption key and then decryption instructions. It then uses that encryption key to encrypt the user's files and then displays the decryption instructions that demand a ransom. Pretty standard ransomware functionality. So let's talk about repurposing this. First we're going to wait for three days. We're kind of in a rush to start encrypting people's files. So what we can do is we can just patch out, knock out the call to the sleep function. Pretty easy to do. And modify the addresses of the servers that the ransomware contacts for the public RSA encryption key and the decryption instructions. As we can see on the slide these addresses are directly embedded in the malware's binary so it's pretty easy for us to hop into a hex editor and change them to the address of our server. Pretty easy, right? And here for testing purposes we just set them to localhost <IP_ADDRESS>. Now we also need a server because the ransomware is going to connect out when it's run expecting that encryption key and decryption instructions. Turns out this is really easy to do. We can just take our own public RSA key and decryption instructions and package that up to an text file and then serve that up via Netcat. So here's a demo of KeyRanger that's now been repurposed. We'll see in the terminal window, this is our server, again it's just Netcat serving up the encryption key and the decryption instructions. The malware has been repurposed though to talk to this command and control server, not the attacker's original one. And then we'll start seeing the ransomware immediately start encrypting all the files on the desktop. Once the malware has done this, again it's launched when the user launched the transmission application. They're not going to be privy to anything going on. You know, the encryption happens in the background. Although they might see all their files being encrypted.  They're going to open the readme.decrypt file and you can now see we provided our own decryption instructions.  It's actually kind of mean, you know, maybe don't do that. The last sample I want to talk about repurposing is called Wintail. Now Wintail is a lovely backdoor written by a rather sophisticated APT group. Here we'll repurpose both their exploit and their own backdoor to use for our own missions, our offensive cyber surreptitious processes. Now one of the most interesting piece of this attack was the Maurer's infection vector. And the way it was able to infect fully patched Mac systems while it was by leveraging a legitimate functionality of Mac OS by abusing something called custom URL schemes. Now I've written a long detailed blog that really dives into the technical underpinnings of this. But in a nutshell, on Mac OS, as soon as an application is saved to the file system, even before the user has run it, Mac OS parses that application and if that application supports custom URL handlers, Mac OS will automatically register it. Once these custom URL handlers or the app has been registered, you can for example from a web page make a request to that custom URL which Mac OS will be happy to now launch. The exploit is realized in JavaScript. Basically they have some code to first download a malicious application that contains a custom URL scheme. Mac OS registers that and then they have some second piece of JavaScript which makes a request to that now registered custom URL scheme to launch the malicious application. In terms of the back door, so again this is the payload of the exploit, they persist a login item that supports capabilities such as file collection and exfiltration. So we fire up a process monitor, as we can see on the slide, we can observe what the malware does once it successfully infects the system. For example, we can see it invoking the built in zip utility to compress the files that it has collected for exfiltration and then via the built in curl command, it sends them to the attacker's remote command and control server. The back door also supports a file download capability. So if we look at the SDF method, a disassembler or decompiler, we can see it decrypts the addresses of some embedded command and control servers. Then it makes a initial request to get a name for a downloaded file and then makes a second request to get the actual file contents from the attacker's command and control server. If we use a network monitor, we can see this happening live and as we note on the slide, the address of the command and control server is flux2key.com and just remember this because we'll get back to this in a second. Now once the malware has downloaded the file from the attacker's remote command and control server, if we look at more disassembly, we can see that it then executes this. So this is done within the same SDF method. We can see that it unzips the file it gets from the attacker's remote server via the built in ditto command and then it executes it via the NS task API. So pretty standard built in download and execute capabilities. Finally the back door can be remotely uninstalled via self-delete logic. So every time the malware starts up, it connects to the attacker's command and control infrastructure and basically says, should I delete myself? If the command and control server responds back with a one, the malware will first delete, uninstall itself and then terminate. It's kind of a neat capability that allows an attacker to remotely disinfect targets. Alright, so now it's time to coerce this lovely back door to do our own bidding. So first let's talk about repurposing the exploit.  As we mentioned, Safari by default will auto expand zip archives. So if we place a malicious application in that archive that implements some custom URL handlers, Mac OS will automatically register that even before the user has run the application. The malicious webpage can then make a request to that now registered custom URL which will cause the application to be launched. Now there will be a pop-up or two, but the average user probably is going to click through these. So here's a demo of the repurpose exploit that will drop and persist Wintail. We can see the page has downloaded the application and then we made that second request. There was a pop-up. Again, if the user clicks through which some percentage of users will fall for, we can see that now the malware has been automatically and persistently installed. Again, since this is a logic flaw, there's nothing really for Apple to patch. So this still works on the latest version of Mac OS. So now we've repurposed the exploit. Let's talk about repurposing the malware because it doesn't really help us if we are spreading their original malware that still beacons back to the attacker's command and control servers. So step one, we need to modify the embedded command and control server addresses so that when the malware is deployed via our repurposed exploit, it connects to our servers so that we can task it. Unfortunately, the embedded addresses of the malware's command and control servers are encrypted. Not to worry. What we can do is we can coerce the malware to always load a dynamic library. And then once our library is running in the process address space of the malware, we can basically modify the malware at runtime to intercept the decrypted addresses of the command and control server before the malware uses them to connect out. So let's look a little closer about how we accomplish this. Again, the goal is for us to specify our own command and control servers so that the malware connects back to our infrastructure. So our dynamic library has now been loaded into the address space of the malware. What we can do is we can swizzle or swap out the malware's decryption routine. It's a method named UPE. This is accomplished via the rather powerful method exchange implementations API. As we see on the slide, once the decryption function has been swizzled or swapped out, whenever the malware invokes the decryption function, transparently the Objective C runtime will actually now call our method in our library. What our method now does is calls into the original decryption routine. And I have no idea what the decryption algorithm is or how it works and frankly I don't care. But what I can do is since it's swizzled and passing through us, I can see the results of that encryption. So if the original decryption method returns the decrypted addresses of the command and control servers, I can just return my own command and control server back into the malware's logic. So this dynamically gives us the ability to transparently swap out the decrypted command and control servers with our own. And the malware is none the wiser. Now I added some debug messages to the library so if we execute this now repurposed malware, we can see and watch this at runtime and view the addresses of the command and control servers being swapped out. So at the bottom of the screen, in the terminal output, you can see that our library, which is called through to the original decryption method, has detected that that response is the command and control server that's now decrypted. We don't want to return that to the logic in the malware that's trying to connect out. So we basically just drop that and return our own. Again this ensures now that the back door will always communicate with our servers instead of the original APT groups. Now we still have to write a command and control server because the malware will now be connecting back to us but we have to be able to talk the protocol so that we can correctly task it. Now the main feature of the wind tail back door is to collect and exfiltrate files. So let's add this first to our custom command and control server. Files are exfiltrated via post requests. So we simply write a few lines of code in Python to handle this saving off each file the malware sends to us from the infected host. So here's a demo now of our custom command and control server receiving exfiltrated files from the attacker. On the left you can see the network requests coming in and on the right all our command and control server does is write these out to the file system. This means we now support the ability to respond to the malware as it exfiltrates all sorts of files off infected hosts. Let's also extend our custom command and control server for this wind tail implant to support the download and execute capability. So we noted the malware makes two requests for this. The first is a request where the expected response is for the name of the downloaded file. And then the second request we need to respond with the actual bytes of the file we want to download. The malware will then save that to the file we specify and then unzip that and execute that. Now again we only need a few lines of code in Python. We can leverage a lot of built in Python libraries to do this. And once we do that we then have the ability to task the implant. So let's watch this in action. We're going to see the malware connecting out to our command and control servers because we've swapped out those addresses. And what we'll do is we'll have it download and execute the ubiquitous calculator dot app. So here comes the connections. We're giving it calculator dot app. This could be any binary and then the malware will unzip that and execute that. Now armed with this remote download and execute capability this means we can install other tools or even second pieces of malware. So it really makes this repurposed implant kind of infinitely extensible. Finally let's add the ability to our custom command and control server so we can support the self-delete command. Because at some point you know if the FBI comes knocking on your door it might be really nice to just press a button and have all these infected systems uninstall themselves. It's trivial to support this functionality. Basically when the malware connects out checks in with our server and asks should I self delete? We just respond with one. This will tell the malware to delete and fully uninstall itself. This is another demo showing this in action. We can see the persistently installed implant. It's that orange logo of the application originally tries to mimic a PowerPoint documentation. Kind of sticks out but this is in kind of a hidden system directory. What's gonna happen is the malware is going to connect out to us make the should I self-delete command a request. We're gonna respond with one which will cause the implant to uninstall itself and then terminate. So we can see it disappear and then it stops beaconing because obviously it's just self-deleted. Alright so we've showed exactly how to repurpose a myriad of Mac malware. We talked about backdoors, ransomware, cryptocurrency miners. Although we still need to talk about how to remain undetected. As generally speaking we're repurposing known malware. Which means both MacOS and third-party AV products likely will block or detect these unless we do something about it. So first up let's talk about MacOS' built-in malware mitigations such as X-Protect, the malware removal tool and code signing certificate checks. Now at first glance these might seem somewhat problematic as they may generically block our repurposed malware on any or rather on all Macs. Again we really need to talk about bypassing these because if we went to deploy a nicely repurposed piece of malware and Apple blocked it, that would kind of suck. So let's start with X-Protect. X-Protect is a simple signature AV, simple signature based AV scanner that's built into all recent versions of MacOS. It's somewhat limited it only scans files downloaded by the user the first time they're executed. If we look at their signatures they're in a file named X-Protect dot YARA we can see various YARA signatures to detect common pieces of Mac malware. Uh for example key ranger the malware we just talked about repurposing. Now at digital we wrote a simple utility called UX-Protect that allows you to explore these signatures via the UI as well as at any time uh basically scan arbitrary files against these built in signatures. Now since X-Protect is purely a signature based detection mechanism it is incredibly trivial to bypass. So let's take a closer look at the signature for a key ranger. We can see it maps to a group of instructions in the malware. Knowing this to bypass the signature we can simply reorder or modify any of these instructions. This will change various bytes which will of course cause the signature to no longer match. So here for example we simply change the number of bytes that may be written to a buffer from 0x400 to 0x300. Now this doesn't impact the malware's functionality at all but as we've changed the bytes that the signature is looking to match on we've completely thwarted the X-Protect signature. Demo time uh here we have two instances of key ranger. Again it was distributed in this infected transmission BitTorrent client. On the left we have the original sample. This is the hacker's sample. And on the right we have the repurposed sample that's gonna talk to our own command and control servers to get the encryption key. And then also it has that one byte modification to bypass X-Protect. So we're gonna launch both of them. We're gonna see the original one is detected by X-Protect and blocked. There's no option to run it. The user has to click cancel. If we execute the second one, it's gonna say hey this is downloaded from the internet. Are you sure you wanna run that? User clicks yes. The ransomware is now able to persistently install on the user system and immediately starts encrypting all their files. Woo. But wait there's more. Next up we have code signing checks. So most binaries on Mac OS are code signed. Signed binaries are not blocked by Gatekeeper thus a lot of Mac malware is now code signed. The downside from the malware's point of view is once this malware is detected, Apple can simply revoke the code signing certificate and this means the malware will not be able to execute on any other Mac. So here for example we can see we have wind tails, the wind tail binary and the original code signing certificate has been revoked by Apple proper. This means even if we repurpose this and attempt to redeploy this to new systems it won't be allowed to run. Well the problem is really the revoked certificate. It's not the malware per se so we can simply remove the code signing certificate. And then either distribute the repurposed malware unsigned or re-sign it with another legit developer code signing certificate which is pretty easy to get. So to unsign it Apple actually provides a utility called code signed. There's an undocumented flag dash remove signature. So if you execute this and point it at the malware Apple's utility will nicely strip off the revoked code signing certificate. And then as I mentioned we can just deploy it as is unsigned or if we want to get a little more fancy we can re-sign it again using the code sign utility passing in the dash s flag. This means the re-signed wind tail will now no longer be blocked by macOS. Finally we have MRT. This is Apple's malware removal tool. It's another built in AV like tool that ships with all versions of macOS. It's similar to X-Protect but scans malware that may be already on the system and if it detects it it'll automatically removes it. Now unlike X-Protect its signatures are directly embedded in its binary. Kind of interesting. And as an interesting aside Apple also uses MRT to remove legitimate software it deems unsafe. Such as the recent zoom application that had some vulnerable components. So giving a lot of power to Apple here for them to arbitrary remove files from your system. Now since MRT like X-Protect uses signatures it's equally trivial to bypass. If we dump the MRT binary because again the signatures are embedded directly in that binary we can see all the signatures including one for fruit fly. Which makes us wonder is this going to detect our repurposed fruit fly malware. Well if we examine the embedded MRT fruit fly signature we can see it's detecting fruit fly based on both the path of the malware and its launch agent P list. This means as long as we change either the path or the name of the malware it's not going to detect it. Like I said, trivial. Now I also want to briefly mention bypassing third party antivirus products because you might run into a system where the user you're targeting has an antivirus product installed. Now there's been a lot of talk about this so I'm not going to dive into it a ton but suffice to say traditional AV products that largely rely on signatures which is pretty much the norm especially on macOS are equally trivial to bypass. For example I took fruit fly and recall it's a pearl script. I ran it through a free online pearl obfuscator which caused detections on virus total to drop all the way to zero. Now that works but you might want to use more sophisticated capabilities especially for the binary based malware so you can do things like pack them, use a custom encryptor or even in memory execution. Now I gave a talk a few years ago about these techniques. I recommend checking that out because that can ensure that even if the user has a third party AV product installed on their system it'll be unlikely to detect your repurposed malware creation. Now for mac users this kind of sucks right? We showed that advanced adversaries are already using repurposed malware samples and that is actually not that difficult to do. Oftentimes we just have to switch a few bytes or change the command line arguments and now you know power to the people we have the ability to take these very sophisticated threats and redeploy them for our own surreptitious processes. Also the majority of detections, those built into macOS or those provided by third party AV products fail to detect these repurposed threats. This in my opinion is rather lame because if you think about it this repurposed malware is basically an exact copy of the original known malware. We've really just slightly reconfigured it. So I want to end this talk, kind of switch gears we'll put on our white hats now and talk about how to generically detect such threats. So first and this is a well known fact that I hopefully just illustrated. Detecting malware based on signatures bound to fail. So instead what we need to do is look for unusual or malicious behaviors. By definition malware does unusual or abnormal things. So let's leverage this fact to discuss generically detecting even repurposed threats. Because again as we just stated, if we are an attacker repurposed a known piece of malware, the underlying functionalities, its behaviors and capabilities aren't going to change. That is to say even a repurposed piece of ransomware is still going to encrypt all your files. A repurposed backdoor is going to persist and then start exfiltrating your files to a remote server. So here on the slide I've kind of listed some capabilities or some actions or events that are good to look for and monitor because they may provide insight into generically detecting even repurposed malware. So let's start by talking about persistence. What I mean by persistence is the way that malware installs itself so that it's automatically restarted every time the system is rebooted. This is something the majority of malware both on Windows and Mac strives to achieve. So what we can do is we can simply monitor the file system specifically observing persistence locations and then if anything modifies those we can alert the user or assist admin and say hey look something just persisted maybe you want to take a closer look at it. So here for example we can now alert and generically detect when wind tail either the original or the repurpose item persists. Next up is mic and webcam access. A lot of Mac malware accesses the webcam or the mic to spot to spy on the users of an infected system. So to detect this subversive activity we can register with macOS. It turns out there is a callback you can register that anytime anybody accesses either the mic or the webcam you will programmatically get notification of that fact. So this will allow us to detect malware even something like fruit fly which we've repurposed or an exploit like zoom anytime it accesses the mic or the camera. Next up we have key logger detection. A malware often times tries to capture your keystrokes to steal things like passwords and credit card information. On macOS the easiest way to capture keystrokes is by registering something called an event tap. What we can do is we can generically detect these taps by again registering for a dispatch notification that will be kindly invoked by the operating system anytime a new event tap is installed. This allows us to very trivially detect key loggers even if they have been repurposed. We talked a lot about fruit fly. This is one of the more sophisticated feature complete malware. One of the neat features that has is the ability to remotely interact with the UI via synthetic or programmatic mouse and keyboard events. So the attacker can move the mouse, click on buttons or even send programmatic keyboard events to an infected system. So this is something we want to detect because this isn't something that should normally be happening on a mac system. Luckily it's pretty easy to detect. We can install our own event tap and monitor for mouse activity. When a mouse click is generated we can then examine a variable, a state variable which macOS sets to one if the mouse event, the click or the move has been programmatically generated. So this allows us to now generically detect synthetic clicks even in repurposed malware. Now at the start of the talk I mentioned I was the creator of Objective C. This is my kind of personal Mac security web page. But the main goal of Objective C is to provide free and largely open source security tools for end users who are using Macs. Many of the detections we talked about are realized in tools on this site. So if you're interested in that hop over to ObjectiveC.com. Now a downside of these tools is they're all separate, right? There's an individual key logger detector, there's an individual firewall, there's not like an all in one solution. So when I started working at digital we decided to combine these into a single all in one tool. And the idea was to monitor the system and then create rules to detect unusual behavior. We then took both those pieces and fed them into Apple's built in game logic engine which would do all the hard work for us and would evaluate our rules on top of our data and spit out a result for us. So now let's talk about some of these behavior based detection rules. And it's important to note you can use these in other security tools as well. So this is not specific to our product per say. So using this system we can generically and rather trivially detect all the repurpose malware we talked about today. So for example, Wintail. We can detect its exploitation vector by observing the fact that Safari downloads and then auto extracts an application. That same application then triggers the registration of a custom URL handler. And finally then that same application is started automatically via custom URL request from a website. So detecting the combination of those events gives us a generic capability. Both against Wintail or any other malware that utilizes this same infection vector. Similarly we can detect fruit fly at install time by observing various events that cumulatively indicate something is suspicious or at least amiss. So here for example we observe the fact that a hidden non-apple binary has persisted as a launch item. There's no legitimate reason that this should ever happen. So that's really shady. Now imagine that fruit fly is already running on the system. We can also detect this by observing runtime behaviors. For example fruit fly does things that are really shady and suspicious. For example as we mentioned persisting and running a hidden process. This hidden process in some scenarios will drop an embedded unsigned binary to the temp directory. Execute that and then start capturing or generating synthetic events. Clearly this is incredibly shady. Now of course we don't have to trigger when all these events occur. We can look at them either cumulatively. We see one or two we might want to take a closer look at the system because something might be amiss. So hopefully I've illustrated that by using these behavioral heuristics we can generically detect even sophisticated MAC threats even those that have been repurposed by advanced adversaries or by Patrick. Now before I end this talk I'm really excited to announce Objective by the Sea version 3.0. Some of you are aware Objective by the Sea is a Mac security conference that I organize each year. This year it will be back in Hawaii. So if you want an excuse for your company to send you to Hawaii in February go to ObjectiveSea.com. Well that is a wrap. I want to thank you all so much for attending my talk. We have about two minutes for questions and then I will also be off to the side of the stage if you want to come up and chat and talk nerdy about Mac stuff. So again thank you though for really attending. I really appreciate it.


Hey guys! I wanted to share some happy mail with you that I got from my dear friend Lindy Lesserd. She's a very sweet lady and I sent her some things. She's had a lot of tragedy in her life especially recently and I really felt for her so I sent her a little something and she sent me something back which Lindy you didn't have to do but I thank you. There's a very sweet note in here which I won't read on camera but I wanted to share with you some little bits and pieces that she made and that she sent me. Some really great stuff. Look at this. I love that and it's all signed and everything. I'm going to put that in my tag box out in the other room. There's also some really great blue trims. She knows I love blue and look at these dye cuts and chipboards. There's an inchy, half inchy, I don't know what you call it. Some plastic paper clips. Really great bits and pieces. I can't wait to use them. Some mixed media. I love this tag Lindy. I don't know where you got, oops, tags this color or if you made it or I don't know but this is the coolest thing. A really pretty couple of doilies. Yeah, there's two or three of them there. Some beautiful hand-painted and embossed papers. That one's signed and dated. I love that. I have to go in my tag books and I love this. I love this card. It's a flash card but look at that penguin-eth attitude. Don't you just love him? This really, look at this. Can you all see that texture on there? It's the coolest thing. Gold paste, green and snow cone gelados. That is cool. Then this little mini composition book. This is like the one Cat Hand does her mixed media morsels in and I don't actually have any of these little teeny. This is cool so thank you and I need to catch up on the mixed media morsels because I started doing them in a different journal and I never finished. Now I don't know what journal that was. I started them in so maybe I should just start over from day one and do them in here. Thank you so much, Lindy. I appreciate it so very, very much. You didn't have to do that but you know we all love you and we support you and if you need anything we're here for you. Oh, she also sent this pen and it says, she says, sorry I didn't find a teal pen because she knows I like teal which is fine. I like blue. This is a pretty blue. It's the Inkjoy PaperMate pen which is a favorite pen of mine and it's the note is on here with some washi tape that says dream so I'm gonna leave that and I'm gonna take that off carefully at some point and use it on a page. So thank you so much, Lindy. I appreciate it so very, very much. A few guys would like to send me happy mail. Join one of my Facebook groups. Support my channel by shopping in the Etsy shop or just sending a tip to my tip jar. Any of those things check out the video description because all the information is down there. Alright that's it. The most important thing of course, go out and have a great day. Do something nice for yourself because you deserve it and I'll see you later. Okay bye guys.


Okay guys, we are going to start our little small video series. I don't know how long it's going to be or how many episodes, but we're going to start it today. So this is Simple Journaling with Art Foamies. So full disclosure, I am a designer for Art Foamies and I do have my line of stamps and things over there. I will include a link to their website down below and you can search by designer. There's some really great folks over there. I think I'm going to stick to three of my designs. That's not three, this is three. Three of my designs today and yeah, we'll get started, shall we? I'll be back. But I used three of my Art Foamie stamps to create the page and that's the finished page. I used this one in the background behind her head, a sort of a halo type of shape and then I used this one for the shape of her body. And this of course leafy one was for the background and after I stamped them I added more paint and painting marks with different colors and paint brushes. Sorry, I missed out on some of that footage, but it's very simple, it's not complicated and yet a lot of fun and a great way to create an art journal page. Art Foamies has a lot of designs by a lot of different people. Go check them out. Don't forget to like, share and subscribe, please if you will. All right, that's it for today.  Bye guys.


So hello, everybody. Sorry for the delay. We had a slight mishap with our presentation Thank you for joining us today We would like to show you how we at SAP are using open stack at home in our shop to service our customers My name is Martin. I'm here with Andrew and Michael. We are all engineers working on our open stack cloud To give you a little background SAP is a software company. So we are like all other software companies currently in transition We are moving from shipping our product on a CD to our customer To a deployment model where we run the application and the customer is just a user using our applications in our cloud and This is where our part the converge cloud actually comes into play So we are what we would call an internal cloud So you can't buy infrastructure services from SAP, but our payload is not development only or in-house applications only, but it's a massively external facing business from SAP. So for us Uptime is a very important factor We are currently Planning to be in 15 regions worldwide nine of those are already up and running and we are shifting gear around the world very quickly To manage our data center build-ups For us performance is very important So one of the key factors is that we want for performance reasons as many as Open-stack services as possible in one network so that we don't have to go over multiple hops and lose bandwidth in between We also want to give our internal customers or our business units a large set of services they can use So we are not limiting ourselves to Just the classical computer and storage, but we also want to use advanced network services like load balancing. We want to offer Secret store and DNS as well and also we have a lot of Services on top that we offer which we build ourselves like our own dashboard and our Billing services built on our HANA database And also we are currently working on Kubernetes as a service so that we can give containers to our customers as well a Little bit to our Scale so in absolute numbers. We are ranging roughly under 100,000 cores We are currently around 1.1 petabytes of memory and We have around 40,000 running instances and a relatively high turnaround. So we are counting around 4,000 instance operations a day meaning Create an instance do something on a deleted We do it again. We have a lot of customers using automated tests So our turnaround in general is very high as you can see from the numbers We are expecting basically a full cycle of all running instances every 10 days We also have a relatively big net growth. So a couple hundred instances a day Remain after all those operations actually a long running We Have two main things which might differentiate us a little bit from what everybody else is doing with open stack And I will shortly try to give you an introduction what it is and why we are doing it So first of all our 10,000 foot view would be we have a Control infrastructure underneath where we run our chorus and Kubernetes, which is fairly common nowadays We have been doing it for quite a while now On top of that we run our control plane infrastructure So the open stack itself our analytic services our monitoring and everything else, but our customer payload we See as a completely different entity. So we don't mix our control infrastructure and the infrastructure where customer payload Actually is running on so we are investing a lot of work into splitting those up properly The one thing We do need a scale and we had bad experience and with trying to run Our network in a way that we use overlay from end to end from device to device So we decided to approach it differently because we need to integrate a lot of services into those networks for performance reasons So we use something which is called hierarchical port binding in neutron and Those are the particulars, but in the high-level view would be We use overlay only within the network and as soon as the packet leaves Any switch there is no more trace of an overlay network. So this is a functionality Which neutron gives us where we can run at different parts of our fabric different protocols. So VMs only CV lines Storage boxes only CV lines network device only sees a villain except for the core fabric It sees the excellent packets and it gets encapsulated and de-encapsulated at the network edge level So this gives us the opportunity with very little overhead on the attached devices to achieve a relatively large scale This is one part where we look a little different And it's also one part which took us a little time to get implemented because it's not the most well publicized feature and the second thing is as I already said we have a strict split between the control plane where OpenStack actually runs and the data plane and We are concentrating our efforts mostly on maintaining a lot of low-level drivers where we can use OpenStack's orchestration to remote control already Installed equipment in a data center or equipment which we have been comfortable with running at scale for a long time already. So Imagine an large enterprise which has been using these centers for the virtualization for ten years It's a huge benefit if you don't introduce a second hypervisor another operating system where it runs on and a new orchestration layer at the same time, but you can leverage your existing operations infrastructure to basically orchestrated in a more seamless way for the user. So as you can see we are running our L3 through hardware routers. We are running our L2 through through a Cisco ACI fabric. We are using established hardware load balancers. We are using V-Centers and ESX servers as our virtualization platform And we use Cinder and Manila to drive our various storage backends And so basically if you Imagine that we can this allows us to treat the SLAs and SLOs for our two parts of the cloud completely differently So we can define that we don't need to care about the Kubernetes run OpenStack site that much if we can ensure that the customer traffic survives an outage in any of the OpenStack controls So if the dashboard goes down for 10 minutes it's less of an issue than if an external facing application suffers a 10 minute outage and To give you a little more insight on how The left hand box works. I'm handing over to Andrew Sorry. Hi, so I'm gonna focus in the top right hand corner, sorry on how we actually deploy the The OpenStack and our own components on on Kubernetes and and how we get it running so There's a slide that if anyone's been to any presentation on OpenStack, this is probably already well known but we're using Helm to deploy and Helm is a package manager effectively for Kubernetes and the package in this case is called a chart and it's providing tooling for install update delete and It allows us to compose a OpenStack deployment based on a series of dependencies so a release in in Helm is a Version deployment of a particular chart and in our case that reflects as an OpenStack region and we can go with Kubernetes and sort of split control data plane from a empty Kubernetes infrastructure to a Running productive environment in about five minutes and we can do pretty much zero downtime Subminute possibly sub 30 second updates So what we've created is a set of charts that allow us to Deliver a group of regions Which are pretty much identical in the top? right hand corner again here And all we really want to differ in those regions are the things that make that region unique so it will be things like endpoints The definition of the hardware passwords the rest of it the functionality is identical across the region So what we do is we create a region chart Which just has a dependency to a meta chart this OpenStack Helm and the region values that we push into Helm that generates the configuration at this level are just passwords endpoints domain names hardware configuration And we rely on the next level down the the OpenStack Helm chart to allow us to deliver a predefined set of unique of a constant function OpenStack deployment below that we then have the individual components So we have Nova Neutron Cinder plans whatever we're putting in there and each one of those then has a set of default values That tell it how it's going to behave So really the the behavior is fixed through the chart structure and then below that we have Infrastructure components like Postgres, RabbitMQ etc and a typical chart would look Like this this is Nova so we have deployments in Kubernetes for all of the Processes that we need to run we have a an ingress and service for all of the Services we need to expose either internally within the cluster or externally through the API or as an API and the things that are slightly different from some of the other sort of OpenStack deployments that we've seen are the the way we're handling the the DB migration and also the way we're Using operators in Kubernetes to scale our hardware. So in terms of the DB migration we we have a job that runs and That runs the migration and in Kubernetes that will stay there until we delete it if we need to upgrade the database We delete the job run a deploy again, and it will it will deploy the DB and for the Compute nodes and any hardware scaling that we're doing we have an operator, which is basically responsible for listening for external events and creating the necessary Kubernetes constructs that allow us to provide hypervisors and all of the configuration and all of the The values that we previously showed are being mapped in as config maps and then mounted into the the containers as as volumes and then transformed into into OpenStack configuration and What we do is we have this big? chart That we just throw out to Kubernetes and it deploys everything and what we found initially was that that led to Timing problems with things like databases not been available or RabbitMQ not been available and luckily We were quite early to the game, and I think we were also joined by Stacking ATIS and they solved this problem with this Kubernetes entry point Binary which basically provides very basic dependency management so what we do is we throw everything out and we define within the specs the Dependencies that we need to keep or ensure that the system is eventually running so we start off with the databases and the Ingress and services been been running and then we wait for the migration job to start So that's got a dependency on the database obviously once that's completed the API starts up. It's dependent on the migration and and then once that's Available the the headless services the conductors schedulers etc. The compute nodes can start to Can start to go in there and it's the and then the operator takes care of scaling up the hypervisors once we've once we've done that and how that That works we have a thing we calling a v-center operator It doesn't follow the the core OS pattern exactly in terms of the the way it's Sort of defined to work in the in the documentation, but it basically provides us with operator type Behaviour so we we deploy that into the into the cluster and it's listening or polling a DNS DNS service and as our infrastructure guys build out a new v-center The last thing they would do would be to create the DNS entry for that for the API on end point Once that once that happens The operator takes over and finds that it's got something new to configure and it takes some pre-configured Configuration for things like the username and master passwords that it can generate the necessary configuration for the compute node And then it will create from a template the necessary compute and compute pods and also the config map to to get that working so what we can do is we can say Create a v-center put it into the environment and it's immediately recognized and configured and up and running And we're using that also for seeding keystone. So things like service users passwords Endpoints, etc. They're all they're all been managed and populated via a An operator and we see this we're not doing it at the moment But we see this as other ways as a way to scale the other components in the in the data plane So new load balancers new routers, etc. Would be self-discovered by an operator and configured or the necessary agents would be configured to bring them online and This is a very small but Black screen which shows the the result of the of the helm install So at the end of the end of that five minutes, this is what we're left with it's basically all of the components we've got running in the In the system and hopefully there's a demo at the end which will put this into a bit more bit more light so what we What have we learned during our two years? I think the first thing is that generally open stack works pretty well on Kubernetes I think that seems to be well recognized In the last two summits the main problems we've seen a headless services are very difficult to health check So things like conductors schedulers. It's very difficult to tell whether they're actually Doing what they're supposed to do and we've also had problems with signal handling and orchestration. So The orchestration that we've solved with the Kubernetes entry point and we're using dumb in it As a basic process manager that will handle signal mapping There's a lot of cases where some of the open stack components don't react to the the signals that Kubernetes is using to manage the podlight cycle The other problem we've had is that some of the some of the components don't They don't behave very well if you just throw them in and they're not ready for the environment that there are the environments Not what they're expecting. So for example a conductor process Tries to start if the database isn't there. It just waits. It doesn't retry. It just waits It looks like it's running, but it's not and there's no way for us to find out whether it's really working or not so there's a few gotchas I guess in the in the individual components that you have to work around and Luckily the the orchestration that we're using is doing that The other thing that we found is that the the big monolithic helm chart that we've created doesn't really scale very well not in terms of technical scale, but in terms of the way people are Interacting in it. So we started with one or two people building it and we've now got a team of maybe ten people interacting with the helm chart and the question of is it safe to deploy is asked probably daily and so what we'd Recommend I think is looking at and what we're going to try and do is to try and break that out into individual components The reason we didn't do it in the first place that was that when we started helm didn't really Help us to do that and then but there's been a lot of development a lot of features in the last few months That will hopefully solve that problem for us. So we're looking to split that up the other thing is the is the monitoring from our previous life we created an open clone or It was actually developed in parallel and one of the things we didn't do right at the beginning was integrate monitoring We put it in at the end and it was not not great so the thing we've We've act we're accepting is that Things are gonna break. There's a lot of moving parts and it's very complicated to understand what's going on And with the nature of Kubernetes is those processors are gonna stop. They're gonna move around they're gonna They're gonna start up in a way you didn't expect so we've We're trying to use a lot of the the sort of core capabilities of of the Kubernetes Seeker system particularly Prometheus to to scrape out a lot of metrics and we've Integrated a lot of middleware into or some middleware into Open stack that will allow us to export via stats D as many metrics as we can find within the open stack components And we've also got some similar middleware that's doing the exception reporting for us. So we're not we're not log parsing where actually capturing the exceptions as they occur which has helped us significantly I think in terms of problem-solving We're also using a lot of canary tests so that when we make a deployment into a region We've got both data plane and control plane checks which very quickly alert us in our slack channels whether or not something's gone wrong, so we're Yeah, we're very We're very keen on monitoring as much as we can the in terms of open stack there's Where it when Martin said we're an internal cloud We're trying to be an internal public cloud so we're trying to make it as easy for our consumers to consume the cloud as possible So we give them a self-service portal a little bit like any public cloud dashboard and we controlling things through things like quotas access control and we're setting up administrative domains in in Keystone and We're then restricting what people can do through policies now all of those at least from our perspective are very niche features in OpenStack because every time we try to change them It's taken a lot of effort to make them work and Those things for us are really really important in terms of being able to restrict what people do or allow people to do things and to control How much people can use? The other thing is and it's related to that is with it We we believe that open stack needs a proper UI for end users horizon We couldn't really give it out to our end users. They wouldn't They wouldn't use it and it doesn't support most of the features. We need to control it. So things like quotas policies domain support they're all very sketchy I think within within horizon and there's no self-service for people for getting people on board and for allowing people to access requests and for letting administrators process those requests. So we've built a dashboard which we call Elektra which Provides pretty much all the features that horizon has but then all of the things we need to allow our users to use the system and What we'll do now hopefully if we've got time I think we have we've got a very short demo which Really talks about that all shows the the effect of the control plane data plane split So hopefully this will work I'm not sure how I make it play So it's going to demo a very simple scenario of Taking down some elements of our control plane. So this is our dashboard here and we've got a couple of VMs which are running a very simple web application which we'll show shortly and We've then got a load balancer, which is basically load balancing between those two servers and This is our website which is very close to Michael and my heart And as you can see it's working Everything's running and if we look at our dashboard here everything's green and and rosy So what we'll do Now is I'll set up a ping to the two well to one of the VMs and one of the and to the load balancer endpoint Sequences have been shortened. So you might notice some skips in the sequence numbers and this is our our control plane that we showed earlier and what I'm going to do now is delete neutron so or delete all the neutron components from from our control plane and And because of this control plane split now everything's still running through the hardware you can see the monitors are starting to Are starting to go red and if we look on the control plane now will see hopefully all traces of neutron have been expunged and if we now go back to the look at the monitors, I think the API is going to go down shortly and It's unfortunately showing the one thing that we're keeping in our data plane or in our control plane Which is the metadata service we have some plans there But we haven't implemented them yet and the API and the LBAT API have gone down for neutron now but our valuable mustache website is still hopefully working and As you can see the ping lived throughout the the downtime and then what I'll do now is just do a helm upgrade Which will now make it so again. So that Predefined release is now going to be applied and hopefully neutron will come back It'll come back up. So that's the helm deployment finished. It gives a nice report of what's happened and if we now look back on the On the pods we'll see that neutrons hopefully back and running for us. So we're back into a Healthy state and if we now go back to the To the alerting screen we should hopefully see that those alerts are going to recover Hopefully that's it. So that's how we're I guess managing a very relatively complex open stack deployment across multiple regions and how How helm is helping us to do that with Kubernetes and Now hand over to Michael. He'll tell us what's happening next All right, so I want to talk about what's next for us with converge cloud Andrew as you heard talk mostly about we how we are leveraging Kubernetes to help us operate our open stack and Now that we have open stack running. We're actually thinking about how we could use open stack to Make our Kubernetes better But first I want to take a small step back and tell you how we're actually installing those Kubernetes control planes When we start out we get completely bare data centers and there's nothing in it We don't have any infrastructure nothing and So we started to build out our own little machinery to actually install those clusters on bare metal machines and I went to the talk yesterday about the Kubernetes club sandwich and The speaker was actually asking why would you want to install Kubernetes on top of open stack? If you could just as well put it on bare metal, you should be in a happy land, right? But my answer to that question is it's not really a happy land Kubernetes on bare metal is just as hard if not harder For starters, we need to build that infrastructure It might be easy if you install Kubernetes on your raspberry pi and you just type in the commands but if you're looking at building up 15 regions and each region consists of 10 or 11 hosts You're in a bit of a trouble so we started to create that infrastructure and You get to a point where you're thinking am I building ironic or should I maybe spend my money on some product and buy this thing so It's a bit sketchy there and we also figured out that we're really missing the under cloud in Kubernetes There is no load balancing which is native to bare metal and we have to figure out a workaround to actually expose our services And there is something but it's a bit of a stepchild. They are using these external IPs We don't have any volume management natively we also have to go back to our vendors and ask them to have Some kind of intermediate component which brings us volume management into Kubernetes or we have to do it manually as we do at the moment and Also the whole networking part in Kubernetes on bare metal is an interesting puzzle with which is left for the interested Admin to figure out or you have to go down to the marketplace and talk to a vendors what they have to sell you So we actually implemented something called we call the cube parrot. It's a BGP Speaker, which is Kubernetes aware and so it talks to Kubernetes and it talks to the underlying infrastructure And it tells the whole system or the then networking components How the Kubernetes networking is supposed to look like but you have to do that your own on your own and Finally Kubernetes is not How do I say that it's not really supporting the bare metal use case that well Google themselves. They are running on GCE and that's their main use case what they are driving and we often run into edge cases where we are thinking like How is this even working is anyone else using this and that's just the reality We are seeing with this whole bare metal thing. So how do we get out of that? trouble The first step we are taking is we are starting to manage our Kubernetes clusters to with Kubernetes So this picture actually looks like a no-pray now. Why are you not are you not doing that in the first place? And when we started we experimented with Self-aware Kubernetes clusters, which could manage themselves, but all of that stuff is getting quite complicated So we are now finally came to the conclusion that we are just managing kubernetes with kubernetes instead of building up our bare metal IPXE ironic clone further That also allows us to use a unified tooling So we're also going to use helm to actually install kubernetes themselves and All the auxiliary systems that we have rounded like the Prometheus servers which manage this whole stack and Some extra components that help with operating the the open-stack cluster But ultimately we're still missing the under cloud The interesting thing though is that now that we have that whole machinery and we actually have a running open stack We came to the idea. What if we take all of this and we stick it on top of open stack? Then we're gonna end up with something like this instead of bare metal. We have the same cluster on open stack and We try to reuse as much as we can from our bare metal infrastructure We can reduce all the the hacks and the bare metal edge cases that don't really work And we can recycle our own procedures and stuff and actually give this thing to our customers So we are thinking about or we are actually implementing Kubernetes as a service based on this concept and we are reusing the same Principles and mechanisms and tooling and software and everything which we have for building up the kubernetes clusters in the first place And what's really cool about it is that what Andrew showed you with the control plane split is also now happening for our customers So if they set up load balancer in kubernetes, it actually ends up configuring An f5 load balancer through albus and it gives you the same control plane split as we are leveraging for open stack itself The we also have cinder volumes then there is native support in And the open stack cloud provider for kubernetes. We get auto provisioned volumes right from cinder we have Native new to networking also in the cloud provider what we intend to use is a flat network where kubernetes just talks to Neutral directly and sets up static routes for the pods From here we then thought what can we actually provide to our customers to really super charge those clusters Now that we have open stack, maybe we can do some cool features which no one else can do And i'm going to show you like two examples that we are implementing at the moment And one of them is an open stack native increase controller To give you a bit of context increase the word fell a few times already It's an l7 reverse proxy in kubernetes and it allows the users to Easily define an increase point for their services Like if you look at the bottom left That's a increased spec and as you can see there's a host name in there and a service name And what's happened what's happening in the background is that kubernetes will pick up Whatever increase controller you have deployed and make that spec a reality There is the most common increase implementation is based on engine x So what's actually happening is that when you deploy that you get a bunch of engine x servers Which listen to the kubernetes api and reconfigure themself depending on the users put into For gce there's a native implementation where the increase controller actually talks to the To be to be under cloud and sets up the load balances And since like a week or two ago, there's also an increase implementation for aws Which our colleagues from cora is provided And our intention is to actually implement this increase controller for open stack And so that we can all use it and what would ultimately ultimately happen is that the spec on the left will be translated to Will be configuring a native hardware f5 load balancer, which gives you like Hardware tls termination and enterprise grade load balancing We're going to connect it to designate. So all the dns names are already automatically being set up So you don't even have to worry about how the keystone cloud sapc name gets into the floating ip of that load balancer And one big problem that we're seeing is the certificate management for all these applications For our open stack, we actually have 30 certificates per regions And give that times 15. That's a lot of certificates to manage So you need to have some kind of automated process to renew those tls certificates That's something we have already and we intend to just stick it into this controller as well So that our users get this whole nice workflow right out of the box and don't have to worry about it at all anymore So that's one example the second example how we can supercharge those clusters Is about the gpu demand that we are seeing Just like everyone else. I suppose we have this machine learning hype in sap And our business units are starting to request gpu resources from our cloud The use case we're seeing most commonly is a tensor flow on kubernetes So for us, that means we have the problem that we need to mix and match VMs and gpu resources for our customers And also here opens that can help us to actually build this up quickly We're just going to use nova and ironic And for our orchestration layer on top, it doesn't really make a difference whether it's a VM or whether it's a bare metal gpu box So with those two examples, I'd like to conclude the presentation And you can find all this magic source actually actually on github Most of it is open source And you can see find it in the organization sapcc for converge cloud With that, thank you for your attention. And if there's any questions, we're happy to help you out Or is there something option can do better to accommodate your use cases or scenarios? Yeah, a lot of it isn't actually Open stack code. I think there's a lot of little bug fixes which We keep trying to get ground to putting upstream, but it's a very hard Long process a lot of the stuff is just completely standalone or it's things like the helm charts which are You know, then there is a open stack helm In open stack now, but there wasn't when we started and it's it's uh, it's very similar, but not quite the same and I think The actual changes that we've made or need to make to open stack there Largely in vendor code and we're working with the vendors in our data plane to To sort of upstream some of the requirements and the edge cases we're finding But there isn't actually that much that we're changing or need to change within open stack apart from You know, there's things like we're horizon. We just We just found it was going to be just too hard to to change so we just wrote our own because it was going to be quicker So I think it's not really Um Yeah, it's not really things we need to change upstream I think there's a few things that we'd probably like to maybe discuss with people and You know in the longer term, but it's not it's not something that we need to do I think at the moment in terms I don't think it'd add any value to Upstream our ui. For example, it's there and people can use it And if we want people to collaborate with us, they they can do until we get some traction Were there any particular challenges with nfs as a service in your environment? Not really Or not I mean to be honest. It's not one of our heavily used services So we're we're seeing that as we scale there are issues coming with some of the You know some of the edge cases that we're finding but I think I'm not aware of any particularly, you know, where it's it's basically playing manila with With a net at back end and so far It's it's working pretty well, I think Is there anything we should be looking for or Just wondering that's all Because there was not necessarily mentioned in manila Yeah, no, I mean there's there's a whole load of we Unfortunately, we probably don't have time to go through every time There's a whole load of things we we would have liked to have gone through but yeah, I mean We haven't really had any Apart from the things I mentioned from an open spec open stack perspective We haven't really had any sort of major things that The Concernings and a lot of that I think is is because of the way we're pushing it Into into the sort of more traditional Hardware space, so we're really just remote controlling The hardware and there are issues and there's things we're working with You know some of the vendors with to solve but they're not I don't think most of them are unsolvable problems Two more questions, right? So Did you settle on ironic for bad about the positioning or I didn't get your conclusion on that part Yeah Other than that so the recent open stack on this one, there's a lot of deltas are on OVNs And people Hardware acceleration for those things If we had to redo this, would you have reconsidered any aspect of it not working for the networking part? Yeah, I mean there's always things you would change but fundamentally no actually it's still from our last iteration where we had You need to come back Where we basically had we're in a situation that We had our own platform doing stuff like this in Completely our own way and we were modeling upstream APIs from amazon to give to our customers And there we already did a lot of orchestration of the back end and We decided against Some overall arching sdn seam We tried to make it simple because in the networking part when you make it complicated It doesn't scale anymore and it's not operatable anymore. And so we tried to Make the orchestration Let's say more complex But keep the actual on-device configuration as simple as human possibly humanly possible to keep it operatable. Otherwise If you have basically if you need to trace a packet through five different software switches and try to find where the Thing gets lost that's You're doomed to failure or you're having a very hard time to operating that in the long term. So the general concept is still I think Something we don't regret yet. I think I think the other thing is we're a very small team where I don't know maybe Even without our extended team less than 50 and I think leveraging our existing support for The hardware-based stuff is really helping as we it's a problem We can sort of Not worry about if that makes sense or that we have colleagues in other areas of the of the organization that can handle the support Without necessarily needing to know that it details of open stack or any You know any specific implementation All right, thank you folks. Thanks