So we are here before we invoke function B. And now if we call into function B, we first place our parameters we want to have for function B on the stack. And then we also place the return address where we want to continue execution after function B also on the stack. So if a buffer overflow happens, you can override this control information. And if you override the control information, you can execute arbitrary code. But only if you do it carefully, if you just do it blindly, you will just crash because you override this control information and the program doesn't know what to do with that. So this is a little graphical representation. So if you do that carefully by first adding a knob sled, so knobs are instructions that don't do anything, not a no operations, and then have your exploit code in the return address and take care that you override the return address that is stored on the stack, then you might jump back to your knob sled, fall through to the exploit code and then run your exploit. That is not the only thing you can override. So RIP and EIP are registers and CPUs that store the currently executed command or the address of the currently executed command. And you're not only able to override this, but you can also override, for example, function pointers. So if you use a function pointer in your code and I'm able to override it, the next time you invoke this function pointer, I'm in control. You can override exception handlers. So an exception handler also has an address stored somewhere on the stack.  Once again, I'm in control. Sometimes you don't even have to override these control information. Sometimes you can just override some application-specific data.  If you're able to write whatever represents true into that field, then you might already have one. So what can we do against these problems? The simple solution would be just use Java for everything and we're done. But based on the reactions, I think you also know that that's not really helpful for happening. First of all, we have a lot of legacy code that we will continue to run. So we can't just port everything to Java. Also, I don't like Java. And the next thing is that for some applications, you just need low level access and that usually requires you to have some ability to work with memory directly. So let's dive into a simple 32-bit exploitation example. So we basically have a main function that calls a function that is helpfully called vulnerable. It has a statically sized buffer and it reads too much data into this buffer. And we will now exploit that. So that is why I mentioned maybe come to the front if you have bad eyes. I'm not sure how readable this for you. It's readable for me, but I'm standing here. So if you can't read it, just come a little bit closer. I mentioned that we have some exploit code or some shell code that allows us to run on commands. So in this case, we have a shell code that adds a new root user to the system. You can either craft that by hand or you can use a tool like MSF Venom. This is a tool shipped with Metasploit that helps you to generate shell code. So this nifty thing here is the shell code. That is what we actually want to run. Then we generate a knob sled.  It's usually helpful if you have more space than you need for your shell code to prepend a knob sled before the shell code because then you don't have to aim very carefully. You just need to land somewhere in there. And the knob sled will not do anything so you fall through to your exploit and then it runs. The final thing here is that with this line, we set a static address where we think our code will be and where we want to jump. And this only works without any mitigations. And then in the last line, we just put everything together and run it. So we jump to the next window. Currently we don't have a new root account on the system. And now we will run this in GDB. GDB is a debugger on Linux. I use a little extension called PETA that's helpful for exploit development. You will see a lot of colors right now. So let me cover that briefly because we will see that a few times. Up here. Oh, okay. That is not great because my screen is actually at a different resolution apparently. Okay, I'm sorry for that. The problem is that I have my screen here for a set for the exact same resolution as the projector but for some reason you'd see something differently. So you should see a little bit more just two characters but we'll have to deal with that for now. So here we have the registers that are currently used in the processor. Registers are small variables in the CPU. They can be accessed very quickly and they are used for various calculations. Down here we see the current assembly instructions. So where we are currently executing and this is the command that will be executed next. Here we have the stack. So the structure we want to overflow and if we now continue with that. So every time I press a key we will step one step further down here and potentially something changes here which will be indicated in red. We step through and at this point you now see that we are at the read instruction. So if you remember the code that I showed you before at the read instruction we will cause our buffer overflow because we read in too much data. So if you look at the stack you don't have to remember what's exactly here but if we step over the read instruction in the next step you see that a lot changed and you also see those 90, 90, 90 hex values so this is our knob sled where we actually want to land. Stepping a little bit further we are finally at the read instruction which returns control to the previous function and the read instruction will take the top most stack value and start execution at this place. And Peter has this helpful feature so it shows me directly where it's pointing to so it's pointing directly into our knob sled. So if I step one step further we are right here in our knob sled and if I continue execution then we will see that we now have a second root user on the system and we're also able to log in here and have root privileges. Okay, so that was the good old days. Exploitation was really easy or the bad old days depending on which side you're on. The first mitigation we will talk about are stack canaries so you see a little canary here. The name comes from canaries that were used in mines to alert miners of the buildup of deadly gases and stack canaries basically do the same for software so it alerts you for stack overflows. The general idea is that the compiler generates some extra code that puts a so-called canary value at a certain location in the stack frame and before we return from the function so before this red instruction that we just saw we can check if this canary value is still untouched and if it's not untouched we can bail out. We have three different canary types. There are so-called terminator canaries. They consist out of a null byte, a carriage return, a line feed and minus one and the reason behind that is that you're usually not able to write those bytes because they are not really friendly to string handling functions. Then there are random canaries so you just generate a random number, write it there and the rationale is that the attacker doesn't notice a random value so he can't write it and then there are so-called random XOR canaries so you use a random value and XOR it with certain data on the stack that gives you a little bit more entropy. We have not three as it stands here but four different variants in GCC so that's the joy if you change your slides right before the talk. We have fStackProtector that will add this code only for functions that add buffers of eight or more bytes on the stack. Then there's fStackProtectorStrong so after a while people notice that fStackProtector and this heuristic was not good enough and there were several other criteria that causes this checking code to be added. I will not go through all of these to save a little bit time but in general if you have for example an array you will get this checking code. Then there's fStackProtectorAll it will just add the code everywhere that is not very great for performance and then there's fStackProtectorExplicit there you can annotate functions where you think you want to have it and then GCC will add the code. So just a short reminder of the example code we have our statically sized buffer and we put too much data into the buffer or we are able to put too much data into the buffer. This is the assembly listing of this original function. If you don't read assembly it doesn't really matter because this is the protected code and now I will show you what changed between those two versions. The first thing that changed here is this snippet. Here we take the canary value the known to be good canary value and place it into a register and then we take this register value and write it onto the stack. So here we place our canary at the point where we want to check it later and then here after we executed whatever we wanted to do in the function we get our canary value back from the stack and then we XOR it with the known to be good value and XOR has the interesting property if you XOR two values or the same two values together then you will get zero. So then in the next line we can check if we have zero if we have zero we jump to our normal instructions to leave the function but if we don't have zero then the stack canary was touched and then we will call this function called stack check fail which is a pretty descriptive name that then bails out. So another demo time. Hopefully you will see what I wanted to show you. So this is a binary compiled with the stack canary it's exactly the code that I just showed you before. If we run it and give it 12 ACE as an argument then everything is fine because we have 20 bytes of buffer. We do the same thing with 32 bytes then it fails and we get this helpful message that stack smashing was detected. Can again look at this in the assembly. So if we run it here we are right now at this place where we take the stack canary from this predefined memory location move it into the register and it will be placed into RIX this is unfortunately not visible for you but you will see it soon. So this is the stack canary that we have now for this execution and if we now continue with our execution we are now at the place where we check the stack canary so in the next step we will take it from the stack back into RCX. So right now we have our stack canary that was on the stack back in RCX and if you noticed the stack canary we had before did not end in 41 and 41 is the hex code for in capital A so apparently we overwrote part of the stack canary. In the next step we will now XOR the current stack canary with the known to be good stack canary and we should receive zero in ECX which we don't because they are not the same and now in the next step we will not take the jump because we didn't get zero as a result and we will now call the stack check fail function and if you continue then we would actually see it if the screen is big enough up there. So this is the interaction I was talking about. There are several limitations. Do you have any idea what the limitations of this technique might be? Yeah, performance definitely is a limitation. Okay, so I will keep asking you because you answered. You can thank those two. Yeah, yep. Yes, you might be able to predict it but in practice that is rather hard because if you have not bad random source then you should not be able to do that. It depends a little bit on your architecture but I mean especially on a 64 bit architecture is going to be really rough to predict 64 bits. But of course it depends if you have a weak source of randomness that might happen and that actually happened with some implementations. So my list is it does not protect data before the canary so you write the canary at a certain point in a stack and if you have local variables for example a function pointer on there you might still be able to overwrite this. To counter this some implementations try to reorder the variables to ensure that sensitive stuff like function pointers are also protected by the stack canary.   You can change a chain a buffer overflow with an information leak so if you're able to get the value of the canary then of course you can write it and there's also no protection for inline function so if you have higher optimization settings then the compiler might start inlining stuff and since this is all related to the function prologue and epilogue of your functions if you don't have that because the code is inline then you won't have these checks and of course you can still use it to cause denial of service. Denial of service is definitely better than remote code execution but still if you run a public DNS server wherever you probably don't like denial of service either so yeah. Okay so first mitigation three to go. Fortify source is a pretty nice one because it transparently fixes so-called insecure functions to prevent buffer overflows so for example mem copy, mem set, string copy get additional checks. I have insecure here in Attelex because there's this little tweet here. We tell people not to use string copy and then they do stuff like that so a string copy in itself is not insecure if you know what you're doing but usually you tell people not to use functions like that because there they can just forget those checks in the other case they can just forget how to use the functions correctly but yeah well. This only works for statically sized buffers or at least maybe for even more situations but the compiler needs to understand what you're doing here. So if the compiler sees you have a statically sized buffer you're using string copy it then can add an additional check but you need the compiler to understand what you're actually doing there. If you do some fancy buffer magic then the compiler will probably not understand that and will not add the checks. You can enable it with d45 source equals two and the interesting thing here is you need to enable optimization because this 45 source magic works somewhere in the optimization stage and if you don't enable optimization you will not get the benefit. We have a different example here it's based on Matthias Gastner's work he's a colleague of mine that uses that for our trainee trainings and basically we have the same structure so we call into a function it has a statically sized buffer and we copy too much data into that buffer with string copy and we have that little snippet here to prevent GCC from removing our buffer because since we don't use our buffer the compiler would say well nice buffer you have here you don't seem to use it so I will just kill everything here. So with that we keep the buffer. This is the corresponding assembly and if we now go to the assembly with 45 source the first thing you might notice that it's become shorter. The reason is that we enable optimization and therefore the compiler worked a little bit harder to generate better code and because of that we have fewer lines and we have two interesting lines here first is the compiler notice the size of our buffer and places that into edx and it doesn't call string copy but it calls string copy check so it replaced the call to string copy to a call to a different function and with that it can then check if we move too much data into that. So another demo. We are again in GDB I will first run that code with an input that is too small to cause a buffer overflow. So once again we are at the beginning of the program we have the hex 100 where the compiler notice the size of our buffer and we have our string copy function with the checking and if we now step through that we reach the part where we call string copy check and if we step over that nothing happens because the copy just worked and if we continue the program will exit normally. If we do that again and provide an argument that consists of 700 As we will now overflow the buffer. We are again at the beginning we start to step through and if we try to step over the string copy check function we will not come to the next line because the function realize that too much data is placed into the buffer and it will terminate. So empty slide again. What are the limitations of that? In general it's just limited to some functions or situations. So if the compiler doesn't understand what you're doing there it can't help out. It can still lead to denial of service same problem as before. You don't get remote code execution but your service still dies. One other problem is that developers might just keep using these functions. So they are thinking well writing all those pesky checks that's rather annoying I just don't do that. The compiler does it for me. So if you're a security guy in a company then you should probably still ensure that people don't use those functions. But it comes with almost no cost so you should enable it. We have it enabled in our build system so that every package that is built is automatically protected with a 45 source. And as far as I know we did not really have problems with that. And also the performance cost is rather negligible because you need those checks anyway. If you don't have them it's not a performance benefit you are just insecure. So the question was why do we need to have optimization turned on for this to work? The reason is that 45 source works in the stages where optimization also works. So if you disable optimization then this whole engine just isn't triggered. I can't give you the detail why GCC does it that way. Just this. Yeah, that is definitely a limitation. Should add that to my slides. Thanks. Okay, okay. So then let's go to ASLR. ASLR stands for address space layout randomization. And this is a technique so that certain memory segments like the stack, the heap or the code segment are loaded at random locations. And if you think about it computers are deterministic at least the ones that we use. And unless you have some sort of randomness in there you will have the same memory locations every time you start a program. So that's the way it used to be. That is pretty nice if you do debugging and stuff like that. But it's not great if you want to prevent security problems. Because if you have randomized locations then attackers don't know the return addresses for their exploit code or for the C library for example to jump to. If you remember the first example that I showed there I hard coded the address where my code will be and then jumped there. But if my stack is moved randomly in memory then I can't do that anymore because I don't know where my buffer will reside later on. So if you cat proc maps for a given process then you see at which places certain memory segments are. So for example for this process the stack is a dislocation, the heap is a dislocation. And if you now do that five times and just grab for the stack segment then you will notice that these numbers here change. So this is run on a system that has ASLR enabled. The stack is moved to a different location every time this code is run. You can check what settings your system has. If you check this file can have three settings. You can either have zero for no randomization or one or two it then randomizes various parts. If you have a recent distribution you definitely should see two in there. If not you should talk to your distribution. I think all major distributions have that. But if you're maybe working in a embedded space or something like that it is definitely something you could check out. To get a full benefit of that you need to compile your programs with fpy. That then ensures that the code is generated in a way that the binary works no matter where it's located. And this is rather important because some binaries assume that they will be loaded at a fixed address and then the loader actually has to load them at that address because otherwise it will not work. But if you compile it like that then the binary is generated in a way that it can be relocated wherever and then the loader can relocate it to that address that is randomly chosen. So what are the limitations of ASLR? Spectrum, yeah. Definitely. Come again please. It works but not really that great. I mean yeah. It's the, I have those points. Okay, so first point for all the systems that had a pretty hefty performance impact. Yeah. Sorry, come again. I can't hear you without a microphone. Sorry. Stack Clash, no that's not directly related. I mean you could have exploitation of Stack Clash without ASLR but you could have a situation where it's becoming easier because those segments are nearer than they would otherwise be. You have limited entropy on 32 bit systems. So at least that is what I know. I'm not sure why you think it doesn't work at all on 32 bit systems. Yeah. Yeah. And of course you can't really use all those 32 bits because you have various memory segments so you can't place it everywhere. That is definitely a limitation on 32 bit systems but fortunately not too many around of those anymore. At least where I work, if you work in an embedded space or whatever then it's probably different for you. So brute forcing can still be an issue if restart is not handled properly. So even on systems where you have more entropy if you restart your process or if you don't restart a process because you get this randomization on the start of the process but if you only restart part of that and still keep the same memory mappings you might still run into problems there. And it can be circumvented by chaining an information leak into your exploit. So that is with regard to the spectra comment. Of course if you're able to leak information from the process then you might use that to gain some knowledge about where those memory segments actually are currently. You also might have some exotic software that relies on fixed addresses. So for example if you have some inline assembly then this might be problematic. I don't see a lot of those problems but that also depends heavily on which area you're working on. And sometimes you might have some usable memory locations in registers. So you might just have a usable pointer to whatever you need in a certain register so then you're lucky because you don't have to find out any addresses you just have it around to use. So for ASLR I don't have a demo because it was pretty clear from the five executions of this cat program what it does. The next mitigation we will talk about is about non-executable memory. So modern processors support the ability to set certain writes for memory mappings, sorry. And they can say that certain memory should not be executed. Another term for that is NX or this thing I don't know how to pronounce. And the most interesting memory regions for that are of course the stack order heap because in well-written software you usually should not execute code that lives on the stack order heap. Unless you're doing some whatever writing a virtual machine or if you write normal code you probably should not do that. So the stack overflow can still take place but it's not possible to directly return into your shell code because your shell code will live on the stack. The stack is non-executable so you can't jump there and execute it. If you try it then the program will die. And when you see my italic of directly there's probably a way around that. If we remember what we saw before so if we again cat this maps file in the proc directory and grab for the stack then we will not only see the address where the current stack is but we also see some modes here and in this case we are allowed to read the stack to write the stack but we are not allowed to execute the stack because otherwise we would see an X here. So what limitations of NX can you think about? Come again? JIT? So JIT stands for just in time compilation. If you do something like that then it's going to be tricky to do that. I mean you can still work around it by first compiling the stuff and then afterwards changing at least the right permission from the memory map but you can use existing code in the exploited program. So maybe there is a privileged operation you want to run. Maybe you don't need shell code you just want to call the delete all database functions or whatever and so you just jump there and since it's part of the code segment it's executable and you can jump there. You can use something like return to libc so pretty much every code is linked against libc there are a lot of useful functions you can use so you just jump to libc and use the code that's already there. You can use return-oriented programming so you structure the data on the stack in a way so that you always jump to small sequences called gadgets that end in a return instruction and try to get your functionality just by jumping to those little small gadgets and since we'll be using that shortly I will show you that that's a small graphical representation of that so you add data to your stack so that you override the instruction pointer with this data which then causes you to jump here and there you only have one useful instruction you just pop EAX that means that you take the topmost stack value and place it into EAX so the topmost stack value will be this after the address here is removed so you will then set EAX to one and then you end in a return so the return causes the next address to be taken and control be transferred here which you then can repeat and get basically whatever behavior you need so you use the stack only for the control information you don't put code directly on the stack and execute it and this is used a lot in modern exploits so we talked about quite a few mitigations we talked about stack canaries we talked about ASLR we talked about no-execute memory and we talked about 45 source so you should probably be safe no more exploits, right? Yeah, unfortunately not I will now show you a counter example it's taken from a certain blog post I changed it quite a bit to make it work on my machine and also to make it a little bit more understandable for this talk I will leave out 45 source for this example just to be able to create an easy exploitable program but I probably don't have to tell you there are enough vulnerable code pieces out there so this is just me cheating to make my life easier it's not because you can't circumvent that so that is the code of the program that we will now be exploiting we have a main function that then calls into a function called memleak which is a memory leak so we have now here our information leak where we get the information we need for further exploitation and then after we call to memleak we call into one function that is a classic buffer overflow so we have a statically sized buffer and read too much data into that so not very complicated since we want to be able to execute our own shellcode we need to make the stack executable again so the default stack is non-executable but you can use functions like mprotect to make it executable again and we want to run our own shellcode so we need a way to call this function it accepts an address, a certain size and then some protections so the permissions we want to set on this memory mapping on the architecture I'm using here the first few arguments go into registers so they are not placed directly onto the stack and so we need to be able to set the registers and to be able to set registers we need to be able to execute code but since we have non-executable memory we can't directly execute code so we need to use Rob return-oriented programming what I just showed before for that we need to find some gadgets so gadgets are those small code sequences that end in return directly there are a lot of programs out there that help you do that I used RobGadget this just prints out a list of all gadgets it finds in this case I just checked the C library and I wanted to have an instruction that allows me to load data into RDI there are programs out there that you can give a binary and it gives you out complete Rob sequences so if you don't want to build it yourself you can just use those programs so demo time again and let's hope that it's actually worked because the last time I gave this talk it did not work which is kind of disappointing so this is the exploit that we will be using we have our shellcode running in a virtual machine and we want to be able to connect back to the hosting machine so that we get a reverse shell the shellcode is generated with MSFvenom that will connect back to my host machine in this line we will open the vulnerable function that is only the compiled version of what I just showed you before the system has ASLR it has non-executable memory and it has stack cookies enabled and first we read what the program writes and then we write something into the program to trigger this memory leak and this memory leak is triggered via a format string vulnerability format string vulnerabilities are not that common anymore but they used to be really nice because you can do a lot of stuff you have your memory leak if you want to but in this case we just use it to leak some memory then we read the memory that we just leaked and start calculating the data that we need so first we calculate the base of the libc that we need then we calculate the address of the buffer and then we extract the stack cookie because we leak so much memory that we have the stack cookie in there next we calculate the address of mprotect in the C library because we want to jump here to make our stack executable again and since this mprotect needs the arguments in the registers we first need three different gadgets to set those arguments in the corresponding registers so with the first gadget we set the value in rdi the second one sets it in rsi and the third one in rdx so then we do a little bit more calculations, not that interesting in the end we add a knob sled so that is the same as I showed you before that just makes our targeting a little bit easier and we put it all together that is the final exploit we have our knob sled we have our shellcode that we generated we have our stack cookie at the exact location it needs to be for the check then we add our gadgets and the data they will add to the registers the address of mprotect where we want to jump first and our buffer address I set up a listener on my local machine so this is my laptop and what we will see after that is just a virtual machine in my laptop and if we now call this exploit we see that this is the result of the memory leak so this is what we are getting out of this other process then we use that to calculate all the data that we need so we get our stack cookie we get the addresses that we need and then finally we send our payload to our program and if we go back now we see we got a connection we are root which is because this program runs as root and we are in the virtual machine so what we didn't cover now is quite a lot there are some other mitigations that we use in SUSE products so for example this Stack Clash Protector really saved us a few weeks ago when the system D exploit broke fortunately we enabled it for all our current products and because of that we were not vulnerable this is also one of those mitigations that are not that costly so it was really easy to employ them then there is something called Rail Row so there are relocation tables and binaries if you want to move them around in memory you need some place where you relocate your symbols those are pretty nifty targets for attackers so you want to make them read only after you do your locations it's also pretty basic there is definitely more that should be done because as you've seen exploitation becomes much harder with those mitigations but it doesn't become impossible and if you look at exploit prices then it just has the effect that exploits become a lot harder to construct and therefore they become more expensive that has definitely a positive impact on us because if an exploit is more expensive to generate then people really think about if they want to burn them just to break into your database server but it depends on what kind of enemies you have the ROP technique that I just showed you before it's used in a lot of exploits nowadays and because of that there are some mitigations that are coming up you have something called Shadow Stacks so we were just talking about the normal stack which is control information and data mixed into each other Shadow Stacks just have the control information and then you can have checks that says we have the return address here and we have the return address here we compare them if they're not the same we have a problem then there is control flow integrity that should ensure that you only jump where the programmer intends you to jump and luckily we will see that in hardware pretty soon or maybe you already have that because if you do it in software it's rather expensive and then there's something called data flow integrity that even goes a step further that really ensures that data is not changed in a way that was not intended by the programmer but that is not something that you see in standard systems that you run nowadays the main problem here is that those mitigations are rather costly especially control flow integrity integrity is something that is rather costly in terms of performance so we have a rather hard time convincing our customers that taking a 10% performance hit is worth it and of course they can also be circumvented so I mean it's the same thing every time you invent some mitigation and the attackers just get better but you make it harder every time and those mitigations are the reason why you currently don't see or seldomly see exploits just posted somewhere because if someone can nowadays write an exploit for iOS or for Chrome they can earn really big on that and because of that they don't post it publicly anymore and the reason the prices are so high is because it's really hard to do that nowadays and that is partly thanks to the mitigations that are now in those products so thank you for your attention do you have some questions in that case duck will come to you for the ASLR you said there is no performance degradation on the x6 64 bit platform is that true or is that just because you didn't mention it to go back to the limitation ASLR has not really a performance issue because you do these relocations once at the beginning of your code or before you start up your program and that costs a little bit of performance at this time but then if you run your code it's not really a factor on modern hardware I'm sorry I have a hard time understanding you I'm not sure I can follow because the ASLR only moves your program in memory and then later on you just execute your code as usual so there is really no performance impact questions so I'm not really sure how to phrase the question but I don't think the bottom line is what the heck do we do now so concrete talk you talked about some hardware features that are coming online there are some improvements in the program languages go with maybe better than Java REST is probably better than C what do you see the system software doing in the next few years say 5 or 10 years so in general it's definitely if you're able to choose a programming language nowadays and if you don't have to use C or a really low level language you don't want to do that because you just solve a whole class of problems or you don't have them there are still enough problems there for you to care about with regard to security I think going forward we will play this game probably forever we will have mitigation then the attackers will become better and in the end it really depends on what you're defending with the mitigations that you have nowadays I think we're in pretty good shape unless you're the target of a state actor or something like that and if you look at what really owns how large organizations are really owned then this is seldom the newest zero day exploit it's usually a user just clicking on a link or executing something because they got told well I'm the IT guy do that so I'm not too optimistic about the future of security because we as humans are just too trusting that is definitely something that we can think about but in my experience unless something is really horribly broken and a lot of people lose money because of that we usually don't change it so I mean programming languages like Rust they are really cool and great but it will take decades until we have replaced all the C code that we still carry around with any others alright thank you thank you testing one two the working kind of sort of maybe a little bit sort of alright no judging on non-professionals I don't do this part for a living alright stop that I have two o'clock on my watch can you get started alright so here we are what are we doing what are we talking about this talk who about me my name is Patrick Schwartz I am a in my first year at SUSE been a systems admin systems architect for the last 20 some odd years for multiple companies that's how you can find me whether it's on twitter or linkedin so what are we here to talk about today what are we trying to solve as systems admins we're responsible for everything right whether it's patches it's configs whether we have to install software account management system security tuning I mean the system admin the system architect we've got to make sure that things are right things are stable things are where they're supposed to be users the developers whoever is using that system the consumer of the system depends on the system admin to make sure it's right can we all agree on that whether it's making sure of software drift making sure the containers are safe I was reading an article recently talking about some massive percentage of the containers that are deployed today are vulnerable they're finding that containers that are finding that are not safe they've got vulnerabilities already in them and we're still deploying them even though they've got vulnerabilities so as system admins we're responsible for a lot of things to manage and today's world and I give this sister presentation of this about our commercial product called system manager our world is getting more complex not less as we have to deal with OOSs we have to work with but also the hardware platforms we're dealing with as well whether it be x86 ARM Z you name it we deal with it in some way shape or form so on the bottom line is what do we tend to do stay in control that's what we're after so the solution I want to present today is called Uooni and you're going what in the world is well I might answer that question for you it's not this Uooni although we'll give this Uooni the credit this one is the largest salt flat in the world why did we name this project after the largest salt flat in the world in Bolivia well it's because it's based a lot on salt stacks salt which I'll talk about in a minute what does Uooni do for us well like this slide reads it deploy, manage all kinds of workloads no matter where they're deployed from from a single UI the gives us the ability to automate audits and reporting capabilities and when I get into the demos we'll do an audit we'll look for a CVE and we'll see what we can do to fix that we can do hardware systems a lot of times we get out there in the world whether you've been in the job for years or you're brand new to a particular position you need to know what's out there and a lot of times we get out there we get into systems we have like oh my gosh the system's been running this for how long we didn't know to be able to maintain standard configurations and that's one of the key issues that we have to fight constantly is configuration drift right I'm going to propose it gives us that ability to maintain that and last but not least and not even this is just a short list but building the deploy physical VMs and container images also from a single interface and I think about that the ability to deploy physical systems, virtual machines or containers from your interface at the same time when you know they're deployed they're going to be deployed the way you meant them to be in a managed controlled infrastructure all right to let you know I'm not a big slides guy so we're not going to have a ton of slides we're going to get into more demo stuff is that all right so where did when you come from other than that big salt flat way down south from here originally my slide kind of cut off here but we started off with a project redhead sponsored the project back in 2005 time frame I believe called spacewalk have you ever heard of spacewalk couple so we get our origins way back in the spacewalk project well as redhead has decided that was the foundation for redhead satellite 5 great product and we took that same SUSE took the same code stream and developed our SUSE manager two and three product line from that but in the meantime redhead has moved on from redhead satellite 5 redhead satellite 6 and completely rewrote the interface the whole project has been completely redone and they abandoned or they decided to no longer maintain spacewalk and it's got an end of life somewhere around 2021 whenever satellite 5 completely goes end of life so SUSE manager or SUSE said well we need a project that we can continue on and continue putting upstream code into so we've decided to fork the spacewalk project into Uuni that will be so Uuni now will be the foundation for our upcoming enterprise product called SUSE manager 4.0 which is due out this summer with lots of advantages to that so this spacewalk origins way back here is continuing its life through Uuni and through SUSE manager 4. All updates will go into from Uuni into SUSE manager 4. So what is really we're talking about here well Uuni it talks about being an opinionated branch of spacewalk to provide a simple installation using salt how many of you are familiar with salt stack? okay good good we use salt extensively behind the scenes underneath Uuni we talk about because the world is becoming containerized and Kubernetes is becoming a huge opportunity within our world so Uuni ties into containers as well we've redesigned the website to use React UI and I'm not a web developer so that's a little bit foreign to me but anyway I'm liking what we've done to it Python 3 and JDK 11 will be the foundation for everything going forward as well as and this is right now the current supported clients open SUSE SLEZ, Sintos, REL and by the summer even Ubuntu so from this single project I can manage just about every Linux flavor out there these down here are the platforms that we support and the ones in bold are the ones that you can actually run the Uuni server on the other ones that can be strictly clients you're not going to want to run the server side on RTA to say you know raspberry pi type a little bit beefy for that but so that's what we are right now alright so just like open SUSE tumbleweed Uuni is a rolling release it's constantly being updated there is going to be there's no community versus enterprise additions it's just Uuni with a rolling release but it is the upstream like I mentioned before going forward for SUSE manager from this point forward the code bases are going up into Uuni and then coming down into SUSE manager 4.0 and on this is what makes up Uuni today like I say it uses salt stack to do all the configuration management and the provisioning with physical virtual or cloud boxes it has the ability to tie into Kubernetes, has the ability to tie into VMware and the next version will even the next version or next dot release will have the ability to tie into KVM as well we use PostgreSQL behind the scenes and to get to it typically you are going to use the web UI most commonly is the web UI however there is a very extensive API back into this you can then use any of the typical API tools that you want to use to get to the back end of Uuni and also from the command line you can do just about everything you can do from the web interface either through the APIs or the command line okay and I'm talking really fast that's okay so like I said those of you who know about salt this isn't from you so much but the others like I said we're backed by salt stack and if you want to know more about salt stack there's the github link to that salt stack isn't a configuration management tool then they'll tell you that they are a remote execution framework meaning that you can do lots of stuff to your systems including configuration management all from salt stack or salt in orchestration automation is the foundation by which salt was originally designed for Tom Hatch and I have had long talks on the foundation behind salt and he's a fascinating guy to talk to and okay so salt though it pretty much runs everywhere it's written in python it runs on Linux you know traditional flavors of Unix even Windows so it runs just about everywhere it can be ran in agent or agentless and the behind the scenes is using the zero MQ to put things out on the bus so you've got a little bit more secure method because you don't have to open up ports directly to every client nor does there have to be behind the scenes issues with clients in firewalls alright so let's jump into some demos and see if I you know did my appropriate this to the demo gods today so what are we going to talk about I'm going to try to show you some salty goodness alright we're going to start back at the command line and work our way up we're going to talk about this thing called formula with forms and to do that there's this whole functionality that I want to show you that uses salt and salt fillers behind the scenes to deal with a lot of complex installs we're going to look at some audits and see them be able to see the audit and fix it alright we're going to talk about these things called action chains and what we can do with action chains and then lastly if I have time which because I'm really fast I'm going to talk about some software channels that we can do with our in world because software the repositories are known to be one of the most complex things to deal with having to manage multiple layers of systems whether it be test, dev, QA pride, pre-pride in an SAP world and having to manage all of these repositories in a unified method without spending terabytes upon terabytes of disk space to do these rc backups of which became really challenging and Uuni gives us that ability to do that so let's see if I can do this alright is that do I need to switch that to white on black um um would white on black look better okay I forgot how to do that I forgot where that's at oh which one oh breeze okay let's try black on white is that any better alright you see that now if I move this that over a little bit and I totally left my sheet of paper back in my room alright so this is all on memory here so step one so what do we want to do here so I want to show you for example um the world I happen to do this without my cheat sheet anyway alright so salt is very easy if you get to see my bad typing I can target my systems and the first step is targeting my box and I'm going to salt first thing I'm going to do is targets and right now I'm targeting everything and I just want to do a simple test without paying to see what I've got out there and boom um that yellow is ugly though um it comes back and tells me hey all these systems are true I got one error though so this one's in blackout mode and we'll talk about that when I get to it that's their own purpose um but just like that I was able to get back uh individual things so if I want to target um and that's one of the things with salt there's a back end called grains and by default salt pulls in a ton of information about the system um so let's look at that for real quick here one new client um um grains dot items all right and I can look at this and I can see if I scroll up back up here at the top um I can see all kinds of data about my system that salt pulled in all by itself without me saying anything um told me it was running you know it's got some it's got an SSD the BIOS release version uh and every one of these things in blue I can now target on if I want to target every box that's x86 if I want to target on every box that has a domain of susan zoom dot com um or target on any you know GPU host name or ethernet address I can target on any of these things going forward and we're going to talk about we're going to show real quick here it's just strictly OS let's target on our OS so salt I'm going to do a dash G um OS colon uh SUSE test dot version let's try something different okay so I'm going to target all my SUSE boxes they came back with a grain of the OS grain equaling SUSE and sure enough I only got back this list of clients now I can start doing all kinds of stuff from that now breaking things down and doing more with these individual systems if I wanted to get even more granular I could go you know is it SUSE OS version you know 15 or SUSE OS version 42.3 I can now start targeting my systems in a much granular fashion and they all respond and what's nice about the zero MQ uh message bus is it puts the salt master which UNI is uh puts the messages out on the bus all the clients are listening to the same bus at the same time can respond immediately there's no more of this loop shell stuff of a list of systems that you've had to compile I mean how many of us have done that right we create a list of systems hoping that that's complete put it in a systems dot list file and then run our you know bash shell loop across our systems list file and it works it's going to hit each one individually in serial format and hopefully you got every system you needed to in that list right well with salt though it puts that on the message bus all the salt minions is what they're called for the clients are listening to the bus and they all respond at the same time so I don't have to wait for this individual one after another uh method to get my data back and I can compile all this data from the command line and actually port it out to a list if I want to port it out to a collection program like Splunk for example or any other log collaboration correlation engine all right so I can do that and targeting my back in so if I need to target something so that's all my targeting I can target all kinds of stuff I even I can put my own grains into the system if I want you know like you know it's in data center 2 rack 5 U32 whatever okay those can be individual grains as well you can put in salt so salt is the foundation by which a uni is built on so everything starts here all right so if I want to do something with my system and I've got a couple of simple um very very simple demo worthy um salt states as they're called then I can show you and let's look at one of them for example of root profile because let me have your own root profiles right I certainly have um root um profile and hit .sls so in this few lines of code I'm able to push down the bash profile that I want from my systems I tell it file this is where it's going to go root dot bash dot underscore profile that's the location that's the destination of where this file is going to end up it's a file that managed these are called modules with insult and there's hundreds of them and each one is uh it can do a lot of things differently where's the source though what file am I pushing down what's in my root of my salt infrastructure of call and it's called bash underscore profile and then of course I'll make sure that my permission is correctly on that box alright so if I wanted to push this if I wanted to actually make sure I got that um I can show you where that's um just so you know I'm not lying there's the this is the srv salt is the root uh the root path and then of course root bash profile root bash profile okay and of course I'm going to push that bash profile out so I can just wrote um salt thanks fel I'm going to target uh two you client one and I'm going to say state dot apply on my root profile okay cross our fingers oh except in this one's in the I think I picked on all the ones I picked on I picked on the one it's in blackout but it goes it helps me go to prove a point though because I've set a pillar on this box it says don't allow any changes to the box okay so I tried to push a config change to this box and it said no um the pillar says you're not allowed okay the one thing I'm allowed to do is actually run a salt refresh pillars on this box it's actually updated to pillar data so let's try a different box let's try to client two and sure enough just like that I push that box out I get a lot of data about that I can see that I get a diff if it happened to be a root profile already there I would have gotten the differences between the two and good get format um and I would have seen that I can see that it was successful it changed one file nothing failed it took 66 milliseconds to run I could have done that I could have targeted all of them and they would have all happened that fast because again it's happening in a parallel to all of them it's putting that out on the message bus and they all just happen okay so that's the foundation behind salt um it's a very very quick overview of salt yes sir well I can roll back to I can push down a different profile there's not a rollback feature per se unless you that goes back into the the question is is there a rollback feature um and it kind of goes to what is your underlying OS support if you're using OpenSusa with butterfs there's a lot of rollback options then if you're using ext3 yeah there's other ways you have to go about it okay um but I could have made this profile a lot more um I could have made the state file more uh rollback proof for for example I could have made it do a copy and make an original file um so I can do a lot more in that state file I made a very very simplistic um on purpose um oh yeah so I could have put more data I could have made this a little bit more complex here to give me some rollback options but in typical I tried to keep things simple alright so good questions though and I don't mind questions along the way alright let's move on and I'm getting that make sure I don't run out of time here so the next thing I want to show you though was the um how this looked then in the UNI itself alright let's give a quick overview of the system oops I don't care about that but it's local I didn't test that my apologies um if it wants connection um let's try this get a network here alright much better fine um alright we're back on track okay so let's look up here at home overview real quick this is the interview this is the interface now I'm having to squeeze it all in to fit this screen up here um and you can see that it's a little bit cut off but that's okay um I can see an overview of my systems I can see I have no interactive systems I've got one pending minion waiting to be accepted I can see my most critical system so it's got updates to them I can see what's happened recently um we're going to come back to this on relevant security patches we're going to come back to these and actually deploy these so this is just kind of an interesting overview I can see I've I'm fairly evenly distributed between my sentos and my open SUSE clients um but what I wanted to show you though let's go back up here let's go to close this down systems system list all and I can see that I've got um a lot of clients where they're at for patches packages where they're at what base channel I'm going to talk about that in more than a minute but I want to scroll down here and look at my configuration channels and you can see I can do the same thing I had I had already set up one called bash profile and I can look at the init file for that and I can even edit this init file from here you can see the same format so I could have pushed this out from the UI as well across lots of systems I wanted to show you a couple more real quick since that's kind of a simplistic one the permit no uh this one I had set up called no root permit now where we can turn off the ability to SSHD so in this 10 lines of code I've basically saying here this service SSHD needs to be running this service needs to stay running but watch this file if it changes restart SSHD and then I'm going to push a new config file down to the box with my permissions and it's going to land in this director in this path right here which matches what I'm watching and if that changes it resarts as SSH so in my 10 lines of code I was able to easily push down and restart SSH without much trouble so for sake of time I just wanted to show you a couple examples of what we can do another one real quick here this one because I'm a lab and I do a lot of stuff in my lab box here I don't want to deal with all the spectra stuff so it won't slow me down so I turned it off and now I'll replace on my grub my etsy default grub and of course I could have put more targeting here but I can actually target saying if it's a SUSE box use this path if it's a red hat box use this path for example but here I'm saying in this file look for this pattern so think of said I'm looking for this pattern that ends with the line show ops quote I'm going to replace it with this line show ops no specs back off example and then if this changes down here on changes of this up here run this command I need to run grub 2 make and fig because I've made changes to my default grub because I told it on changes here run this command and this changed so run that command so again very simplistic only 12 lines of code actually if I take the one out the one space line out alright so that's a very simplistic look there's some that get very very detailed but I want to give you a quick overview of what we can do with solves from the UI alright so let's go back up and one of the things I want to show you let's pick on a system here oops I don't have this one and I can go down here and there's this idea of we talked about formula with the forms and you can see that I've already rolled out this one formula called Splunk what are these formulas well this formula is a very simple one and it's asking me whether the install is either a universal forwarder or a full version of Splunk very simple but behind the scenes the formula behind the scenes says if it's a UF install from this path with these parameters if it's a full install with this package from this location and do all these other kernel changes that I need to do for a full Splunk install but from this simple UI I just all I had to do was pick whether it's one or the other so let's look at a different formula though it's I'll say it's oh that's right because I didn't I need to put it back to where it was alright so let's look at a different one we could have added I can get my mouse to work here there's the one I talked about on that U1 we have blackout but let's look at one a little bit closer it's like a DHCP save that go up here now I've got another menu item up here you can see this one's a little bit more detailed and I can add my domain name domain name servers my interfaces so all the things that you would configure within a DHCP server I can now do it via this formula okay via this form and push that out and if I add it to a particular group DHCP would be a bad example but all the machines I add to that group now would get this formula and get these parameters okay so let's clear the values there alright any questions so far I want to turn that off that's a little bit longer talk yeah actually there is a presentation at SUSECON coming up first week of April that we get into that a whole two hour session on formulas I will show there's a topic on not topic but a web page form salt on formulas and then there's another the documentation on how to create the form portion of that as well I'm doing okay yeah half an hour alright so let's go back up here and look at our systems real quick and from here actually I want to go yeah let's do this go back to him so I talked about my states and I'm sorry do I want to do this yeah configuration channels so here I can do my configuration channels and see I've already pushed this one called zoomer host well because of the way my little lab is built on this one single laptop and the way I'm doing DHCP and here I had to actually push that file and edit my local host on each box so it pointed back to my laptop for him so I had to do a config file there but if I wanted to apply I could search for other configuration changes SSH and can search I can see that I can add for example I'm going to click on that I can even see what it's going to do I can see I'm doing SSH running I'm changing the message of the day and I'm pushing out a banner okay so I can add that I can even edit it from here if I wanted to I can apply that I'm going to assign that save changes I can then rank these depending on which one needs to make sure they get done first if there's something that has to happen before the other one and confirm it and now the config channels change has been scheduled if I go up here to high state and when it builds the high state I can then see where that's validates and if this builds cleanly I can see that sure enough there's my changes that I'm going to be pushing down to the system as soon as I apply this high state these banners this stuff would change and we just go ahead and do that and I can see that my high state has been scheduled and from here I can see what it's been doing what the details are and when it completes it will show up here I can see it's in progress or you know when it fails it will show up here and it will give me a little bit of a clue of why okay so that's our schedule one thing I wanted to do oh golly yeah so I wanted to show you on our audit for example two ways to go about this one I can see from my overview page that I've got some relevant security patches on different systems they're updated with the most current ones at the top as recent as two days ago this one opens to 2019 there was a docker change I've got one system that's affected by that okay so if I click on this one I can from right here click that apply patches and he's done that security vulnerability has now been remediated for this system okay and boom he's now been scheduled oops he's been now scheduled for that box I can see this patch down if I wanted to see what that patch did the details about that patch I can see what it's fixing I can see the CVE that's relevant to this that's also relevant and I can see what it's fixing in that I can see then what packages it's pushing down as well and where I what repository I'm pulling those packages from and then of course the affected systems and ooh it failed okay let's find out why okay again I picked on the blackout box typical so let's go to that box let's get rid of the blackout and then we will be able to do things to him and now if I go back to that I can then quickly come down here and see my actions this one failed this high state failed because I had it in blackout mode if I click here now and go back and do it and quickly apply patches confirm schedule and that's old okay so anyway we're moving on another way I can do it though if I need to see particular audit on across all my environment I'm looking for particularly let's say 2018 I don't know 365 I think it's one I have in my back of my head and from here you can see that I'm allowed to not only audit my servers that I already have but I can also audit my container images to make sure that before they're deployed they're clean for this particular CVE audit okay I don't have any containers set up right now so we're just going to audit our servers and you can see I've got some that talk about I've got a couple of options here some that are clean and one that has the audit against it is available alright and I can do the same thing I did then click on this guy I can see the clients are affected details, packages highlight the system I want or all of them and apply patches okay alright now let's talk about repos real quick so in our world of managing you've got all the repositories whether it be SLEZ11, SLEZ12 OpenSource to 42, OpenSource to 15 Sentos 6, Sentos 7 and soon to be Ubuntu all of these repositories and they're changing every day right you know there's new patches being deployed to the main repository upstream every night practically seems every night well you don't want to apply changes to a system today based on today's repository the upstream repository and then apply another update tomorrow to a new system and it have newer patches in the one night before you need a baseline you need to be able to say and I can guarantee that from this point and I update my pride it was dated March 1st and every box I apply then to March 1st is going to be exactly alike agree well that UNI gives us that ability to create those baseline we can do it either from the command line or from the UI and you can see here for example I've got several out here I created from February 5th baseline and February 18th baseline February 22nd, January 28th March 4th I've got all these different baselines and I can then roll I can push my system to those individual repositories and so that when I push it to that repository I know that they're going to get the same patches as the one system prior to that and then I have just barely scratched the surface of the interface of what we can do here is our images ability we can deploy systems from within UNI whether they be virtual or containers and I don't have this this is one area I didn't plan on demoing but I wanted to show you the interface for it I can deploy either a kickstart file use a kickstart file or auto-yass file to do it and I have a presentation we're getting ready for for SUSECON on how to do system upgrades from SLES 11 SB4 to SLES 12 all automated all through SUSE Manager and SUSE Manager today is synonymous with UNI okay? so UNI is a great place to you've got that development space you've got that lab and you need to be able to have system control over lots of machines you can do that effectively from UNI so let me jump back to my slides real quick so encourage you guys let's get involved and how do we do this in lots of ways the project itself page is here in which all of these are then you can find them from here our GitHub page the open build service page the mailing list the user follow and of course IRC all the developers are on IRC so I really encourage you guys it's a brand new project there's a lot of functionality here that we've already brought in but there's a lot more we want to get to it's a very active community I want to build it more you guys get to be the first US based UNI talk so let's make it happen let's get some US people involved in this project just like the couple of talks we've had today already about getting the community involved this is a great project to do that whether it be in documentation because there's a lot of documentation that's been modified from system manager to UNI that needs to be updated and we need people to jump in I'm not a developer so I can't commit code but I can commit to doing documentation updates and changes and that's highly needed in this project as well the project also needs testers people to go out and try this find out where the bugs are where are things missing what features are needed that are critical getting involved to find those things from a user's perspective instead of just a bunch of developers writing code that don't do this on a daily basis that's where we come in as the admins to make things happen so get involved that's what we really need so that's my presentation I'll open up to questions again the typical disclaimer have any questions? I was just wondering is this dependent on an OpenSUSE environment or will it work on any? the server today requires OpenSUSE 3 the server side actually I think it will run on LEAP15 now as well on the server side the client side can be Red Hat all the way back to 6 as well if you want to use Salt if you want to use the traditional client which I didn't talk about you can even go back to CentOS Red Hat 5 as well as CentOS Red Hat 5 Support 2 but it's still available today but the server itself needs to OpenSUSE 15 or OpenSUSE 42.3 so I walked in late so I missed the first part but is this just a front end to Salt or any configuration management solution? this is our front end we use Salt underneath it's not a per se front end to Salt other than the fact that we use Salt and do do a lot of fun and work for it it's not a replacement for SaltStack Enterprise we're not trying to compete there that's a completely different market and different mindset they do things we do, we do things they don't do right no that's agreed it's meant for it runs Salt underneath and only Salt however there is several SUSE developers we have an internal Salt team that writes code for Salt and several of them have pushed up stream code that will allow us to run Ansible Playbooks within SUSE Manager so we can run Ansible Code via Salt inside SUSE Manager so you don't have to take all the Ansible code you've already written and throw it away and start over, you can continue using those Playbooks other questions? thoughts? comments? rotten tomatoes? eggs? alright well, thank you very much thanks guys we're on the cusp of getting, oh my voice is much louder now that I've turned on the microphone probably don't even need the microphone because I do this a lot but for the recording purposes they told me to keep the microphone on they also warned me not to dance they told me to stand in place try not to wander around which is impossible for me I'm completely incapable of standing still repeat questions from the audience and don't forget to tip your waitresses I think that's an important one I was just telling everybody this is one of my favorite places on the planet this is Henry's Lake just outside the Tetons West Yellowstone it's in Idaho it's called Island Park Idaho, Henry's Lake I used to spend a lot more time there than I do now but that's not my presentation this is and does it work? good it does thanks for coming today I know that there is a high competition for time everybody's got lots of interesting presentations they want to see or booths they want to visit or Simon Cowell out there with the America's Got Talent show but I'm going to spend the next 40, 45, 50 minutes talking about software to find storage and if you don't like my presentation Alan Ott will be giving a similar presentation on Sunday I don't remember when and he's a lot smarter than I am but he doesn't have a beard I up until recently was one of the engineering managers at SUSE for the software to find storage product that SUSE has called unimaginatively SUSE Enterprise Storage and now that I'm no longer one of the engineering managers for that team I feel a lot less qualified to give this presentation but I'll do my best I'm currently writing technical training materials for the SUSE Enterprise Storage product so that still qualifies me to give this presentation in addition to that I'm an adjunct instructor of computer science at Utah Valley University which gives the only credential that gives me is that I can stand up here and talk to you fairly comprehensively does not allow me to do interpretive dance though but for a long time people have kept records and wanted to keep track of their stuff and if you can imagine, you know, Thag you know, we're still alive the day and if he didn't ever delete his data you can imagine the huge storage of communications data that he would have stored up how many caves of data do you think Thag would have, right? and they used rocks to communicate things and you know this is some sort of Meso Asian writing script, some script oriented base of language stored on stone really incredible stuff that we have about storing data and this is Greek this is some sort of official edict from back around 300 BC and this is my office right here and this was 5 get this, 5 megabytes of IBM storage around about 1956 so you know, we've been dealing with this data thing for a while and this is nice data stored digitally and we're not much farther along than that but it's a big problem obviously we need data and we're keeping lots of data and whether we want it or not we never delete the stuff and from 2009 we're some down here and here we are in 2019 and the data growth is about 40% per year ouch that just blows my, if you think that blows your mind wait until you see the next slide and now although it's interesting though that storage costs are declining, that's good we're grateful, however, comma you can see the disparity that's the word I was trying to fish for right there although the costs are going down at roughly 25% annually, the rate of growth is 40% annually so it's still very expensive to store our stuff alright so now that if that doesn't blow your mind it's alive it's just nobody ever delete stuff, you're not allowed to delete anything anymore and IDC predicts that there'll be 10 times the amount of stored data in less than 6 years about 6 years and by 2025 there'll be 175 zettabytes of data being stored on this planet now if that doesn't blow your mind this is a zettabyte a zettabyte is 1 trillion gigabytes or 1 to the 21st that's one with 27 zeros of bytes and I forget how many DVDs worth of data that is but you can't watch that many DVDs, trust me and just to make it all the more complicated last year the same IDC folks said by 2025 we were going to be at 135 zettabytes and so they're already inflating the number where are we storing this stuff that's a complicated answer there's a shifting paradigm in how we store data and we have to do that if you're a mom and pop shop or if you're at home you probably don't need to listen to this talk we've got the standard today model of storing data on legacy storage systems and that's fine and that's probably going to do us for our lifetimes but if you're in any sort of business any sort of organization that is not allowed to delete data you've got to figure out larger and faster and more comprehensive ways of not only storing it but making sense of it keeping track of it, knowing where it is just because you've saved it doesn't mean that you necessarily know where it is and can get it back you've got to be able to manage and organize all of that ridiculous data in our current day to day of storage our legacy approach to storage storage silos with a direct attached storage and if you're lucky enough to have something that's at least a little bit greater scale than that at least you have the means of expanding that direct attached storage and having some sort of array within what do you call those things a closet, a cabinet that's the word I'm looking for a cabinet to expand and then when you get to the point where you have to have now two cabinets it gets a lot more complicated it's hard and it's a lot of work and it's a big headache then you've got to back up that stuff which is really nasty and the process that's behind it is a very slow reaction to change and as a matter of fact the way most proprietary vendors handle their software their hardware for storage these days is what we call in the industry a forklift you don't come in and add storage although you can you come in and add a shelf and then you add another shelf and you can at least get some scalability there but then when you get to a certain max size for that architecture then what your vendor does is they do a forklift change out and replace they basically come in and take your cabinet out go out and throw it in the dumpster and then they bring in another cabinet that's got higher density disks, higher capacity disks and it's hard to scale that kind of stuff and it's difficult to manage that kind of growth and not that I haven't put this particular detail on the slide but it ends up getting really expensive, really fast and the hardware vendors know this right once they've got you they've got you for life what people want is not that what people need going forward in long term is something that's far more agile and we end up calling this a software defined infrastructure software defined everything infrastructure and as a part of that you've got software defined networking and you've got software defined this and software defined that and in that whole hodge podge of software defined everything you've got software defined storage where your ability to deliver storage to the people who are using the storage ends up being more agile, more flexible, more along a dev ops delivery mechanism and more ability to move the data around and change the size of the data and it typically ends up being more business driven your ability to react to the business needs is far more facilitated and you can be faster in terms of deploying what it is that those users and customers need for what it is that you're keeping track of and this is where most organizations want to go and even if organizations don't know that that's where they want to go when they try to tell you what it is that there's their use case they're pretty much describing this which is a software defined storage approach now one thing that a lot of people start looking at as they're going this direction they say oh yeah well they've got cloud storage and great cloud storage is awesome and there's a great application for it it's very flexible you don't have to do any capacity planning because all you have to do is just tell your cloud storage provider hey give me some more disk and they go hey we'll be happy to give that to you yeah here it is you know what they're doing in the back end software defined storage but that's another story right and and from a hardware perspective you're really happy that managing all that hardware is somebody else's headache yeah good stuff right you don't want to be in the business of managing all that hardware but there's always a but that comes at a cost and I mean literally a cost right remember growth is about 40 percent per year today for two petabytes that costs you $50,000 a month and of course prices are coming down but your growth is going up and they don't intersect and so it just gets more and more and more expensive for you plus in your cloud environment although there are a few options typically speaking you're talking about object storage and maybe you need something different than object storage what if you need block storage what if you need file storage so once you get hooked here too if you ever want to change has anybody ever tried to get out of the cloud if it's your your mp3's that's pretty easy to move out of the cloud right but if you're talking about all of your medical data for your hospital getting that out of the cloud that's not into impossible so there are some challenges with cloud storage solutions there's a lot that's appealing about it that's fine that's good and there's a good reason to be there just be aware that this isn't always what everybody wants so how do I store all my stuff well along came an open source project that was free and open source and was a clustered distributed model of software defined storage it predates the cloud movement right the first stable release of sef was in 2012 by a guy it was his phd project somewhere he was here in california I think he was at yeah there you go and his name is sage sage vile sage vile good guy really smart guy starts a company called ink tank within a few years red hat they come along and they say look how valuable this is they buy up ink tank so now all those sef guys all of those ink tank guys have moved to red hat but they're really good friends of ours and I work for susa and we're also involved with the development of sef and I'll show you a little bit more about that when you get in the next slide sef has a lot of really good goodies really incredibly powerful and useful features for a distributed storage cluster right now of course it stores all of its data all back as an object store but then makes it available in three different formats that we'll talk to in a moment another important feature about sef is that it excels at durability so the redundancy and the fault tolerance of the storage data is second to none very very rock solid making sure that you can get to your data when you need to get to your data it allows for a very efficient and fairly effortless scale out to the tune of exabytes if you need more storage you just add another node and it's built on commodity hardware a lot of those proprietary software vendors today they're selling you their hardware that will only work with certain disks that only have their controllers and you have to be using their software and it's all proprietary stuff whereas with sef keep in mind there's limits to this you don't want to go out and buy the cheapest controller card that you can find you don't want to go out and buy the cheapest hard disk that you can find but generally speaking it's all on commodity hardware that you choose you don't have to buy the vendors hardware you can go out and buy the stuff that you like that you have the confidence in HP, Dell, EMC whatever it is you want and it's all open source it's entirely open source it's almost entirely C++ although there are some other bits to it that are other bits of code and to be honest it is the most it still is year after year the most popular choice for the back-end storage mechanisms for OpenStack now I have a reference down here the 2017 survey report from from the OpenStack summit they didn't do a survey in 2018 for some reason probably because it's expensive to do surveys but it is still even today unofficially there's no report to back it up but OpenStack pushes the fact that SEF is the way to go to store your stuff in an OpenStack environment that's a big testament to how robust and how relevant that SEF is just in 2018 SEF was removed from direct control from Red Hat and the organization now is governed by the Linux Foundation that's a really cool thing right nothing against Red Hat right but even the the old ink tank guy Sage they're all saying as much as we like Red Hat and Red Hat's paying us very well it provides greater flexibility and greater openness to be managed by the Linux Foundation now that doesn't mean that Sage is employed by the Linux Foundation now Sage is still a Red Hat employee but SEF as a community is governed by the Linux Foundation so we have a new SEF Foundation and its founding members are listed here I of course because I work for SUSE I'm biased and I put SUSE here at the top it's interesting that you'll see over here on the number of lines of code that are contributed to SEF with the most recent release of SEF which is the MMIC release in 2018 we're currently working on the Nautilus release which is actually on the list of being released it'll come out probably the beginning of April but as far as we have for lines of code contributed for MMIC you have 537,000 lines of code contributed by Red Hat and close second to the tune of one third the amount but the second greatest contributor to SEF is my company SUSE and a lot of other great people right this is a bunch of people that are unaffiliated just their own everyday run of the mill garage contributions right a company known as Morantis still they've kind of left the game Morantis is a great company but they've de-emphasized their involvement with SEF so although this number in the early part of 2018 was significant it has dropped off a lot since they made that announcement and then a company called ZTE that's one of the other surrounding members and a bunch of others I just picked the top five that were here SUSE produces we have about 30 engineers that are involved in contributing to SEF Red Hat has I don't know what was the last count they've got more than 100 that are working on and not exclusively SEF right they're still committed to Gluster but their storage solutions they have significantly more people that are on board the development of SEF so they have a lot more contributions here and of course they've got Sage there who's running the project and a lot of the other original ink tank guys that are senior officials in the governance of the project but it's a great cooperative effort those folks at Red Hat holding tank guys and we at SUSE we work very hard we work very closely together and I'll mention that here in a moment actually I'll mention it here one of the great details that are coming along first is sort of a preview in Mimic and now it's going to be seriously supported for Nautilus is the SEF dashboard which is now a graphical interface to do the management and the interaction the graphical interactions with SEF now that SEF didn't have before Mimic and we at SUSE had a product adjacent to our SUSE Enterprise storage product called Openatic and we got together with the Red Hat guys and said you know what we really need to do is bring these two things together and so our Openatic developers ported all of the functionality from Openatic to make it native within SEF called now the SEF dashboard so that's a really great cooperative effort between both Red Hat and SUSE at SUSE we make two variants of SEF or that is to say we're involved in making sure that we have an Enterprise product which is called SUSE Enterprise Server but we also in the efforts that we go through with Enterprise Server we publish all of the bits that we are working on as well as all the rest of the upstream SEF efforts compile it release it on Open SUSE for both LEAP and for Tumbleweed on LEAP we're currently shipping the stable SEF MIMIC but actually available in Tumbleweed today we have the latest and greatest pre-release bits for Nautilus which is going to be released here in the next few weeks all the work that we do and says gets rolled down into the open source the open distributions of Open SUSE LEAP and of Tumbleweed of course the Open SUSE bits community supported whereas for those who want to buy into a support model where we'll help you fix all the bugs that you may have then there's the SUSE Enterprise Server that's available and I have the links here that you can get to the LEAP and the Tumbleweed bits of SEF SEF is composed of a storage cluster which makes perfect sense N number of nodes within this cluster it's basically self-healing self-managed that's kind of misleading isn't it what it's trying to make sure is that it's keeping track of where it's storing all of the stuff and all of the technical details it's keeping track of and just says hey when you ask for the file you don't care where it is I'll get it for you generally there are no bottlenecks there are obviously caveats to that because you can implement or design a poor design of any cluster and have the whole cluster itself be bottlenecked but when properly designed according to best practices according to what it is that we recommend for it there is no bottleneck it'll just find the data wherever it needs to the equivalent speeds no matter what it is that you're trying to find and as I mentioned earlier it excels in data durability let me make sure I'm clear about that that's how it was originally designed that has always been its particular focus to make sure you don't lose your stuff there are proprietary solutions that are going to be much more performant than something like Cep let me mislead you Cep is fast it's clippy those little cephalopods if you've ever seen them swim man they can get around where they need to go but there are faster alternatives if your value is speed right if you want the IOPS you can buy a product that's faster but for all the combinations of things that exist with the durability the scalability and the open sourceness you can't get any better than this this is absolutely the best there are basically three different ways to access the data it's got its native object store which you don't access natively but it does provide an interface for object access for Amazon S3 or OpenStack Swift there's block access if you've got some native linux application you can just use LiO to attach to the data that's there or a few different gateways that are available right you can get to these the storage store through KVM or iSCSI a variety of different ways and then we have in a recent release of Cep now the introduction of distributed file system storage which is called CephFS that's really a unique name right everybody loves it it's great CephFS you couldn't have come up with any better name than that and associated with that CephFS you can also access that file system data with the likes of NFS through NFS Ganesha there's lots of flexibility in how you get to your stuff a basic Ceph architecture looks like this you have the baseline object store that's been called RADOS or RADOS and it is much more imaginative than CephFS right you've got the reliable autonomic distributed object store or RADOS you can see here someone like you know like the Arby's commercial guy right saying that with a really deep voice we have the meats right RADOS and on top of RADOS you have these basic interfaces for those three access methods you've got the RADOS gateway which provides for object storage interface for S3 and Swift and it's a nice restful interface you can get to the block storage through the native LIO access obviously but with a gateway to allow provisioning and snapshots for ice guzzy and the like and then you have the CephFS that we were just talking about that provides a POSIX compliant file system mechanism that's very tantalizing for the likes of Hadoop or just regular file system storage that you might want to have if you don't care so much about your own EXT or butterFS kind of environment. Three great ways all rolled up together so a quick look at how this is right and this is I pilfered this from an old set of slides that I have this is actually slightly wrong and I didn't have the disposition to actually change it yet but I need to on a node on a storage node you have three basic sets of functionality you have the physical disk itself and then sitting on that disk you have some file system and this is the part that's a little bit misleading because in the recent past we just took advantage of XFS because we thought hey we don't want to be in the file system business we'll let somebody else take care of it and that's good right we let it be somebody else's hey but then we thought for performance reasons maybe we don't want some of the overhead that's associated with XFS so we're going to build our own and we're pros and cons with that but we created now a new file system mechanism that's exclusive to SEF that's called BlueStore and of course it takes advantage of a few other not ours technologies you know this whole database is based on everybody's favorite Facebook database what's the name of their database ROXDB right so we use ROXDB to help manage all the metadata that's on the file system but it's our own proprietary proprietary it's open source what am I saying proprietary for right but it's our own invention of a file system that provides for effective SEF usage please ask a question correct yeah it's part of the SEF development here I'll try not to say stuff that's just SUSA specific which is a SEF discussion here and then sitting on top of that is a demon called the object storage demon and for each disc in a node they all have their own OSD that runs that just keeps track of hey what am I storing in my database and what am I storing on this disc so that's the OSD process that runs so if you have eight discs attached to a node you will have eight OSD's that are running on there and these OSD's all know how to talk to each other amongst all of the different nodes they're talking to each other they all know what they're storing and then they are able to serve their stored data out to whatever client is it says hey I need my file oh I've got it here here's my here's the file you're looking for and if anything would happen to disc N inside of this node then all the peers know that oh some disc has gone bad where's the other replica for that let's now make up for the fact that that doesn't exist anymore that disc has gone bad so they all are able to talk to each other to keep the data relevant and replicated all throughout the entirety of the cluster those discs usually are made up of three different varieties the most common is still our everyday run of the mill cheap HDD's that we affectionately call spinners there's also the solid state devices SSD's which nowadays are really getting less and less expensive and more and more affordable and and I'm told that we're on the verge of another SSD innovation a revolution there that's going to bring the capacity up and the costs way down yet again so that's good now don't be fooled right if you're going to buy the SSD that you can get at Walmart that you're going to put into your home system no that's not going to do it for you you've got to get enterprise class SSD devices and a matter of fact Kifu no is it Kifu it's not Kifu somebody in the community keeps track keeps a web page up the date on a weekly basis about what SSD's are available and which ones are going to provide you the meantime between failure results that you're expecting to get and the performance that you're expecting to get so make sure that you're getting enterprise class SSD's and the cheaper these become the less you'll be inclined to use spinners but there's still a good case for spinners that we'll get to here in a minute and then the most current technology that we have which is kind of similar to SSD's the nonvolatile memory express flash capabilities that exist still pretty dang expensive but if you got the money this will give you the performance that you want the best thing that you can do though is that you can use all of these things in concert you can tear these or you can say keep my raw data on the spinners and just keep all of my metadata and my journals on the faster devices and you end up having a great combination of storage capabilities there so you have a bunch of disks and their associated file systems and further the variety of OSD's that are running and they're all represented as a particular node that you're going to see on the next few slides this is a node with the disks and the file systems and the OSD all combined together each physical disk has an OSD and of course you don't want to use all of the disks on your system because you have to have an operating system on that server so save one disk or maybe two if you want to mirror the OSD if you want to mirror the OS that's a good idea, that's a good best practice there to have some redundancy for the OS on that node but then you just give all the rest of the disks to Ceph and it takes care of it, self-managed it just takes the disks, sucks them up hoovers them up if you're English and makes them all part of the storage cluster there's also one other kind of node in these clusters called the monitor it's the brain of the cluster and it's not in the data path at all none of the OSD's really care about what the monitor does but the monitor just keeps sanity going on in the cluster it knows what the OSD's are it knows which ones are healthy it knows which ones are sick it knows which ones are overworked and it just keeps this map of what's going on with all of the different nodes within the cluster and it has to be a a it has to form a quorum it uses the Paxos protocol to make sure that the data that it's managing is authoritative so you have to have at least three monitor nodes in your cluster and you need to have an odd number of them now strangely enough you don't have to have more than three monitors in your cluster there are some customers that come around and say oh yeah well I'm going to have this really ginormous cluster there are five monitors and between Sousa and Red Hat we've been monitoring this monitoring for the sorry that was a pun we've been monitoring this for a really long time and we still even with CERN one of the largest clusters that we're associated with they only need three monitors if you want to have five go for it yeah you got the hardware hey hey knock yourself out but you basically only need three monitors they'll handle the load so you put those nodes the storage nodes together with three monitors and you got yourself a nice little handy dandy self self cluster I can say that and maybe we want to get some data right of course you don't want to store data if you can't get there right so you've got some sort of client that's running maybe this is a a CFFS client and it's trying to then write some file system data and it's going to go over to the cluster and it's going to say I want to write this file and the cluster says hey baby we got it leave it to us and so somebody in that cluster grabs the data and it makes by default three different copies replicas of three that's configurable and of course you don't even have to use the replica mechanism you can use erasure coding that's one of the different types of technologies that we implement also in SAF but for the purposes of this demonstration we're showing replicated distribution here and so he takes the copy and he makes two friends and now we've got the stored data and then as soon as the cluster has verified that it has the data and it has given that instruction to three nodes to write the data it sends it back says that's ours we got it your data is safe in my cluster and so without clustered storage solution would be complete without the ability to get the data back out and so you got a client that comes along and says hey you know remember that file I gave you a little while ago I need that back so you talk to the cluster the cluster says hey I know where that is let me give you the one that is fastest and so this guy answers here it is clients got it now do you need a little more storage just add a few more nodes need a lot more storage I'm simplifying it obviously right all of a sudden you're in two cabinets three cabinets you've got to have some robust networking in the top of the rack there to work effectively but you think it up we can do it you get the right networking you add as many nodes as you want you have three monitors and you're good to go now can you do data center replication on this yeah you can it's not great performance but it's hugely robust you want to put a backup set of your data in a data center across the hall piece of cake now if you want to put that backup data center across the country or across the planet things get a little more complicated but it certainly can be done and it's because of this robust mechanism of these clusters all talking to each other if you want to use CFFS instead of block storage you do have to have one separate dedicated node called the metadata server that provides all the POSIX capability now it's entirely technically possible to have a CFFS system without a metadata server but anytime you want to do a find anything you want to do anything POSIX you've got to have something that's keeping track of all that POSIX metadata for each of the different files that are stored and that's what the metadata server does and you could conceivably for high availability purposes have N number of metadata servers and have them fail over one to the other it's not necessary to have multiple metadata servers for performance at least not that we found yet you don't need to have multiple metadata servers but we do allow for the duplication of metadata servers in case you want to have that kind of redundancy now let's see, what do I have on this slide for RADOS it's native object store this is its base design this is exactly what objects were designed for is for large scalability but even so with CF that was exactly what SageVile was after so with the object store we just simply keep everything together and then our native object store and then can certainly make it available for what other object application that it is that you have whether that's S3 or Swift and it's better that we're dealing with large-ish objects but it's okay you can have small objects that's certainly not a problem and of course we're avoiding single points of failure everything is very robust, redundant at least three replicas or with erasure coding you've got that spread out in a kind of way out throughout the entirety of the cluster and it's entirely distributed so you can configure it to have replica one in cabinet one and replica two in cabinet two and replica three in cabinet three and then you have those three replicas so that you have a lot of that robustness that you expect to have to be able to do all of this stuff though then you do have to have some robust servers with some RAM and with some cache and with some CPU and of course with some disk and so you don't want to go out and just buy the cheapest hardware that you can get you want to have some enterprise and if you're toying with this in your garage you're just making a SEF cluster for your storing your DVDs in your basement then you can buy whatever you want but if you want to have something that's going to do what SEF distributed software is intended to do make sure that you're getting hardware that's equal to the task as well so there's three essential components to storing and managing all this stuff and we'll talk a little bit about pools, placement groups and crush maps here I'm checking the time pools are just a way of logically grouping your objects you could create a pool for department A in your business and a separate pool for department B and a separate pool for department C that then those different departments have control over their own sets of storage for their particular department and typically these pools are all organized in a very comprehensive way that makes sense to you you have to give the pool a name you need to tell it what kind of redundancy that's associated with it whether the number of replicas you want to manage your K plus AMI ratio coding mechanism that you've decided is the best for your organization and you have to identify a number of placement groups I'll talk about placement groups in a minute you also are going to have a bunch of crush rules you could take the regular crush rules as they are by default or you could customize I mean we'll talk about crush groups crush rules in just a moment and then as part of that pool you also need to say an owner by department A you're going to make sure that department A is the people that have access to it and the department B aren't going to have write and read access to what's in the pool the pool for pool A and of course as any good storage mechanism is that set of core functionality is going to allow you to create objects, read them, write them and also you can create snapshots for those objects now for placement groups it's a weird thing that kind of trips some people up and that's okay if you don't understand it it's actually not going to cause you any problems you will have to understand it a little better but I'm going to give you a nice promise that with Ceph Nautilus you have to understand it less we're going to do some automated placement group manipulation if you want to if you want to give that kind of control over to Ceph and just let Ceph handle the headache for you you can do that if you want to be cool if you want to have more control over what's going on with your Ceph cluster and control those placement groups more manually you still can do that you have more options here this is just the way that Ceph balances the data across N number of OSDs and there's a kind of a ridiculous math algorithm that you use there's actually a tool on the Ceph.com website that helps you to calculate the best way of doing that and also a lot of other groups that you should create for a particular pool one placement group will typically span several OSDs like shown in this diagram right here and one OSD typically serves many placement groups and it's entirely tunable based on what's going to happen in this particular pool of storage if the data in your pool is going to change really fast you should have a larger number of placement groups and of course these formulas that we provide for you or your consultant provides for you or the website provides for you helps you to make that calculation appropriately if you're going to have really really static data you're just going to write it once and then people are going to just read the brains out of it then you don't need a large number of placement groups for that particular pool and then there's this other acronym called CRUSH controlled replication under scalable hashing it's cool right somebody got some money for that I'm sure that is the way that you as the storage administrator now makes designations of how best to organize the data in your data center so that you can decide that for this particular group of users or this particular application you need to have that in a set of nodes that have the fastest disk and the most CPU and RAM so that you can have the fastest IOPS for that particular application or you can customize your CRUSH map so that you are spreading it out over a specific set of racks in your data center or that you can organize your data so that it has the most optimum disaster recovery resilience right the CRUSH map is the system administrators of how all of the data is supposed to be organized across all of the different nodes and the users don't really care how this works they just need to be able to get their data so when the client system comes to fight out where the data is the user doesn't have any idea what any of this weighting is going on and all of this measurement is going on and all of this access is going on it just comes and asks it and the CRUSH map and the monitors all know okay go find the data over there and boom the client goes over and gets the data and brings it back so with all of the components working together there's some kind of object some file that you're trying to store you're going to go through the client it's going to get lined up in some pool spread out over a bunch of placement groups the CRUSH algorithm says well the best place to put that is in this placement group that placement group says yeah well then because I'm talking to these bunch of OSDs and that OSD is not doing anything right now put that data over here on the OSD and then the OSD says yes now I have that data let me give that to my two brothers over here for three way replication and then it sends all the information back says hey we got your data and the object says great I'm safe safe and safe so a few bits with the time that I have remaining here right let's talk for a minute about legacy storage arrays now if you go to your NetApp organization they're going to have a different list and I accept that I'm biased and I accept that they're biased right so we've got these kinds of limits associated with these traditional storage mechanisms it's a tightly controlled environment it's not open at all there's some scalability but it's limited compared to what it is that we can provide in a clustered system like SAF it's relatively expensive don't be too fooled by this right once you get comp what's the word I'm looking for competent once you get competent hardware in your software defined system it's also going to be expensive but not as expensive as your proprietary guys are and you don't have as much flexibility and as many options that are there now they're going to tell you something different and to be honest with you you know the folks at EMC at NetApp they have a lot of functionality they've been in this business for a long time and they're good at it I'm not faulting them at all boy they're good at it they have got really well-designed hardware and software I love them they're really great but you can only use their specific drives you've only got what it is that they can tell you in their shelves for those number of slots you don't have as many variants and what it is that you can do with RAM you just stuck with what it is they're going to provide you you can't use networking hardware you can't just go get commodity networking you got to use NetApp's networking technology and you're limited to what CPU and RAM that it is that they can provide for you the benefits are though they do have a long track record they're really good at what they do you know that you're going to get value out what it is you know they're going to give you the support as long as you're paying your support contract they're going to take care of you they're really going to take care of you and it's a very predictable pricing structure to find storage and of course NetApp's list is going to be very different here than mine is but the biggest complaint about software to find storage is it's slow and I'm going to tell you that's not true SEF is really fast it's just that it's not it's forte that's not what it was designed to do it's emphasis is durability of the software of the of the system so don't think that just because we say that it's slower than NetApp or slower than other proprietary systems that it's slow it's not if you design it right you can get good performance out of it that's one of the challenges though about this I should have put this as a bullet point right who's responsible for making sure that it gets tuned right you are if there's something about your NetApp system that's not tuned quite right you pick up the phone and they're going to tune it for you aren't pros and cons the best thing about software to find solutions like SEF is that you've got basically infinite scalability infinite adaptability just a nearly infinite number of choices and just all the flexibility in the world what it is that you want to do with the beast when you're thinking about a software to find storage system like SEF though make sure that you have a good idea of what it is that you're trying to find because maybe your legacy every day run of the mill current day shelf technology is the right thing for you depending on what your use case is you need to think about the pros and the cons the balance that you need to strike between availability of your stuff and the density of the stuff that you're trying to store SEF does not do well at 90% you got to keep it 80 82% full or you're going to start seeing problems your data is going to be rock solid SAFE it's going to be there it's never going to get lost it's never going away but it's going to slow down you need to think about your performance in terms of IOPS now whenever anybody talks about storage they say oh IOPS I need IOPS I've got to have the fastest performance that's not true you need to really think about that's your data usage what's your use case because you might not need the very fastest access that you think you need if you do the calculations you'll have a much better idea of how you need to balance these kinds of things and then of course you always need to consider what the cost is if you just want the the latest net app stuff that's going to give you the best iSCSI performance and you've got the coin for it do it but if you want to have a lot more flexibility and if you want to have a lot better long term growth and if you want to be open then net app is not the way to go I'm not bad mouthing net app I love net app I've been a net app customer off and on for a long time boy they know this they know the business they're good at what they do but it isn't everything right and I'm going to mention that here on this last bullet point right here you have a lot more hardware options you can just buy anything that you want but you need to also understand that now you get into something like software defined storage and now you're in control of that software environment and you are the one who has to do the management of all of that so there's tradeoffs with that one size does not fit all you may not really be the best use case for software defined storage software defined storage does not just say we're going to solve everybody's problems on the planet it's just not the case seph is not the best answer for everything one thing that you need to keep track of is the network the network is just as important as the nodes in your cluster you've got to have a competent network in terms of hardware you have to have a competent configuration of that network in terms of the software and you've got to have some competent people that aren't going to screw up your network configuration so you as a storage administrator have got to cooperate with and have some coordination with your data center network people and if you're able to do it make sure that you get in your agreement with your data center people that they don't touch your hardware especially your network I'm in control of my network because I don't want you guys coming in and messing up my storage network then nobody can get their data and who are they mad at me not the infrastructure guys not the data center guys they're mad at me because I'm the storage guy you let them mess up my data as with everything choose the fastest network that you can afford please don't buy a network that you can't afford it's not worth it you'll need to have two separate networks this may be two separate physical networks but it could be two separate logical networks it's perfectly capable of managing that sort of thing but typically you need to make sure that the back end communication between all of the OSDs the private network the storage network is roughly two times faster has two times the capability of the public network the public network is the network that's where the clients are coming in and of course the fatter your infrastructure the better of course I want to use gig E can you use a hundred yes hundred is better than forty forty isn't necessarily better than ten because forty is just four tens and it's just kind of weird but anyway we can talk about that more outside if you want to because I'm running out of time you need to have competent hardware you need to make sure that you've got good CPUs and good RAM but to just get what you can afford you'll want to have good storage controllers and you don't want to just get some storage controller that can put twenty disks on it because then where's the bottleneck and where is the liability the storage controller if the storage controller goes bad did you lose data if you've got it all crammed into a shelf of twenty disks you've got kind of a problem don't you so spreading it out is better and to have good storage controllers is a good idea use your SSDs and NVMEs for your journaling and for your ROCS DB your metadata taking care of all the metadata on the disks and use your spinners for just your raw data and it's easy to configure this with Bluestore it's just simply a matter of saying yep put my data on the HDDs and now I've got one SSD with these four or six spinners and now that one SSD is the journal and the metadata server for all the rest of the spinners that are there it's a great performance combination that's what I meant to say more about journals spinners good this is something for a different talk on a different circumstance if we had more time we'd talk about these kinds of things the great takeaway here is that SSDs as good as it is and as much as I'm trying to sell it to you it's not a golden hammer it doesn't solve all problems in small simple environments if you only have terabytes of data and if your data is fairly static and it's not going to change very much then just go get yourself a nice shelf from the bell or from HP and call it good and particularly for single threaded ultra high performance requirements SEF is probably not your best choice today we keep working on higher performance, higher throughput kinds of changes and tweaks to it but the emphasis is on reliability and durability of data so software defined storage in SEF best accommodates long term growth to give you the best opportunity to increase capacity to manage more complex data over a long period of time is the best but it's not the best if there are other things that that are in play might have you think not to use SEF so sizing considerations and then we're here at the end so thank you for letting me spend some time talking about software defined storage like I said if you don't like my presentation go see Alan Ott on Sunday I forget what time it is but I think he's over room 107 I can't remember look him up in the schedule Alan's a really smart guy he'll tell you all the stuff that I didn't tell you today I'll open it up for a few questions, Doug's got the mic I have two questions the first question is you mentioned if it's designed poorly that leaves a lot of problems you hinted at it with how you set up the components and how many you have but what are the big three things that people do wrong when they set it up that causes problems and then my second question is isn't SEF good for certain types of workloads as is Gluster and then there's other workloads that it's just really not suited for and you shouldn't go down that road great questions let me ask this answer the second one first he's talking about specific workloads definitely true and if we had had more time I would have drilled down into that in a little more detail Gluster, so EDC the consulting company the analysts company guys they call storage mechanisms by two different types in fact they called type one and type two and type one is the traditional variety that is more what Gluster is after and that's the more legacy common kind of storage traditional storage that we see today and then EDC calls type two the software defined approach that's intended to be more flexible more agile and greater scalability capabilities so if you just have a type one application for your business for your work need for your use case Gluster is great or just your everyday run in the mill EXT4 on a Dell shelf and you're happy right but if you have long term scalability needs with high demand for durability and reliability and manageability then Ceph and SDS is the way to go likewise if you're talking about a bunch of small files then Ceph is not the best Gluster is a better choice traditional storage is a better choice if you're talking about large variety of files particularly large files and rapidly changing files then Ceph and software defined storage is a better choice for you there now back to your first question which now I've already forgotten what was it just shout it to me real quick oh design yeah what are the three bad things that people do yeah too wrong is that they forget to enable big big big what do you call that jumbo frames they forget to enable jumbo frames and you have to have jumbo frames node to node has to go all through the data path that's one mistake that people make another one is they're just too small right they try to put their cluster on 100 megabit network cards and switch and that ends up being a problem and you'll be surprised right one mistake that a lot of customers make particularly in terms of hardware is that they don't realize that cables matter you know they go into the closet and they pull out those old dusty cat 5 cables that aren't being used and they plug them in and you're not going to get what you need out of those cat 5 cables you need to have and you need to test your infrastructure hardware and software going into it those are the kinds of mistakes that people make next question just a question on SSDs you mentioned that the enterprise SSDs and I remember researching this when I did my cluster setup I haven't used CEP yet but it seemed to me like the main advantage of enterprise SSDs was the over provisioning and it was actually cheaper to buy just more prosumer SSDs than to get the over provisioned enterprise ones the only other difference I saw was in the firmware of the FSD and that was related to rate controller but it seems to me like better off using a JBOT setup than letting step manager fair enough so I'm still going to tell you to use enterprise class and if you go look at Kifu I can't remember if it's Kifu or some other engineer I can't believe I'm drawing a blank here but one of the engineers keeps this website of all the performance data of all of the SSDs and you can see that those consumer grade SSDs perform great for three weeks and then the performance just crashes and we're not talking about tailing off, we're talking about spike straight downward and it's just because they're not engineered to the same specifications I highly recommend for a few pennies more that you get the enterprise ones next question so actually we're going to blanks up next and if you have questions for Craig you could test test ok let's get started because I have too many slides for not enough time I have to be really fast my name is Frank Kaliczek I'm an open source guy for a long long time for 20 years it's a bit scary but yeah I started to contribute in all kind of different projects KDE at the beginning part of the open source community obviously like everybody in Germany I guess and then founded own cloud successor next cloud this is also the topic of the talk so I want to talk a little bit about the next cloud obviously what it is how it can be used how to get it, how to contribute and then time for questions hopefully so who knows next cloud ok so that's like half so I want to give you a little bit of an introduction to the elevator pitch at the very beginning so next cloud is an alternative to Dropbox to G Suite and to Ovid 365 with fundamental differences obviously that it self hosted you can host it where you want you can run it where you want you can host it on a tiny tiny Raspberry Pi you can run it in a big data center you can actually run it on AWS or somewhere else you can choose where it's running 100% free software so all the components are serviced everything is 100% free software and it's all built around distributed and federated ideas I will talk about it later so this is the web interface it has a web interface if you log into next cloud you see something like that it looks basically a file manager if you have files in your folders you can search them and you can filter them and you can in the sidebar you can do all kinds of things you can manage your files with it but a lot of the users of next cloud actually never see this they never see the web interface because they use our clients so because you have really rich clients on the left hand side is an Android app for smartphones and tablets where you can do all the same thing natively on your phone and your tablets in the middle you might think this is the iOS client we actually have an iOS app and this is the normal file manager from iOS itself where we have a plugin you can see that there is iCloud from Apple and just below it is next cloud so next cloud is a first class citizen on iOS you can actually open and save and store and do all kinds of stuff from all iOS apps directly inside next cloud very similar to the same as with iCloud from Apple and the right hand side you can see a tablet in the face also iOS in this case where you can see that there is more you can do if you have bigger screen you can do more stuff manage your files, look at PDFs and other stuff so we have very good clients and also we support also all kinds of other rich use cases on modern devices and here for example push notifications we have rich push notifications where you can get a notification if someone wants to talk with me in a comment or if you are running out of quota and all kinds of other things and even works on other devices like smart watches or televisions and all kinds of things that support this kind of stuff and we also have desktop clients of course for Mac, Windows and Linux this is built with Qt and with C++ it's basically for synchronizing folders from your laptop or your desktop to your next cloud server very similar to OneDrive or Dropbox but it's more flexible because you can actually synchronize to different servers, you can say I want to synchronize my photo folder to my private next cloud and then I want to synchronize my work documents folder with my work, next cloud and so on and also there we have very rich integrations, you can just click on the file and directly from the file manager and again this works on Mac, Windows, Linux and Linux we support GNOME and KDE Plasma you can directly share with the person directly on the desktop you get this nice dialogue where you can configure expiration dates or set passwords and all kinds of things and you also got push notifications on the desktop so this is all basically all possible on all platforms and you never have to use the web interface you can do everything the web interface you want but it also works natively in all common desktops integrated you can have all my files in next cloud that's great what you can do with them first of all you can look at them of course, here for example it's our built-in PDF viewer you can look at PDFs if you want if you upload pictures then there automatically picture gallery is generated from all my pictures so again all on my own infrastructure self-hosted might look similar to Facebook and Flickr and other stuff but this is all on my infrastructure and you can also share dialogue on the side always share with other people set expiration dates, set a password and so on support two-factor authentication so usually you integrate into some kind of LDAP or Active Directory in the backend and this can do some kind of advanced security but we also support two-factor authentication with hardware tokens or second-factor wire SMS or OTP protocol so it's very secure we have this Federation feature which is very flexible so let's say we have many many of those next-cloud servers, let's say one of the next-cloud server is my that one is my next-cloud server at home and this is my work next-cloud server and here's not next-cloud server at a service provider and we can all share folders with each other so it basically behaves from a user perspective as one big service it has the same usability and can do the same thing we have a single drive or Dropbox but there is no central server there is no central components or completely federated and distributed we use the OpenCloud Mesh API something that we redesigned but other vendors support this too this was built together with CERN and other organizations so it's a fully distributed system another concept is data access engine where you can mount other existing data sources into your next-cloud DriveShare or SharePoint as three FTP storage something you can all mount this into your Dropbox into your next-cloud and consolidate data like Dropbox into one thing a lot of organizations have legacy Dropbox accounts or legacy Google Drive accounts and you can say look employees you're no longer allowed to use it but you can mount it into your next-cloud and then you have a migration period of let's say a year or something over one after another and then we kill the connection completely so this is useful for that but if you have your data then all aggregated in your next-cloud you can do useful stuff like full-text search where we can use elastic sea