irst of all, it was the first time that I realized how much I love DevOps, and the second reason is was that it was the first time that I ever said out loud that I believe that every developer should practice DevOps, and now after that I said it, now we can really start talking, so hi, my name is Noah Barkey, I am developer advocate and full stock developer for about seven years. I am also a tech writer, one of the leaders of GitHub, Israel community, which is the largest GitHub community in the whole universe, and I work at an amazing company called The Tree, where we help developers and DevOps engineers to prevent communities mis-configurations from ever reaching production. Now, why am I telling you all of this is because before we launched The Tree, we wanted to learn as much as possible about the common mis-configurations and the pitfalls. Yep, works. Good afternoon, scale. Welcome back to the cloud-native track. It is, these are the final few sessions for the Friday, for today, which is, I think, Friday, and if it is, in fact, Friday, it is my deep pleasure to introduce Fritz Holhland, here from the Netherlands, to talk to us about basically, and this is I actually, you see it on the screen, memory usage and file.io, and yeah, and I think there will be actually exciting demos here he already mentioned, so I'm going to be very excited. Please, please be kind during the demonstrations, and I look forward to your talk. Thank you. Thank you. Thank you all for joining me. My name is Fritz Holhland. As you could hear by my pronunciation, I'm from the Netherlands, so like you said, please bear with me, English is not my first language. I work for a company called Yugabyte, and Yugabyte is a distributed database, which is a very interesting technology and actually has a great match with the cloud and with stuff like Kubernetes, because we created it from the ground up to use it. And one of the things that I have been looking into and have been talking to with a lot of clients in my role as developer advocate is on how to configure things. And that is what this talk is about, is about Linux and disk.io and disk.io performance with relation to memory. And that is actually really interesting. And if you look at Postgres, Postgres uses I.O. in a way that memory of Linux is really important. So I think this is a talk which is really good for Postgres people to look into too. And oh my God, what is it? If you look at cloud native applications, so meaning running in the cloud, what you typically see is that running in the cloud you try to use as small machines as possible. And with as small machines as possible, it's the lowest amount of CPUs because that's cheaper, but it's automatically also means you have lesser memory, which is really interesting because these machines have with it a limit on the amount of I.O. they can perform. And that is a really interesting topic which I think has a really important relation to this presentation, which is on one hand a bit philosophical, but on the other hand is really important to understand. Think about this. When I started doing database technologies, we disks were all the thing you absolutely had to look at and understand that the disk rotates at a certain speed. And because of this rotational speed, it meant it could give you data at a certain rate. So disks, the rotating disks were absolutely a bottleneck. And if you think about where this is going with cloud, please bear with me because I think this is really important to understand if you want to understand and learn about disk I.O. performance and cloud. So we have this rotating disk and they have a certain limit in how much bandwidth and I.O.s per second they can give you. Luckily, in the 90s came rate set. And with rate sets, you could use multiple disks as one logical disk, which meant that you could use more bandwidth. The latency didn't go lower, but you could use more bandwidth of the disks. When that started to happen, at the same time network attached storage and storage area networks came into place. And with these, you had specialized appliances running I.O. for you, which meant that they could produce even bigger amount of I.O.s because they could host more disks than your server probably would have been able to host and the especially SENS and NSS too would have had caches, which meant that they could actually lower latency. That was all great. And then came SSDs. And SSDs were fantastic because SSDs were disks which were sitting with didn't have any rotating things, which could sit in your own server and in your server, they could provide you really low latency. Then came NVMe access to the disks, which meant that you could use a lot of more bandwidth, which was essentially limited to the system bus on your computer. So you had huge bandwidth and very low latency. And then came Cloud. And with Cloud, if you take such a small machine, and as I talked about earlier, that machine is automatically bound to a certain amount of I.O.s and a certain amount of bandwidth that you can use. And with that, if you look at the bandwidth and the amount of I.O.s per second that you can do in a virtual machine, it's essentially, I see the same limits as with a disk which I first saw in the 1990s. That's mind-blowing and weird. What does this have to do with this presentation? Well, first of all, it's important to realize that disk is actually, again, a bottleneck, especially if you use smaller cloud VMs and using smaller cloud VMs is what you probably want to do in the cloud to reduce cost. So now, going back, if you're doing I.O., probably the I.O. you're doing is buffered. And buffered means that if you're performing an I.O., the stuff you obtain is stored in the memory of your Linux server. And if you request it again, it comes from memory. There is another option, which is called direct I.O., which can be set using a specific flag. That is possible. That will completely bypass the cache. And that has advantages, that you don't store this data in two places because if you want to get it in user space, in your application or for your database, in your database cache, then it doesn't make sense to also store it in the operating system. A lot of databases use direct I.O., but Postgres doesn't use it. And Postgres even explicitly in still today, I'm not sure if that's going to change, asks you to set a reasonably small amount for its own cache and use the operating system caching. Now, if you're confused, if you have nothing to do with a database, but just want to understand how your application can use I.O., then, and you're not sure whether you're using direct I.O. or not, probably you're using buffered I.O. You can use LSOF as a utility to just see how your open flags are, but probably you're using buffered I.O. So, where does buffered I.O. go? Well, somewhere on your Linux system, obviously. Linux does not have a true dedicated cache for these blocks. Traditional Unixes like IBM AEX and HPUX did have a dedicated cache. So a shielded area where this would go. Linux doesn't have that. And there is another interesting topic. If you're doing buffered I.O., which is probably most of all of your I.O., then it must be stored as a cached block in your server. Even if you're completely running low on memory, it must be stored as a cached block at one point. Even if it is flushed immediately, writes are different. Writes are special and I will get to that. The important part is, and because I've done Oracle for a long time and Oracle, you should set direct I.O. for Oracle. When I started looking into buffered memory, then I realized that if you're doing buffered memory, it competes equally with applications just trying to take memory for the purpose of growing their heap for performing their work. They compete evenly, which is really interesting to think about. And I know numerous cases where people, in this case, again, Oracle, but I've done Oracle for 25 years. So I have a lot of experience and references to that. I know a lot of people doing Oracle where they very carefully crafted all their memory areas. Very carefully because databases are really sensitive to memory and especially shortage of memory. And they would work and then all of a sudden, they would find that their system had swapped, which they totally didn't expect. And I, for a lot of times, didn't understand it, but it wasn't a problem because the system wasn't actually swapping but swap was taken. And now when I investigated buffered I.O. In hindsight, I realized what happened. The backup is done and the backup is probably done buffered, which means it competes with normal applications and therefore it would push out certain pages which haven't really low touch count. The Linux buffer cache keeps a touch count of all the buffers and it will push out the one with the lowest touch count if you're running low on memory and push these to swap. But if that is a bootstrap, a page which contains your executable code which is only used for starting up your application, this page will never be used again. So it's not a problem. So that is another point.  And I've seen a lot of tickets which people try to lower back their swap because that was what's supposed to be how it should be. Don't, that's not necessary. Okay, so where does Buffered I.O. go? So I haven't said anything about it. Well, Linux provides an insight into how memory is divided in Proc Mem Info and messy sounds a bit negative on the slides. I say it's a messy gathering of statistics but it's actually just a whole bunch of statistics and some of the statistics have figures in it which are also doubled by other statistics and some statistics aren't kilobytes but are pages. For example, for the large pages. But very roughly, if you look at the statistics for cache dirty and mapped, these are roughly what is that. What is the cache of these blocks? That's not really precise, what I'm saying. I know. What I'm trying to say is it probably is not useful and handy to try to exactly figure out how much of these pages you're caching. It fluctuates, also it fluctuates all the time. But you have to have memory available for Buffered usage and actually the best indicator which is very much non-set. I cannot find a lot of reference to this but the best way to assess if you have enough memory is the statistic Mem available. It's a, I've forgotten the exact Linux version but it's Mem available is not a really ancient statistic but Mem available is what is actually potentially available at that point in time that you are querying Proc Mem info. One of the things that I get asked quite frequently is what about Mem free? Well, Mem free is not free as in available. Mem free is actually a small amount of data which Linux tries to keep free for the purpose of very quickly handing out freed memory. Linux, and this is a really important rule, probably like most operating system tries to do as little as possible. So if you're dirtying, if you push in data it will just be there. It will not be freed only for a bare minimum which is Mem free. So it has nothing to do with memory being available. It's a tiny part of memory which is explicitly made free for the sake of providing memory pages to a Linux process which requires free memory. Of course, just after startup there is a lot of free memory because you haven't touched all this memory so it's by definition actually free. But like I said, if you start doing stuff it will just put stuff in all these buffers and it will not do anything with it unless it absolutely has to. If you actually need to make memory free you're using the swapper and at Ugabyte we use virtual machines with no swap allocated which I had a hard feeling about it but actually it works rather well. And what you sometimes see on a system with no swap allocated is that the swapper threat of the Linux kernel gets active which is really weird because we don't have any swap. Well, the swapper is actually not really good naming in the Linux, in Linux. The swapper is actually the page demon and it's the swapper who frees memory if a process actually needs it. So, and I've got a link. This presentation will be made available. I've got a link who tells a lot more about it. So, mem free. It's the kernel estimation of memory that is available without requiring swapping immediately. So why would this be important? Well, buffering IO can do miracles for performance and equally it can do miracles for application which require to do IO. And let's test this. Let's actually just see how this works and I provide really simple comment lines. Test it for yourself. If you're having doubts, do the actual good thing and test it for yourself because everyone uses memory and IO in a different way. I'm doing it on an Amazon EC2 machine which is of the shape a C5 large and I've got these weird numbers here. Oh, you don't see my pointer. I've got these numbers there which says C5 large VM, 20,000, 4,000 IOPS. The first number is the bursting number and the second number is the base number and that's also for the megabytes per second. How does that work? If you have a small machine, then in a lot of cases you're allowed to run with bigger limits for a small amount of time and that is specified in the documentation how much it will reduce it which is called bursting. If you're using a virtual machine professionally, you should not use bursting. It can absolutely drive you nuts if you're doing something at a good rate and then for reasons unknown because the Amazon platform reduces this so you can't see it. Linux doesn't give any indication. It just magically gets lower. If you're bursting limits, you're bursting credits I should say. Amazon, it works based on the credit system, goes lower.  There the limits are, these have limits too. So you have to combine these limits and obviously the lowest of the two is what is applied. In my tests, I'm not running into my bursting limits so then the lowest to apply so I can do 3,000 IOs per second and 125 megabytes per second. These are my actually limits. There is a page, if you know, even if you know what you're looking for, it's, I think it's hard to find that page but it's really important to understand the limits so you can use this link to see your EC2 limits especially if you see weird things and now when I say this, you think, hey, maybe that's it. So I have a C5 large, it's a four gig machine and what I'm going to do is I'm going to use FIO and I'm reading two gigabytes in this way. The switches are all there. A word about FIO, FIO is a brilliant tool. You can do anything with FIO but in its brilliance, it has so many options and so many options have been totally thought through that it sometimes is really strict in what it's doing which means that it might not actually be doing what you think it's doing and one of the things for which this is true is I have a switch which you might not have seen a lot which is a dash dash invalidate zero. Dash dash invalidate zero means to not wipe the cache before running it because I want to run it two times and I want to show the advantages of using the cache so it would not be helpful if FIO would wipe the cache for the second one because then I would be doing exactly the same, right? If you're doing things with FIO, validate, cross check that FIO actually does what you think it is doing. It's really important to do that. I find that I want to look into specific behavior. It might not do what I am thinking it is doing. So this is how it looks like and I hope it's a video so I hope it's big enough and I, yeah, it is doing stuff. This is a utility and I'm afraid that I cannot enlarge it because it's a video so this is, I wrote a small utility in Rust to show the memory sizes and I will show the end output in a bigger font later on so essentially I'm running stuff so I'm looking into this and then I'm pasting FIO and by pasting FIO it will start reading data and then it produces the output and this is the highlight. I'm doing a 2,609 IOs per second which is 21 megabytes per second. My limits are higher so why can I not reach the limits here? Is Amazon lying about its limits? No, it's the latency. The latency is really good. It's 0.3 milliseconds per IO but still I need to do them and these are all sequential and I'm reading it for the first time in my cache so I'm still bound by the latency despite the fact that my latency is really low so this is really great but still I'm bound by the latency by doing IO. I do not hit the limits of the device. Now, if I perform the exact same run again, well, this is how am I doing it but it returned immediately and these are the figures. My IOs per second now doing it a second time with the Dash Desk Infill at Zero Switch so I actually can take advantage of my cache. I'm doing 580,000 IOs per second at a rate of 4.5 gigabytes per second. I promised you it can have magical results. I think these are magical results. The first time was okay. Well, actually really good for doing physical IO and the second one is marvelous. This is what you want to see. The reason for it, well, kind of obvious. I haven't done any physical IO actually at all and FIO does show this with the IOs line. It didn't do any physical IO. It just read it from memory. Now, how about four gigs? And mind you, my system is four gigs in size approximately. So I'm doing the exact same thing. The only thing I changed is setting it to four gig. Well, I'm looking at the same stats again. So the summary of the run is this first run is actually alike the other first run. I'm bound by the latency of doing the IOs which are really fast but still you need to read it. So I'm bound by the latency. So the rate is identical to the two gig. Now let's perform the exact same run again. Look at the option dash dash no random map. FIO is really smart. At first I try doing it and running it again but because FIO, if you're doing random IO the IOs are truly randomized but it keeps a map in memory and it very explicitly touches every single block truly only once. To truly only take each block once for reading it which is not the random in my definition because an application which would fire random IOs would touch some blocks multiple times. You can achieve that by setting the dash dash no random map flag. Again, what I said is FIO is so smart and well thought through that if you think, if you want certain behavior please validate that it actually does what you think it is doing. So it is random but only random in choosing the blocks not random in choosing blocks multiple times. I was very surprised to learn that. And if it does, if I don't do this and just touch every block once and I've got lesser memory than the file it cannot keep it and it will actually push through all the IOs so I will not have any caching effect at all. And that is more or less what is achieved in this run. I'm doing loads of IOs because the file I'm trying to read is bigger than memory so I cannot cache the entire file. And just because I touch some blocks multiple times I will get some caching effect but the caching effect here is way less. It's, I'm doing 4200 IOs per second which is where I'm bound by. And that is because I simply touch some blocks multiple times but most of them still need to be read. But the question is I've now done some tests but are these tests actually reality? If you're most of the time if you're doing stuff with IO you have an application which drives them and this application likely will use the data you're trying to IO and will keep that in memory for the sake of running the application. So what if I do the same tests but occupy 50% of the memory? Well this utility I wrote which I actually borrowed from a C program from this URL and reprogrammed it in Rust. I take 50% of memory and I touch the memory so it's actually allocated. And then by that take 50% I map 50% of the memory so it's not available anymore. And then redo the same tests. What is happening? This, I have no idea what happened. So this is running the utility. The first run is identical to the one with without memory taken because that just reads the IO, reads the blocks and the IO it's unbound by the latency and the latency doesn't change. So, but I have, so the, it's generally equal. And if I do it again, because I have lesser memory in my server, I hear now with 2GIG, I already have to add dash dash no random map but if I'm doing it again, well this is too small to be visible, sorry about that. I cannot get the same figures as I had previously. Well, I think in this order, this is really obvious but now think about a scenario where you have an application or a database. I think it's quite common to have a database in an application stack. You just start up the database, nothing is actually allocated and you run a test and the test can take advantage of the caching and you say everything is good. Then a year later if, or whatever the time of the project is and you're actually, the whole application is done, you hook up all your connections to the database and the database is actually being used and you're doing exactly the same and then the performance is totally different. This is what I tried showing here and this is actually a real-life situation which I've seen an enormous lot that people tested actually probably without realizing in an absolute good situation and then in reality, when memory is actually taken and stuff is actually run, performance is drastically lower and nobody understands why it's low because we tested it. Well, this is the reason you have to understand your memory footprint and your actual IO amount that you're doing and you can see it here too. I'm doing the two gig and FIO shows me that I've performed a lot of IOs. You see IOs there at the bottom line. Now let's look at writes. Here I start too with an idle machine and I write something and this is summary of the run. My IOs per second, for writes, obviously I do not have to do two runs. For writes, I just perform writes and it's done. For writes and writes are special. I just said it in the beginning and why I write special. If you're writing in Linux, any buffered write is not done to disk. A lot of people think it's done to disk, it isn't. You're writing into memory and then the kernel will decide at a certain point in time to write the blocks you've dirtied to disk. And that's a really interesting concept and it's not obvious because you think normal human thinking is that if I'm writing, I'm writing to disk immediately but that's not what is happening. Another thing, sorry, yes, there is. Yes, yes, the remark is even if you're writing then the disk control will cache it. Yes, absolutely. I do not want to go down to that level because the amount of stuff in Linux is already a lot. Writes are special. The first thing which is really special in Linux with writes which you have to realize because then it becomes a lot more logical is if you read something and then the memory is needed, this read can be discarded immediately. Nothing is harmed by discarding this block and it can be reused.  It cannot, it will not even put it to swap, a write to swap, it will write it to disk and it must do it because if it doesn't, then it will corrupt the file system. And that is what I meant with writes are special. Therefore, the limits for writes are way lower.  That's the whole purpose of buffering them in the first place. So therefore there are kernel parameters set which are VM dirty background ratio, a VM dirty ratio. And here is a thing which is really not well documented and which a lot of people think these ratios come from the total amount of memory. That is not true. I think it's even documented in that way. I'm not sure if it's in the kernel documentation but there are a lot of blogs describing that these ratios are from total memory. These ratios are from available memory which means that if you take an application and you start allocating memory, then the available memory obviously lowers because you've taken memory and use it for a certain purpose. So the available memory lowers automatically your dirty background ratio and dirty ratio therefore also will lower. And that is really important to understand because that means that then a write which could be buffered in a certain way at one point in time, if your application is really hungry for memory and takes a lot of memory, then at another point in time your threshold for needing to write these dirty pages might be way lower. And actually what the kernel does is if the kernel finds that you're swamping the memory with dirty pages based upon these ratio, if you're performing a write system call, the kernel in the write system call will evaluate the amount of dirty pages and just put a sleep. This is really what it does. It calls balancing. It will put a sleep in the write call and just put your process or threat in a sleep state. It will truly just schedule it off CPU for a certain amount of time in order to try to push back on the writes to try to balance it. And it's called actually balancing. And I've got a link here too to a blog post where I wrote about it where I go to the exact lines in the kernel where this is happening. And the reason it's actually good even if you can't read C to go to the kernel code because there is a lot of explanation about these ratios in the C code of the kernel for this. So I'm writing and I'm writing 500 megabytes. And this is the summary of it. I'm doing 500 megabytes on this system and I get really great figures. It's 193,000 IOs per second and 1.5 gig per second. And why am I getting this really good figures? Well, I've got a system which has no applications taken a lot of memory and therefore this write of 500 megabytes didn't actually produce any IOs whilst running it because I was below the limit. And that is what is on the lowest line is my available memory is three gig and the ratio is set to 922 megabytes. So therefore the 500 megabyte I'm trying to write fits in memory as dirty pages. Brilliant. Now let's look at reality too and I think you understand what the reality stuff here is doing. If I take 50% of memory then my available memory will lower obviously and therefore my percentage of the limit will be lower and therefore if I produce the same write of 500 megabytes and do it again in pretty much the same sense as my reads now all of a sudden this write will be way slower and that's not because the system has magically gone slower. No, there is a very good explanation. We've taken a lot of memory. We have way less cache and therefore the kernel will do exactly the same if you cross a certain threshold it will determine that you're trying to flood the system with dirty pages. So therefore it starts throttling your write calls in order to balance the memory. And well this is all doing that and well write two gig I don't think I can add a lot if you know this and if you understand that these limits gets lower. It with two gig the first run of two gig will already go over the limit. So with 50% taken it will also go over this limit. So what am I saying? If you have a system running in Kubernetes in a pod which is probably has a certain amount of memory and you rely on IO performance at a certain rate do you and you know you're doing Buffert IO do you keep track of available memory and do you understand whether what you're trying to do fits in with how your memory is sized? Do you know that? Because I don't know a lot of people who actually actively keeping track of available memory whilst if you're relying on performance this should be the parameter to look at because that is what is most telling about what is available for it and do you understand the amount of reads and writes that you're doing? And this is a really tricky one and in my past as an Oracle database administrator I saw that a lot too. If you're doing a certain amount if you're using a certain amount of pages now it doesn't mean that you will be doing exactly the same if you're a few months or a year further. Probably if you have a successful application there'll be more users, there's more data so you're maybe are doing more IOs. So even if the application absolutely didn't change if everything kept on being the same you could still hit a limit simply because you're using more data and your memory amount is fixed. So therefore you hit the threshold and therefore your fast IOs from cache cannot work anymore because you're requesting more IOs. And it's important to understand the differences between reads and writes. And this is active data set that's what I just talked about do you understand how much data you're requesting all the time and do you know if it grows or not? Well I said this at the beginning and this is actually my last slide so then I'm open for questions or any remarks in that sense. The tests I did were performed on a system with no swap device that is how we do it and for simplicity and understanding where if you have a memory shortage that is, I was kind of surprised but it's actually refreshing because you do not have to swap because I see swap as kind of a question for falling. If you have too little memory you have a problem period or your performance is influenced let me put it more friendly your performance will be influenced. If you do not have swap you will hit kind of a wall because your memory is limited. If you have swap you will be hitting the wall too. No question about that. I know if you're running out of memory you're running out of memory period but you have discussion that it is a little less painful within certain limits. So I'm not advocating totally against swap but I'm also not advocating for having it all the time. You have to think about it whether it makes sense and if you want the complication especially if performance is critical. So well and then the last remark is Linux ages buffers based upon an LRU mechanism least recently used. So if things swap out it's not used in a long time I can guarantee that well it's based upon the LRU state of all the pages obviously but it isn't used in a long time so if you swap out some stuff it might not really be well it means you demanding memory which you don't have so that is the reason why it would swap in the first place but it's not really a problem if you have some swap being in use because the Linux corner will try to do as little as possible so if it's swapped out pages and you're not requesting them they will be sitting there swapped out as long as the application from which they came from is still active and it's not a problem. A lot of people think it's a problem it's not a problem. So that is the end of my presentation so hopefully you've enjoyed it and it gave you a lot of thoughts. I will repeat the question. The question is how does the right throttling work? I think that is a really short version. The right throttling works but essentially it just tries to add time to prevent the system from getting swamped with pages like I explained so it will do a little and I think it will go up to two seconds or something like that but you have to read the source code if you go to the link you can see it it's really interesting. There is a Perf you can set a Perf probe on the kernel how is it called? It's called a probe but on the kernel function which executes this which will then tell you whether it did the throttling and for how long. So that is how you can see it but there is no that is a really weird well for my mind Oracle is really well instrumented there is no other indicator that this is happening and that is the reason why it was kind of surprising to me that this is happening in the first place because there is no statistic that will tell you it happened. I will get, I think, yes well the way thing which happens in the right call is actually the Linux scheduling of CPU for a certain amount of time. No that is, sorry, no sorry IOS that shows the block device statistics so that has nothing to do with that. IOS that will show you how it communicated with the disks not how the process performed right. So, yeah, it's really close but it's really two different layers. Prox, so the Linux kernel has statistics and that's Prox, oh PSI. So the question is the Linux kernel has PSI and does PressureStallInfo give more or more useful information than ProxMemInfo? I think they serve two different purposes. ProxMemInfo gives you a lot of details but it really doesn't tell you anything about performance and PressureStallInformation is all about if I'm executing IO, what is the impact on running because that is what I think the stall information is showing you so in my mind they show you two different sides of doing IO or well actually ProxMemInfo doesn't say anything about IO just how memory is divided between all the pages in a lot of details and PSI will tell you that but I haven't looked too deeply at PSI because it is a fairly recent feature. There hasn't been a lot. Well, there is enough blog posts about it but not an overly lot and I haven't seen a lot of people actually using it. That is what I've seen which is a shame because I think it's really interesting and if you have the opportunity to enable it I would certainly do it to see if you can make sense of it because it gives you actually information based upon the perspective of the process running the IO instead of just figures for an amount of IO's that you're doing. So it's a really good thing to mention but I haven't got a lot of experience sadly. Okay, I think I understand your question. Your question is okay, you've talked about it but what should I do now with this information? So the question is well I think this question still is okay. You've told me a lot of details about how this works but what should I do? What can I do with it in a practical way? Well, a lot of these details would mean... You talked about the examples with 50%. So does the swapper process get running at the point that both of those examples are read and write when you don't have a swap available, a swap file or whatever you use, a swap space. So could you see that process running at those times? I try, so your question is if the problem happens, what can I see? What I was going to say, which I think like you said is quite closely related to your question is what can you do? Well, if you want to get down to the bottom of it, it will cost you time and you need to build up some experience and probably if you have a whole form of service, you do not have this time. Well, get a lot of the details and just get them. Prometheus will get them for you so you can research and go back to these details in time because that is a thing which Linux doesn't provide to you but with, for example, Prometheus or with a lot of other tools, you can get the statistics back in time so you can figure it out if you're running into a problem. One of the things which I think is really important to understand is you need free memory if you have an application doing IO, doing buffered IO, you need free memory which for the perspective of someone who is trying to size a system is memory not doing anything, you need free memory for the sake of this caching and this is what I see being left out all the time. People would carefully size up saying I have an application which takes so much memory and I take so much for the operating system so this is the amount of memory I need and what I'm telling you is if you're actually using that memory, then there is nothing left for performing this buffering so then your IO will fluctuate all the time, it will be good one time and then be absolutely crap the second time and it's kind of hard to figure out why. Yeah, well the first realization is you have to understand that you have to size extra memory which from the perspective of sizing memory is that memory doing nothing but you actually need it. Exactly. We have time for one more question and over here was I saw Henry's over here. So you've switched between two things. One in my mind is that you have an Oracle database that pretty much owns the machine in certain cases and then but then you also talk about Kubernetes where my pod that is doing read and writes is fighting with other pods that are all using the combined resources and machine so my pod though it could be memory limited the caching and the read write layer is really being shared by all the other pods so how do I determine what my pod needs because I don't know what the other pod owners are doing with their pods as well so once again I understand that if you had an application that owned the box you can make these calculations but once you say Kubernetes pod you're fighting with other people. That's a very good remark and this is something which I generally find lacking in a lot of talk about Kubernetes because indeed you have a single physical box and you run multiple pods, multiple machines which do not see each other which try to use these shared resources which they all share in the same machine and I think this is a very big under an aspect of running Kubernetes which is greatly underappreciated on looking into because what should be there but that is an opinion I'm not saying it should but simplistically thinking if you're having resources and you know the amount of resources you should divide them and just say okay well for CPU I know you can do that you can already divide so you have a fixed amount of CPU memory is also divided per pod as far as I know so you can do that, figure out if you have enough memory but then indeed your physical IO devices are shared and actually in my opinion your physical IO should also be fixed per pod so that you can carefully cut out and say I've got 3000 IOs and especially in the cloud and with a lot of other machines you know your amount of IOs per second that you have and the bandwidth that you have and say I want 20% of this IO to be dedicated to this specific pod which is running here so this is a resource which need to be there if this pod is running on this physical machine but that is an opinion I have and it's not that I want it the point is if you don't do that then exactly what you say and again this is what I see as a big problem your usage and your latency will come across as very random because at one point in time you can do a certain amount of IOs and 20 minutes later you can't do it because you have a few other pods who also try to share the same resource and therefore you get a lower amount and this is a big problem if you try to do quality of service and say I want to make sure that it's running at a certain rate absolutely I think that is a missing part so yeah I wholeheartedly agree with what you see and absolutely true Thank you so much for feeling all these questions Fritz Really awesome talk Please feel free to continue the conversation I'm sure Fritz is happy to catch you later or outside we will be back in about five minutes at 16.30, 4.30 p.m. for delving into more cloud native database related concepts That's fine Looks What do you say? Geneva right? Plenipere All right Good afternoon everybody Come on in Settle down and I'm excited to introduce Rog Srinivas who is here to share about Cassandra running on Kubernetes and just a little bit more of a deep dive on databases that are cloud native so please give him a hand and welcome him to scale Thanks for being here Thank you, Geneva Glad to be here but it's always kind of dicey to be the last session Oh, there's one more I guess after this, right? So I guess I'm okay So in terms of history I've submitted several times to scale but this is the first time I'm speaking at scale Really excited How many of you have already attended multiple scales? Looks like all old timers But treat me gently In any case I'm here to talk about Cassandra running on Kubernetes and multi-cloud Kubernetes so a lot of different topics coming together Has a little bit of a story because it started at KubeCon U actually in LA about a year back, October when I did a multi-cloud or a multi-region actually Cassandra on GKE because GKE networking is probably the easiest among all the other clouds, right? But fast forward through Reinvent and KubeCon EU and all that Now I'm in a position where I've been able to install a multi-cloud Cassandra and you will see that in action today in a lot of demos today, okay? A multi-cloud Cassandra running on GKE and EKS which is installed from AKS, okay? So I think it's pretty cool you'll see that it took me a few months to actually make that happen and I think it'll be even more exciting now that Satya Nadella and Larry Ellison are talking about multi-cloud, right? So everybody is talking about multi-cloud but my question to the audience is how many of you are already doing multi-cloud? A few people and I know multi-cloud has different connotations, right? You know, if I use different clouds it means it could mean multi-cloud but in this context it really means my application is spread across different clouds. So with this new definition where my application is running on multiple clouds at the same time how many of you are doing multi-cloud? A few, okay. And all of you, I mean I assume that three of you are using Kubernetes or something else or kind of, yeah, yeah, okay. Okay, so let's talk about this and we'll see where it takes us, okay? Oh, by the way, my name is Raghavan Srinivas. I work as a developer advocate for data stacks. You know, I was a mechanical engineer but since then I've done a lot of distributed systems middleware, some of the things I like to see is code in action and that's why I'm gonna do a lot of demos today. And I have a colleague of mine, Matt Overstreet who is much smarter than me so I'll delegate all questions to him, right? But again, feel free to talk to us if you have any questions about Cassandra. He did a great talk. I recommend you kind of go take a look at that as well. You know, where he talked more from an application patterns perspective rather than from any infrastructure perspective. I myself, I live at kind of the confluence of both infrastructure and applications. I just want things to work. I come from a Cloud Foundry background. You know, I just want things to work. You know, I don't know if you guys have heard of the famous CF push and everything is stood up.  So I'm also a big fan of kind of the inner loop. Inner loop is when you have a kind of a lightweight CI CD cycle and you know, some of you used scaffold, jib and kind of those tools. It makes it a lot easier for somebody who's an application developer on Kubernetes to be able to do this, you know, mundane tasks over and over again, like maybe 100, 200 times a day. With that said, how many of you were able to attend CAD cost gross talk this morning about Kubernetes? Yeah, that was a great talk as well. So I recommend, you know, if you're new to Kubernetes, how many of you are new to Kubernetes? I mean, like really new, okay. Okay, that's about half. That's kind of what I expected as well. I would strongly recommend you attend that, I mean, not attend the talk because it's already done, but you know, take a look at it or if you have a chance, you know, kind of see her at some point. We have a great crew behind us. We do workshops every week. They're all free to attend. We have a tier which makes it possible for you to actually run some fairly sophisticated production workloads and you get a $25 per credit. I mean, $25 per month credit without even having to put any credit card or anything like that. The main goal of our data stacks developer crew is to up level, you know, your learning of your experience or whatever that case may be. And I myself learned a lot in those workshops. It happens every Wednesday at 11 o'clock, 11 Eastern, I need to be specific about that, which is a little bit early for Pacific, but we try to get here to a worldwide audience. We get about anywhere between 200 to 800 people. So feel free to attend that if you want some more basic information, because today I'm gonna be going into a little bit more advanced topics, okay? But even though I'm gonna be doing a lot of advanced topics, and by the way, I've been criticized for being a little too loud, okay? Which you can see, it comes from excitement, okay? So if I'm too loud, you know, just let me know. I will pipe it down, okay? I'll try to anyway. So the agenda for today is just do a quick intro, which I already did, right? And then for those of you who have never heard of NoSQL or Cassandra, a very quick intro of NoSQL and Cassandra, and then a little bit about Katsandra, which is a play on Cassandra and Kubernetes names, right? And I think it's a pretty cool name. It's Cassandra running on Kubernetes. And then of course, the multi-cluster, multi-cloud, using the Katsandra operator, which is introduced very recently. So when I started my GKE adventure, there was nothing as a Katsandra operator. There really wasn't. So I had to do things manually. But now I don't need to anymore, and I'll show you how we can do that. You know, it uses, and maybe I'm giving away everything right now. But basically what Cassandra does is uses something called as a gossip protocol. And as long as one node from a Cassandra perspective can talk to another node, you enable the networking to make it happen, then essentially it can form a larger cluster on its own. And that is all that I'm gonna show today, okay? Basically it doesn't matter whether it's multi-region, multi-cluster, multi-region, or multi-cloud, it works exactly the same. And the gossip protocol is pretty cool because what it's saying is it's just not kind of like a heartbeat kind of thing. It actually exchanges more information to the level that each node has an idea of what the other node is doing at any point in time. And not too much information, it's not too chatty, but at the same time it's not really basic information. And as long as you enable the networking, you can make it happen. And you will see one or two instances of that. I won't go into my first adventure, which is with GKE, but I'll go into EKS where I use something called as EKS cube fed. Unfortunately, cube fed is not, anybody using cube fed anyway? I like to talk to, yeah, I don't see any hands at all, unfortunately, but EKS cube fed is a great project and I'll talk about that. And then I'll do some demos and then finally, kind of point you at some resources and hopefully we can get out of here much before half past five, okay? All right, so no big deal, no sequel was a, somebody came up with this catchy hashtag, but I was corrected that somebody actually had invented this term, but I think it got popularized a little bit more by this meetup on June 11th, 2009, roughly 13 years ago, okay? So really if you think about Cassandra, no sequel and all that, it's not fundamentally new. It definitely isn't, but if you think about the cloud and no sequel, there are a lot of commonalities, okay? One of the biggest things about no sequel is that it's really about horizontal scaling, commodity hardware, cheap hardware, but tons and tons of them so that even if failure happens which is gonna happen in distributed systems, you can still deal with it by having a replication of the data and you don't need to worry about any of that as a developer. It's all automatically taken care of for you. That is in general the philosophy of no sequel. Of course, each of the different platforms vary a little bit, but we will look at all of this in just a second. There are a lot of examples, Cassandra, Mongo, Couch, HBase, Couchbase, and so on and so forth, okay? But in general, what it is is about horizontal scaling. It's because with a relational database, you can go up and up and up and up. At some point, it's gonna keel over and fall, right? And really, there are ways in which you can do, you can spread your data on a relational database, but it's very clumsy, very awkward, and a lot of times you have to do it manually and it's never foolproof, right? Whereas no sequel, on the other hand, was really conceived with the idea that distributed systems failure is gonna happen, so what I'm gonna do is I'm gonna spread the data across. I will worry about how to spread the data across. I might use some kind of consistent hashing algorithm. Don't worry about it, I got it for you, right? And what happens when some piece of data that you're trying to access actually is not available at that point? Those are the things that the no sequel database is gonna deal with. It's really meant for the cloud even before there was a cloud. That's kind of how I put it, okay? Some of you might have heard of this famous cap theorem or sometimes referred to as a Brewer's Conjecture, but essentially what it says is in the event of everything running fine, everything is fine, okay? So in other words, I can have consistency, I can have availability and I can have partition tolerance and I can, well, failure means even partition tolerance, but in essence, when a failure happens, you have to pick two of the three, okay? Unfortunately, you have to give up one of these, right? And that's the way the distributed systems work, okay? So either you have to give up consistency or you have to give up availability or you have to give up partition tolerance, right? So you can't have all three at the same time. And turns out giving up on partition tolerance is probably worse than giving up on consistency because those kind of consistency issues, even my grandmother can notice that that's a problem, right? On the other hand, in consistency, there is this concept of eventual consistency where things might seem to be off a little bit, but it's really not a big deal because eventually they'll become consistent, right? And turns out even for some of the fairly sophisticated distributed applications, eventual consistency might be good enough, okay?  And as you can see here, most NoSQL systems do not give up on partition tolerance, they may give up on availability or consistency, right? And really it's not like it's eventual consistency consistent either, right? You can actually make it what we call as tunable consistency and some people don't may not like that term, but essentially what you can do is you can say, I want strong consistency, but now the system will not be available at that point because if I want a acknowledgement from all of the different replicas, right? Then typically what will happen is, the system will not be available because it may not be able to take care of that particular transaction or that particular right. One of the key things about Cassandra is that there is no single point of failure and there is really no such thing as a master. Basically, you can call it a peer-to-peer architecture if you will, a lot of terms, a lot of times you might have seen the ring associated with Cassandra and essentially every node is kind of treated the same, okay? And all of these is important to keep in mind when you're building this multi-cluster Cassandra because it makes it so much easier to build a multi-cluster given this architecture. And in a lot of ways, remember again what Kelsey Hightower said, Kubernetes is really not a platform, it's a platform to build platforms, right? So likewise, the multi-cluster concept really depends on the application of the platform that you're building on top of Kubernetes and whether you can take advantage of it or not. In the case of Cassandra, actually it's pretty straightforward. Why do you need to partition? Because you can't fit all the world's data in one node, right? So you distribute it to multiple nodes and you shard the data, but sharding is hard, right? And that's why the NoSQL platforms shine because a lot of times you as an application developer don't need to worry about any of this, right? There were some NoSQL systems at least in the early days where you had to worry about the sharding but really, friends don't want friends to shard. It's hard, okay? All right. Cassandra, like I said, is configurably consistent which is probably a more palatable term than tunable consistent, right? So there is number of ways you can do this and like I said, you can make sure that you get consistency all in which case everybody has to reply and so on. This is not so much about kind of putting in logos and a marketing slide or whatever, but the point to be made here is that some of these large, large, large scale, thousands of nodes of Cassandra are running as we speak in some of these bigger companies like Netflix, Apple and so on. Apple is still a contributor to Cassandra and really, it's really about scale and one of the things that Cassandra shines is linear scalability. There was a study done when you went from 100 to 1000 to 10,000 whatever and it was just linear. No fall off, there is no drop off at any point of time which is very hard to do even in a horizontally scalable system, right? So, what about Kate Sandra? Kate Sandra is Cassandra running on Kubernetes, right? And a colleague of mine, Chris Bradford, a great guy again, was a former skeptic of running any database as a matter of fact on Kubernetes and it's probably true in the early days of Kubernetes because there are a lot of things which wasn't quite production quality or whatever you wanna put it, right? And there are a couple of things that even with Cassandra is not easy to do but long story short, Chris Bradford now actually works for data stacks.  So, what is Kate Sandra? Again, it's a cloud native scalable data tier as Matt talked in his talk yesterday. Basically, it's not just about providing the day zero tools but also providing day one tools like backup restore, being able to repair nodes, being able to get some metrics like Prometheus Grafana and so on maybe able to do ingress, you know, using traffic or engine X and so on. All of these are provided in a easy to install and easy to manage and easy to administer, okay? So, as you may expect, what is the installation? You add the repo, you know, Helm repo add Kate Sandra and just provide the repo, right? You update and then you install and that's it. As simple as this, right? You can tweak some values, very straight forward, right? You know, you can say, well, I want repo, I don't want Stargate. Stargate is a unified API and you will see that in a second. So, you can pick and choose the components that you want. You can say, well, you know, in a case of a multi-cluster install or a multi-availability zone installed, right? You can say, I want my data centers to be rack-aware and I want it to be spread on different racks and provide a higher level of availability. So, even if a particular rack goes down, right, you shouldn't have a problem. And then, of course, you know, you can go further up from there and do a multi-region or a multi-cluster or a multi-cloud as well, okay? But the philosophy is generally the same, okay? You can install it on pretty much anything. I think I've done pretty much everything, right? You know, I do have Kind, I do have Minikube, I do have Civo, I do have GKE, I do have AKS and I do have EKS as well.  It's very cool because, you know, if I want to do some local development, you know, I do mostly on Minikube or Kind, right? Mostly on Kind.  So, you know, pretty straightforward. In fact, our Kate Sandra operator, our docs, actually talk about installic and Kind cluster. And then, you can kind of extrapolate it and do it on multiple regions, multiple clouds and so on and so forth, okay? By the way, does anybody have a Kate Sandra installed? One. I hope you did. Yes, here's a thousand note cluster which you'll be glad to talk about. Yeah.  Obviously, Kassandra at the heart of everything, right? And then we have a unified API called Stargate. Okay, with Stargate, you can do, as an application developer, really cool because you can do document API if you want, you can use SQL if you want. SQL is our Kassandra query language. It's kind of similar to SQL, but it's a little different, right? And then, of course, you can do GRPC or GraphQL as well. Okay, it's a unified API, which is, again, very, very cool. And then we have two components called Kassandra and Repair, Repair, you know, which basically does the repair. And then, of course, Prometheus and Grafana. All of these are installed as Helm charts. Okay, all that I do is I install the CAS operator. And then, all of these are automatically installed for me. You can see here, I install the CAS operator, and then I get a REST or document API endpoint plus Swagger plus GraphQL endpoint.  And we have some sessions from KubeCon. You know, if you're interested in install, you can kind of walk through that, but the Kassandra at I.O., and I'll point you at that, has wonderful docs, and all that I do is kind of refer to that. So, that was a lot of introduction about Kassandra and Kassandra. We don't even get into the multi-cloud aspect of it, right? So let's get to the multi-cloud cluster in the multi-cloud. The reason this was born was because we kind of started hitting some limits with Helm. Can you bend and make Helm work? Probably, but it just wasn't right to do it that way. So we went ahead and built another operator called the Kubernetes operator. And there's a great discussion by Jeff Carpenter and John Sander, who is basically our tech lead for Kassandra, and they walk you through this. If you have some time, it's just a discussion, talks about why Helm, what are the limits that we hit with Helm, why a new operator, what is an operator, because for some time, the Kubernetes community kind of looking down on Kubernetes operators in general, because there was a lot of perception issues associated with it. But now it looks like the Kubernetes operator has come back in style again, right? It's still pretty relevant to be able to do that because it's really all about the same Kubernetes concepts that you talk about. It's really about self-healing. It's really about being able to watch what's happening and taking appropriate action based on that, whether it's corrective action or whether it's some other action, you can do all that. So if you have some time, take a look at this, why we pushed Helm to the limit and then built a Kubernetes operator. It's a four part, so you will get all details about that. So why multi-cluster? Cassandra has always been designed for multi-region. We thought about multi-region right from the beginning. And each node of the cluster maintains the full topology. It talks to the other node, where a gossip protocol and all that, so it routes the traffic to the other neighbors. It has an idea of what the neighbors are doing because it's talking to each other and all those good things, right? So in a way, it seems like each cluster understands the other. Kubernetes, on the other hand, was not designed for multi-region and a lot of those goes back to kind of the networking design that was made where you kind of assume that each part is able to talk to another part, right? And more so, they all need to be in kind of a flat network, right? So it becomes a little hard to kind of be able to do that. And actually for my multi-cloud, there are a number of ways of achieving multi-cloud for sure, right? What I did was I used something called as a, you know, something called as Aviatrix. I don't know if any of you have used Aviatrix or are aware of it. I'm not getting paid by Aviatrix. I just liked it and I have actually submitted a proposal for KubeCon North America to be able to build a multi-cluster to actually do this in action in 90 minutes, okay? What I'm gonna show off in a little bit, okay? So the K-center operator is really designed to make it very easy to kind of do this multi-region, multi-cloud, I'm missing, always missing something, right? It's either a multi-region, multi-cluster, or multi-cloud, okay? It doesn't really matter. K-center operator will handle that, okay? I mean, if you're not doing multi, also it will still do that, but really K-center operator has a lot of advantages. The CAS operator on the other hand is great for a single cluster, right? From a Kubernetes perspective, right? But the moment you get into a multi-cluster scenario, the K-center operator has this concept of what is referred to as a control plane and it's able to take care of the data planes, you know, kind of similar to how, you know, many of the Kubernetes services operate. It supports for multi-data center, multi-region K-center clusters. It consists of a control plane and data plane, right? And the control plane creates and manages the objects. Again, using API, right? The control plane right now, unfortunately, can only be installed in a single cluster, so it's not highly available, which is, you know, coming, but it's not there yet. Do we know when it's coming or no, okay. And the data plane can be installed in any number of clusters, you know. In this case, what I've done is I've used AKS as my control plane and I've installed the two clusters on EKS and the other one is GKE, okay? And you will see that in action today. So how does that actually, how does this magic really work? The magic of this is that basically what happens is there's a little bit of a chicken and egg, right, where the control plane needs to understand the data plane, right? And so what you do is you inject the configuration of the respective data planes into the control plane. Okay, so there is a way to inject the configuration and once you know the configuration, then things become a lot easier from an installation perspective. And then, of course, once you install Cassandra, you really don't need to do anything because they know the other nodes, they're able to talk to them and gossip around and build a bigger cluster, okay? So here's an example of a Kate Sandra cluster object that is used to, you know, basically build a multi-region Cassandra cluster. You can see here there are actually two conflicts that I'm using, or two Kubernetes context. One is East and the other is West, okay? So essentially what I'm saying here is my data plane is going to be on East and my other data plane is going to be on West and this is the context that you need to use. Okay, and you will see all of this again, you know, when I kind of dissect my installation, okay? So essentially what I, there are two contexts that are injected into the control plane. One is called East, the other is called West. And once it has the Kubernetes context associated with it, it's able to go install Cassandra there and then, of course, by the way, by the nature in which Cassandra operates, it's able to build a bigger cluster. And this kind of shows the same thing as well. And this is kind of how the client config looks, but we don't need to worry too much about this, okay? And what we'll do is we'll jump straight into the demo. So remember my journey, right? My travail started with GKE where I set up, you know, the multi-region Cassandra, very easy to do because, you know, networking in GKE is a lot easier. You know, it has, you know, addresses reserved for parts and addresses reserved for services and all that I needed to do was take the address from the seed service from one and inject it into the other and then it found a big cluster, Kumbaya, you know, it's all done, right? But I'll skip that demo, but I'll skip to the next demo, which is this, this is the EKS demo, okay? And hopefully you guys can see it in the back, kind of, otherwise I'll explain it. Essentially what EKSCubeFed does, and it's a fantastic project, really. If you have not used it before, and really, you know, it provided me with exactly what I wanted. What I wanted was I knew I was gonna make mistakes. I knew I was gonna stumble. So what I did was I wanted something that was repeatable, right? And at that point, there was no CadeSandra operator.  So I was lucky to find this EKS project, okay? EKSCubeFed, which is actually maintained by Amazon.  But what EKSCubeFed does is it provides me with, you know, these two VPCs. As you can see here, in this particular example, you have 172.21 running on region two and 172.22 running on central in, I believe it's in the EU central one. And the other one is EU west one. So Frankfurt, right, is running 172.21 and I believe it's Dublin, which is EU's, sorry. EU west is Dublin and EU central is Frankfurt, whatever, okay? And we'll see all that in a second. And it provides me with a bastion host. A bastion host is a way in which I can connect to these respective hosts if I want to, right? So the bastion host is installed for me. It's on 172.20, okay? It's all opinionated but that's fine for me because it works and I'm not a networking geek by any means, I just wanted to make it work, right? So I had a 172.20, 172.21, 172.22 and then what I did was I installed Cassandra on 172.21 and injected the seats into 22, exactly the same thing and they formed a bigger cluster and, you know, good to go, okay? And you'll see this in action. But with the introduction of the Cassandra operator, I don't have to do any of that, right? Because it, in principle, does the same thing, right? You know, it knows the seats and essentially is able to, you know, do the gossip protocol and make it work, okay? So that's exactly what the Cassandra operator does and we'll see that in action as well, okay? So this is, I don't, basically it talks about all of these different components and what are the different contexts and so on but let's keep all that, okay? So I don't know why this came in here. I need to get rid of this. But let's go into the demo. Let me know why you're too much about this. Okay, so here's my, what I want to show is a little bit of everything, okay? So let me start actually with the EKS demo, okay?  Okay, because my scale public fast did not work. So talk about, you know, anyway. All right, so let's go into my instances, okay? And, okay, so here are my instances. I have my co-pilot, my controller and all that which I'll get to in a second but we're still looking at the EKS demo where I did the manual injection of seeds myself, okay? So I'm gonna try to connect to it. Remember what this is doing? This is the bastion host, right? And I'm connecting to the bastion host, okay? That's all I'm doing, right? So let me connect to that, okay? And I'm good to go, okay? So now I am gonna do a few things, okay? So let's look at, you know, kind of the, I'm blanking out for a second here on the cube comments. So let's do get context that'll get me going. Okay, so here are the two contexts, okay? One is one, the other one is two, very, you know, rags NS fed to one and fed to two, okay? So let me set the context or use context, whatever that is, right? Is it set context, use context? I can never figure that out, okay? And just to make sure, get context, okay, that's good, okay? So now let's get the nodes. And you'll see here, this is running 172.21, okay? This is the US that I was talking about, which is basically running in Dublin or whatever the case may be, okay? Now if I do the same thing with two, you should be able to see that as 22, okay? And all of this is automatically done for me through EK's cube fed, which is really cool, right? So let's look at some pods, okay? And it doesn't really matter which cluster, okay? I can take a look at all of this, right? And you'll see here, you have the, you know, kind of like the Grafana, you have Prometheus, right? You have the Reaper operator, and you have basically all of this done using the CAS operator, right? And pretty straightforward. Then you have a bunch of CRDs and all that, which we'll take a look into in a second. You'll see here there are a number of different racks, right, rack one, rack two, and rack three, okay? And how I was able to set that up was, you know, let's take a look at EKS West, or DC West, okay? And you'll see here, I injected the seeds from the central, okay, which was the other install, right? And then what I did was I also specified the racks, and you can provide the affinity. So basically I'm saying that, you know, though one node is running on US West 1A, the other one is on 1B and the other one is on 1C, okay? So basically it's a three availability zone cluster, okay? And that's pretty much how I set this all up, okay? For those of you who are Cassandra admins, and still don't believe that this is a multi-node cluster, right, or multi-cluster Cassandra, okay? You can run something called as node tool, okay? And basically what this is saying is I'm running the, you know, basically the Cassandra container, and I'm running the node tool utility within that, okay? And I'm providing the username and password, you know, don't ever use, you know, admin and password like I do, but this is just for illustration purposes, right? So you will see here that I have a two data centers, DC1 and DC2, okay? Can everybody see that, right? And you can see here this is running on 21, okay? U stands for up, and D stands for, I mean the N stands for normal, okay? So basically what it's doing is it's up and it's normal, okay? And you'll see here that it's running on 172.21, 172.22, and if you actually do the nodes as a wide, I think this should give, no. Is it gonna get topology? I forgot, but essentially you can, if you dig through or describe, we do this, hopefully this will. So you'll see here that it's on 1B, 1A and 1C, okay? And if I do the same thing with two, right? Or with one rather, you'll see it a little bit different, right? Not a whole lot, but it's running on U West, and it's A, B, and C, okay? So it's already rack-aware, and now what we've done is we have made it region-aware as well, okay? So if something, some calamity happens on West, I can all this shift to central, and you know the drill, right? So this is as far as the EKS cube-fed experiment was concerned, right? Now I wanted to go really multi-cloud. I wanted to kind of do it on a multi-cloud, and you will see here what I've done, and this is using Lens, okay? I have three clusters here. One is on AKS, the other is on EKS, and the third one is on GKE, okay? So these three are actually used by the Kate Sandra operator, starting on AKS, you know, I just picked one, right? And essentially it installs it on EKS and GKE, okay? But to be able to do that, I used something called as aviatrix, okay? And I use this Terraform scripts, and I'll point you to this as we go, okay? So basically it's AVX, multi-cloud, Kubernetes, and what it does is it does an opinionated implementation. You just specify the names for the AWS, Azure, and GCP accounts, and it will go ahead and create the appropriate things, that need to happen. For example, if you look at, I believe it's variables.tf, you can see here, you know, the AWS Cider is 10.1, the Azure, do you want me to increase the font? Yeah, sorry about that, I forgot about that, okay? So you can see here it's 10.1, okay? Then you can see here for Azure it's 10.2, and for GCP it's 10.3, okay? And you'll see that as we dig into it. So it does all of this, it stands it up, and then of course the nice thing is you can tail it down as well, right? So I wanted something that I need to be able to do. And then, of course what I do is I, I think the readme talks about this. Essentially all that I need to do is point it to my account, to the name of the controller, okay? So what we can do is we can take a look at the controller, okay, and to go back I'll go back to EC2 instances, okay, go back to the instances, and look at the AVX controller, okay, and I'm gonna connect it, our action, open the address. Okay, not secure, that's fine, I don't care. Okay, and it'll give you an idea of, you know, what are the accounts that I've onboarded, you know, AWS, Azure, and Google, what are the gateways, and so on and so forth. So it provides you with a nice, what I want to show, here is multi-cloud transit, I think, or multi-cloud gateway, one of the two. I can never remember this. So if you look at the gateways, and if I just cider filter on 10, okay, then you'll be able to see AKS, EKS, and GKE, okay? And you'll see that AKS is on 10.2, 10.1, and 10.3, okay? It's basically an opinionated implementation, okay. So now that we've seen how I set it up with AVX, right, now what I'm gonna do is I'm gonna go into my cluster, right? So to do that, I'll just go to Lens, which is kinda cool because I don't have to do any of the cube cuddle commands, right? And here's my Azure, right? And what I'll do is I'll take a look at some of the pods, okay, and take a look at the Kate Sandra operator, and you can see here, somewhere here, it should have a control plane of true or something. I mean, a data plane is, I mean, control plane is equal to true. You can never see that when I want to. Let me, departments, let's look at that. But believe me, somewhere in there, it's there, okay? I'll let me look it over here, that's not good. But, you know, you should be able to see this. I'm sure I'll be able to see that in a second. Actually, let's look at custom resources. And you'll see here, this is the Kate Sandra cluster that was created, okay? And the name of the Kate Sandra cluster is Rags and S, okay? And you'll see here, these are the context that was injected DC GKE and DC EKS, okay? And these are the data centers that are there. So for example, if I go to EKS, let's go to EKS, right? And if I look at the pods, you should be able to see a DC. So you can see here, this is the DC that was injected, okay? So it has two, you know, here's the DC that got injected into the configurations. In fact, if I go back to the Azure, which is really where I started, right? And I can take a look at the, look at the client conflicts. You can see the two clusters being injected here, right? So here's my EKS AVX cluster and my GKE X cluster, okay? And that's kind of how I kind of made it happen. And again, I think a lot of this was already built into Cassandra, so I didn't really have to do much other than working on the networking. You know, once I got the networking going, everything was pretty straightforward to, you know, to make it happen. And now with the Kate Sandra operator, which is really about multi-cluster, multi-cloud, and so on, makes it a lot easier to be able to install on Kate Sandra, okay? So with that, I'm almost at the end of my presentation. Give you my slideshow. Let me go randomly to 30. Okay, that's good. All right, so I'm done with the demo. We saw the EKS CubeFed demo. We also saw the multi-cloud demo where I installed Anik, and actually I probably should have shown you a little bit of the command line, but I guess I'll skip it today. You know, we really don't have too much time. And with that, if you wanna head over to Kate Sandra.io, it has pretty good docs, actually. Surprisingly, for something which is pretty technical, it's pretty straightforward to install, whether you wanna install on Minikube, whether you wanna install on KIND, whether you want to install on Siebel, really anywhere, you know, you can do that. I myself wrote a blog about multi-region Cassandra on EKS, so you can take a look at that using CubeFed. You know, it essentially goes through the same demonstration that I did today. But like I said before, we also run workshops every Wednesday. You know, you're welcome to attend. I tell everybody that, you know, we're living in the golden age of developers, right? We really, you know, we have all the power, right, as a developer. So it's really up to us, yeah, exactly.  But do attend this, we also provide badges. And for whatever reason, these badges are extremely popular, okay? We also have a pretty robust discussion going on in Discord. So, you know, something a little bit more permanent, you know, in the workshop, we have a lot of chatting going on, but that's primarily on YouTube. But if you want something a little bit more permanent, you know, jump onto Discord and try that out. Like I said, whole bunch of these, if you are interested in introduction to NoSQL, or if you're interested in building your own TikTok clone, or if any of you are interested in doing a Swift workshop, Swift is Apple's open source, you know, client and server-side programming, you can try all of those, okay? So, like I said, badges extremely popular, okay?  So try it out, and with that, thank you very much. Great. Still have 10 minutes. Thank you very much, Rugs. I am so happy you were here. If there are, we have time for maybe one or two questions if you are able. Absolutely. Are there any questions in the room? Oh, yeah, I'll come over. Thank you for the presentation, Rugs, and for the multi-cloud, multi-region deployment. So, as the organization planning to go to production, do you have any advice to reduce costs as like if you're going to one provider to other? So, just you'd like to hear your thoughts. Yeah, yeah. I jokingly refer to this as I'm a developer advocate and I really don't concern myself about costs, which is really a bad thing to say, you know? But maybe somebody in the audience has better, you know, like guidance, especially from a cost perspective. Anybody wants to jump in and kind of provide, especially from a multi-cloud perspective, as I understand, right? Anybody wants to jump in? Otherwise, I really don't have good guidance. My apologies. No? Okay, sorry. So the question answer session didn't start very well, but really we'll keep going. Well, sounds like we have time for one more question. All right, in that case, I would like to thank you again. Thank you. Thanks for this awesome thing.  Hope to see you there. And as usual, it's scale. Talk to people. Everyone's friendly. That's hard. I know. I know. That's kind of it. No. That was perfect. As usual, that's kind of available. Time is weird when you're performing, when you're on stage. I saw that. Hold on. What, that works for you? Yeah. Hello, hello. Yeah. Check, check. Hello, hello. Test, test. Hello, hello. Mic, check. Test, test. Mic, number one. Check, check. Hello, mic, check. Check, check. Check, check. Mic. Check, check. Hello, mic. One, two. Check, check. Check, check. Are you guys, I was told for the last one that we can all do. Oh. Do they want to do, I mean, I can do the live. This is not a panel. This is just a single speaker. So maybe we can put it on the day. So thank you. Check, check, mic, check. Come on in, have a seat, make yourselves comfortable. We're just waiting for a little bit of AV setup before we dive into our last talk of today, Friday at scale. Yes. Hello, test. Cool. And with those final setups, yes. Thank you so much. Welcome, everyone. This is the closing talk for the Cloud Native Day this Friday at scale. I am super excited to introduce Daniel Hicks, who is going to show all of you four ways to spin up a Kubernetes cluster. We've been talking about having clusters living on our machines all day long, every single presenter. It's like, I already have a cluster. Well, here we are. This is how you get to that stage on your machine. Please welcome Daniel. Thank you. All right, hello, everybody. As she said, my name is Daniel Hicks. I am a software engineer at Digital Ocean on the storage platforms team.  Just a quick, who am I? As I mentioned, I work at Digital Ocean. I'm on the storage platforms team. We do a lot with Kubernetes on my team, hosting sort of an orchestration platform for our storage products. I graduated from West Texas A&M, go buffs. I'm also a current computer science student at Georgia Tech for my masters. And I also really enjoy open source, contributing back some of my first projects were contributing back to the .NET runtime, as well as like GitHub desktop and the external secrets operator for Kubernetes. So, jumping right in. Often, companies, we measure things, sort of your Kubernetes maturity, per se, in terms of a day one versus day two operations, where day one is sort of, you're getting your feet wet, booting up a cluster, deploying applications into it, testing things out, seeing what works, possibly automating some of that. And the more mature model would be day two, where you have sort of things like automatic logging, monitoring ingress and egress, as well as sort of backups, hardened security, things of this nature. And so we're gonna focus more on getting from day one to day two, since the process of getting from day one to day two can not only be days as in the name, but it can also take weeks, months, or sometimes even years to get there. So we're gonna talk about how to sort of speed your way or accelerate how to get there. And we're gonna do that by using things like a managed Kubernetes service, as well as picking up the best technique for your process to spin up your clusters. So, this is what we're gonna cover today. The four different ways. We're gonna go over first the web interface. Most clouds have a web interface, so we're going to specifically use digital oceans, Kubernetes interface. We're also gonna use a command line to boot up a cluster, as well as look at things like Terraform and Pulumi to boot up a cluster. And my disclaimer is everything here will be focused on digital oceans, Kubernetes platform, because it's the one I have access to. I will say, though, that every method that we're using today, or demoing today, is also available for any other cloud platform, AWS, Azure, Google Cloud, all have their own web interface and see Lies, and they also all have their own providers for Terraform and Pulumi that are openly available. So this applies for anything, not just what we're showing today. So, first off, let's do the web interface.  And that's awesome. Speaking of, all right, so we're going to keep this small. This is just doing the UI via the web interface. You can see this is just going through and picking the various options on digital ocean. Slightly hard to see, but since it's smaller, but you can do things like choose your own cluster size, how many nodes, how big do you want the nodes to be? There's several other options such as region or if you want the high availability control plane. And so this is sort of a very quick way to do it, a very visual, you don't need to be super technical to do so if you're working on a proof of concept, it's very simple to do. And it's just a real quick way to get into it. And then from the rest of this, this is just showing it booting up, as well as how to get connected. So normally you would download something like the kube config file or something that you would put on your machine that allows you access into the cluster, as well as most cloud providers will allow some form of UI to interact with the cluster. On this one, it specifically shows just using the regular default shipped Kubernetes dashboard that would get if you say booted up Kubernetes on your home machine or something of that nature, which is what we provide at digital ocean. And then here's the dashboard that we provide from that.  Let's see if we can get the slideshow to work. All right, so the pros and cons of this sort of way of doing things. Like I mentioned, it's very fast to get started. You can boot it up right away. It's just a couple clicks of a button, nothing too fancy. It's very visual for people who like learning visually. I myself really like graphical interfaces, but that's just how I roll, how I learn, how I do things best might not be great for you. But like I said, it's very easy to spin up and down. A huge downside of this is it's hard to share this among various people. How I click the buttons or the options I choose, you only know about that from me. I can tell you how I did it and you can try to do it the same, but you might get something different. You might end up with a different configuration. And in that sense, it's very hard to repeat the same thing over and over again. Also clicking through the buttons, it takes time. So it's very hard to do over and over again, as well as like the configuration for it. Like I say, it's stuck in my head. If I go on vacation, you have no idea what I did. So there's no way to get it. And so it's just sort of stuck with the creator. It's best for prototyping something very quickly if I just need a cluster to show you something cool or to add something or just to check the install of say, I wanna check out a new operator or something of that nature. 100% perfectly fine for that. It's also good for comparing the different cloud providers. Say I wanna know what does AWS's interface look like versus Google Cloud or DigitalOcean versus Azure. It's very good for that. And you can sort of see what fits your needs the best. And it's also okay if you're willing to accept the technical debt that comes with not being able to store that configuration is something you're going to automate later or something that you're okay with up front. And so next we're gonna go over the command line. Most cloud providers provide some form of command line to interact with their services. We are no different. We provide a tool called the DO-CTL tool or DO-Cuttle tool or however you choose to pronounce that word. So here you can see I'm just loading up a interacting with our QBDND service. I'm saying I wanna create a cluster called my awesome CLI cluster minus some screen jittering. And then I want a count of three nodes. And so this does take a little bit but through the magic power of editing we'll see that this is going to speed up fairly fast and we're going to jet right to the end. And then the DO-CTL tool will also do some extra fun settings where it will actually download the Qube config onto your machine and change your current context for Qubectl to your cluster that you just created. This is usually beneficial just for getting quick access to things but then we can just run Qubectl and you can see that we have pods in all of our namespaces and things of that nature. So the pros and cons of this very similar to a UI. It's fast and easy. It's also really developer friendly. For those that like working with the terminal this is the way to go. It's also slightly possible to automate it. Bash scripts are a thing and you can very easily put this in just a file that somebody can run and boot the inputs. The downside, it's still slightly difficult to share configuration. While it is an automated script there is no state per se of what is actually up and running. Say if you have developers booting this up every time they enter the development mode there could be 10 clusters running. The command line doesn't know this. You can run, the cloud provider might provide a way to list all the clusters or they might not, depends. As well as if you're going between cloud providers or switching contexts fairly often their CLI tools are gonna be vastly different from each other. They all have different interfaces. They all provide different services. Everybody calls them differently. It's just, it won't be very consistent from provider to provider. AWS or Google Cloud all do different things than how any other cloud provider does things and so on and so forth. Since they have very similar pros their best four are very similar things such as quick prototyping, proof of concepts, understanding the cloud provider's API. So you saw where with DOCTL we did Kubernetes all of those command line arguments are almost identical to our REST API that we provide behind the scenes which is literally all this is calling. So if you need to get familiar with a cloud provider's API CLI tools are usually a very good way of doing this. And as I mentioned it's also very good for writing very basic automation. Spin up a real quick cluster by running a bash script et cetera et cetera. And so next we're gonna view a slightly more automated, consistent way of doing things. The, very nice. The first two were sort of a day one more operation where we're gonna be focusing more on day two for the next two. Yeah. All right, let's figure this out. It really hasn't. It's when your computer's working real hard. Let's find out. All right. Do not move.  Please hold tight. All right. Oh, yeah, you're welcome. Quick poll. Who pronounces it cube cuddle? Who pronounces it cube CTL? Wow. Clear preference here. Any other pronunciations? Cube control. Cube control, right? Why didn't I not think of that? Anything else? Cube cussle. Cube cussle. Oh, yeah. Well, because sometimes you just have to cuss at your computers. That is very. That is super reasonable. Let's see. All right. Why did the robot cross the road? Why? Because it was programmed by the chicken. I learned that from my 10-year-old. How are we doing over there? Do we want a different computer? Last time it was a computer. Well, we had some issues with like a 2015 earlier. I don't. Anyway, thank you all for your patience. And see, I tried to record the demos just so I could avoid technical difficulties. Go figure. All right, so as I was saying, the first two, the web interface and the command line, they were focused more on day one, sort of model of maturity in terms of Kubernetes. So the next two, Terraform and Pulumi, they're gonna be focused more on sort of the accelerating to day two, slash day two maturity model of booting up Kubernetes clusters. So this video, let me, well, if it's too small, I can't really do much. So first off, we're just gonna cat some files, look at them. This is just showing the sort of the provider, Terraform file that you would do for a digital ocean provider in Terraform. It just mostly says what the provider is and the version that we're gonna be using. The next one is going to be, I believe the actual Kubernetes cluster that we have defined in Terraform.  I'm putting it in the region SF03, as well as declaring its name. This is the name that shows up in the UI, as well as a node pool specifying specific sizes, as well as how many I'd like. And then this is just sort of showing, for this method, you do need a digital ocean API token, which is the same thing as what you would need with the CLI, but I've already done it, so we're not gonna worry about that. Usually first, you have to run Terraform init, so there's some initial steps. This goes and downloads the provider, anything else that you might need that Terraform can configure for you. And then right from there, we run for Terraform plan. The nice thing with Terraform is it actually shows you what it's going to do, shows you what it's gonna create, the things, the state that it's gonna collect, as well as it'll tell you things that it's gonna find out after it creates the resource. So this will make an API call to digital ocean to create a cluster and return any state that it might need, such as like the private networking ID and things of that nature. And so we'll run Terraform apply. It'll ask us if this is what we truly want, and then we'll of course say yes. Again, through the power of editing, we're going to fast forward through this since provisioning Kubernetes clusters sometimes takes three to 10 minutes, depending on the options you chose, the size of it, et cetera, et cetera. So you can see us speeding through it, but this is going and getting the state behind the scenes, waiting for it to be ready, and as well as just finishing it up. So we'll skip the last couple of seconds. So once it finishes, it'll tell you that it's created a resource, and then we can just run the doctl command that we did earlier and save the kube config down to our local machine and merge it with our kube config that's local on our computer. And then just very quickly kubectl the show, I'm not fibbing. We do have a cluster up and running. We're just going to get all the pods inside all the namespaces. You can see a bunch of pods. We install Selenium or Selium by default, as well as just some DO agents that we put to monitor the cluster. So the pros and cons with this method, one of the cool pros that I think is that you can preview your infrastructure. You can put fairly complex logic in here. So sometimes what you're booting up might not always be super obvious.  The initial state that it's going to be that Terraform can know at that time. This also introduces the concept as infrastructure as code. Your infrastructure is no longer something that sits inside the cloud only. You can check these files in as well as the state to source control. You can also store the state out in something like S3 or digital ocean spaces if you so like digital ocean products. As well as it's very repeatable and auditable. So there are plenty of tools out there that can audit Terraform as well as monitor how much you spend in the cloud based on your Terraform state and things of that nature. So you can get fairly diverse with Terraform and monitoring things of how you go. The downside, this does require slightly more effort than just writing in a simple command into the command line. And you also have to have somewhat technical knowledge of the cloud provider that you're using. You could see that I specified specifically the slug for the size of machine that I wanted for my node pool. You have to know these things about your cloud provider if you're gonna do that. It might not always be obvious that I need to create a node pool with a Kubernetes cluster because if you're doing it via the UI, it automatically does it for you as well as the same with the command line tool. So you have to have somewhat knowledge. You have to dig through the docs slightly more and you have to go do things like go and find their provider and hope that there's good documentation on it. But this is usually best for clusters that are used by multiple people. So teams of people, this is very good because you can share the state among them. So they can all share if they go to boot up a Kubernetes cluster or say change something. Terraform, if you're using a shared state, will tell them, hey, I don't need to change anything, this already exists. If also if they need to say scale up the cluster, Terraform can tell them, hey, this is what I need to change from what exists to scale up your cluster.  As you can, like I say, get fairly complex with Terraform and also chain different resources together and feed them into each other. So you can build sort of a very complex setup and it's also very good for disaster recovery. So say if I have an entire region go down, it's fairly trivial to say change one or two variables and boot up an entire another region in any cloud that you can think of as most of all of them have providers and can do that fairly quickly. So last, one of my favorites, Pulumi. We're going to start with a code snippet because this is the meat and bones of this. Pulumi is very much like Terraform and can actually piggyback off of Terraform as well. If there exists a Terraform provider for it, Pulumi can piggyback on it. Pulumi is more of a library for whatever coding language you would like, whether that be Python, Java, C-Sharp, Rust, Golang in this instance. You can define your infrastructure as actual code. This right here is literally just a method where I'm defining a new Kubernetes cluster the exact same way that I did in Terraform with the exact same options. With some extra fun go syntax with the ifer equals nil. If you've ever coded in go, you know exactly what that looks like. So running a real quick video, you're gonna notice that this looks very, very similar to Terraform. It is the same thing as just run up. The difference being that they store a stack and so I will actually pause it super fast to cover that. You can do things like define different stacks. So you can have like say different environments. Your staging environment could be a stack, production environment could be a stack. This is very similar to a Terraform state. This is the state of the environment and where it's at. You can store these locally or inside a Pulumi or whatever kind of backing storage Pulumi supports. So it does something very similar to Terraform but it also tells you this is what I'm going to create. Are you sure you would like to do this? You can go down and view the details. This is very similar to just Terraform's plan. So this shows you sort of what my code would go and do. And then you can just say yes, I would like to perform this update. And then this is super sped through again but this is just going through and doing the exact same thing, calling the API, requesting to create this resource and then spitting out that we're done. We also have things where we can do outputs with Kube configs in order to keep that code snippet small. We're not doing anything with the output for now but you could realistically say take the output and go put it in something like say vault, like HashiCorp vault or something of the nature to share among other developers. And so I don't know if you ever typed in front of somebody but it always seems to get 10 times more difficult. Recording demos, oddly enough, have the exact same effort. So you can see me struggling very hard to type in the command here. But, so this is just the same thing again.  We actually went and created a cluster. Pulumi does add a small identifier to the end of your resources so you can see I had to go grab that and come back. So, and then we'll just do the same thing that we've kept doing. We're just going Kube CTL, get all pods and all namespaces. And though and behold, we're running a cluster that has only been alive for two minutes. So, what is this good for? This is infrastructure as code supercharged. It is quite literally just code. Terraform uses the HashiCorp language which looks like a derivative of JSON. While it does have logic, the logic has to be pretty basic. So, it's not something that you can boot something up if you need complex logic for looping or like say include and exclude this. You also get all of the benefits of the language you chose. As it goes, coding languages are very strong and being able to do very complex logic, whatever you can tell it to do. Pulumi takes advantage of this and you can do very complex things and they provide a mocking and integration test library for you. So, you can write things like unit tests and integration tests against your own infrastructure. So, like say I want to make sure that if I provide this CLI interface and I say I want five nodes, you can write a unit test that says when I do that, it will give me back a cluster of five nodes. This helps provide sort of confidence in your infrastructure or your infrastructure as code and sort of gives developers more confidence that what they're doing is actually, it's doing what they want it to. The downsides though, this is fairly complex. A more unusual requires an engineer, someone with technical prowess in whatever language you've chose to do what you need to do with the infrastructure. This is no longer, I need someone to just go in the UI. This is, you need in-depth knowledge of how this library works to go and use it. So, this is usually best for teams of software developers that are booting up clusters for whatever they're doing it for. And when you need something that's testable, so if you need to test your code, you need high confidence and that is a requirement for you. This is very good for that. And one of my main strong points that I like the most, if you have a certain language that your code base is in, you no longer need the context switch between it and Hashicorp language. You can make it be the exact same language that your main code base is in. The perfect example is my example was in Go. Most everything we write a digital ocean is in Go. So, I no longer have to swap between the two and it can live side by side. And it's fairly good for complex infrastructure. A good example is we were doing something that required a cross-region deployment to multiple regions and they all had to tie back together. This, it also involves serverless functions. So, this combined with serverless functions proved to be fairly complex in Terraform to where it was taking multiple days for this small little tool that is gonna be used a couple of times to boot up. We instead switched to Pulumi and could define the complex language, replace things with variables and got it up and running in like two hours. So, when you have something very complex, sometimes this makes it a lot easier rather than messing with a language that's not really meant for that sort of complex setup. So, in summary, we went over these four ways to boot up Kubernetes cluster. The first two, the web interface and command line. Very fast, simple, quick way to boot up clusters sort of focused on that day one model of maturity for Kubernetes clusters and the Terraform and Pulumi sort of the day two, we wanna automate stuff and go fast type of setup. And so this just sort of shows the, we're focusing on making that the weeks, months and years, we wanna make that a lot shorter and make it go faster. And I'd like to say thank you for coming in to the talk. Thank you for DigitalOcean for sponsoring me to come. On most everything, my name is Adusti Oldmuffin. Fun story where that came from, you can find me on GitHub and Twitter on that. And as also, if you want to, you can find me on LinkedIn as well. If you would be so kind, we have time for a few questions. I'm gonna come around with the mic. This time I'm gonna hold it though. Hey, I think you said the Terraform one, one of its pros was it's auditable. But then Pulumi is not auditable or is it also auditable? I mean, like in the sense of external and security auditors. Yeah, so the question was, is Pulumi auditable just like Terraform? In the sense of I believe that you can write tools to sort of audit it. The main reason I mentioned that Terraform is auditable is because I know that there are specifically plugins for Terraform, that that is their sole purpose of auditing sort of your infrastructure to make sure that it's hardened in terms of a security sense. Sorry, so you could include that in the Pulumi. Yes, more than likely. Okay. Yes. She works at Pulumi, so. I promise I did not slip him some cash to put Pulumi in the presentation. I had zero idea, so I promise. Questions, questions. Comments. Actually, yes. I joined a little late, so I don't know if you mentioned this, but will any of the tools set up the control plane of Kubernetes? Sorry? The control plane of Kubernetes. Is it going to, like any of the tools that you presented, any of them are going to set up the control plane or just spin up a bunch of hosts inside your Kubernetes cluster? I got you. These are all focused on sort of booting something up in sort of a cloud provider. I did talk really early that every tool I mentioned or every way I mentioned has a way in various cloud providers to do it in a very similar way. Most cloud providers will boot up, they will boot up a control plane for you, but usually will not give you access to the control plane and instead will give you access to the worker nodes. DigitalOcean does that very similar where we boot up control planes for you and we give you options for highly available. HAA control planes, but we give you the worker nodes and that's like in our model that's what we charge you for is the worker nodes. Awesome. I see a question in the back. Could you quickly put up the references slide again? Yeah, sure. Thank you. And 100% if you want this slide deck or any of the references that I have on this page, feel free to contact me as well. I'll be more than happy sending the slides. The references are Kubernetes on DigitalOcean that's just our, just links to our product page, as well as the DOCTL docs, Terraform, Pulumi docs, as well as the Kubernetes starter kit that I believe is provided by the CNCF. I don't remember the name of it now, but wasn't there a project in Kubernetes to have like a cluster API or something where you could, do you know what I'm talking about? Yes. What happened to that? Is that still happening? So are you talking about cross plane? No. Okay. I believe there are multiple tools. This was originally, there was another one which was called cross plane. They provide something very similar where you can boot up resources as Kubernetes resources and it will spin them up in the various clouds as you needed. It's very similar. Too many technical issues coming into this talk since I was last minute sort of prepping for this. So I went with Pulumi instead but there are other ways as well. So as far as I remember cluster API was a sub-project of the Kubernetes project. So I would go start there. Could you briefly talk about the difference between the state in Terraform and the state how Pulumi handles the state real quickly? Sure. And I think she can probably cover, might be able to cover more in-depth on the state in Pulumi. The Terraform state is generally more, usually what I would describe is just a JSON file and it stores sort of the state of like what is, everything that you see with Terraform plan is exactly what's gonna be in that state file. You can just specify different things in Terraform to spit it out to like say S3 to where it can be shared or you can do it locally. And I don't know if it's best practice to check it in a source control but you can check it in a source control. Pulumi does things in sort of stacks and so the stacks is just a very similar concept. They just sort of expose it via their tool a lot better as well as they will natively send it to their SaaS product. I do things locally instead. That's pretty much the wrap up. I mean the Pulumi state is a big JSON file. Yeah. The question was in Pulumi, can you keep the state locally and I would have to get back to you on that because I work there so I get a free account. I can actually answer that and say yes you can. That's what I did for this demo. Yeah, I didn't feel like pushing it so you can configure it to just push locally. No problem. So I see people are bringing in chairs and drinks and all this sort of thing so I will thank Daniel once more. Thank you so much. Thanks everyone for being here. What an awesome, awesome day and I hope to see everyone at upscale. All right. Have a great conference. See you around. Yeah, okay. Good evening, scale. Are we, are we ready? Let's do this. Okay. So our first speaker tonight is Seamus Blatley. He's going to talk about the independence of the importance of independence. Seamus needs no introduction to those of us that follow him on Twitter. He's lived more lifetimes than all of us can combined imagine. He's worked on some game development, some physics on games. He's worked on some advanced math, some baking. He said there would be bagels but there aren't any. So I'm disappointed about that. And I also had him come here yesterday just to do a full on dress rehearsal to an empty room. I think he enjoyed that. But anyway, without further ado, Seamus Blatley, guys. All right, let's do this. Hey, everybody have enough drinks? The most important thing when you hear me talk is that you have enough drinks. All right, everybody drink up. Then you're not going to enjoy this. I'm sorry. But try to get into an altered state anyway. I might end up putting you in an altered state called boredom. Today, we are going to talk about innovation and freedom. Freedom is something I think very important to this crowd and to free software, which is the thing that's been geared on my heart my entire career. I am Seamus Blatley and today's date, although Bala told me otherwise, is not 728. It is 729. Yesterday when I came here, this was my talk. It was less stressful. It was a perfect presentation. We're going to have a less perfect presentation today, probably much worse, about freedom, tinkering, and innovation.  So hopefully I'm not bore you too bad. I have some housekeeping to get out of the way first on this.  There are not going to be slides for this presentation. Here is the main slide for this presentation, the subtle absence of slides. All right, so listen. Here's the deal. Almost everybody in this room, including me, is in the business of making something from nothing.  And they are never going to understand what you do. It's just not going to happen.   I have this thing called cryptocurrency.  Just pay attention. So, anyway, the thing about it is that what we have been doing this whole time is creating value from vacuum.  It's similar. And yeah, it's right, it's the spirit that matters. And a lot of people can't understand this and it's specifically strange because guys from my generation and old guys, you know who you are, we come from a world where everybody was a banker, everybody was in real estate. All the cool kids were making money and manufacturing. And in those worlds, you are playing a zero sum game. Some guy minds the minerals that the other people can't get to and he wins over them. Somebody sells a property for more money than the other guy or sees some kind of tax benefit better than the other guy and he wins. The other guy loses. And the thing that these people do not understand about what it is that most of us in this room do is that we're not playing a zero sum game. We're playing a game where there's nothing. And then you work really hard and if you can stay sane and pay for food, after that, there is something of value that gets sold to somebody and your sister's boyfriend is super confused about why you have a nicer house than him. Okay. And then to add incredible befuddlement to these people, you start to talk at Thanksgiving about the fact that the software that you write, you are giving away to other people for free.  They really don't understand what you're talking about. But it's interesting because the free business model has been around for all of human civilization. I am sure the first drug dealer gave some free samples. Kickstarted his business that way. But we do, most people do work for free all the time. They just don't notice it. And what I want to talk about tonight with respect to freedom, creativity, innovation, tinkering, all these things is that it's that free part, the part when you're playing around, the part when nothing's on the line, when you're not playing with $100 million investment or you're not like driving a super expensive truck. It's that moment where you're actually gonna create something new because you're free and you're not worried about shit, right? You're not concerned about anything.  But for most people, that moment where you're free is the moment where you achieve this maximum creativity, okay? And that's the magic moment. That's the moment that the old world didn't understand. And the trick is, how do you get yourself into that space often and aggressively? How do you manufacture a situation where the largest number of people at your company are in this creative space, for instance? Generating revenue from nothing.  You have to do this to succeed. And the answer is, you try to put them all into this place where they're playing, where they're just screwing around. Better, better, even better, where they're playing and showing off to their friends, okay, where you're showing off the stuff to do to your friends. I've had some interesting experiences at places like Microsoft and other unfortunate employers where I've tried to explain to people that we need to give, I will quote the anal class, the class of people who make something from nothing. You have to give this group of people as much freedom as you possibly can if you want to make any money. Okay, and money guys will immediately tell you that you need to constrict their freedom as much as you can. We must make a walled garden. I love that term, because, you know, I don't think the prison exercise ground is a walled garden. They want to have a euphemism where you own everything that somebody makes, but you still expect them to be really creative and able to show off to their friends. It never works. It never works.  But in fathering, a plastic box filled with transistors, I had to figure out a way to get the most important people in that ecosystem to feel freely creative, and that's the game developers. You're going to make games for the box, and they don't work for Microsoft. They work for a thousand different companies. In fact, the best ones are working for a company that doesn't exist yet. They have an idea nobody's thought of yet. The idea of better than the big game publisher is better than the other console manufacturers because he's working alone. She's in her room at her mom's house developing something that's the coolest thing you've ever seen.   How do you get it there? Well, you have to convince them about this model that I'm talking about. You have to tell them that freedom leads to creativity, which leads to the creation of value from nothing, right? It's something that I think about all the time.  And he explained to us something that stuck with me for a really long time about all this. And that was this concept of showing off to your friends that if you didn't have, and it was really obvious to all of us then, because I'm an older guy, we started programming in the late 70s and the early 80s and there was no GitHub, there was no internet, there was nowhere to get any kind of code that you could copy or look up, maybe in magazine. And there's no way to distribute anything you did. So the only joy you got out of writing code, aside from kind of a masturbatory joy of seeing that you could do it, which is I don't want to discount that. That's another talk entirely, but you get the joy of showing your friends and hey, I did this amazing thing. And then something magical happened which was totally new to me and I think very new to culture in general, which was automatically your friend would be inspired by that and maybe do something even better. And then maybe you do something even better than that. So we were steeped in that culture. And guys from my generation, I'm very proud to say, it made so much of the infrastructure that we take for granted today. Incidentally, all running an open source software, right? And the reason it's all running an open source software is because that generation of people who wanted to show off to their friends were encouraged to do things better and better and better, more secure, more real. When everyone can look at your code, you write good code.  When some software is operating a nuclear power plant, I'd like to think that more than one guy had looked at it. And furthermore, I'd like to think that the guy who wrote it was having a good time writing it, that the person who wrote the controller for the cooling towers, right? She was really excited to show her friends how there was no scenario in any test that could cause it to melt down, right? That place of play and creativity. It doesn't matter how serious the job is, it's play and it's creativity. And there's a positive side to it, which I've been talking about. And game development for me has been a tremendous source of a lot of this because the product of game development is fun and happiness. And so you kind of want to make sure everybody's fun and happy making a game. Tell electronic arts that, but separate talk also. There's the dark side. And there's a great story that I often tell that I probably shouldn't name names in here, but it involves a large European passenger aircraft corporation and a certain model that they had. And the way that modern jetliners work obviously involves a tremendous amount of software and automated control systems. And we have moved past this idea that there's a physical connection between the thing the pilots hold and the control surfaces of the airplane, right? And this was a big tenet of Boeing for a really long time. They wanted to make sure that there was a control surface. There are a couple of accidents, unfortunately, that showed that no matter how enthusiastic a pilot was, they couldn't apply 180 pounds of force on the stick for the hour to get back to the airport. And so people have been in this idea and they said, okay, maybe it's a good idea to have actuators that have software running this. And of course that's dangerous because programs crash. A friend of mine at MIT worked on the F-22 fighter control system. He told me the F-22 could reboot at 22 Hertz. I'm like, I don't know that that's an awesome thing to brag about, but that's pretty amazing. And so the way that people thought about this is voting. And most of you guys know this and a lot of systems do this. They vote. There's a point to this story, I promise you, so just stay with me, okay? So at this large European passenger aircraft manufacturing company, they had the idea that they would have separate teams of people with no contact, make separate systems that operated with completely different control laws to decide what to do with the control surfaces when the pilot moved the stick and the throttle and that they would devote. And whichever got the most votes is what the control service would do, right? And they isolated them from one another. So the feedback that they got was only to themselves. Worse. Worse. They told them they couldn't talk. So all the fun was gone. Nobody was showing off to anybody else. And then they had a shared cafeteria. Ha, ha, ha, ha. It is not that funny because the first Paris Air Show debut of this airplane resulted in the airplane doing a fly by, the airplane control systems deciding this must be a landing, extending the gear and everyone died. There's a really dark side to this and it's a lesson. And I think it's something to also keep in mind when you talk about the many benefits of free software, when you talk about the evolution of Linux and why it's so important, why it runs the world. So understand that the openness and the playfulness, the acquisitiveness that causes people to want to work on free software to work with free software to share software to work together on projects, where the value is the input of the ideas and not the control of the license is critical to make systems that matter. So you can go all the way from halo to crashing jetliners and find examples of why it's important to drive what you do with this playfulness and this inquisitiveness and go a step past that. And I think understand the point of this entire exercise. And when I look at a conference like this, the point of the conference is to learn more about how to engineer into the process of making things the spirit of openness and sharing. So that baked into these systems is a measure of this safety and a measure of this fun. And it's also important for ensuring that the world isn't just sort of a grim hellscape. And part of the reason I say that is that interfaces are miserable at the moment.  You touch it like, that's almost a second.  Like, okay, what if I touch twice? You know what I'm saying?  And I'm imagining the team that put this thing together and on different airlines, if you become a connoisseur of terrible in-flight software, as I have become, you'll imagine what the working environment of these software teams was like. Like, what concrete basement were these poor motherfuckers in?  And you're like, okay, clearly these guys were given like the last generation of like video player that's somewhere and has a terrible response time. And like, you know, it's libraries are like in Fortran. And they're like, okay, what are we gonna do?  Just like, ah. And then they're good ones. And you know, I've programmed so much and worked on so many projects now being ancient that, and maybe some of you will agree with me here and tell me that I'm not totally insane, but you can kind of feel this when you're using it. And the airline entertainment system is a really good example because it's a completely captive audience, right?  And so I enjoy trying to figure out what's good and what's bad but the overarching feeling that I get, you know, trying to use these systems again and again is the same thing. It's that the joy that the team of people felt making it is the joy that it gives me. And if the team of people making it had a shitty time and a bad boss and were miserable and didn't have enough resources, I'm gonna have that experience. I usually have that experience. Drinking helps a lot if you have a couple of beers. Yeah, the response time becomes less of an issue, but the trick is that even when you're trying to be entertaining, you can be totally un-entertaining. If you haven't built the software and built the system in such a way that you have this play involved in actually architecting the entire thing, okay? And I think that in the early days, in the UNIX days and the pre-UNIX days, and when I started programming, it was completely extraordinary to find jokes in the manuals for software and for computer systems.  And I thought, you know, when you're in that culture you think this is really special because we're working in these computers and we're special people and we don't have to follow the same rules as everybody else and we can have fun, we can tell jokes and we can be different, we dress how we want and we can have long hair and we can be whatever gender we want and there was really a true thing and it was a completely special time but the thing I didn't realize at the time because I was too young was that the reason all those things were true was because it was a group of people who were creating value from nothing in a way that the other business people couldn't figure out and they reacted to it by working together and having fun doing it and so the jokes and the joke names for everything and things like having a penguin logo for your operating system and all the other stuff that we fucking love came from that playful spirit and you think, oh, that's just because we're underdeveloped kids, we never grew up, we're not serious like the bankers and this is, again, back to your sister's boyfriend who looks at you because you dress like this and you don't dress in a suit like he dresses to go to his fucking job as an investment banker. Okay, fine. But the... Brother.  That's a big deal.  I would say quite the opposite.    They were like, oh, here's your job, here's your boss at Steven Spielberg.  Okay. But years later, when I was doing interestingly entertainment finance, which is a totally separate story, I was talking to him one day and I had to give him some piece of information about our business. And I called his attorney, Harold Brown, who's also an awesome, fun, loving guy. The entertainment business has a little bit of this. I called Harold and he wasn't available. All right, so I called my boss who ran this big company that worked for Steven. I said, hey, I didn't talk to Steven and he wasn't, boss wasn't available. All right, so I called Kathy Kennedy, who was his producing partner and his old assistant. She wasn't available.  Picks up first ring. And I was like, hey, I tried to call Richard and Harold and they didn't answer.  I had all those people so I can sit around and answer the phone whenever I want. Like, okay, I get it. Very good story. And I told him about this problem and I was saying, you know, it's kind of weird. I deal with all these financial people all day long and they're really concerned about like whose name is first on an email and who gets invited to lunch first and like who makes slightly more money and which model year of car they have and it's starting to really freak me out. And he said something really wise. He said, hey, you know, when you're walking down the street and you're walking past a tree and their ants climbing up the tree, you think to yourself, like, what do they, do the ants see me? Am I terrifying to the ants? Like, are they just talking to one another? Like, are they living in the ground there? Did they come a long way? Do they all know they came a long way? Like, what's it like to climb? What's up the tree? What are they thinking? It's not that bad a story. It's really, it has a good punchline. Okay, so yeah, anyway, so he said, listen, those guys in the other world, they don't think that way. They're not having that imagination. They're not walking down the street, imagining a page of code, or walking down the street imagining some way to use a piece of technology to do something magical. They're walking down the street thinking, who else is going to the lunch I'm going to tomorrow? And can I be in a better spot than them? And he said, you should feel bad for those guys. You should feel bad for them.  All right, so I'm gonna, I'll leave some room to ask questions, and hopefully nobody asks questions, and I'll get to finish early, which would be great, and have a beer. But, the last thing that I wanted to go into is kind of the corollary to all of this, which is something that, I figure you guys think about quite a lot, and I know you come from a very diverse background of different industries and experiences in life, but you're all here at a free software conference, and so you all have this kind of experience in common. And maybe it's something that nobody tells you enough, and it might sound really cheesy, but you know, the entire world basically operates right now because of free software and how it started. There's no part of our lives that isn't touched by the philosophy of Stallman, by Linus, it's nothing. No part of your phone, this presentation running on this Apple laptop, and what is the Apple really running? You know? That's right.  But, I think that we get all tied up in shit around what's the newest thing, what's the newest trend, what's going on, what do we need to learn, what's happening, who am I talking to, what's my project, how do I find a boyfriend who's not abusive, all these things.  And it's just true, right?  That's exactly right. And then what do you do for fun when you go home is you try to put that same operating system on a Raspberry Pi Zero and freak out for like eight hours and then finally get the kernel up and it doesn't really work, but you still think it's victory. But any computer that needs to do anything important that's in real time, any system that needs to do something that's important or connect to other systems, any time important data needs to be transferred every stock trade, every crypto trade, every contract, the transmittance of every contract, smart or not, every flight schedule, every space mission, every robot on another planet, all of them depend on the continuation of free software and the spirit of play and innovation that built it all and try to keep that in mind through the times that suck, okay? All right, I'll take questions now. I have a slide for this.  Okay, all right. I'll give you a copy of the slides for sure, freely available. What's my favorite open source software? I have to say my favorite open source software is the numerical libraries for Python.  And making the world safer and better. And my favorite video game is, yeah. What's your favorite song?  Like, the first functional build of Halo is my favorite video game, okay? But I still have on an Xbox dev kit and play and it runs for almost five minutes, but it's a glorious five minutes, man. Do new developers? You know, it's boring. It's find a place where you can play.  And it's not always fun, right?  And I know that's like an Old Testament kind of a message, like a thou shalt not kind of a thing, but that's what it is.  I should put them in a gravestone. Over here, your X window comment was straight up. Oh, next step, oh, even better. Yeah. I have stories about next step. I have not.