optimizes after its startup time So it takes a while to actually hit, you know Can I get to its maximum throughput where it can take as many requests as it's actually able to To take so there's there's some discrepancy there So in that case we'd have to always over provision our java applications, which isn't so ideal But again, like I said, there's solutions to that And one of the one of them is the quarkus project. So this is a project that was started. I think about Four no five years ago. We the quark has just had its fifth anniversary So quark is uh, they say is supersonic subatomic javas the supersonic so very fast subatomic So, you know an atom, right? It's very small And subatomic so smaller than an atom. So the idea is that, you know, it's java, but you know very fast and very small So the way that that works with quarkus is that If you compare it with a regular kind of normal Java application running on the jvm You'll see that still do their packaging and then during the runtime during the startup time It's going to do class path loading and doing a bunch of optimization But that's already when it's started up, right? So imagine a container it starts up and then the jvm is loading everything and then finally starts up so Which makes sense in the traditional java world where we had big monolithic applications on dedicated servers And the idea was not that they would start up, you know and stop all the time, right? The idea was to give them keep them running as long as possible Um, but in a container world, of course, or especially serverless We want things to start up fast and we don't want the same optimization to happen every time for every container that starts up Why don't we just do it before and that's exactly what quarkus tries to do So it does all that class path scanning and building its model of the world Before it packages the application so that it doesn't have to repeat that over and over again when it starts up So that's kind of how it works It's more sophisticated than that but this kind of gives you an idea And then what's also interesting with with quarkus and this is not just quarkus You know other java stacks can can do this too But what's interesting with quarkus is that enables this Out of the box, which is a native compilation of your java application So it compiles it down to a native binary. So typically a linux You know the target is is a linux binary for a linux container And so there's no jvm actually in this natively compiled binary So it has a whole bunch of optimizations thanks to the project grolvm And that's really interesting for serverless Because we're going to have a very fast startup time. There's some downsides to native compilation because the jvm actually has Some some interesting optimizations that you lose by using native compilation, but Maybe we'll get to that. Maybe not. Anyway, here's like a little bit of a comparison between A traditional cloud native stack, which I guess in this case it was a spring boot application That started up in 4.3 seconds and then the exact same application with quarkus on the jvm because you can also run quarkus on the jvm. You can already see that's quite a bit of difference, right? So especially if we're thinking of serverless This is this is great. And then you can see, you know, if we compile it down to native Binary 16 milliseconds, I think I can wait for that I think that that would be okay. And then yeah in terms of the footprint and the memory usage It's also quite a bit less So going back to that Java warm-up time like I said native compilation is interesting because There's no more optimization to be done after it's Compiled its binary. So you can see here this yellow line is basically Fairly constant, right? So that's with a native compilation of our job application But what's interesting is you see the red line, which is you know happens to be the quarkus application on the jvm Even though at the start it does not have the highest throughput It does thanks to the optimization of the jvm end up having better throughput So it kind of depends on what kind of application are you running, right? So if you have serverless applications that just need to do something very quick and then that's it native compilation is great if you have longer processes That need to handle a lot of throughput then you're probably better off running on the jvm So that's a little bit of a sidebar on how to optimize You know java for serverless. There's actually A lot more that you can do But yeah, there's the talk is only so long. I'm actually writing a book about serverless java And so I'll go into a little bit more detail in there and I'll I'll share the details. It's not published yet But I'll you know, I'll share what what we're up to Later on so anyway, so we have our java application that starts up fast now. We want to deploy it with serverless, right? So most people when they think of serverless, they think of functions as a service. So, you know, we can start with that So functions as a service So I think amazon lambda was pretty much the the first one or at least the one that I saw the you know the first time Is uh, basically the idea is that you have your code and you package it and then you give it to Amazon lambda and it takes care of deploying it and and scaling it based on The usage and then same with azure function same with uh Google cloud functions and IBM cloud functions and all that Which is great. It works pretty well. Um, the only downside is that uh, we have um To create these functions as a service Amazon for example or same with azure and same with the other ones is like, oh, yeah Here's an example of a function that you can write for uh, you know to deploy to aws and then you can see that Oh, that's nice. They have import this from amazon aws import that And you can see that, you know, you have your handle request and then there's a context and stuff like that And that's actually all uh, very specific to uh, amazon lambda So if you want to redeploy this function to azure functions, well That's a problem.  So that's not so super great, right? So yeah lock in again and this one in my opinion is worse Than a lock in in terms of infrastructure I think if you have your code where you need to rewrite and refactor That's a lot of man hours and that's very expensive, right? So we want to try to avoid this as much as we can so, um quarkus actually has some uh, some interesting solutions for that too. So you can actually write a cloud agnostic functions um With uh with java and quarkus so, um, one of them is uh, There's a project called quarkus funky and you can write, you know, you can see down there at the bottom a basic uh function you just annotate it with this funk And um, that's pretty much it and then the only thing that you need to do is you add a uh, a quarkus extension, which is basically like, uh Like a smart dependency. So you just add that to your palm dot xml It has all the knowledge of how to package and deploy your application. So you can add for example, um An extension for quarkus funky amazon lambda And by adding that extension I can deploy my function I can build it and deploy it to amazon lambda And then, uh, if I then want to change it I can change my quarkus funky amazon lambda to quarkus funky azure or azure htdp I think it's called And uh, basically that's all the change that you need to make. So it's a much less painful process to do that um Now i'm gonna have to be a little bit creative because of we're you know, usually I mirror my screen so I can see what What i'm doing if i'm demoing this Um, so i'm gonna have to think for a second how i'm gonna do this. So maybe i'll get This screen over there And so here is Uh, a project that I've created with a few different functions. So you have The quarkus funky aws. So we'll start with that one Um, and you can see that we have, you know, a regular palm dot xml and I hope this is big enough For you in the back I have to look over there to see what i'm doing. All right So you can see there's not much to this palm dot xml other than you know, it's using the quarkus platform To add some convenient stuff and then you can see here quarkus funky amazon lambda as a dependency And then I have a function here source main oops No java there we go Uh, and I created this function For my fictional company that's Creating some, uh, you know some space stuff. Anyway, so this is a function to ask for a landing request on a given planet And so you can see you pass in some landing details And then it'll you know reply with approved for a landing. It's not a very complicated thing, right? But you can see this is just plain java And then you just see this uh at funk annotation and things to that we can deploy it to aws Lambda so let's try that here for a second Oh, so when you compile your application what quark is what the quarkus extension will do is it'll add Uh some stuff in your target folder And this is where it becomes interesting because you can see It has a function dot zip and that's exactly what amazon lambda is expecting you to send So it's expecting you to zip up your your artifact and and send it and then there's this manage Script and that's a little convenience script to deploy to amazon lambda without having to know exactly the syntax of their Of their cli so with that we can do Target and then that managed script Maybe I need to recompile let me see if I'm in the right directory here Quark is right. No, I'm not that's why CD go back and then quark is funky amazon. All right. What is it? Quark is wait This is so hard to do from here What was aws? Okay. Thank you And then uh if we now do targets manage dot sh we can uh create our uh our function So I would do that except uh the one that uh that gets shipped has by default Sets the runtime to java 11 and I happen to be running java 21 So what I've done is I've created a copy of this manage sh In my main folder so I can just call it from there and then use java 21 So I can do manage sh and sorry. This is probably a little small for you and then create And that's going to create my function on on amazon lambda and then You know then I'm up and running so I could invoke it from there and then I'm Using amazon lambda then at some point if I want to change to Uh funky to uh to azure I can create the exact same code. So you'll see main Java and we see the exact same landing request function Uh Scroll this over so you can see here exactly the same uh function and then When I compile this We'll see that in the target instead of that Function dot zip and the amazon stuff. It creates an azure functions folder With all the stuff that you need to deploy it to azure and so then you know, I would do kind of the same thing To deploy it to azure, but i'm not going to spend too much time on this because otherwise we'll we'll run out of time I think But that gives you an idea right so I have examples of these And I I'll share the github so you can see exactly what the difference is between the different ways to Deploy to the different functions as a service um Now the thing is that even if you create a function on these Providers and you do it with something like quarkis and you don't have the dependencies right in your code um Chances are you might still get some lock-in right because uh if you look at Lambda, maybe you want to add some events to it, right? And you want to tie it in with a messaging system or you want to back it up with a database Like dynamo db or you want to have some storage or something Well be careful right because we're going to have the same problem of having lock-in with uh with the provider So you need to be careful with that too. So function of the service is cool. You get uh, you know kind of on demand billing and uh in scaling and you don't have to worry about the infrastructure But actually serverless goes a little bit further than just you know functions Serverless becomes even more interesting if we combine it with uh containers. So um, Again, the cloud providers provide some solutions for that as well. So you can see that amazon for example as far gates azure has container instances, I think and google has cloud run And uh, so you can basically create containers and push them and it'll uh, it'll scale for you as well And then there's some other projects as well out there such as uh, k native which we'll visit in a second Now the thing is you know, this is for example, azure because I don't want to always show aws and and uh, Say that aws they're locking you in it's the same with all the cloud providers, right? So here you can see you know, this comes from uh, from the azure website Uh, some examples of how to use uh, azure container instances. It's like, oh, yeah, this is really cool And then you add an azure application gateway and then uh private endpoint and then you call lists from azure and blah blah blah And then yeah, of course the same lock in right so um So that leads us to a project called uh, k native. So k native is an interesting project uh, because enables you to uh use serverless Um together with kubernetes. And so it's basically an open source project working with an open source project To build serverless and event driven applications on kubernetes. So the idea with k native is that It tries to make containers easy. So it helps you with Building the containers pushing them and deploying them all at once. You don't have to create yaml's in in kubernetes. It kind of does The heavy lifting for you Then it has you know, autoscaling out of the box. It also has this revisions concept. So every time you create a new um You change a configuration or you push your application Again, it's going to create a new revision So it makes it really easy to roll back And forward between different versions kind of out of the box Or you can do traffic splitting and enjoy things like canary deployments or ab testing and stuff like that So that's also something that comes out of the box with k native, which I find interesting And then yeah, the idea what's what's cool about k native too is that because you can run kubernetes Not just in the cloud. You can actually run kubernetes in your private data center It allows you to use or enjoy these serverless capabilities of autoscaling and easy deployments You know kind of in a hybrid cloud Situation and because it's containers, you know, it supports any programming language, right? So as long as you can run it in a container you can use serverless with it Um, so, you know, you can create Functions with k native as well. So you can you know, again kind of the same concept as before with with lambda um or azure functions you create your code and then K native can just package it up deploy it for you and it will do that in the form of a container But to the developer that's all kind of abstracted away, right? They just do k native func create to create a function and then funk deploy is going to build the java application in case of java In case of no js, it'll do whatever npm build and then it'll create that into a container and then Deploy it so it'll use by default build packs for that if you're familiar with that project Or you can also just Have your Applications built as a container and you can deploy them as a k native service As well. So you can just use containers as a serverless Wait, so we've seen kind of the evolution from functions to serverless containers Um, but where serverless is really interesting is kind of with uh event driven architectures, right? So you have events coming in that triggers You know a certain scaling of your application and then you can Create really cool kind of applications or architectures that are going to scale up You know this part and that part and you know you can use you know exactly only the resources that you need And you can do that of course all the cloud providers have their solution for this But you can use it with with k native as well. So k native supports Uh The cloud events protocol, which I'll tell you in a moment But maybe first to give you an idea of how k native works. So on the serving side, that's where uh, you have Basically a control plane that's going to monitor the traffic If if requests are coming in it will scale according To the requests that are coming in so it's going to scale up and down Based on exactly the demand of that moment Um And then there's also an eventing part of uh, of k native to handle the events Going from one service to another so you can see there has Brokers and it has uh channels um And so the way that it passes those events between uh different services is using the cloud events specification Has anybody heard of the cloud events projects? See one hand, uh, so cloud events is uh, is is pretty interesting project. So it uh, recently graduated as a cncf project Um, and the idea is to have a comment event Schema standard, um, you know If you're familiar with uh, sending messages between different systems I can be you know, like in this case, we're using kafka and we're doing avro and there we're using amqp and there amqt and we you know the consumer and the And uh Provider and the consumer the producer and the consumer have to know exactly from each other like hey I'm sending this in this protocol in this format And then the consumer can do something with it, right the idea with cloud events is that uh, you you add that kind of information to your payload Um as in the form of a cloud event Specification so everybody knows exactly what to expect so it's protocol agnostic and uh, it's it's an easier way to communicate between different Systems and this is something that k native supports So you can see here a usage pattern of uh, of k native here is uh, you have an event an event source And you can send that to a sink that's gonna See that hey an event is coming in and then i'm gonna scale my k native service Based on that but you can make it more interesting right so you can create a channel Much like uh brokers, right Where you have a channel and then different services can subscribe to the events that are being sent through this Channel so you can in this case have two different Subscriptions being subscribed to a channel and then you can also have like a full fledged broker where you can have Subscriptions and filter the messages that they're interested in so you have a whole bunch of uh Interesting scenarios that you can build with k native and again using serverless to scale up and down your uh, your resources So with that, um I have one little demo to show and uh, you can participate in hopefully because I need some help Uh to generate some load to scale up my my uh, my applications, right? So I have this uh, yeah, I'm terrible with ui. So this is really the best that I could do Uh, but you can vote for your uh, favorite ide so Some people like vim other people like sorry, I don't have emacs on here by the way But I have intelligent a vs code uh eclipse So yeah a little bit focused on the java developers and then uh, yeah because I'm from red hat You can also vote for open shift dev spaces Which is uh an ide that runs on open shift and that you can use in your browser and basically is uh You can use intelligent or uh vs code, you know kind of in your browser, which is cool But yeah, I just added that kind of uh to be funny. Anyway, so Now I have to get this window over there There we go. Okay. So here's my application running in a kubernetes cluster. Yeah, it's uh, It's an open shift cluster in this case and we can see that we have uh, a few different uh components and actually what I'll do is Go back to this slide because I think I have a diagram of this uh of this application So basically I have my ui right that uh scales down to zero if there's no request coming in Once you start uh voting. Well, actually once you go to the url You'll uh, you'll see that uh, this is going to scale up because it's going to wake up And then uh, every time you vote uh, it's going to send requests to my uh, Ingestor and then that's going to send the requests over uh through a kafka topic in this case To a consumer that's going to you know update my database and then my ui is then going to uh, Read those events back uh into the ui. So you know, this is kind of uh, A very over engineered uh solution for for this problem, but you know kind of demonstrate how it uh, how it works with Um with serverless So I don't think this qr code is going to work because I forgot to update it Actually, uh, no, but I'll I'll just cherry the qr code. So uh, I'll go here to my ui So right now you can see uh, if there's no blue circle around That means there's no uh actual no pods running so we can see here. There's no pods. It's autoscale to zero And so once I go to this url Uh, it's gonna wake up you see and it's gonna start a pod and then my ui is now up and running um, and so I really have to go look over there. There's a qr code thing, but I can't see it from from the stage start uh, oh, I think it's in uh It's not there. It's in chrome. So sorry apologies I was hoping I could mirror my screen So I'm going to go to chrome here this and then you see a bunch of tabs apologies for that and then uh We'll go here again to the thing and then now we have a qr code So go ahead and scan that and if anybody uh feels up for a challenge Uh, you can vote as many times as you want Um, you can create a quick little script if you If you uh, if you want to and then uh, we'll see that the votes are coming in And then uh, let's see what uh, what our open with our cluster is doing right So let's go here and keep voting and we'll see that right now We don't have a ton of uh votes coming in but we do see that our pods are scaling up Right so we can see more pods are being created because you're creating more load and I you know, so you can configure k native um Based on how many requests it should Scale to how many pods so I have it set up that every time there's concurrent events It should handle those by a separate pod which again is a little bit over engineered But you can see that it's going to scale up and down based on whether you start or stop voting. So, um Kind of up to you if you want to see it scale up or down But uh, yeah, so we can see now it's terminating some of the pods So, you know, some people are voting a little bit less and then if you start voting again It's going to scale back up and so it just keeps going based on the demand of the moment. So Let's see what our votes are at here So we have IntelliJ winning and then vs code. That's the the usual suspects, right? Uh, and we have, you know, but is it 870 votes for IntelliJ. So at the last conference I was at somebody was uh had A script running and I think we got up to like 200 000 votes for for IntelliJ or something So and that was funny because you'd see like 50 pods running, you know, and then uh, yeah And then yeah, if if uh, let's say, okay Let's say that everybody stops voting and let's see if it's going to scale back down to zero. So if everybody stops voting Uh, we will see that uh, it'll terminate in just uh, in just a moment And then uh, it'll go to sleep And just wait for more requests coming in so we see that The dark blue circle means that everything is terminating and then uh, in just a moment We'll see that no more resources are being used Excuse me. Yeah, unless Jamie wants to be smart and then All right, so we see that it's uh, going to sleep And that's exactly how I feel. I'm a little bit jet lag All right So yeah, the last one is terminating There we go. Now it's zero and now if you start tapping again Or you know you hit refresh on your ui because actually what's interesting is that the ui Is only gonna wake up when you go to the url But once you're already on there, it's not gonna reload, right? So but now we see that it wakes up again Anyway, so that's uh, just a quick little example of uh, how that could work Uh, so I'll go back to my slides. I think it's this one. Yes Um Yeah, and then of course we have to talk about ai or every talk has to say something about ai, right? Um, so serverless in ai, uh, actually they're a pretty good match because Um, we have in ai we have for example Uh, somebody's using uh fancy chatbot or something So every time somebody types in once a question or something that creates load, right? That creates requests to some uh server some service that was built by by your model There's many kind of use cases with ai, but serverless makes a lot of sense, right to When my model needs to be created I'll scale it up and then when it needs to be used and stuff like that So I added the link here of a use case from the k native project If anybody's interested in uh, that's uh, it was it's an interesting read. Um, and then yeah, just sort of wrap it all up So, you know cloud providers they offer a lot of cool stuff, right? So, um, I would say use it but Yeah, keep in you know, like keeping an eye on what's what's happening in the in the open source world in the community Because if there's a solution for it in the community, um You will lock yourself in less and you will have less pain later on Um, oftentimes what the cloud but the cloud providers offer for services might at the face Seem like a better solution an easier solution. Uh, but yeah, you need to be mindful of that lock in Um, and you know serverless if you thought that it was just functions as a service It's a lot more right you can actually use it Uh for many use cases in event driven architectures or just you know With containers that you want to scale up and down and just use the resources of that moment That allows you to have more density in your nodes That use them more effectively and you might be able to use less nodes Saving you a lot of money and saving the planet just a little bit too, right? And then yeah use open source when you can and then yeah provide Proprietary services when you must because It does end up being a little bit inevitable that you're going to end up in a scenario where you do Have to resort to that But the more we use open source and the more we push these cloud providers to use That we like open source the more they're going to start investing in it, right? So we've seen that Microsoft AWS IBM They're all doing a lot more open source lately and that's because we are using it, right? So please use open source if you can Um, and then yeah, I would say if you feel yourself limited by what uh, what you find in the communities Contribute, right? Uh, that doesn't mean you need to write the whole solution. But you know create issues Provide some ideas. Uh, that's what open source communities are for so, uh quick kind of, uh Link if you're interested so our team so I'm part of the red hat developers team So we write also books, um, and red hat is nice enough to sponsor us Or sponsor those books at least, um, and then you can download them for free So it's kind of a nice little thing so you can see that there's like for example a quarkis for spring developers So if you liked, uh, what you heard here from quarkis and you're a spring developer Well, there's a book for you right there If you want to modernize your applications if you want to work with get ups and then, uh, yeah And then one last kind of, uh Thing for for myself. So I'm writing this book serverless job on action. So what you saw today Um, there's a lot more that goes into that. Um, so that's what we're writing the book is not, um Available yet, but it should go into the manning early access Program in the next few weeks. So keep an eye on that. So, you know, if you if you're interested Follow me on, uh, linkedin twitter, uh, masodon, whatever And then I'll definitely be posting about that because I'm pretty excited about that book And with that, uh, I thank you And I would like to know if there are any questions Yes, uh, we have a microphone. I don't know if it works Yes Yeah, I wanted to know, um, what jvm were you doing when you were doing the benchmarking with some of the quarkis stuff you had earlier? Uh I don't remember because it was a few years ago. Yeah, but yeah, you'll see, um Overall, you'll see that if if you compare like for example spring boot or, uh Like java e or something and you compare that with quarkis, you'll see the the difference pretty quickly Um, the only thing is that, you know, you you need to kind of Measure the right thing because for example a quarkis Uh, when they say they measure the startup time, it's actual When it's ready To accept requests whereas spring boot it'll say like, uh, it took this long and it actually is to, uh To start your application, but not being ready to actually receive requests. So yeah, but anyway, thank you Uh, any other questions? Yeah, I've been running the serverless framework. Um, and I'm wondering why I would want to choose The tooling for quarkis and k native natively versus using serverless to Basically set up a yaml and I don't I don't have to change a single line of code Except the yaml file for deployment Sorry, uh, I didn't get the question. Oh, um, why would I just use quarkis and k native? natively versus using the serverless framework That creates a uniform and unified CLI and mechanism and I don't need to change my java code because general it'll just package up And add the the packages for lambda or for whatever we have For each native cloud platform, right? So why would I it's like so why would I want to use that versus like one more layer on top of that via serverless? Yeah, that's a good question. Yeah, so I'll I'll ask because there's basically two questions in there, right? There's a question about quarkis uh, so Um, the thing with quarkis is that you'll get your performance boost And some integrations with the different providers the serverless framework is interesting. Um, however, I've seen it kind of I don't want to say fizzle away, but it seems like the adoption has has kind of stalled. So I would be a little bit Um, I mean, I don't know if you're part of the serverless framework people, but Um, I mean, it's it's an interesting project. Uh, what I like about k native is it's that it's uh Back by cncf and uh, and so you know that this project is going somewhere Has a lot of adoption and you can use it with kubernetes. So it lends itself to a lot more use cases than just Uh, than than just with the serverless framework Um, but yeah, I mean if you uh, if if it fits your use case, absolutely I just talked about k native. Yeah Right. Okay. So in other words, yeah, it's still you know, it's still a uniform Uniform mechanism to deploy I'm just wondering why it's it's I'm lazy I'm extremely lazy and and I just want one one mechanism and I don't have to change any line to code It's like and and just have one piece. So yeah, I can package it up either for google or for azure Or it's like whatever I want uh, and lambda. So I'm yeah, I'm like curious. Um, yeah It can still use quarkus. It's uh underneath as well. Um, uh and and caught land but the question is is Is um, should you and what are the you know, what are the bonuses the one thing I do see is Quarkus is native is really interesting That'll run much faster, which I guess eight of us lambda doesn't provide very well at all Right. Yeah. So for example, eight of us lambda. They provide snap start, which is an implementation of project cracks like checkpoint restore at something I can never think of the name but it basically makes a checkpoint of Your application on the jvm that has already started. So, you know, it's kind of makes a checkpoint and then starts up from there Uh, and amazon lambda kind of uses that implementation. Anyway, that's kind of beside the point, but yes, um, I'm not like I'll be honest. I'm not super familiar with uh, Latest developments of the serverless, uh framework. Um, that they support k native Um is interesting Um, but to me then it's an extra layer in between quarkus and k native because I can just use quarkus that has extensions for k native So then I don't need that. Sorry. I have a another question there But yeah, I mean if you are interested absolutely use it, right? I mean if it works for you Yeah, I have two questions like first question is um regarding like language support Like like javascript guys are using typescript like that java guys are using kotlin And curious like kotlin encourages using coroutines and flow versus quarkus encourages its own framework for Non-blocking code. I was curious. What is the direction? Is there any possibility to support Kotlin native? Yeah Um, so quarkus supports kotlin. Uh, I'm not sure for native though so It could I'm I'm just not sure myself One more question. Yeah, sure regarding the cloud events Uh, generally cloud events are I'm using it with uh with kafka And I was curious like how it works for the htp transport than kafka. Uh Can we use the cloud events with the htp server? Context also or not? I yeah, I don't see why not. Um, yeah Because it it basically Supports that protocol out of the box, right htp or whatever. Um, so I yeah Any other questions? All right. Well, then thank you and have a nice evening All right, I guess we'll get we'll get started. Um, hopefully more people will trickle in We've got matias, uh, geese. I hope I didn't butcher your name Director of technology at venify learn something today. It's not the benefit. It's it's venify Matias has been part of the jet stack effort behind cert manager as well and besides that The fun anecdote about him is that he's uh, he's quite an accomplished marathoner sub three. What 249 249 So we'll we'll talk afterwards. Maybe I'll get some good hints. All right, it's all you Thanks. Thanks for the introduction. So today I'm hoping to help you The mystify a bit about spiffy and how it can solve secret zero or what some people have Talked about and and mentioned as the bottom turtle problem um Maybe a lot of people already might be using spiffy. Um, it's the underlying Technology that is powering the identities within is to your service mesh. For example Quite recently linker d has announced support for it as well Sillium is using it for their mutual authentication and authorization system So more and more systems are starting to adopt it But like I'm trying to wanting to demystify a bit around what spiffy is spiffy is a workload identity Thing and to be able to talk a bit about workload identities We kind of need to go back to a higher level and like Where doesn't spiffy is a potential solution and like you kind of have to a kind of identities Now you have machines and you have human identities human identities are well understood And you have countless solutions that help you solve the management problems of them Examples are google sso Enter active directory Octa and all of these kind of solutions But the lesser known identities are the machine identities Machine identities exist out of two categories device and workload identities For devices think about tpms that you might even have on your laptops and other solutions But with the sprawl of microservices We have seen an increase of workloads and it is important that we can uniquely identify workloads To be able to govern and authorize communication Research has proven that machine identities will outcompete user identities with 45 to 1 Currently there doesn't really exist Many full blown solutions to manage those With workload identities and we get a mix of secrets api keys Tokens certificates job tokens or even cloud native or cloud solutions that give you the workload identities But they're not very much compatible with each other. It also makes it very hard to govern and manage them so why should we care and This is the reason we should care by because of this man Aristotle An entity without an identity cannot exist because it will be nothing To exist is to exist as something and that means to exist with a particular identity So that's why we care about workload identities So the purpose of this talk is to guide you through the basics of the spiffy framework And showcase how it can help solve you secret zero or bottom turtle problem But let's start at the beginning. What does spiffy actually mean? Spiffy stands for secure production identity framework for everyone. It's really a mouthful It's an open source project designed to establish a standard for securely identifying and authenticating software services In a distributed and dynamic environment such as cloud native and microservices architectures The project aims to provide a framework that allows different services to securely communicate And trust each other all the identities without needing to rely on traditional network based security mechanisms Like firewalling IP table rules and all of these kind of things Joe Bida, who is one of the co-founders of Kubernetes first proposed spiffy in 2016 At the time Joe was still at Google and here I find that specification of spiffy framework with security experts from other organizations like Netflix and Kind of like these kind of big organizations with the aim to have an interoperable framework for workload identity management There is an actively maintained open source implementation of the spiffy framework, which is called spire It is actively maintained by many organizations and we can see public use cases of the spire usage from the likes of uber pydance and many others There are some really great blog posts out there from uber on how they are running spire at scale and It's definitely worth doing a quick google search on this to like read them like they're very well in depth uber talks about how they run spire at scale They also talk how they use it together for example with kafka for authorization to kafka services like very interesting blog posts so Over the last two years we have seen spiffy taking up a lot more traction First and foremost spiffy itself is fully open source It's part of the cloud native computing foundation and it's a graduated project in there We've seen It's being adopted by and standardized on by the industry So like for example, azure has announced that they're working on a product for spiffy Google is working on a product for spiffy and ados has already support for like spiffy workloads More on the ados use key is even later during some demos that I have so We also have seen that Due to the failure to protect identity leading to supply chain attacks like solar ruins was a really great use case So like it's becoming more and more important to be able to uniquely identify all of your workloads So that like you can protect them And also regulation and zero trust mandates Play into like this more awareness around spiffy because like spiffy can help with like your zero trust strategy So it's all fine and well that we know this what spiffy stands for But what problems and challenges we face can spiffy actually Help us with so The first use case and probably the one I get the most excited about Is removing the need for api keys Imagine a world where you no longer need to have long lift api keys Or tokens you don't have to deal with securely storing them Or having a fine the way to retrieve them from your secret store Which on its own also requires an authentication token One of the major hurdles of adoption for spiffy will be to make your application spiffy aware There are some great SDKs out there in the different Programming languages to help you make that really quickly to implement If you can change your applications for example with databases You can always opt to put a proxy in front of it. That is spiffy aware Like really great use cases of proxies are the envoy proxy, of course, which powers like service measures like Istio But there is also for example Really great proxy for post-gresql that already has spiffy support as well for example A second use case is to authenticate from one cloud to another cloud Through the usage of spiffy An example of this could be that I can write a file to an advas s3 bucket From a google cloud instance or even from a non-premise VM And the third use case is to give all your steps in a cicd pipeline a unique identity That can help with allotting of your software supply chain And you would be able to use your spiffy identity to authenticate To your signing provider and then sign your release artifacts Or any evidence that you provide as part of that release artifact like an software bill of material and sbomb So implementing spiffy is a major step in your zero trust security story By giving all of your workloads applications and machines a unique identity this spiffy identity Allows you to do explicit authorization between your workloads By using your spiffy for authorization between workloads and services You prevent that the api keys can be shared between multiple applications without your knowledge I've seen countless times Where one api token gets shared between 10 different applications making it impossible to track where a request is coming from By using spiffy identities It allows you to have an improved auditing system on your authorization layer As you can explicitly log from which spiffy identity a request is coming from On the previous slide I talked about that you don't have to think anymore where to store your api keys or have a way to retrieve them A spiffy identity in most cases will be automatically provided as part of your platform Where your application is running on a great example of this is kubernetes When your application runs on kubernetes The platform thing can automatically provide a spiffy identity to each workload running on kubernetes This makes the life easier of the application teams as they can focus more on adding extra features improve reliability And security as well as other levels of their applications One of the big challenges with long lift secrets is most cases a regular manual rotation is required To stay compliant with policies and control Spiffy takes this all away and it's even recommended to have shortly spiffy identities of between one hour to 24 hours maximum That automatically get renewed as well as rotated So this is a quote from joe bida again And it summarizes exactly on the previous items we talked about It's very important to deliver a great developer usability experience to improve the rate of adoption Spiffy is also a solution that works everywhere on premise And in the cloud serverless and in mainframes And the advantage that spiffy gives on observability and policy enforcement are unprecedented So we now have talked a bit about what spiffy is the problems it can help solve Spiffy is an open framework with its specification published to get up for everyone to see use and implement We are now going to get into the depth and you're going to discover how a workload can get a spiffy identity and what a spiffy identity actually is So to be able to Demonstrate this I have created a small architecture of two servers with each of them two applications running on them And as part of this we're going to now start with the current status quo that we probably all know Where we have a secrets manager manager running somewhere that can be hush-core vault That can be like one of the cloud providers that have the secrets manager So the way this kind of works is application b for example Will retrieve a secret with its client id And then it will initiate a connection to application x and it will send over the client id that application x then can help verify With that of course application x only knows that the connection comes from a customer that has this client id The client id can be shared between multiple applications without the knowledge So you cannot be 100 sure that this Connection comes from application b unless you're using ip's But like we all know in clouds ip's change quite a lot One of the other downsides as well this connection is not secure. It's not mtls or tls encrypted So the spiffy framework exists also five distinct components And each component sets it has its own function unlike as part of like this presentation. We're going to go over one by one We're starting with the spiffy id To a human that is comparable to however our name and how we are called So the spiffy identity looks very similar to what the website's URL would look like And it exists out of the following The standard scheme that identifies what follows will be spiffy identity The second part is a trust domain. This is the to identify Distinct domains of trust one enterprise can have multiple trust domains And common example is to have a trust domain for production And another trust domain for development because you want to keep them completely separated and not necessarily trust each other But even in production you can have multiple i've seen for example if you want to have distinct trust domains per data center For this example, we're using venify.com as our trust domain And the last part is the path and that's how we identify our workloads Each workload should have its own unique path The path can be a unique uid that's being randomly generated Or a more meaningful one as this is the case in this example We can see our workload runs in data center one on node 10 and it's a web server of the front end So that's like allows us to uniquely identify our workloads So we're back to our application And we have removed the need of our secrets manager and now application b gets a spiffy identity We can see it's spiffy venify.com server one application b and Application b is sending over its identity to application x and application x can verify this and it can It can authorize it based on that spiffy identity So we know that application b is that application x gets like a connection from application b one of the downsides of this is still like our Network traffic is still not encrypted because like it's an identity In essence, it still can be forged So that's why we need to go to the spiffy as fit the spiffy verifiable identity document It's comparable to our passport that we carry around and that's how we can for example, we can get verified at border control So a spiffy verifiable identity document or as fit in short is a cryptographically signed identity document That can be used to verify the identity of a workload within a specific trust domain It's a fundamental component of the spiffy framework Which aims to provide secure and standardized identification and authentication for services in distributed and dynamic environments Such as microservices architectures as well as like cloud native applications Currently job tokens nx5 or nine certificates are supported as cryptographic key materials Which can cover a lot of use cases And also like the all these keys should have a short lifetime that in case of compromise the exposure is very minimal This is an example of a spiffy as fit x5 or nine Certificate and in there you can see in the uri part You can see the spiffy identity in there So like this is the as fit and that has the identity encoded in it And it's signed by our ca which then can be trusted in our environment and it's kind of encapsulating our trust domain Similar example between the job token and again, we have our spiffy ID in the job payload So we again have our Application on two distinct servers, but now instead of just sending over the spiffy identity We're going to send over this spiffy as fit as part of an x5 or nine certificate So application b is again initiating the connection to application x And this time we're sending over the public key of our 519 certificate Which has our spiffy identity in it And application x can verify this based on the common root of trust based on the ca And it verifies this it and it authorizes this And from now on our connection is mtls secured as well because it's the public key You can start doing mtls with that one The biggest missing things after all of this is how can we get applications as spiffy as fits From our central system and how can identities be issued in a trustworthy way And this is where the workload api comes in like requesting our passport at the passport office We need a way to get the spiffy verifiable identity document for each workload And this is done through a standardized api the workload api The spiffy workload api that preferably rents on a local socket Or at least only listens on local host of a note as it's unauthenticated This api is the way to retrieve spiffy as fits It is responsible for the automatic renewal and rotation of our spiffy as fits as we want them in short lift It is quite common for us pfs fit to only be between 1 and 24 hours of a lifetime The workload api allows you to create integrations for your different platforms that deliver spiffy as fits directly to your applications on a well-known location Currently it already exists for kubernetes, but imagine a live that wherever you run your workload You have a spiffy as fit no matter where you run from ados lambda functions to on-premise vms to even mainframes The spiffy workload api is an unauthenticated api endpoint and hence why it only should be reachable locally It's also responsible for validating and attesting the different workloads that turn on that note This happens out of band Otherwise, we wouldn't really be solving secret zero a bit more on the attestation and verification later We are now back to our two servers, but this time we have added the spiffy agent Which runs locally on each of the server and it exposes the workload api through a local socket on that server So Application x will never be able to reach the workload api on server one. For example, it will only be able to reach the workload api on server two So we are going to request and spiffy as fits from the workload api my application It's going to get validated and attested in the background again more on that attestation and verification process later The workload api is going to issue an as fit for application b And application b can then start using that one We're repeating the same process for all our other workloads so that each of them have a unique identity And then we're doing the exact same thing as before application b is initiating the connection and it's sending over the Over the public key which application x can then verify and we're still empty less So we now have solved almost all of our problems And the only problem is remaining to be able to attest and verify that we can issue x5 or 9 and jot certificates through our workload api We can get as fits from a central place But we also need to make sure that those workloads are with this adr The verification of workloads can be happening in band as that would require api keys And we wouldn't really be solving secret zero or the bottom third problem verification and attestation of workloads need to happen out of band and asynchronous Verification and attestation happens at two levels. First, we need to trust our node where the pf agent is running And then after we trust the node we can start trusting the workloads running on that specific node And to be able to build up that trust we need to gather evidence of the environment The nodes and workloads are running on for nodes in cloud provider For example, this can be done through querying the instance meta data, which is locally available to each ec2 vm or tcp vm Retrieving facts about it and then a central system can verify this information out of band In on-premise system, this can be done for example through tpm or other environmental verifiable data Once the node is verified the workloads are running on a node can also be verified This is again done by verifying the environment proof can be gathered there for example from the windows or unique sockets Or when running on in kubernetes by gathering proof from the kubernetes api through the kubelet for example I'm going to talk a bit about an example of this again To kind of make this better graspable. So we have now added a spiffy server As well as a node attestation endpoint From our spiffy server and our spiffy server is also the one that controls the issuing ca for Each of the x5 or 9 and shot tokens that are going to get issued On the servers itself We also have like a node attestation and a workload attestation process running That's helped with that and and the way this kind of works is when a spiffy agent comes up for example in server run It's going to gather some proof of note and it's going to send some proof of note to our spiffy server Our spiffy server is going to validate that information by doing its own checks Like for example by querying the cloud api endpoints and valid validating that the node Is with this it says and it's going to then issue a spiffy identity for the server It's going to send this back And the reason that the spiffy agent also gets its own identity is that from then on out The communication between the spiffy agent and the spiffy server Happens over a secure connection based on that identity as well So like all communication between the agent and the server after its initial verification is happening over a secure connection So we now have like an identity for our servers Now we can start verifying our Workloads we send a request sfit to the workload api And the spiffy agent is going to verify that like it can issue an sfit for our application based on unix Processes or like for example when it's kubernetes on the kubernetes api querying that again Verifying all of that information And it's going to then issue An identity once it's verified and from then on it's exactly the same as before like we send over our certificate We validate the connection and it's still like secure So we now have like actually solved the true secret zero process that can be used for multiple use cases One thing I still want to talk about a bit is spiffy federation It's a More advanced topic, but like I still want to briefly Talk about it and very high level cover it So spiffy federation is important when you're in multiple spiffy servers or multiple trust domains A common scenario is to have like a spiffy trust domain per environment But in certain cases for example, you will have like development and production But you might have a shared environment that then Needs to be able to talk to both dev as well as production Like think about the icd servers for example So you want to federate the shared with production and the shared with dev But not dev with production to be able to have like distinct domains and like reduce the blast radius And that's why we need to start thinking about spiffy federation because Two workloads from different trust domains still sometimes need to be able to talk to each other So each distinct spiffy server must have a trust bundle endpoint that can be queried by other spiffy servers A trust bundle it's comparable to a ca root chain that publishes the public keys of that spiffy server When spiffy servers want to federate and trust another spiffy server They need to explicitly define it in their config And based on the refresh timings that are set in the trust bundle The spiffy servers knows when to re query the trust bundle of the other spiffy server to get an updated one By using those Timings or three first timings it will allow allow like the different spiffy servers to do rotation of their ca's as well in a Nice way without causing down times The three most common use cases for federation is to segment environments with different levels of trust As I talked about with the shared and the production and the shared and the death for example A second use case for spiffy federation is between different companies It is similar to the first one as federation will happen between different spire spiffy deployments But there might be difference in implementation and administration of the spiffy framework And the third and final use case is to enable consumers that don't have a spiffy setup yet People without it can fetch the trust bundle and use that to authenticate their callers without having to commit to a setup of a full blown spiffy deployment And like for this start setup This is for example, how you can do federation with like lots of cloud providers lots of cloud providers support oidc open id connect And spiffy or the open source implementation spire Supports this so that you can do the federation with like a non spiffy full blown setup And afterwards the spiffy server is responsible for distributing the trust bundles to each workload So that a workload can verify the spiffy as fits from another spiffy server This was a very high level overview And we're back to our application. This time we have two distinct spiffy servers We also have added a trust bundle endpoint and server one will Get like be part of the trust domain of spiffy server one While server two will be part of spiffy server two and you can see Each of them have their own ca's that are not linked together. They have their own trust domain So by default they wouldn't be able to talk to each other because x519 certificates wouldn't be able to trust each other So what happens is when we define in our spiffy servers that we want to trust each other Is that they're going to query the trust bundle endpoints of each other and they're going to retrieve those trust bundles And then afterwards they're going to distribute those trust bundles It's a bit of a simplification, but this is kind of how it works And then we get like our spiffy identities again like by doing all of the normal rounds that we talked about earlier And because even though they're in different trust domains because the trust bundles have been distributed Like application x again will be able to verify and authenticate and authorize this connection And set up mtls even though they're part of of separate trust domains so I'm going to do try to do a small demo. It's going to be very basic On like showcasing some of the the spire stuff kind of thing. So if I So spire have Is an open source project And it's part of the spiffy community And what they have done is they have created the helm charts For installing spire and I kind of wanted to quickly take you through some of the things This is like my helm chart and you can see it's it has quite a lot of information It's mainly exposing some of the Domains that I have running like for example, I have tornjack is a ui for example for it Although like it's a very basic ui for spire, but like it shows you something Like for example here, you can see that I also expose my oidc endpoint And and set up tls for it And then also one of the things I do is is As I talked about the workload api runs on a socket I will write a socket to a specific socket And the reason I need to do this is to make it work with istio for example istio works together with spire And I'll show that also A bit later So once after running this kind of thing what you get is spire running inside your Kubernetes cluster So this is my kubernetes cluster and what you can see in here is I have my spire server Currently, it's not highly available setup for production use cases. I suggest to do that, of course And we have a spire agent And the reason I have two spire agents is when I do kubectl get nodes You will see that I have two nodes running in my gke cluster I also have two csi driver ones These are the ones that make it easy to map the workload api socket within my kubernetes spots and again more on that like in a bit And I also have the oidc endpoint and the tornjack kind of one running in there So to be able to showcase this I have a spire tool server set up Within my kubernetes cluster that has the workload api socket running in them and if I now run the spire agents And I'm running the command spire agent api fetch So I want to get my identities And I want to write my identity to a the temp file path and I define the socket path So this is the socket path where I can get my workload From so when I do this And I run this it has like Do the validation in the background and the attestation in the background and it has Gotten my keys And written them to temp. So when I now do This one the bundle And this is the ca bundle you will see this is my trust domain spire.internal.muchasg.be And It's signed by itself. It's a self signed one that I have like currently using because to make it very easy but one of the things I can also showcase you is This one And here and you will be able to see here. This is the spiffy id of my workload It's quite hard to readable, but like you can see for example, it's My trust domain is spire.internal.muchasg.be This workload runs in the namespace default with the service account default and you saw me Getting into the pot. It was running in the same namespace One of the nice things you can also do is you can add extra dns entries to it as well Like you can see here. It has a full pot name In a bit you probably see later the demo that it can also have like service names for internal kubernetes ones Like it's it's fairly moldable to do that One of the advantages of being able to add this extra information is that you then can use this as well for Applications or other things that don't support spiffy yet, but can work with like certificate authentication Like databases and stuff like that. They're supported by default So I'm and for the last bit I'm talking a bit about Istio. So Istio natively can work with spire So Istio has its own As workload identity generator. It's part of citadel But it doesn't necessarily do the same in-depth attestation and validation Then what it's it's really simple attestation and validation spire can do way more in-depth validation of it And that's like you would be able to replace this And it's it's fairly easy to do this. You can see here. I need to inject some sidecar webhook and Besides that like it's it's it's a very basic config like I deploy my ingress gateways. They also have like those workload sockets in there And after I've deployed My bit I can So this is an istio ctl command to retrieve some secret and the certificate from it And this is a And is the standard book info application that I have deployed And you can see in there that it's like runes in the namespace book info with the book info details And it's also quite clear in there that the extra dns name I was talking about earlier like all of like it has services in front of it So I have auto populated them This is all possible with like the the kubernetes integration and it's very easy to start playing with that one spire itself Is is a really great open source project, but when you start running it in production, it has certain challenges It's store. It needs requires a database. So by default it uses a Local file database, but it needs to be sequel based So you can back it with postgresql or my sequel for high availability if you have high availability requirements And it requires a bit of setup and understanding to to start running that But like as I said the cloud providers are working on support for that There are a few other startups that are working on it So there is all at this moment quite a lot moving in that space that hopefully is going to make this a lot easier to run And soon I will also talk a bit about like how to make it How to get like something more simpler running than spire Also like before ending this talk, I have also one more quote from the head of security and czo of notable For him spiffy is a game changer and it really helps in simplifying tls distribution to hosts To end this talk with a bit of a summary spiffy provides foundational identity It's going to reduce the need for api key distribution It's going to give you short lift identity that automatically gets renewed All of the attestation and validation of your workloads happens out of band It is hopefully going to make the life easier of developers like they don't need to go to secrets manager anymore Populate secrets in there like it's something that should be foundational on your platforms And spire is an open source implementation of spiffy I also promise that I'm going to make it a bit easier to get started with spiffy Two weeks ago. I did a webinar With for the cncf where we demonstrated how you can use serp manager Serp manager is the de facto kind of certificate management tool within kubernetes And it's very easy to set up and serp manager has simple support for spiffy It doesn't have support for the full spec, but it allows you to get spiffy certificates And together with another open source tool called authorize You can use that to fully set up the authentication authorization to a double s across clouds This is the link to the webinar page And in that webinar page, we're talking you through like how serp manager works together with authorize It also has a demo that you can fully replicate And what is this going to do is uh, it's from google cloud for example from a cheeky ecluster You can talk to a double s as three without needing to do anything like it automatically gets like an a double s Authentication it all sets this up and all from within kubernetes So after the initial setup together with authorize like all of your workloads are getting a spiffy identity Thanks to serp manager and then authorizes doing the authorization bits to a double s like you add some annotation to your workload You declare an intent in your kubernetes cluster and that intent in your kubernetes cluster will automatically set up For example the ados im policies for you On the ados site and this is all can then natively happen within kubernetes And this works in a pure multicloud one unfortunately This serp managers spiffy implementation currently only works with ados as ados is at this moment I think the only cloud provider that allows authentication with x519 certificates And serp manager currently only does x519 certificates If you want to do make it work with other cloud providers You need to more look at job tokens and then you need to look at spire at this moment I want to thank everybody for coming here and i'm open to questions as well So if you have any questions i'll pass the mic over to you So we can record it When you had the two different servers serving trust domains and they would swap the trust bundles How can they authenticate the other servers trust bundle if they don't have a shared cryptographic source? So the trust like if I go back to the To this thing here So what the retrieving of trust bundles does is it gets the public key of the ca And then the public key of the ca then gets distributed to for example the spiffy server one Uh spiffy server two retrieves a trust bundle of spiffy server one, and then it's uh Populates the public key of the spiffy server one in here And because you then have the public key you can verify do the verification of it That's true. So Yeah, so so yeah So the question is can this uh trust bundle distribution endpoint be man in the middle If you don't use an htp htps connection Yes, of course like you might have like for example other means already of like public key distribution in your Images for example where you have like a ca thing then you can of course put it in front of htps And it's it's a lot harder to uh man in the middle that So it can be an htps endpoint your trust bundles and as long as you then again have the trust some in through some other way Uh, yeah, this can be secured and you can get rid of man in the middle Or prevent it So you mentioned this makes the developer life easier What does this look like so I know where to get my passwords. I know how to generate one of those. Yep. How do I generate a spiffy? our spiffy id so That that's indeed where like spire for example comes in which is the the production ready implementation of the spiffy framework Or like start manager with the start manager sees is spiffy store spiffy driver that Generate those spiffy ids for you Because like a big part of the spiffy identities is also doing the verification and attestation of that you would be able to Mint your own x519 certificates kind of in a manual way And you would be able to do the own verification attestation and then just mint with the spiffy identity in the right place on that certificate Of course That would be the very rudimentary way But most of the cases like you're going to go to an automated way and look at like tooling like spire and start manager As well as like maybe the cloud providers that are coming that are going to give you that spiffy identity that you then can use for authentication authorization between different workloads but like it's indeed it's it's a great question and like I've been thinking quite a lot around like from when does it start paying off having spiffy everywhere and Like because for a really long time you're going to have like both of the solutions And one of the nice things for example is hashiq or vault As an example, as I know, it's a very popular secrets manager supports authentication with spiffy So like a really great way For example to start is especially when I look at VMs VMs don't have like basic identity like Hashiq or vault with kubernetes clusters can already be used through the the short endpoint, but like VMs don't have this But once you have like spiffy for example on your VMs through like system like spire You can already authenticate to a hashiq or vault and then slowly start chipping away Kind of thing and getting rid of secrets and doing more spiffy native But like there is for quite some time going to be A time that you're going to run dual kind of thing Can I use can I use spiffy in a model model app? to Have the mobile app identify and authenticate with the server Sorry, what was the question can I use it in a mobile app can configure in a mobile app on on a phone to Have it authenticate itself and exchange Trust information with the back end with the cloud um Nothing stops you because like spiffy itself is a framework so in like we've been Debating about it like can it for example be used for user identities as well and where your user gets a Certificate then In theory it's possible because it's a specification Uh, I haven't seen it done like as an actual implementation kind of thing or like any of the open source have have implemented this But like I don't think nothing stops you from kind of doing The development or like somewhere and then Doing it out. I think it's probably would be a nice and pure So as of about six months ago hashi corp vault doesn't exist really for a lot of us anymore At least not as an open source project. So are there any open source? secrets providers at all that Be done with this fire One of them actually is quite recently and I'm not sure how That's also like what what the future is of that since the acquisition, but Uh, vmware created vmware secrets manager, which is powered by spiffy for authentication in the background And uh, it will be an alternative I really haven't really tested it But I've seen a bit of like buzz around it and and I really need to test it out As like the people that grace it that are very active in the spiffy community as well And and it sounds like a quite neat solution indeed like where you need to run, uh, jewel I also know there are some efforts on Having like an like an open tofu for hashi corp vault, but like Yeah, I I haven't really looked into that story. So But I mean so that vmware thing you said that that that's A replacement for spire or it's actually just a cb provider that spire can work with Uh, so so it's a secrets manager That is natively using authentication off off along the spiffy side. So it works together with spire natively for example Is is it yet another piece in the tensu product portfolio? Yeah, but like the the vmware secrets manager itself is fully open source and and you can run it without being as part of tensu Yeah, I forgot where you mentioned. There was a uh a database, uh that the users. Where is that in the agent? It's in the spiffy server. So the spiffy agents itself are fully stateless the spiffy server Or like the spire server because spire is a production gradient implementation Is has a database uh, of course like they're like The spiffy itself is a framework is a spec um, I know like again Knowing a bit about what's what's being built out there Not all of them are going to have a database on on the on the on their spiffy server, but spire definitely has a database Thank you Matias. Thank you. Thank you for the questions. They were really good Okay Okay, okay, this is all I guess we'll get started a few more people may trickle in so We welcome Carlos Sanchez To discuss lessons learned around migrating existing app to a multi-tenant environment Carlos is a principal art scientist at adobe and He has been very much involved in open source for over 15 years including His focus on Jenkins kubernetes plugins And he is also a member of the apache software foundation. So welcome Carlos Thank you. Thank you for having me So i'm gonna talk to you about what we did Real-life lessons what we did. Well, what did it wrong? moving migrating to multi-tenant cloud native And hopefully you'll get some ideas some things that will be useful for for your for your projects So let's well first. Thank you for being here and instead of I know there's a leakers game at the same time So thank you for coming here And who's here who knows about kubernetes? Okay, who's who's using kubernetes in production? Okay, so almost everybody Okay, so let's talk about well, this is all down And because we are in a Small audience let's you just can interrupt me whatever you want ask me any questions you have I'm gonna talk about a little introduction about what this other experience manner it is So you understand the challenges here So it's an existing distributed java osgi application. So it was Before we move it to the cloud. It was already a distributed application So that had already some benefits people could run this on-prem across multiple VMs multiple Machines and so on it's it would scale horizontally So that was something that made it our life easier It uses a lot of open source components from the apache foundation And it has a huge market for extension developers people that write code to run on the em platform So this is gonna be interesting later on So what we did was let's take a m and run it on kubernetes because kubernetes kubernetes kubernetes, right And we are currently running on a sure we have 45 more than 45 clusters and we keep growing over time And because we are a content management system. We run across multiple regions Whatever the customer or customers want to have the content closer as close as possible to their customers So we have u.s. Europe, Australia, Singapore, Japan, india and whatever new region that comes up will probably take it And another interesting fact that the job is that we have a dedicated team building The clusters for us. So what is called today a platform team? because platform is the new trendy word and That also limits what type of things we can do, right the cluster. We don't own the clusters We have another team that provides a cluster. I think this is gonna be if it's not already It's gonna be very typical in any big company where there's gonna be a team that is gonna Give you access to the clusters or even a cloud provider where you say, okay, I'll take the cluster but I cannot Do everything I would be able to do in a local cluster that I run For the for good reasons and bad reasons you do not you're not gonna be able to do everything you want you you could do we have 17 000 environments and What an environment is is a Set of deployments that we give a customer So a customer can have multiple environments And this comprises multiple Kubernetes deployments and services and Kubernetes objects and so on And that means that we have more than a hundred thousand deployment objects in kubernetes clusters That's also That means also we have over 6 000 namespaces So that's more or less the scale we as am Just the single product of the W the scale that we have An environment for a.m. Is Something that the customer self serves So they come to a ui. Well an api. They say, oh, I want a new environment a new environment Can be a dev stage production environment for them And so they can have Multiple dev environments one stage what's one production that means at least they have three environments And each environment has their own It's a bit like a helm chart with their own deployment services and so on and These environments are also separated by kubernetes namespace for isolation So each customer means at least they're gonna use three kubernetes namespaces And each environment is what I like to call a micro monolith So we took the job application that customers were could run On prem or on a vm or we could run on the cloud for them already on a vm And now we take this and we run it in kubernetes containers in kubernetes spots We use namespaces to provide the scopes For the multi tenancy part namespaces in kubernetes give you network isolation quarters permissions So you can say, okay, I don't want a namespace to talk to another namespace I don't want a namespace to grow over this much amount in case You misconfigure something and and the scale is uncontrolled And uh, yeah, I don't want a namespace to see things in another namespace So that's all what you get for free in kubernetes when you use namespaces We have multiple services multiple teams building services and different teams have different requirements So we kind of leave Let people do whatever they want in a more of a you build it you run it mentality So we help them Because i'm sitting more on the on top of the coroneris part on the on the layer on the more a bit infrastructure Not so much application side we tell people building the the services on top Okay, you can you can do a bit of whatever you want just have these things into account You want to use golan? You want to use node? You want to use java? That's fine and The model we are following is api patterns. So services have apis and operator patterns where we build operators That will do actions on the clusters on the environments and everything So the operator pattern kubernetes if you are not familiar with it is Of a managing state You create a custom resource definition kubernetes Then you have an operator that is a service that is continuously running and monitoring these custom resources and saying What is the desired state and are we in that desired state or I need to make changes So the operator keeps this reconciled reconciliation loop forever Checking when there's changes to one of these custom resources What things need to be done? So this is very useful when you have For instance the helm operator the helm operator you create an object that defines I want to install this chart with these values in the cluster and in this namespace The operate helm operator goes and always looks okay. Is this installed? No. I need to install it Has this changed? Yes. Oh, I need to update it So that's the operator pattern and we use it for a for a bunch of services On the environment side We use init containers and many sidecars to do division of concerns So on kubernetes you have the concept of init containers Things containers that run before your main containers run And sidecars containers that run alongside your main container And this has been there's a new feature on kubernetes where you can have More init containers or sidecars that start as init containers and become sidecars That's in the latest versions And that is useful for things like logging So you when you want logging you want the logging sidecar to start as early as possible And start shipping logs somewhere And you want it to continue to run all the time Before this I think this was in the last version Before that you will have init containers and Main containers or The main container of the sidecars and they were separate Now you can have one that spans from the very beginning to the very end of the life of the pod On the sidecar part Yeah, the division of concerns Model that we follow is instead of adding more things to the main container and to the java application that is already a big Micro monolith or whatever you want to call it We create sidecars in its containers that that do specialized things So we don't we can separate them and it's better different teams manage them. They follow their own release cycle and so on So we have service warm-up storage initialization We have an htdpd server front in the java application for caching and for other configurations Sidecar containers that export the metrics to primitives fluent bit for logging Um to in java we can collect threat dumps and ship them also and store them For uh, net for more advanced networking. We use envoy. Envoy is our proxy and I'll talk about I think I'll Following slide and and another is the auto-updater. So the service warm-up for instance is uh, it's a service That when the pod comes up It starts hitting the most requested URLs So they are warming the cache Before the pod is receiving traffic So we manage the readiness pro in Kubernetes So the when the pod come up comes up before saying it's ready to accept traffic. We warm it up We warm the cache And then it it can mark itself as ready and start getting traffic And it does this lazy caching without having to do very expensive starts Fluent bit is a very typical Solution to do use you run it as a sidecar You have a shared volume Where your main application or all your? Containers are writing logs to fluent bit resource logs from the file system and ships them wherever you want I mean you could do this from the main application main container and whatever But this makes it easier to change it without having to deal with the application So it's the separation of concerns And we can configure it independently. We can say oh, we need to upgrade Fluent bit we don't have to Make a release of the main application. We can just do behind the scenes and update the fluent bit without Changes to customers Envoy proxy every who knows envoy Okay, just a few people envoy is a A very Well used proxy Engine on Kubernetes. It's used for by a lot of service measures behind the scenes So if you use Istio if you use Service and pretty much all the service measures use envoy as a proxy inside And we use it Because we have customers that say Oh, I need to connect to my internal vp and to do to get some data or I need to To go out to the internet Using a dedicated ip because I don't want to get affected by other tenants in the clusters And I want to use a dedicated ip just for myself because Maybe You have one tenant that is doing a lot of requests to a service and you get in throttle So we have this ability Having dedicated ip is vp and connectivity what we do is in the pod we run an envoy sidecar We send the traffic from the job application to that proxy that envoy proxy that envoy proxy does An empty ls tunnel over hcdp To a vm that goes out to the internet with a dedicated pipe So that's how we we implemented it now you have more options And there's tools by cloud providers that will make this easier to say Oh, I want this specific pods to go all through this specific network route And that network rock and have vp and can have other things So there's the the cloud providers are giving you more Out of the box functionality The out of data is is a sidecar or any container that we created Anybody heard about the log for j cv? Well If you were not under a rock you probably heard about it So now suddenly we are running these thousands of deployments and we are have to figure out How do we upgrade log for j in all of them? And because the environments are in control from by the users so the user can say Oh, I want uh, I want to upgrade now or I don't want to upgrade or they're using a specific version of am We have to go and say Uh, okay, whatever version of am you're using log for j has to be upgraded So what we did was add in this unit container that on a startup Does changes on the file system before your main the main container starts So different people have done it in different ways But we chose this because it was a bit transparent, but not too magical So every time your pod starts this unit container Can do whatever you want right and go into the file system and set a file change something And that way we can control this without changing the the the main application the java application At all and this allows us to patch the whole cluster for life. So if tomorrow we have another issue We can go and say, okay, just changing the the container for the sub top data. We can do whatever we want Any questions so far? Yes This probably isn't worthy of being recorded but just so In theory for log 4j. Are you running through and like going through every palm file in am and Dating something. Okay. No we we patch we don't rebuild We patch it in the in the main application container file system. Gotcha, right Just wanted clarification because we I mean we we run the same application is just different versions and different things So we know where to go and it's like oh in the file system log for j Is here in this path and the version is all one affected by the cb then just Copy the other file over on the operator side As I mentioned we use a bunch of operators. So we started with or we created one That I think on the operator side is what makes business sense or what makes functionality sense for you We created an am environment operator that goes and You create a custom resource that defines an environment And that operator goes and looks at that and because that makes business sense When you create an environment, you just create an object one object You don't need to create 20 different objects You just go and create one object and then the operator will look at that object and the The parameters and then it will do other things But from your business point of view is oh, I'm creating one environment and The the unit is the environment. So I create one object And this manages this life cycle of environment. So when I create this object this custom resource I'm going to end with an environment running if I delete it. I'm going to end with the environment delete it and it makes this Semantic sense And this operator instead of doing everything inside the operator. We also delegate to other operators So for instance, we have to launch jobs before an environment is created This operator launches the jobs and then it uses all these other internal operators to reconcile And reconcile the status of the of the environment An example open source the flexibility helm operator. I mean flexibility is a github's operator and Yeah, it allows you to to deploy things to the cluster similar to is pretty much like cargo cd But also they have inside the the umbrella of flux They have the helm operator. So you can use this operator separately. You don't need to adopt the whole github's approach for each For each of just to take advantage of the helm operator So the helm operator Allows you to manage helm charts using declarative state. You create an object. There is a helm release The operator is going to watch the helm release object Oh, if this is created is running a helm install if it's changed i'm running a helm update And so on And it always keeps them synchronized So we when we create a hell A m environment cr the operator behind the scenes create a helm release cr another ones And this change the operators So this way we don't have to implement everything inside one operator But we take advantage of other open source operators and operators that already exist and divide the functionality across multiple operators And it also when the helm operator reconciles the helm release It writes the state into the helm release and we can take that state And put it back into the environment in a way that makes sense so It's a change of of things that happen some of them happen in parallel some of them happen one after another Helm install hem updates and a few other things launching jobs and so on and then the main operator says You can go and see the main operator The status of the main environment resource and know what happened to all these Like subcalls and change of workflows, let's say argocd we also use argocd Again is the whole you can use the whole github's Services that they provide And it applies github's state into the cluster. It's very widely used at adobe and we contribute back to it And if you have different things inside workflows events, so you can also use part use parts of it We use it for some namespaces So because our platform team has established argocd as a standard so when we create new services, we can just go to git and say Okay, just deploy this with argocd and behind Well, we don't even have to say that we just onboard that project that git repository and that gets deployed with argocd and argocd Or argot the argot umbrella project includes argorollouts, which you don't need to use argocd at all But argorollouts, I think it's very cool because it allows you to do Advanced deployment techniques progressive delivery So you can do canaries, blue, green, savey testing, whatever There's a bunch of different things that you can set up And the very cool thing that it does is allow you to do automatic rollbacks So you deploy you configure your argorollouts Object you set some metrics. Well, that's what we do. We set some metrics and say A successful rollout means that Less than 10% of their requests get an error If there's more than 10% of the request that get an error Argo will automatically roll back to the previous version And you don't have to do anything. There's no manual checks. No Not nothing Obviously that requires you to have nice wood metrics and to have some sort of confidence on On doing this, but it's very useful If you don't if you use a service mesh, it gives you a lot more Power where you can say I want 1% of the traffic to go to the new version If you don't use a service mesh, you can still use it. We don't use a service mesh But it's just you are limited. You can only play with the number of pods. So if you have 10 pods You can have one more and have 10% of the traffic go into that pod But you cannot do 1% or 5% right is You are a bit limited, but you can still do nice things, especially the automatic rollback is very nice on the Moving on to the how to scale and how to automate resources when you have big deployments And when you are coming from a bit of a monolith So I mentioned that each environment each of these 17 000 plus It's a micro monolith. We have multiple team teams building services So we need ways to scale that are more orthogonal to to the developer teams So we don't have to go to each developer team and say hey change this change that change this other thing On the Kubernetes wall, uh, there's two important resource concepts Requests and limits Request is how many resources you have warranted? Limits is how many resources you can consume And depending on what they're applied to The result is different the the the action is different You can apply them to cpu to memory to ephemeral storage And when you apply them to memory The limit is enforced So if you go over the limit So you have warranted the request that you asked for but if you go over the limit, uh, your, um Your process and the container is going to get killed On the ephemeral storage part The limit is also enforced. So if you use more storage Your pod is evicted Pod eviction means that pod is removed from that node and Kubernetes will schedule it in another node So you probably are going to lose all the data that you have in the ephemeral storage Well, probably, you know You are going to lose all the data you have in this ephemeral storage and your pod is going to start fresh somewhere else A very interesting one is the cpu the cpu resources And on the cpu request side The requests are used for scheduling So you are saying oh, I'm requesting one cpu for this pod Or this container So Kubernetes is going to find a node that has one cpu available and it's going to put that there And after but after that is still used as a relative weight So it's not the number of cpu's that can be used Is a cpu number of cpu cycles that the process can use So if you have two Two containers running in a node And they just request 0.1 cpu They can all Both use At the same time 50 percent of the node cpu time So this is a bit tricky because it it got it got us We were figuring out what the hell is happening And it tricks a lot of people On the limit sides this translates to cgroups quota and period Some people said containers are just processes containers do not contain So they are just processes that have cgroups Kernel cgroups enable So cpu limits are cgroups quota become cgroups quota and period The period is by default in the kernel 100 milliseconds So the limit is the number of cpu cycles that can be used in that 100 milliseconds If your container is going over those Over that limit your container is going to get throttle So imagine you have one Container that has one thread only And it's using one core You can and you request one cpu if you have a limit of a cpu You're going to be fine. You're going to use one thread as much as you want In one cpu that's that's going to be fine It's going to run for the 100 milliseconds and then it's going to get other 100 milliseconds another 100 milliseconds and so on This is a bit challenging for java applications and multiple thread applications For instance, if you have if you request one cpu a hundred a thousand milli cpu's In Kubernetes and you have four threads Now suddenly If each thread uses all the cpu In 25 milliseconds, you are done. You don't have more cpu time So hopefully this makes it a bit Clear you have four threads each threading one core different core The period is 100 milliseconds, but after 25 milliseconds you already consume 100 milliseconds of time across the four threads So you're going to get throttle 75 milliseconds And if you are doing the same thing, you're going to have over and over and over again Every 100 milliseconds You're going to only use 25 So this is something very important if you are like serving web requests You're getting a request and you look oh Typically it was well, but now suddenly the response time is going like through the roof what's happening You look at the cpu throttle in metrics that Kubernetes provides And you realize that your container is being throttle So that's something that also tricks tricks us and tricks a lot of people Yes I can repeat the question Okay, so that's the that the million dollar question is what I suggest using cpu limits in general And the answer I don't know if I have it later, but the answer is In production you should not use cpu limits Because if you use cpu limits, you are artificially limiting The amount of cpu that you use so you are leaving cpu unused for no good reason So typically Uh because the request is a relative weight you have warranty that two processes Are not going to starve each other One process is not going to starve the other one. So there is a relative weight between processes If you set up a limit imagine you have one two processes one is doing nothing The other one wants to use a lot of cpu If you have a limit To one cpu that process is going to use only one cpu But you have maybe a 16 cpu 32 cpu node you're wasting nodes you're wasting a course So there's it doesn't make any sense Even if you have the two so you just remove the limits One process is doing nothing the other one wants to use a lot of cpu it can go to 32 cores The two processes want to use a lot of cpu's if they have the same request they can use 16 cores 16 cores So you don't have the problem of masturbation between each other For us we One consideration you have to do like We haven't made this change we have it planned But we want to remove the limits in in the in the production environments But also imagine we have stage environments for clients and production environments But for us it's all production but for them is oh, I have my Stage website and I have my production website and they may run Performance test on the stage If you remove the limits and that Pod happens to be or those set of pods have happened to be in empty nodes that don't have anything else The performance can be great But then the production pods are happening to have some noisy neighbors and running on busy nodes Now the performance is not going to be as good as in the stage So the thinking here is like maybe in the stage we want to have limits So people have an expectation Of what are they going to get the best? The minimum they're going to get in production, right? Because otherwise like oh my stage test was great, but no production is not So this is going to be very confusing. Yes, I guess I've got a follow-up on that one So what do you enforce some sort of a resource code or Do you enforce a strategy more upstream? Let's say whether using scaling groups or you're using carpenter type of implementation where you've got specific node groups for specific workloads Or do you get more close to the namespace level resource code or limitations? No, we don't we don't do separate node pools for separate workloads With one caveat that I'm going to talk about in a bit That's a there's definitely a possibility that you may want to do if that makes sense It just complicates things a little bit But there's tools like now carpenter carpenter the auto scaler Uh, I think it's supported now. Well, it's supported by aws. It's open source, but it's also adopted by assure And and you can run your own Carpenter We'll look at different considerations and do autos smart auto scaling where It will look at uh, oh, okay. So you are using You you have pods that require 64 cpus Okay, I'm going to start a node that has at least 64 cpus Oh, but you are now you have a mix of pods that have a memory cpu ratio one to four and now another ones that have one to 32 And I'm going to okay. I'm going to start different types of nodes so you can Optimize the cost and optimize how the workloads are scheduling Kubernetes So this is carpenter what it does and also the price you can It can also start Well, it will try to start the cheapest Sizes that you need So it doesn't make any sense if you if your queue of Kubernetes pods waiting to be scheduled Is uh, oh just Five cpus is not going to start a node that needs has 120 cpus. So it's going to do some smart Things without because before that you have to Manually go and say okay I want a node pool that has this size with this type of VMs with this memory cpu ratio Or if you have many of them you have to have define all of those Yeah, but you can still introduce discipline when you're defining your carpenter definition, right? So you can say Do not do this In carpenter you can configure it to do different types of things Uh, and what I was talking about we use a different node pool with arm cpus So we estimate that we get like 15 to 25 percent savings for the same performance And it's very easy to switch if especially if you're running java You just switch the base container To an up to another jdk That is built on arm Done nothing else to do.  That's another benefit. So that's that's a wing wing Anybody here doing java? Yes one two Okay I'm gonna skip a bit through the quiz because it's very java oriented but in java The the way the java jvm And this is similar for other for other Languages you have to be aware of how it decides How much cpu is going to use how much memory is going to use like in java The default heap size When you run java in a container It depends So depending on what's the size of the container the jvm is going to take more or less memory So it's very hard to figure that out. I mean to to to get it right And if you use the defaults you are wasting money because the default if you have a Normal container the default is going to just use 25 percent of the container memory for the heap size Uh So that's a waste of of money um This was improved late in the in previous versions of java But let's skip that but you can the thing is you can configure the the ramp that you want for the heap And you should do it because otherwise you're running just with 25 percent of of the memory of the container and you're wasted A lot because for us. I think we are running around 80 percent So that's that's a lot of money and that you are wasting Yeah, typically you can use 75 percent in java Unless you have things that use off heap memory like elastic search And spark But it's it's kind of similar for for other for their languages On java, we also have the garbage collector and depending on How big the container is the jvm is going to pick one or another So it's also tricky that um, and if you're running some tests in a In a smaller container This those tests may be wrong when you run in a slightly bigger container because java decides that oh Because I have a little bit more cp. I'm gonna switch the implementation of the garbage collector So that's also tricky And you can also configure it on java with with uh with flags And there's there's a garbage collector's table that uh microsoft released, but it depends also on your use case um The cpu is that java we the java the jvm will see also is intriguing because um Depending on which version of java you are running and uh It's going to use a different number and depending on how many you assign to the container And they changed this lately to to do the as many as there was allows because before It was it was calculating it with cgroups and it was not quite right What it meant before if you say to between zero and 1,023 Millie cpus you get assigned one cpu if you set to 10,024 you get nothing no limits If you say and then it was more or less normal But now uh, yeah, there was uh, there was an improvement to the jdk And Yeah, basically you can set the active processor count if you want To tell the jvm hint the jvm on how many cpus it should see And that's um And and this is about the the request on the limits that I was talking about before So let's say you have a 32 cpu and this applies not just for java for any process that you run You have a 32 cpu host two jvms or two processes with the same request What is the maximum cpu that you can use if you set the limit to eight cpus The maximum they are going to use is eight each so you're wasting 16 If you set it to 16, well They can use maximum 16. So if both are busy, you are okay. If one of them is not busy. You are wasting cpu But if you don't say set any limit so that the question that that came before Uh You can use uh Any any process any of the two processes can use up to the 32 So that's It's never going to start one another But you can use the whole cpu of the of the node Which at the end of the day is money The other important bit is how do we scale kubernetes, right? How do you scale the pods? How do we scale these things so we don't have to manually go and change things? Or we don't have to have huge humongous clusters all the time On kubernetes, there's three types of autoscalers You have the cluster autoscaler the horizontal pod autoscaler and the vertical pod autoscaler The cluster autoscaler is Increasing the number of nodes of the cluster house based on cpu and memory requests So that's an important bit there and We said the well, it's important to set the maximum nodes that you want at the cluster level because uh Bucks can happen and this of course I have a lot of money because you don't you don't want to run at full capacity all the time So a typical scenario is You get we get more requests more scale up and then kubernetes scales the number of nodes And then when there's not traffic, there's less cpu usage, whatever the number of nodes goes down. That's the typical seesaw pattern but another interesting one that happened to us here is Suddenly this went up to 100 something 140 something or 150 And this didn't grow up more because we had a limit on the number of nodes. Thank god And this what happened is we had there was a bug here on the autoscaling process If not, this would have like keep going up So the bug was fixed and then things went back to normal So that's why it's important to have a max node or a very big credit card to pay for your cloud spend The vertical pod autoscaler will increase or decrease the resources for each pod. So Um Suddenly you say, okay this pod would Because it's getting a lot of requests it would benefit from having more memory. You can define that But Until the very last version of kubernetes this request that will start of pods and you can set it to automatic on neck or on next start and this is now a alpha or beta feature alpha Of the of the last one So that's tricky. Um, we use it to to only We used it not anymore To developer environments to scale them down if unused and the If you set it to automatic well, you take the risk that your pods are suddenly to improve And then you may get a notice. So that's be very careful of of what you're doing there And the horizontal pod autoscaler is having more pods whenever you add more traffic more cpu More whatever metric you want to you want to measure it for and we scale on cpu and hdp request per minute and It's uh, it's obvious when you're reading now, but not always obvious for everybody You cannot use the same metrics as for the vpa You can have both configured at the same time But not on the same metrics because then you're going to have a mix of scales at the same time and it's not going to be great It's going to be confusing cpu is a bit tricky. It was a tricky for us because You can have this periodic tasks or a startup cpu spikes for java, I think it's more very common The pod is starting now. There's a lot of things to be done at the very startup And then the cpu goes crazy. And if you don't configure it right Is the hpa is going to say oh your cpu went down Very high. I need to start another pod the other pod the starts goes very high on cpu. Oh, I need to start another pod so this happened and so you need to be careful on this because You can configure a lot of things like what is the ramp up time the startup time all these things Because you are just doing The nail of service on yourself and this is what We have a bunch of these problems where we We are running at a scale that we can easily Do a distributed the nail of of service internally There's some horror stories there But yeah, I mean you you change something but it's running across thousands of places and now suddenly One it doesn't one is not important, but a thousand times It's it screws up things So, yeah, that's the space on the startup can can do the the The cascading effect So to sum it up Three things that I If if you want to remember three things is that on kubernetes is very easy to start and then optimize So it's very easy to do lift and shift to kubernetes existing application. Just put it in a container run it There's some things to consider like database, especially state is is the tricky part You can use patterns to decompose the application sidecars in it containers new services operators So you don't have to add things to the monolith And on the resource optimization part because money seems to be important for people if you heard kori Last night money seems to be moving the world And things are expensive in the cloud You can all tune this jvm cpu. You can tune the memory on This on java the garbage collection and so on and you can use all the auto scaling capabilities that kubernetes provides So if you have any questions about a couple questions one You talked a lot about jvm resources and management of those How much discipline did you guys did you guys have? Because of the extent of the number of clusters you have in the upstream base container images For these java applications. That's one question second how much sorry the base container image like that the size of it like how Oh, the the size of the images of the container of themselves, right, right? So that's one one question Like did you have a strategy for using certain types of container based images second question is You didn't you didn't talk about kubernetes jobs in your clusters. Did you have those scenarios as well in your ecosystem? And how did that Into the role So the image base images we don't I mean we use some JDK images that are built internally at Adobe for the main application for others services whoever builds the service picks up an image We don't care too much For now about the size of the images, but there's some approaches that we are looking at at improving the The download time and the startup time Obviously is caching the images at the cluster level at the regional level and so on There's some features that I know for sure as sure has About streaming images. So if you have a very big image that you need it to be so big you can Configure it so it will stream the image and get the layers The main layers that you need to start up early start up early while the rest has been downloaded But so far, yeah, we're looking at some of the messages, but we didn't give it too much thought or too much importance Because there's was there are other things that are more in the critical path and for the question about Jobs, yes, we run some jobs But there's nothing really the only one of the tricky things is When you use Prometheus and you use jobs because the jobs are ephemeral You need to use the Prometheus push gateway to send the metrics instead of getting the metrics So making sure and also the logs making sure that your container Terminate idle termination period is a bit longer. So you have time to ship the logs seems like that Thank you Yes, I'm just curious for the amount of Infrastructure and stuff for caring for your Kubernetes clusters Can you give me like Approximately how many human resources you need to basically maintain and operate this at this scale How many people? Yes I have no idea Ah fastest of people no no, I mean so we have We have teams a team that builds the clusters for us and Maintains more of the core of the cluster. So those does the upgrades So they are doing this for the whole adobe or for a lot of different products And so there's this I guess nowadays it's a platform team, right? They provide Kubernetes as a service internally And so this is a bunch of people also building new features I mean you have to to consider it's not just maintaining what it is There is building the new features having new things coming in new services and so on and then as On top of that we run the services I mean the whole product is probably The the total thing is is hundreds of people but It's it's not just maintaining is building new features Building new stuff providing new business value Yeah, if if you run on if you run on top of a cloud provider Kubernetes service and we are only talking about purely the Kubernetes service You don't need a lot of people But purely the Kubernetes service So okay making sure the cluster is up the cluster is not rushing Upgrading the clusters Yes Yes Yes on the scale. I mean if if if we we have 45 clusters ourselves And keep growing them If we run on premise there's there's a lot more things that we have to consider We already have to ask cloud providers saying Hey, we want to run in this region. Are you going to have capacity for us? Are you going to have arm nodes? Are you going to have these? Are you going to have that? The reserve. Yeah, we have reserve capacity also but Cloud cloud Yes and cloud providers There are some is Infinite capacity. There's something that is not infinite about it So what they sell about infinite capacity is not that infinite I know a case of a cloud provider that is Migrating people off of a region. They're saying like this region no longer can onboard anything And you have to move off And we're giving you time to move off because this region is gone. It's done or Oh, there's a launch of a new region and You need I cannot give you 100 VMs Because probably they have 100 for from somebody 100 for somebody else or whatever and they have to measure that capacity so that's At some scale, there's there's that's something you also have to consider Yes So it sounds like you chose namespaces to do the multi-tenancy. I'm just wondering if any other strategies were considered Yeah, so the namespaces Isolation is the few things I mentioned is not total isolation We also look at cata containers for So cata containers is a open source project where you can run containers both as a VM So each pod is effectively a VM with micro vm with the hardware support that vm's have So that makes it harder if you want somebody to Explode getting out of the container and so on um We have issues for instance if we if you overcome it the nodes if you say Okay, I'm gonna put I have 32 gigs of memory I'm gonna put 32 containers with one memory one gig of memory request each But the limit I'm gonna put it in 10 Now any of if any of those pods go Over the limits or if all of them go a bit over the limit that node is goes into a kernel out of memory And now things get a bit weird for a bit until that those pods get killed and reschedule somewhere else and and the kernel is doing things So you can still have noisy neighbors problems Same thing with the cpu If you don't account for the requests And the limits correctly cpu memory even disk space if you have Another thing that will happen to us is There was an issue With the cleanup of all the images Because we were getting a lot of different images and big and so on And the disk space the nodes were running out of this space And then suddenly is They have to enter in this loop of I have to delete something I don't load something But then that something gets evicted because there's no room for the image and now it has to be rescheduled somewhere else So it was this continuous loop of Downloading the image filling up disks Kicking out things for other two other places that also get filled And because you are kicking out things and the loading Scheduling a new pod now you are filling it more. So we have for instance that issue. So we you still have The multi-tenancy corner is not Just pure multi-tenancy, right? You were talking also about quite a lot about like optimization of the cluster and everything around that and I was wondering Have you looked into the shadowing of workloads kind of thing to improve the bin packing of of the nodes Of of the nodes or like Yeah, but like currently when communities decides where to run a pot it's that pot is is tied to the node and due to the lifetime like say for example Deployments have happened like your clusters can be less optimized and I know there was some project out there that that de-shadowed like Did some optimizations and did some de-shaddling into the things but I've never seen that go anywhere and like at my previous employer for example We wrote our own controller to try to de-shadow workloads to run on spot VMs. For example Have are these kind of things that that you're looking into with adobe as well? Yeah, we use the de-scheduler So this the de-scheduler is another controller operator That will look at how your things in the cluster are and will kick out pots of the nodes. For example You are saying I think If you say, uh, I want to scale down the cluster Maybe you cannot scale down because the nodes are a bit busy So the de-scheduler will go and say, okay these pots that are in Nodes that are not busy out So that node can be deleted another one interesting one is when you have multi availability song Uh spread So you say I want to run three pots of this service And I want one of them in each availability. So So coroneris will go and schedule them Um, well, let's let's say six spots is what I mean the sample better So coroneris is going to go and put two in each availability song Okay, now suddenly the hpa or the Node needs to go away because your cluster is scaling down or whatever And two pots in this availability song in one of the ability songs go away because whatever reason this can happen in Coroneris pure coroneris is not going to care about that So you're saying oh, I need to scale down I'm going to take this node And this node and this these two pots are in that node. They're gone or because anything else So you end with two pots in one availability song And two in another one and nothing in the third one The de-scheduler can do you can configure the de-scheduler to say Correct the spread across availability zones So it will kick out pots from the other ones To make sure that they are spread out correctly Because by default coroneris is not going to do it. So if you have hpa and availability song As hpa can go Send up and down the number of pots hpa also doesn't care about spread After the pots are scheduled So there's nothing really that cares about spread after the pots are scheduled You have to rely on the de-scheduler To kick out things If they're not correct No, it cleans up the pots is just that you end with a spread. That is not what you want So it's if for whatever reason Yeah You you have six pots running across three availability songs the hpa says oh, you don't need six You only need four it may decide to kill two in the same availability song There's no real measure of which one to kill So you may end with two in in four in two availability songs or all in one Things like this happen And you needed the scheduler that to go there and say Oh, the spread is wrong and want to kick things out So cuberneris It starts in your pots and when when it schedules the pots is when it looks At what in which availability song should I put it? Yeah, I will put it back in And correct it.  You don't care if it's across availability songs or not at the ploy the A deployment or a stateful set can be across availability songs you have to configure the pot topology is called And configure it in the way that you want So you can say I want a minimum of one pot in each availability song or I want No more than two difference The difference you you you configure this cue So if you have one pot in one availability song Do not have more than three in the other ones. It's a bit complicated, but you configure it with pot topologies And that's when cuberneris schedules Looks at the pot topologies and decides in which nodes to do it And this is all with labels on the nodes So it's not I mean, I don't think he's aware of the concept of an availability song You just set no labels in nodes For instance for arm We are building multi multi-architecture images that can run on both arm and uh Intel And we say in the pot topology we say Schedule this in arm if available and if not schedule it in Intel Because otherwise you It may not be able to be scheduled essentially that requires your your nodes to have Selectors on them like all the nodes have to have the labels the selectors everything configure correctly So it seems like you ran into a few issues when you're migrating to cuberneris One obscure one that I heard about when I was in another talk Was that in clusters where there are I think hundreds to thousands of nodes it can Applying it applying a deployment can be inconsistent Due to at cd not being scalable enough. I'm wondering if you ever ran into anything like that So we also have that issue not because of the number of nodes, but because of the number of objects At cd has a limit of eight gigabytes When you get close to that limit, uh, it becomes read only And you cannot do anything and your cluster is pretty much gone And you have to recover it from a backup Yes, and then What happens is One is the that that would be like the worst-case scenario, right? You You have too many objects You fill it up It's another thing that may happen is You have so many objects that The cuberneris api takes a long time to respond You can have the watchers in cuberneris also consume memory So every time you mount a secret or you mount a config map by default that that's a watcher And those watchers consume memory on the api if you have a few it's okay if you have thousands That's a problem And that limits how much you can scale the cluster because now suddenly the api every time you make a request It can take longer and longer and if it gets over one minute The api cancels the request So once your api calls get close to the minute You are in big trouble because nothing will work Like the auto scaler is not gonna work the scheduling is not gonna work the The leading pulse is not gonna work things like that the traffic The ingress controller is watching the api to know where to send the traffic your external traffic is not gonna work So you gotta be careful about that too You have to keep You have to watch for the number of watchers the number of memory that the api is using the number of objects that you are using in the cluster so First advice, don't you set cd as a database Don't use cuberneris api as a database don't use the cuberneris objects as a database So that's that's the one because it's very convenient. Oh, i'm just gonna store things here in this secret on this config map Don't do that That's the first one and then if you have a scale where you have thousands of secrets thousands of config maps objects and so on Try to refactor it in a way or limited size of the clusters. That's the only way out We are going we're not growing the clusters way bigger because we don't We have this scaling problems. We try to go now with more smaller clusters Big Using postgres There's there's no the only Supported really supported way to to put a storage backend is for cuberneris is at cd Yeah, well k3 k3 s is not something that you would want to run in production for loads No, there's I mean you If you if your problem is i'm storing too much data in config maps or something Yes, use a database to store that data. Don't use cuberneris api There's no reason why you should be storing gigabytes of data on the api Now there's also some tricks like you can limit the defaults in cuberneris the default for the number of pots That they are kept in the history is 10 You can say out or other replica sets things like that If you have many of them there's you can say don't don't keep 10 just keep three Help help is the same thing history 10 by default don't keep 10 just keep three There's small tricks like that Yeah Okay, all right Cool. Well, thank you carlos. Thank you Now you can you can you can go Enjoy the rest of your day night One hour and eight May not be very smart and it's like just ask away Yeah, listen to you in person how many total that came Um, I will find out. I'll let you know Yeah, yeah, I'll let you know I'll definitely let you know we had I think call for papers We had I can tell you what I because I was part of the committee that reviewed all the calls We had over 400 CFPs That's that's a lot. Um, so Yeah, yeah, yeah, but then some dropped off because they also got into kube on paris So then they started Oh, cool. Good luck You're flying out of lax great I am based out of san diego down south