Hello, hello, can you hear me? Okay. Yeah, I usually don't need a mic, but I think we're recording. So I'm gonna use one How is life?  So wow amazing I think I think it's also isn't the first talk of the day or like maybe the second I think like but not all tracks. So you had time to sleep, which is good Yeah, sometimes that happened in conferences where like it started like 8 a.m Like people are still lying over from like the conference party or they didn't slept enough So no, I realize people Seems to have a little bit of life within you. Maybe I'm gonna kill that life talking about communities We'll see so super happy to be here My name is Frédéric Harper.  I'm a principal Developer advocate at Cube first.  I'm here To talk to you about what the heck is Kubernetes By the way during that talk feel free to Share stuff on Twitter. I don't know if I still call this Twitter. There is no way I'm gonna call this X So feel free to share things on Twitter that you agree. Disagree things that I've said that are good bad Take pictures of the slide of my beautiful face. Whatever. I'm on Twitter if it's half Harper feel free to connect also Other thing if you have any question You don't want to ask publicly Twitter is also a good place to ask question Obviously, I'm not gonna watch my Twitter account during the talk But after that I'm gonna take the time to check if there's any question comments or insult so What the heck is Kubernetes before we start about that topic? I want to tell you when I Decided to write that talk like write the abstract the title I was like that's a brilliant idea like that's the talk that I wanted to have when I started my cloud native journey and When the talk got accepted and I have to create that talk. I was like that was a bad idea That was really a bad idea. Actually. It was not a bad idea It's just because like there is so much thing about cube for a cube first Kubernetes that I was like What should I talked about like what would make sense within only an hour with those folks And the thing is that that talk could be like a full-day workshop And I'm not even sure if we would have seen like everything that is Kubernetes. So I Did a lot of like thinking talking to myself a little bit too much And I decided that like that talk is gonna be a mix between like a high level introduction about communities So like the fundamental of Kubernetes so you won't know let's be honest You won't know everything about communities at the end of that talk, but I also want to do some demos and Hopefully help you get started when it comes to communities or at least get you a little bit more excited or a little bit more like What the hell is that to understand just a little bit more the technology to maybe maybe try it at home Maybe try to move your traditional cloud infrastructure to cloud native or communities If that makes sense for you so Copernic what? What is Kubernetes but first? Let me give you a warning actually nobody's gonna listen to me for the next 30 seconds because you're all reading what is on the screen Except the people doing stuff on your computer that are not really listening to me right now, but the other people it's still burnt basically, what does that say? Like basically at the end Communities won't solve your issues or all your issues. So it's not a magic solution if your application is not good Not gonna work if your infrastructure is not Well-designed it's not gonna work also. So it's not a magic solution I know it's used like that all the time when you heard about communities like everybody wants to Like move to communities it wants everybody wants to be cloud native It may or may not be for you depending on your need But again, it's not a magic solution. It won't solve all your needs. I just thought it was a funny comic and it's really like You know the reality for some of us. I'm lucky right now my bus are not like Gilbert Bus, but I used to have folks like that were like, yeah They're just touring like keywords and like you should have like solve every issues we had or every problem With that said how many of you felt like that when you started to like war You wanted to read a little more about communities and we're like, yeah, I have no exactly idea what it is You can raise your hand. I was like that The other people you're just lying you're just liar you were probably like that because in the end communities it's not a lot of new things, but it's a lot of like best practices that we put together and It brings a lot of new terms that we never heard before which is the complicated thing. It's a Powerful but complex product for some complex cases that you have so I felt like that when I Was like, okay, I need to do some cloud native stuff I need to learn about that and that was a little bit too much So hopefully this talk won't be too much. I tried to cram a couple of things within the hour But at the same time I tried to like keep it low So you tell me that's the first time I'm giving that talk So you tell me if like an at the end of a format of one hour was like good enough for people starting But so what is Kubernetes and you've probably seen like k8s? It's just another way to say Kubernetes It's basically like the number eight is like the number of characters between k and s So it's just the it's just for the cool kids. We say KW k8s I just community is so long to type like anyway, so you can see Kubernetes or k8s Which seems to be longer to say k8s than communities, but like it's faster to type so it's what is Kubernetes? It's a open source container Orchestrations it helps you orchestrate container like that written there. This is what it does. It's really gonna improve scalability For your whatever you need to deploy in the cloud It helps with high availability availability, and I know I'll struggle with that one in English high availability It is resource efficient Depending on how you use it depending on like if you know how to use it, but just like every other technology and It is also There is also self-healing Capabilities that I'm gonna demo you at the end of the talk really interesting.  You really make it easier there's also Way of doing things the communities that you don't have to do is call get ops I'm not gonna talk too much about get ops But the point is is that it becomes your source of truth So everything you put in your cluster should be in your git repo and it even makes it's not part of communities per say It's just like one Like way of doing things when it comes to running a communities cluster and it also make even easier The stateless disaster recovery model because it becomes your source of truth and everything is in git So it's really easier to save the day when something is going wrong And there's portability and I put it in it's a lick I know the the screen is a little bit small for that size of room and like it's a little bit low I did not anticipate that I put portability in it's a lick because communities per say It's a portable technology where it become less portable Even if all cloud provider tells you like there is no vendor locked in it's like when you go with the bigger cloud Like AWS Google cloud Asia Depending on the technology you're gonna use because they're free manage communities cluster for you depending on the technology You use within your infrastructure. It may make it a little less portable But it's not Kubernetes fault is like how those cloud provider implemented it Which is usually not the case with the smaller player like digital ocean or CVO Where the offer is really straightforward and they don't have like 10 to 20 to 30 services to help you manage your communities cluster So there is a little less vendor locked in but Kubernetes at the foundation it's created. So it's easier for you to move from Unpremises to public cloud to one or the other public clouds because Mostly everything you do it's gonna be defined in YAML file with that said brief of story just because I find it a little bit fascinating and Because I like to understand when there is a technology like that that will basically take over Everything in my cloud like where if I decide to replace my more traditional cloud The way that I hosted my application to communities I want to understand a little bit like is it technology that makes sense for me So just a brief history because I liked it In 2003 Google created Kubernetes, but it was called board that was Someone said the yeah, you need a small these stickers I need like, you know those stickers that you give to kids when I give you the great answers That's good. Yeah, so it was called board. I think at some point a year after two years Changes for Omega Anyway, it doesn't matter the initial name was board, but it was an internal project which I assume I don't know hold the story, but which I assume was Because they needed a technology to be able to scale because you know, it's Google. I don't know Which services were available at that time, but like just like the biggest like Google search just YouTube Probably take a lot of like cloud power just to make it happen So they love this internally to their own need, but in 2014 so a couple of years ago They're like, you know what? That's a great technology and a couple of engineers I Google where I like yeah, we need to open source that So they decided to open source the project to change the name to communities And this is really when we started to hear a little bit about it That was still not popular. That was the early beginning, but the year after in 2015 It started to become a thing There is that CNCF cloud native computer foundation that is like a kind of like a sub-horror of like Linux Linux Foundation that Partner with Google to really like make communities what it is to open source So Google basically and I'm like paraphrasing but like gave the project to CNCF and like they become the honor of Covernity, so it's not on per se per Google. I assume they're like probably their team are like The biggest contributor to the project still but it is not it is not a Google project anymore So for those of you that were like hey, I've read we're like in the Linux slash like open source or like free software conference You're talking about Google. Do not be afraid now It's under the umbrella of the CNCF and the CNCF doesn't have only communities They have a lot of proud native technology projects that are under the umbrella of the CNCF But it was not the only thing they also start in 2016 Actually, I'm missing one thing I think it's 20. Yeah, okay. It's 2016. So in 2016 There's one technology that I'm gonna show you that's gonna become one of your best friend called elm that was created and released The first version was really was released in 2016 They also create the first QCon conference, which is the only grail of conference when it comes to communities because it's organized by CNCF it is huge and I assume was not that huge in 2016. I was not there But now it is pretty huge. It's twice a year once in Europe once in North America It's the place to be if you want to learn more about communities. There is a lot of vendors there There's like a huge haul of like Exposin so it started in 2016 so it's become it became a little more serious because now we had a conference In 2017, this is where the thing really started to pick up. It was enterprise Enterprises started to like go and jump in the community strain. So AWS started to give like a public offering like managed communities offering Docker when all in when it comes to community so within Docker desktop you can like create communities cluster like they really went all in And a company like get up so pre Microsoft acquisition. Am I right? Yeah pre Microsoft acquisition get up like get up was running on community So just to give you an example of one company that trusted communities like had the beginning hish was get up and We probably hold use get up Or maybe get lab or other technology, but like I would say many people use get up So it's it was probably his still working on communities which bring us to today It's kind of like the de facto now for many medium to large-sized company using communities But also mostly every startups right now or actually before in a previous life a couple of years ago I was working a digital ocean and part of my job was to work with startups And every startup technical funders were asking me like okay, like I'm building my infrastructure for my application Like I need to go cloud native So like every startup like it was the kind of like the cool new things to work on as I saw before It may or may not fill your need what it was telling to those folks at the time is like Maybe don't spend time in the fort right now to understand communities because it's complex thing while you're still building your product You may not have users. You may not even have paid users But if there is a way for you to architecture your application, so it's gonna be easier after that to move to cloud native and Kubernetes So just a quick brief a story if I want to create a cluster So this is the easiest part when it comes to working with communities So you tell me it's gonna be big enough There's a couple of ways to do that So there's a couple of tools where you can create cluster locally on your own machine So you don't have to pay a public cloud to try to test it most of those tools give you the same ish Experience that you're gonna have in the public cloud Some popular one. There is like kind There is mini cube and I use key 3d for different reasons. So air what I can do. I have a creativity key 3d It's a CLI install on my machine and I can say key 3d create cluster and Demo key 3d and actually it's the other way around. It's key 3d cluster great Yeah How do you call demo key 3d? And I don't know how to type Technology is hard Plaster. Oh, there we go. So what's gonna happen here? I have my firewall asking me once in a while But if I go in Docker, so it's using Docker desktop here. Oh, it's over there if I look for a key 3d You're gonna see now. It's like creating some containers for me. It's running is gonna run Kubernetes version in my Docker machine. So now in Turi actually not in Turi in practice I have communities running on my machine. What does that mean? Not that much. I have a cluster Yeah, I'm nothing really running on it except like the Kubernetes technology that makes the cluster the cluster I'm gonna tell you a little more. What is exactly a cluster? So if I use and I'm gonna tell you about cube CTL after but if I get pods and I'm gonna explain what the pods are Just bear with me right now, but just to show you that something exists in my prompt here You may see I know if your people in the back. It's probably too low You may see here. I'm connected to my cluster automatically for different reasons And I'm listing like all the pods that are inside my cluster So those are the things that are installed by default when I use key 3d to create a cluster So that's one way to create a cluster if you want to test your team After that, you can also go in public cloud. As I said most public cloud they have managed communities Offering for you the prices the price is different from cloud to the other I like CBO for two reason and transparency to just acquire us So there's one reason why I like CBO but like I love them before it's just a sample cloud that you know What you're gonna pay for at the end of the day and I can go get go here and say like create a new cluster make it a little bigger and I'm gonna give it the name Nope click back Gonna click create cluster again CBO demo Can choose the number of no we'll get back to this and Here I can define like what's gonna be You know the resources that I'm gonna need on my nodes, which are basically the hardware or virtual Hardware that is running my communities cluster. I go here I'm gonna skip all this because specific to CBO But like if I create here in one minute or two minutes I'm gonna have my communities cluster running in the cloud. So at that point you're like, okay. Good job Fred You have a cluster now what? Now let me explain you before I start to show you how to install vacation Let me explain you a little bit. What is our communities cluster? I think it's really important to understand the for them fundamentals Again, I'm gonna focus on what I think is the most important to know right at the beginning But I'll show you there is a lot more to know about communities like I was saying at the beginning of the talk So communities in a nutshell There is that cluster which is everything Like it contains my nodes to work my application So the cluster is what I created with t3d cluster that create common. This is what I created on CBO This is basically my communities teams running locally or in the cloud, but within the cluster I have some nodes. I have in every cluster I have one node called the master node and the master node it's basically a bunch of shit to run communities So I won't go in details for dad because like there is a lot of like Kubernetes specific technology that helps me to run my cluster in my application I Debated with myself if I wanted to talk about that I was like, yeah the first time I heard about this in a talk They lost me totally and that was not fundamental for me to understand how it's working like I'm driving my car I knew don't need to know like how exactly every pieces in the motor is working for me like in the engine Sorry, I don't know why I said the French word with the English accent, but in the engine I don't need to know hold hold every pieces working together. I know it's working My car is going from point A to B So it's a little bit of the thinking that I had when I say like, you know what? There's a lot of things related to communities Just know that you have like that master node in every cluster that you create Where the fun begin because so far it's not that fun Where the fun begin is when you go to the other type of node, which is the worker node think about the Machine or the resources either the hardware like physical hardware or virtual machine that is going to work That's going to be basically where I'm going to put my whatever I need to run my application So within my node I Have two specific things related to communities and those I'm going to tell you just a little more So you're going to have cup couplet, which is basically the interface between the master node and the worker node So you don't have to worry that much about that. It's there. It's working when you create your cluster It's there. You don't have to do anything But it's still good to know that it's there and there was a second thing called the cube proxy Which going to be what the known master going to use to connect to your worker node or nodes and To be able to talk to cube lap or if you have Part of your application that you want the external users to access which is probably the goal at the end of the day They're going to go through the cube proxy, but everything is transparent. It's just there. I just wanted you to know that The exciting part at least for me. You may not be as excited as me But I am is the container runtime. It is the smallest part of your communities cluster They're called pods and this is where within your pods you're going to run container or Containers so you can run one container with your application two three four how many containers you want? And it's depending on how you want to deploy your application Let's say I want to deploy WordPress WordPress. It's a PHP application. It's used my sequel by default as a database I think you can change further databases, but like it's my sequel by default and maybe what you want to do is have a pod We're going to deploy WordPress and I'm going to show you how to install an application In a pod how to deploy an application So what do you would do in my example with WordPress? Maybe what do you want to do is to have WordPress so a web server that have PHP support a PHP extension that serve WordPress in one container And you may want to have a second container which have you're gonna have a second container is gonna run my sequel Which is the database that WordPress WordPress need so what you want to do Maybe is in your pod in one pod you install WordPress with the web server engine next apache And on the second container it is to running my sequel So you would be able to do that or you would be able to have to pod with each one container One container in pod one which run WordPress with the web server Second second pod with the nutter just one container that run my sequel Which is probably a best way to do that But you have option for different reasons depending on like what you want to do So just understand that the pod it's a smaller part of your community's cluster Container is not a community's technology per se But like this is what you're gonna use to run your application and you can have multiple within a pod And you can have multiple pod within your node worker, but on top of that There is a lot more objects that you can use so pod are one But there is a lot of things config maps where you want to have like different Configurations about your different resources in your cluster. You can use deployment to deploy new application I'm gonna show you an example about that Ingress controller where it helps you like there's a engine x ingress controller Which is quite popular that helps you to like basically offer your application to the rest of the world You know my WordPress Installation that I was talking about it's probably for like people That are that don't have access to my server to access like any other application you publish in the web You want people to be able to access it so ingress controls gonna do that you have job and you have a lot of like Objects that helps you in your community's journey and all those things are part of your node or of your worker node And you can have multiple nodes too So that is communities in an out shell But I told you there is those other objects that I won't spend time to talk about today because I think since the latest version There is like 104 six objects part of communities So there's the list of other objects that you can use I'm gonna go fast ish So maybe you have the time to read a little bit But like there is a lot of objects that you can use within your node to do different things There's job. There's chrome job. There's ingress network policy They're in the namespace. I'm gonna talk a little more about that because that's a really critical component of your Community's journey and there's a lot more Secrets secret store so you can store secrets within your cluster There is like a lot of objects that's gonna help you so again I'm not gonna talk about all those because we're gonna be there at until the end of the day for sure and not All are exciting, but just know that the little I Don't like to say that it's kind of like saying my talk is bad But like the little you're gonna know about you for communities when you leave that room There is a lot more of other stuff you can do to help you within your journey But now today we're gonna start with you're the beginning For your communities journey You need best friends, and I'm not talking about your co-worker or people that you go have drinks after the day I'm talking about Principally, I selected three technologies. That's gonna be part of your journey Those are the three technologies that I feel you absolutely need to be successful So the first one which the logo means with all respect for graphic designer doesn't really tell you what that is It's cube CTL. So the common line that I use Before to show you the content or part of the content that was part of my cluster The second technology is helm. I'm gonna explain a little bit what it is after and the third one is Canines, which is also another great tool that you can use to help you So let me start with the first one cube CTL Cube CTL basically let me go back cube CTL basically is the default CLI to do whatever you have to do with your communities cluster It is Part of communities project it is on their CNCM CNCF There's not a lot of things you can do without actually there's probably nothing you can do within your cluster without cube CTL And I say cube CTL some people say cube cut all because it's k u CTL I call this cube CTL. I think the other people are wrong, but that's another discussion So a cube CTL a cube cut all is the common line. That's gonna make your life Better and worse at the same time because there is a lot of things you can do on the cluster But anyway, that's the common line too that you need to do most of the thing within your cluster We're gonna see also and you may see this in the demo even in the docs They suggest to do an alias from cube CTL to just the letter K So if you see me type K, it's just because it's a nab it and it's basically cube CTL with that said What I want to do now is I created some cluster I want to deploy an application because my cluster is basically useless right now It's just a communities cluster with nothing in it so I want to deploy an application and Mostly everything you do within the communities world is YAML How many of you know YAML? How many of you liked it? There's a lot less people. I understand where you're coming from. I was a Jason person I hated YAML with passion when I started to do a little more community stuff I did it for whatever reasons, you know, like As much as I love like my team well format and everything like they were complaining because you were missing a space and stuff like that I just hated it it took me time and That's the story of everyone.  Yeah, YAML. That's nice That's the best technology ever with all respect for the people who created at the beginning you hate it But at some point it becomes the best thing in your life Because imagine if I'm trying to reproduce this with Jason I'm a big Jason fan. I have to put up brackets everywhere and double quotes everywhere and it's a little less readable again You try to Jason But now I'm a little warm the YAML team for some of the stuff and this is one of the thing because it's it's readable It's easier to read the YAML fight So you need to get used to YAML because that's your new life now if you go communities way So here I'm using one of the other objects that I told you Deployment object That's gonna be the definition of my deploy like the kind of deployment That's gonna be the definition of a deployment that I want to do in my cluster here What I'm gonna do I'm gonna say you always need to have like API version haps Slash V1 the kind in this kind. It's a deployment object. So I'm using this to deploy an application I put some metadata the name of my deployment is gonna be engine X The labels is gonna be engine X to you because I'm you know, I'm a simple person There is a specification part where I can define the number of replicas. So how many pods do I want to? Deploy in my cluster in that case. I'm gonna keep it that one for the demo But what's interesting with the pods when I was saying that like you can have multiple pods It's not just for a separate application. It's part of the scalability and the high up High availability of communities where if I install my server now and let's say I still running WordPress and I installed an engine X server and I'm gonna put WordPress in it and I have I need to have multiple pods because what's happening with the first pod like there are so many people going on my website and the First pod like the resources like cannot give like back the HTTP response to people So there is a way with like a low balancer or community is gonna be able like to point the users to different pods That's still gonna run your same application. What happened if I have an issue on my first pod? It fails got killed didn't work Like you have other replica. So that's super interesting If I skip to the end, this is the interesting part I don't people in the back if you can see but there's a speckception within the template where I Mention the containers what I'm gonna deploy to my application So let me let me show you may be in a better way for the people in the back Yeah, I use VR not cat on person but I can see I have yes code which Which I like to so there's a containers again name is engine X But what I'm saying here. I'm saying, you know what like the images that you use when you use Docker I'm gonna do the same thing here I'm gonna say hey what I want to deploy is the engine X image the latest version You should probably not use latest you probably should define the version for different reasons But like you can do that and I'm gonna say hey I'm gonna be able to access the engine X server on port 80 within my pod. So what I'm gonna do now I'm gonna go here. I'm gonna do cube CTL apply Dash F which is like basically a file the next thing I'm gonna share is a file That is the YAML file that I've shown you and before I do that. Let me just show you where am I on key 3d? So let me do keep pods a so pods is again the smallest in it within my cluster Dash a is because there is a thing called namespace. I'm just saying like show me all the namespaces I'm gonna tell you just a little more about that after be patient So just to show you like there is a bunch of like default stuff that were created with key 3d. No engine X there, you know I'm not a magician And I'm not faking it. So Cube CTL. I'm gonna do apply That half engine X and what's gonna happen the deployment got created. So in Terry if I do K pods dot. Hey, you're gonna see now if it's big enough Gonna say now that in the default namespace I if I have a container that is being created and my engine X Server is being deployed So the third tool I'm gonna talk to you about is gonna help you to show the status of your cluster and stuff because like always Getting like K get pods to see the status. It's like a little bit like annoying, but now you see it's running Another object I could use right now because my engine X server is not accessible for me It's not accessible outside of my nodes right now So what I want to do is not the best way to do that for something in production But just for the sake of the demo I can do some port forwarding Gonna be lazy on this one Actually, I Need the exact name here. So what's happening with the name here is that I call it Call it engine X and there is some kind of like UID that were created for me Because within a same namespace you cannot have the same name So Kubernetes created like a UID for me after the name. So here what I'm gonna do. I'm gonna use K again Cube Ctl again, and I'm gonna say I'm gonna use a common code port forwarding And I'm gonna say for the pod call engine X dash whatever whatever it's written there I'm gonna say hey Locals on the local us on my machine. Let me access the port 8080 That will point to my pod my engine X pod to the port 80 So if I go enter here, it's kind of like blocking my terminal because now it's it's in port forwarding mode and if I go in my browser and I do what close 8080. I have engine X working. Is it not the most beautiful application you've ever seen? Yeah, don't be too excited about that So that's one way of deploying the second way of deploying an application is through helm Actually, there is a lot of there are some other ways to deploy But like this is your second best friend.  So if I Go here What I do I have a not my email You have an application another application a good website to check the most of the packages that are available publicly is Artifact hub that IO and I go here and I say, you know what I want to install engine X Is there like already packages of engine X? So I don't have to create my own YAML file and usually they are better build with different objects that you need to be successful So if there's a bit me that is like probably the company that is like I guess Oracle doing that like they're publishing so many So many packages out there and I'm so like hey, you know what there is an engine X package So there's the install here that explained me how to do that. So I installed the elm CLI on my computer What I need to do first. I need to tell the CLI hey like now have access to that chart repository So I'm gonna do helm repo had bitten to me. I forgot to remove it before the show So now it's not working because already there but let's say it was successful It had this repo within my own personal local ecosystem the next step I should do and again now it's not gonna work because I did not remove it the first thing you need to do after that It's helm repo update because for whatever reasons when you had a repo I always thought that you should update automatically so you have already access to the content doesn't work like that So you do helm repo update It's gonna go through all the repos that you had update them to see the last version. Let me kill this one After that What I want to see it's like okay engine X which I don't know how to write engine X. I Can search to do say search repo I Can search for engine X like now do I have like engine X chart? Accessible to my elm to so now I see I have three and we're like, you know what this is the one I want to install is called bit me dash engine X This is the chart version and this is the application version those can be different meaning that like the application version is the Version is the version of engine X the chart is like hey I can deploy the same application quite often. I made a bog. I I had an object like an ingress Controller to make engine X Automatically accessible from the outside world or stuff like that So I can deploy chart as many as often as I want which give me That kind of version number so first Let me Remove the engine X that I installed manually. So there is another common call QCTL delete Dash half again for file engine Xiamo so QCTL is gonna look at my ammo file It's gonna look. Hey, does he have a deployment that is called engine X should I remove that boom should not be there anymore If I do kick at pods Kick at pods Actually kick at pods Hey, you're gonna see that like engine X disappear now what I'm gonna do I'm gonna be lazy again. I can do Elm install. I Call my installation whatever I want again. I'm a simple person. I call it engine X I say this is the chart that I want to install This is the version that I want to install So the flag I'm using the flag version if I don't want to I think it's if our members gonna install the latest version I can define the namespace where I want to install it I'm gonna tell you just about that right after but bear with me and I use the flag creating them space Because if I want manually if I wanted to manually install engine X Like I did with the apply on the ammo file and I decided to do apply dash dash and or dash dash namespace engine X Before you would have to tell me like hey, you know what the namespace dots not exist There is no engine X namespace Elm give you the opportunity to create it if it does not exist so I click enter here and Now there's a lot more things because that chart contains the deployment that I showed you plus a lot more other things Even though some a little bit of explanation about like how to use the chart once it's deployed. So if I go back to Okay, let me go K get pod without the dash a If I go hey here, let's say no resources found in default namespace Because right now I'm in the default namespace a namespace is a way to Group and isolate different resources together and you're gonna see this about everything and naniting and communities It is super annoying at the beginning really useful when you kind of want to kind of like get to use to always like Either use like the option to get all namespace or switch namespace or if I go K get pod dash a I'm gonna see all namespace, but I can say Dash and engine X. I'm gonna see just the pods that are in the engine X namespace So really useful feature now engine X is running Let me show you that is running, but by doing that. Let me show you the third tool that should be your best friend So it's not a CNCF project. It's still open search.  I was like, yeah Make way more sense. So canine s or canines when you run it. It's a good mix between Now, let me maximize that a little bit Can you see in the back Yes, yeah, and that too is not super good though when you zoom that one is better Yeah, it's fine if you don't like read all the words what I want to show you is that this thing makes me use tube CTL Mostly never for most of the thing because that's a mix between like a UI tool for the terminal So here I'm showing all the pods within my namespace and I have all the information plus Some other information that I didn't have when I was getting get pods So there is a way for me to get more details But like the default common doesn't give me everything and what I can do I can navigate between object show me just the namespaces So I did Cullen namespaces now it's shown me the namespaces and I can say like hey, I want to see only What is in the engine x pod, but there is other objects that are chart deployed So if I do Cullen and I say hey, I want to see services I Click there. There's a low balancer that was installed for me So this is why sometimes I'm installing the chart is just make your life easier because you don't have to write all that YAML, you know for default application that for sure you're gonna use again. You want to deploy WordPress There is a chart for there instead of you trying to find the right image write the YAML Which is nice at beginning you want to do that to learn but at some point you're just like Hey, I'm just gonna elm install my application and that's gonna be there and that tool is great if I go back to pods Actually to pod. Oh, I'm in the The Elgin X namespace and I can also search I'm gonna search everything that is and Jen, I don't know how to write engine X most of the time Engine X and now it's gonna list everything. So that tool is really amazing But the other thing Remember when I did the port for it was blocking my terminal It's a little bit annoying and you can do one port for it at the time if you use cube CTL unless you need to like run the Port forwarding as a background task and like it's just a pain in the ass to be honest So here what I'm gonna do I'm gonna go in the service because that's my low balancer now and actually not this one let me go to the service and I'm gonna go in engine X and now you don't see it because that interface is not super good when you zoom in But there's an option called shift F on my low balancer and now I can say like, you know what the low balancer is running on 8080 compared to like the server running just on port 80 and I'm gonna say I want to Port forward and I can't do other stuff after that now I can go back to namespace and the port for it Actually namespace and the port for it is still working. I don't know how to type today So if I go here again, I have engine X again working Another amazing application That deserve a round of applause now that is the most beautiful application you have so anyway That tool is really amazing because that let help me do that and the other thing that it helps me do if I go to let's go back here and What I want to do I want to understand what is that pod like what is what is that but I can click D And that's gonna describe that pod so every information about the pod is gonna be there everything I can do everything in it.  So it's canines as I said, it's one of the tools that I love the most To be honest right now I don't remember half of the Cube CTL comments that I should do to do the same kind of things most of the time Because now I use canines. So it's still important.  Oh, is it K? I Cannot see I think it's K.  So canines is one of the tool that I like the most The main contributor Italian guys super nice super sweet. No, actually, it's not for that technology I don't know the main retainer of this one, but like just a great technology is open source. It's again It's the CLI you need to get used to CLI to terminal. I don't know for you. I'm a big terminal lover I love everything that I can do in the common line not everybody do and I respect that But the thing with communities that you don't really have the choice because most of the tools that's gonna help you To be successful within the cloud native ecosystem are things that run in the terminals You have some UI that can helps you to do stuff There is some actions that you can do within your public cloud provider But again, most of the thing run on the terminal So there was a third tools that should be or that could be your best friend if you decide to go the The cloud native way actually, I'm gonna go back here There's another tool. It's not in the talk but I don't even because I use that tool. I don't even remember how to switch firm context within why is that what is crudy? Anyway, I should not do allow but I don't even remember how to switch like context because what is great is that usually you don't even you don't have just one cluster you can have multiple cluster and One way to connect most of the tool from your CLI to your cluster is File called cube config which get you the information about your cluster and how to connect to it. So Has an example Earlier I created a cluster and CVO that should be ready. It's called CVO demo Two ways I could get the cube config is that there is a download button here But what I can do I have the CVO CLI install and there is a comment called CVO You don't have to know it's it's not about the talks is not about CVO, but just to tell you I do CVO communities config CVO demo, which is the name of my cluster I'm gonna save the cube config and you're gonna see my prompt would change now I'm connected to my CVO demo cluster the one that I created before let me Switch that again a little bit higher if I go in canine now My firewall is gonna ask me is it okay? It's okay, and now you're gonna see it's a little bit different because now I'm connected to my CVO cluster that I created before If I want to switch there's a real nice tool called Cube CTX That is also a way where I could have said like cube CTX cube first Fred and where I switch the context for me to my New cluster or there is that little UI thing where I can say like hey I'm gonna connect to another cluster. I have called cube first Fred and here if I go in canine It's a cluster that I created with you first. I'm gonna see there is a lot more thing in that cluster So it was just to show you a little more actually Like the canine interface where now I have like all those namespaces We install like chart museum cross planes cert manager vault Hargo CD, which is a CD pipeline That can be really useful for your cloud native journey and just to show you the interface But also for my next demo. I wanted to finish with hopefully something fun something. I like Tell you a little more about like the self-ealing part of communities Do you know what cast engineering is? It's basically what is that it's chaos engineering.  So basically what happened I think it's Netflix who coined the term Basically, they're just like killed services right or wrong like randomly and like they like Remove some some stuff in the firewall like they do a lot of things that you should not do on your server and by doing that Preventing like is it work preventively prevently the word in English. Yeah, I don't know Prevently it's worth it. Okay. Yeah. Yeah, so by doing that before it happened Yeah, I should have said that by doing that before it happened It helps you to see if your system is gonna be still working or at least ending well issues that could happen So instead of like it happens when there is like millions of people watching a show You know before that like if there is part of system like if my API server goes down for Like I don't know getting the description of the movie or the list of the movie you have access in your country What's gonna happen like if everything is gonna be on fire or is there like a service B That's gonna be able to take over and do that kind of stuff So anyway, it's about creating chaos in your ecosystem to understand where are the point of failures within your ecosystem? so Real nice thing to do and painful thing to do also because that also show where you have witnesses within your ecosystem So with that said I'm gonna use another open source tool called Cuban Vader and Before I do that let me go back in canines Actually, let me try to zoom that enough Okay, can you see good enough in the back actually like that? What are you gonna look here? It's the status of the pods running in my cube first Fred community sloster What I'm gonna do here. I'm gonna use Cuban Vader that I install on that same cluster before the talk and I Love that thing. It's a game It's like invaders like there is space invaders, but the invaders there are the resources within my in that case I selected my development namespace and now I have like all those. I don't know if you see them It's not zooming for whatever reasons, but anyway, you see those little halion At the top and I have my spaceship here and I'm gonna kill part of my resources in my cluster But look canines Hopefully no, it's not a good Example, I don't know where I'm right now. Let me go to That is so big Let me go to development And okay now I'm seeing all the pods Let me scroll.   I'm showing you all the pods. They're being killed They're being terminated and communities is just taking every say like hey, Fred Something's wrong, and it's just turning the pods again So in the end if I go back here, I killed a lot of them. I think I was not bad You're gonna see like most of them Should be running now Obviously some are completed because they were like just like things that needed to do some stuff and like they're not needed and the more But the one running there's still running and I killed them like I was really not nice with them I killed them. They're still running. They're back It was fast because doors is like smaller application k takes you can take a little more time But what's great is that in the case like here if I kill I have one two three four five So I have a replic acid with five Development actually four because the other one is API Development pods. So if I kill those two three the way I Put my I created my communities cluster communities were like a life is good I still have pods running and I'm they're gonna send people to the right pods So this is the this is an example of the self-healing part and the high availability of communities that I really like to demo On that note, I hope you're not like Fred I don't know if I still I don't know if I understand communities and it's kind of okay I made peace with the fact that like not everybody at the end of the talk would be like, okay Now I know communities because again, I shown you a small part. We only had an hour together But it's only the beginning hopefully understand a little more what it is how to deploy application How to get started to do the basic thing you needs to do and the good thing is that with any complicated technology You start at the beginning you try it a little bit you messed up. It's fine.  It's way easier than I don't know for how long you're being in tech but like It's a little more painful than like just connecting with a FTP client to my server and uploading new version Which we're doing like a long time ago. That was so fast. That was good But like if something was not working on the server That was terrible. That was like a pain to fix things working on the server right now Something's not working. I created a new plug. I delete my deployment deploy it again because it's working within my container It's a lot more easier. So some resources communities that I owe. It's your main place It's the website the official website the documentation is there.  No, you won't be an happy person that docs is like so huge It's good, but like it's so huge. So I suggest you if you want to know more try it or Find some tutorials online a Udemy course There was a lot of good things out there if you need help you're a little bit lost in your journey There is like three main slack places the CNCF one the communities one because there is two there is like in CNCF There is all the project, but there is a community's channel I think it may be a little bit better to go to the community slack itself because there's channels for specific things About communities and the last one. It's the cube first one even if you don't use our product We have a communities channel. We're just friendly people if you need help sometimes. It's a little less You know Asking a question in a channel where there is like a couple of thousands people versus we have like three or four hundreds members It's a little bit nicer in the communities channel is really low because most people in the community has for a cube first question But anyway, we're just just saying that we're friendly folks the tools that I share about today except cube CTX, which is also Open source tool I didn't put the link here because I didn't thought I was gonna use it But just look look for cube CTX on get up key 3d canines Cube CTL it's part of again It's part of the docs when you read about the docs to tell you to install that Cube Invaders that I use helm and I put cube first here and this is my one-minute product pitch What do you've seen before its? Cluster that I created you first the easiest part of Kubernetes is to create your cluster is after that You need something to manage your secrets You need something to manage your CD pipeline You need something to manage the certificate you need something to manage the external access You need something to manage like if you do Infrastructure as code you need all those things to manage just to be able to deploy your application with cube first It's free as open source that helps you to create a production ready cluster in most of the public cloud That have all those tools already installed in pre-configure, but also use something called get ups Speaking about the CD pipeline if you're still here tomorrow I give another talk in this exact room at 1145, but it's about our go CD our go CD is again A open source, but clown native base CD pipeline that you want to use if you go to the community's journey On top of that it's built with get ups in mind Which I'm gonna talk a little bit more about today, which is one way to manage your communities cluster So there is a talk about this tomorrow and on that note My name is Fred. I Think I'm gonna have like time for maybe I don't know one question But if we don't have time that much for question I'm gonna be out of the room right after my talk if you have to if you have question You can send me an email Fred at cube first the diode does my cat look so nice. It's one of my cat However, she's so nice. Anyway, always Twitter connect with me on LinkedIn if we don't have the time to chat if you try cube for Kubernetes in the future and We have the name is so close like everything This is the thing you need to get used to in the community space every two is K something or cube something so sometimes it's just like you just lose the name But anyway, I offer free coffee chat you can schedule at Fred the dev slash coffee 30 minutes call with me video chat We can talk about communities. We can talk about our go CD. We can talk about cats. We can talk about whatever you want Just to get to know each other. I'm a friendly person. I know I doesn't look like that But I'm a friendly person. So on that note, hopefully it was not too mind-blowing It was helping helpful a little bit and thanks for your time and have a good rest of the conference everybody And actually I'm gonna be respectful for the next speaker I know there is 15 minutes, but like it's always annoying when the speaker before you stay here and like take all the time So I'm just gonna remove my stuff and go outside of the room if you have any question comments and salt. I'm gonna be there Yeah, I think let let's begin. I think it's time for this session and Hi everyone My name is transit time and today my team made like Sean and I would like to talk about like cloud native data and model access management for machine learning in the AI domain So for here, I would like to give you folks a quick introduction about Sean and I currently I'm working at the Starvisa scientist at Alasio And I also is a presto commuter contributing to the open source Distributed SQL engine and Sean is currently the software engineer at Alasio and he is also the fluid Apache product commuter and Here is the agenda of our session today January like we we have a pretty long long list of Points that we want to cover at first. I want to talk about the machinery in the cloud What is the current pattern? What is the current architectural design about it? And then Sean will come to discuss about like say how to access data and models in the cloud and he will present the existing Solutions and evaluate the pros and cons of each design And then he will propose a new design with Alasio for the unified data access and the model access and then he will also talk about the cloud native Kubernetes operator and CISI fuse driver all on Kubernetes and then I will come back and talk about the data access Management for PyTorch and Ray and see how a unified data access could help to improve the training efficiency and Finally will give some use cases that we have learned from partnership So let's begin the first the topic we want to cover is machine learning in the cloud And here is the general architectural design about machine learning system in a hybrid cloud or multi-cloud system Like say there are generally like three parts in this diagram on the very left There is a training platform and it could be in on-prem data center or in a cloud And then in the middle it is the storage layer it could be like different object stores or like say HDFS is kind of archival storage and Generally during the training the training cluster will read the data from the remote storage and run the the necessary training jobs in the training clusters and after the training job is completed They will write back models into the remote storage for the model storage and then on the right side There's a serving platform for the serving they will get I'll say fetch the models from the remote storage and serve them in the serving cluster and There is a very clear like Signal of the separation of compute and storage tiers in this kind of a design the the training cluster could be in one data center or in the cloud a and Then for the serving cluster it could be in another data center or in another cloud so that's the design and Here we want to talk about the data and model access patterns like say in the machine learning domain What is what are the traffic patterns to access the data sets and the models? And here is a pretty large table and we want to focus on mainly two parts of it The first one is about the model training and what is the access pattern about that? Generally, we put them into three categories computer vision jobs net natural language processing jobs and checkpoint rights and for the first the two types though Mostly they are like a rating rating data set for computer visions Mostly it could be some images or videos So the typical pattern of this type of data set is there's a very large number of small files and Generally the files are read sequentially and for the natural language processing What we have observed is that for this type of data set there could be like Not that large number of files But there could be a medium level a medium number of files but those files could be extremely large and Generally during the training the access pattern wouldn't be a sequential rate But it's mostly a random rate especially when using like parallel to read these types of data sets and Generally for the training side our target is to maximize the throughput Maximize the rate performance from the storage and the maximum the GPU usage here and What about the model access pattern especially used in serving and here for this? Diagram we zoom into the model access pattern a little bit here And we have like model deployment model inference all in the like model serving domain And generally for this type of access it is still read and mostly the sequential rate that we need to read a whole Like like model into the serving serving cluster And so but for this type of thing what our target would be like we want to minimize the Latency to load it load the data set. I want to get a high concurrency in the model loading part and Then I think I will introduce Sean to talk about like say how to all exercise the data and models in the cloud And what are the existing solutions and their trade-offs? So let's first talk about what are some existing solutions for accessing data and models So for data access we apparently can directly read them from the cloud storage and second We can also before the training job we can copy the data set from the cloud to the local machine and the third way would be using a local cache layer and We can also use a distributed cache system and For model access the most common way we see is just to pull the models directly from that's cloud storage after Training is done and the model has been written into the cloud so now we look at Zoom in and look at them one by one So for for the case where we always read from cloud storage. This is the easiest way to sell up But at the same time the performance is not ideal For the model access apparently because there there could be multiple say More deploy multiple servers for the models so each time a server is starting the model has to be repeatedly pulled from the cloud storage and for training data Because because the training does sometimes as transition mentioned can be small spiles In fact, we sometimes see reading data can take more time than actual training And this is a screenshot of our one of our experiments done by using pytorch We can see the data loader actually takes 82 percent of all the time And apparently we don't want to see the bottleneck appeared on the IO path Now to look to make the date look to make reading data faster. We can certainly instead of reading from cloud if we copy them before training to the local now We can have a lack of much faster access and also less cost because The data is there. We don't have to always read them across different app books But then the manage data management is a hard problem because The disk space is apparently limited After using the data set or after this model is outdated We always must manually delete them. Otherwise, there's no space for the next batch of training or Reading data But at the same time the local storage is kind of limited because the data data said if data is growing The side of data set is growing faster. It's like very very huge The local disk may not cache all the files. So Although we do have some data can be faster of can be accessed faster, but it's kind of limited Now if we use a local cache layer for data reuse for example S3 has their assets S3FS built in local cache and Luxo has few SDK Now the reuse data is local. So after reading once Those data are cached locally so when we reuse them We get faster access and less cost and now because of this Cache layer they can help with data management. They can do Basically cache eviction after the cache layer is full So there's no more manual deletion of the data or any supervision But the same problem is the cache space is limited because we are depending on the local disk We have we have also use a distributed cache system. So for a Luxo 2.x We use a traditional master worker architecture so the data are cached on worker nodes and The metadata is cached on master nodes. So whenever there's a client needs to read data We the client needs to ask master first for where the file where the data is and then ask worker for the data now for both training data and Models we can keep them in the Cache system in the worker nodes So this is a perfect unified solution for both reading training data and train the models and A cache system to have some additional data management functionality. For example, we can preload the data We can also there's more like pin pin the data so that it won't get evicted Some more functionality like those But then we have the problem that masters now, although we do have a high availability functionality based on roughed But there are still single point of failure What I mean is if masters is down The client when client needs to read data it cannot get the metadata or say where the data is in the worker So there's no way for the client to get the data anymore so now masters are the single point of failure and As the file number grow fastly This problem is more and more severe than makes the masters the bottleneck of the overall performance so to sum up a Few challenges we see in this accessing the data in the cloud the first is performance if we So apparently putting data from cloud storage is hurting both training and serving and the cost repeatedly requesting data from cloud storage is costly both Both metadata and data APIs would cost money to if we need to read them and the third is the reliability because availability is the key for every service in the cloud and Data management is another problem where manual work is unfavorable Now let's talk about the new design with a lot so so earlier. We see that master is the bottleneck master is the bottleneck for the performance both because When there's huge amount of files retrieving metadata is very slow on masters and because of the reliability of the master part So now we use consistent hashing to cache both data and metadata on workers instead of the method on masters so by say consider hashing basically you the client can calculate where the where the data is on which worker by the file name and Worker knows because we can scale them up Pretty easily. They had they have plenty space for cash So training data and the models only need to be pulled once from the cloud storage and later on whenever Either training or survey needs them again. They can directly reach from workers so this is we are using less cost and There's no more single point of failure because master is now Not serving any Traffic we get better reliability and there's no more performance bottleneck on masters This greatly increase our performance on reading data and We also get the data manager system So in implementing this in the Luxio 3x We get higher scalability So one worker can easily support 30 to 50 million files and they scale linearly and we can easily support 10 billion of files, which is Nowadays pretty common in the machine learning and AI workloads We also can achieve high ability but high availability we can guarantee a 99 for a nice uptime and There's no single point of failure in the system and we can get faster data loading in the preload stage and for cloud native Kubernetes scenarios, we also have Operator and it says I feel for the data access management and now let's talk about these two parts so for for a luxury operator on this diagram on the bottom we have cloud VMs and Kubernetes installed on those VMs and on top of those we have the Luxio operator Which manages the life cycle of a Luxio clusters and datasets so on top of operator would be a Luxio cluster and Then on top of the Luxio cluster we have training frameworks Which will read the data from a Luxio cluster and the cluster really sits outside of this More of the come sits outside of the compute site So for a Luxio cluster CRD the custom resource definition It follows the Kubernetes operator pattern So on user side they first create the cloud a Luxio cluster and datasets there are CRS Which basically means they submit a configuration to the Kubernetes API server Which then inform those configurations to the Luxio operator Upon receiving this configurations of CRS a Luxio operator will in term modify and manage Kubernetes resources which involves creating a Luxio cluster monitor a Luxio cluster Creating a dataset bounded with the Luxio and do Depending on the state of the current current cluster will do corresponding stuff and then we repeat this cycle or we call it reconciled and Basically repeat repeatedly compare the current status of the cluster with the desired state of the cluster And if there's a mismatch Will do something And with the Luxio operator tool we can achieve zero downtime upgrade high availability and also scaling So with the operator we can achieve a fully managed cache With this simple Python code we can easily preloaded the data Either train data or models into a lot of a lot of cluster And all data will get evicted if the cache space is full According to the eviction policy though, so there's no more manual work of deleting the existing or unused the data and a lot of server will take the job to read and cache from the cloud storage if there's some any cache miss So now let's talk about the Luxio CSI fields driver The first about a lot of fields a lot of fields is able to expose the Luxio file system as a local file system and So users can directly access the cloud storage just as accessing a local storage for example We can easily do a cat or LS on the inside the directory and We can simply easy do an open file. This is a pie a simple Python code. We can just treat the The cloud the files in a cloud storage just as a local storage This has very very low impact for the end users and here's a star. It has a screenshot of This is basically my S3 bucket. You can see we call LS and it just shows on in the terminal just as a local file system and For CSI for Kubernetes CSI is the container storage interface so before we have CSI for Before we have CSI for Kubernetes every storage provider for example AWS EBS and order they have to write their own code to implement the Kubernetes volume interface so that Kubernetes pause is able to use them as a storage But then this reason the problem of their code can only be used after Kubernetes releases so this make the Made a collaboration between the memory cumbersome because one side has to follow the timeline of the other side and there's a lot of communication between but after the CSI comes in the CSI is Basically one type of volume and then if anyone implements the CSI interface They can just plug in as the CSI and become the volume for Kubernetes pause which separates the development cycles and would make the The middle in the implementation and The release much easier for both sides So now there's more than a hundred existing CSI drivers For example, AWS EBS after this after file and of course we have the Luxio CSI driver So if we combine them together for data access Luxio fuse is able to turn the remote cloud data set into local folder for training And CSI is able to launch Luxio fuse pod only when the data set is needed So which means If the training is now started or the applicant we don't need the fuse process to be there Sometimes this can take some resources that the cluster manager The cluster maintainer doesn't want to see So with this setup We have three layers of caching. So first with the kernel fuse. We have the kernel cache This is the fastest and then with the Luxio fuse we have the local cache for the training node and And Lastly we have the local server Distributed cache that sits inside the compute inside the compute cluster Serving Data as cash, but it's still much faster than accessing data from cloud So on the right side this diagram shows the architecture of how application pod can access the Luxio as the data from a lot of views So on this host machine the application part is basically just the training part where the training Python or Whatever training is running and It uses the persistent volume and person volume claim To be mounted inside itself So that they can see the data inside this persistent volume claim and Then a lot of fuse container runs on the Luxio fuse process Which mounts the data inside the Luxio on to the host machine so that by this Two-step mount the application pod is able to see the data stored the data exposed by the Luxio fuse process All right now I'm back to transition to talk about the data access management Thank you. Thank you Sean and So Here I come back again and talk about like like Sean has talked about a unified caching layer for data and model access Like how would that be really helpful in production use cases like say could we get some results from some evaluation? And that's my job here to talk about like say there are some Experiments that we have done with this new design and we have already gotten some like Significant results that we want to share here today So the the first section about the experimental result We want to share is about using it integrating. It's this kind of solution with pytorch so here is the general design that we Integrated a lot so with pytorch for the training and say on the very left side There is a training node and for example here We have a pytorch cluster to run the training job and then we have a cache client installed in this training node with pytorch so the cache client has a Affinity block location policy so it can help to when a data would be needed by a piece of data It's needed by pytorch. It could know like which target cache worker it needs to talk to to get that this kind of file and then We have a service registry in the middle in the cache cluster to serve for the service recovery discovery So when the the cache client could gather cluster information from there and then talk to that Specific cache worker to get that data file if that data file has been cashed in the cash worker The data file will be directly returned to the cache client and then feed it to the pytorch training node But if it is not there it will get the file from the end of storage Like different object store or different storage medium and then get it back and return it to the cache client So this is a general design about like how to integrate a lot so with pytorch And we we want to say here that this is just an example of a lot so but if we use another type of cash generate the idea would be the same and Here is the some data loading performance that we have collected Still as we have discussed there are two categories of data set for training One is computer vision and the other is natural language processing So still we compare those two types of data sets here on the left side We compare the computer vision like training data loading I OPS So the data set is the very popular image net and we use a subset of it And then what we could Gotten from the result is that a lot so fuse has the highest the performance then S3 fuse and AWS S3 and for the AWS S3 here We use the bottle API the Python API here and for the right side We do some we did some comparison about different types of API's different types of data loading for Natural language processing data set and we use the Yelp review data set here And what we have observed is that a lot so rest for API and the a lot so S3 API Provide a better performance over the others And also like when we do the training we really care about the GPU usage of the training job Because nowadays GPU is a very important resources that we really want to maintain a high usage of it So here is an example that we ran the Render pie torch rest net mechanism on the image net data set that will train this kind of job there And we use the S3 fuse to help us to fetch the data from S3 and Here is the result We could see that the GPU usage here is only around like 17% So most of the time actually the training the whole training most of the time is spanned on loading the data set from S3 So what we can observe on the middle like green block is that like say around like 82 percent of The total training time is spanned on the data loading section And with the a lot so fuse we can improve that dramatically like say still the same training pipeline Just that we used a lot so fuse here And we could see that the data loader rate the proportion of data loading has been reduced from 82 percent to only around 1% here and the GPU usage has been improved from 17 percent to 93 percent here so five times improvement So that's an example that we integrated this kind of data access solution with pytorch And here we want to give you folks another like example of integrating it with ray So before we jump into the experimental results, I want to give you folks a quick idea about what is ray Ray itself current nowadays is gaining like popularity in the machine learning domain It is designed mainly for distributed training distributed compute. So itself uses Distributed scheduler to help to this trap Dispatch training jobs to available workers. There could be CPU. There could be GPU So for users generally we can just write some like still like single-threaded Code but real will interpret it and distribute this kind of job in a distributed pattern So ray help us to horizontally scaling for the training jobs across multiple nodes and Also, it provides a data abstraction for customers to use So for users we can very easily write some ray jobs for distributed machine learning training and Here is how we integrated like a lasso into the ray ecosystem Generally ray as a unified like distributed compute layer It uses like different machine learning training frameworks like a pytorch tensor flow Etc etc. It's mainly have to go in ray. It mainly have like a train train serve and tune modules and then When on usually the data is all the models is stored in In the story player that the very bottom layer in this diagram and then a lasso could provide a unified like high-performance data access layer To read the data to load the also preloaded the data and models into a lasso and the feed it to the upper like training or serving framework and This is the design like how we integrated with Integrated a lasso into ray so integrated a unified caching layer into the ray ecosystem in ray January the data load loading is Is taken care by a module called ray data loader? So and ray data loader uses pyro to raise the data files and with pyro the pyro itself uses the FS back as an interface to talk to different data sources and so what we have done is that we like created a lasso's FS back implementation We which we implemented the interfaces like a rate file with open file a list of directory et cetera et cetera and This implementations talks to the alasio Python client and alasio Python client talks to the rest of all API servers in the worker nodes of the alasio cluster and Here is we also have a EDC cluster here It is responsible for for alasio workers They register their worker information into the EDC cluster and then the alasio Python client could talk to the EDC cluster to get the worker addresses so when Like the ray wants to load a specific data file the Python client who could determine Like which worker it should talk to to get this piece of like data file So this is a design and how about the results? What about some Evaluation results whether we could get some improvement with this kind of design and Here this is a benchmark on some small files on the small small files here Mainly referring to some small images like the experiment was done based on the data set of the image net data set we use the 130 gigabyte Image net data set and we there are some settings on the on the right side like say we have like four ray train workers nine process rating and General what we have observed is that the active object store Memory ranges from four hundred to four hundred megabytes and on the left side We show a diagram here about the throughput the images per second that we can load in ray So we compare we directly rated from S3 So that's the blue bars and also like say what if we low use ray to load a data set from alasio That is the red bars and what we could observe is that when the object store Memory is pretty high like say four gigabytes Directly from S3 versus alasio. They have pretty similar performance And if we have a limited memory here, like say only have a few hundred megabytes for for ray to for the Rays object store we could see that alasio has a higher performance regarding the throughput So that's about some small files. That's only one type of data set like what about some the other type What about the files are pretty large and they could be in different data formats like very Popular column oriented data format like parquet, right? So here is another benchmark that we have done like say for the the data set still the image net data set but but we Put them into some large pocket files in general each pocket file is around like 200 megabytes And in total there are around like 60 gigabytes of the data set So there are like 28 ray train training workers 28 process reading and what we could observe that from the diagram is that we could see that generally Alasio directly loading the pocket files from alasio In ray has a better performance have a better performance over directly Loading them from S3 into into ray. So alasio is a red Red line here and the S3 is a blue line here. Okay and Also, if you folks remember that Shawn has also talked about some cost Say when we fetch say in a multi-cloud platform Let's say we have a training cluster in the on-premises environment But we have a cloud storage for the data set and the model storage Then every time for the training or for the serving we trade the data set all the models from the remote storage That will cause a egress cost. So that will be a data transfer fees like Needed by the cloud provider caught by the vendors So generally with this kind of a caching layer it can also help us to reduce the egress fee here So here for example, there are generally two diagrams It's mainly about the cache itself One is a cache miss and the other is a cache hit So it shows that one we could have a high pretty high cache hit Then we could save a really not large number of API costs from the cloud So because with a very high cash Cache rate a percentage Generally all the calls here like the list of status call to get calls all those API calls will only access the Caching layer instead of the cloud storage and the caching layer has Already cash or say preload the data sets and the models in the cache So they could directly feed the data and models back to the training cluster or the serving cluster So that will save a very high number of API calls from the cloud side and Finally, I want to talk about some use cases Actually, this is not a something that's very new that we have collaborated with different Companies in this kind of design and in this type of evaluation So here we want to share a very practical use case a very practical story Like how to use this approach to really fit it into a production level machine learning pipeline So here is a diagram of the basic like a design Architectural design of a rail like a machine learning pipeline in a company So generally this is a design before we use the caching mechanism. This is the they're like a previous design Generally, they have two different model training clusters one model training cluster mainly runs the pytorch jobs and the pytorch jobs like fetch the training data from the object storage and they also have another training cluster and that training cluster ran Rans spark jobs and from the spark jobs they get the training data from HDFS and the both those two types of training jobs like pytorch jobs and spark jobs Write their models back to HDFS So that's about the model training and then on the right There's about like a model serving a model serving they have a model deployment cluster that will have to fetch the latest the models from the HDFS and Then serve it to for for like continuous serving like say model inference, etc, etc And the the they have different applications in the company And those application will send some API calls to the model serving cluster to use the to use the models to for prediction and forecasting and There are three issues that they have reported in their previous design The first one is a very low GPU Utilization in the pytorch site. That's the one problem because they have to repeatedly like read the data from the object storage And the other side the other the next the problem is about the overloaded storage in the HDFS side So lots of things though. This makes the HDFS cluster a bit unstable unstable during that time And another problem is about a network congestion here because the model deployment cluster will need to Very frequently try to look repeatedly load the models from the HDFS side. This cost some like that network congestion in there in their like this data system and Then what we have done with them is that through the collaboration we Adopted like a lasso into their production environment and it solved those problems say for the pytorch Training jobs and spark training jobs We they both now have a lasso as a unified data access layer to help to Cache the training data and feed the training data to the training jobs And also for the model serving side We also have a lasso as a model cache that it can preload the models into a lasso As a cache layer and feed the models to the the the model deployment cluster So what we can we have observed from this production environment is that we can get higher GPU usage Around like two times. There's no network congestion now for the Model serving and also we have seen like a faster Model rollout here. There's around like ten times compared with the previous design Yeah, I think that's that's all for our like session today any questions or anything that you want to ask us It's open. Yeah, sorry. I didn't quite understand how the data path works from your caching layer to To the worker products like the the pod process. Is that going over a network? Or is it going through like I like the actual IO subsystem of the system like very basically Are you copying to some sort of block device on the system when you're caching and then that's being loaded by the pod? And in which case, how do you make sure that? the data that that particular pod is training on you had like You know multi replication of the cache layer But it looked like maybe as possible for a pod over here would be trying to load a piece of data That's on the cache on this other worker. Is that correct? Let me repeat Yeah, let me repeat your question first So from my understanding your question is I say we have a large number of worker nodes and on the same time we may have a large number of training nodes and then the training data may want a Piece of data set and then we have a worker to help it to load it And then there's maybe another training training training pod that may also need this data set and acts another worker to To load it so there could be some like duplication or miss a communication between those those nodes And you want to know how we handle this kind of problem. Am I understanding it correctly? Yeah For that. Yeah, I think the second question is about like how we communicate What is the communication protocol or the channel between the the worker the the cache worker node and the training node? Okay. Yeah, so I can help you. I think I can answer the first the question first about the architectural design about the between the training node and the the Cache worker node here, so the general idea is that when we have the large number of cache workers there so for Eat for only for what only one data file It will be only stored in one specific worker unless the the customer configures the replicas of this data file and so say we have like 10 training nodes and the training nodes have like a They will have like a Python client there So actually that will answer your second question the communication is Communicated via the HTTP like channels like say we have the Python client talks to the rest of all servers in the cache worker node so and for the The Python client side its uses a consistent hashing because the worker nodes are listed into a hash rain So and then for if you know the hashing there could be different workers and for one data file It will be only stored in one worker and the Python client could know for a specific data file Where is that data file is stored so it will just to talk to that specific worker node So that's the basic mechanism there Is it open? I have two questions one one about the performance of s3. I'd like I Would have thought that you would improve also like on the smaller file size there But why is it not improving is that basically because that the array itself? Caches that this has enough memory as the first one and the second one is kind of more curiosity I mean because here we're not like is very technical, but You could have a lot of these pods are that they are creating the claims Are they all creating the claims against the same volume? So you create one volume and they all have claims Against that volume Could you repeat the first question? The first question was about when you were comparing the performance of s3 for the two different values of of memory on ray Why doesn't it improve for the when you have a large memory on ray? Is that because ray itself caches? Oh, I got your point. I think yeah I think that the second one is to you Sean like do you get did you get that? Yeah, I think I can answer the first question the first question. I think you asked about You asked about the first case all they this case the second case the first one this one Okay, I think the The The case I think what so yeah, I think your question is like say From understanding your question they say for example one the Memories is pretty Limited like a lot so it has a higher performance, but why when we have a large amount of Memories the performance is here is pretty similar. Yeah. Yeah, I think from our Experiments like say because for ray it also have the object stall here as what you mentioned ray is Object stall you can we can consider it as another type of cache Yeah, so it's it's will catch the data in the object stall here So but for a lasso it can help one there's a very limited Memory here because for the ray object stall it mainly store the data set in the memory side And when the memory is limited the data will be spilled to disk But the performance there is not that like significant that there will be like a Versant performance, but for a lasso side we mainly use SSD there So if SSD for the data storage that could be pretty fast. Yeah, so that's the reason Repeat the second question as well it's it's more a curiosity and because it doesn't have anything to do with the performance, but You had those parts that access Alexia Create these PVC's the claims on the storage are they all creating different PVC's on the same PV? so it's a PV and PVC they are like a pair one-to-one pair and When Alexia is created we have this PVC created for them and PV and PVC pair for them and then they just need to mount that PVC into the their application part but when we say PVC enter the hood is just a Path on a host machine so there's there's a Mount between a lot of fuse part with the host machine path So they are like this basically, you know, that's the same thing and then this part is also mounted into the application part But they are the same thing. So this is how like the two paws can communicate with each other Yeah, I think maybe I didn't catch for the SSD layer for your storage there Are you using LRU or some other cache scheme and you know Analyze what the impact on the SSD drive rate, you know burn rate is and whether that costs Basically, you're burning SSDs in order to do this or Yes, I think I think the question is mainly about like say the cache eviction policies in this design and Also about like say why we choose SSD for that, right? So so for the first question for the cache eviction policies Yes, we have like different like choices here like say L IU faithful L a few et cetera et cetera But what we have observed in the current use cases that this is a very basic mechanism like L IU has already been pretty well Working pretty well in the in the production use cases. So fun at this moment. We didn't like Implement like many more complicated mechanism because the straightforward approaches have been pretty nice I have pretty pretty well and the second question is about the SSDs, right? So why we choose SSD over for example, memory here is from some very like Critical observations from industrial level machine learning workload. That's nowadays in the training workload For example, we have different training nodes, right? And the training nodes usually have both CPUs and GPUs and some some companies they only use CPUs for the training because they don't have enough resources for the GPU provisioning and no matter which type of design memory is usually like a very limited or say a Very important resources for training because training itself will have already consumed lots of Memory resources there. So for the caching mechanism, we have a limited amount of RAM that we can use So but SSD has also a better performance over like HDD over like a previous like Disk design and at the same time usually on each training node They have a pretty large amount of SSD unused there So that's that's why we choose SSD for the cache storage for data and models. Yeah So any any other questions? I Think yeah, and thank you. Thank you Test test cool, and I can just switch this on and off just turning it on test test Great. It's working. Yeah Hey everyone Thanks for joining us Welcome to the cloud native track of scale 21 Our next talk is bridging open source developer platform backstage meets coder presented by Ben Potter head of products at coder.com and Long time scale attendee and now presenter at scale Please become Ben. Hey, thanks folks. So yeah today. I'm gonna be talking about two different open source developer productivity platforms backstage and coder For us to give a quick intro. So I grew up in San Diego Currently living in in Austin, Texas Like you said, I work for product at a coder.com and I spent the last five years working for various dev tool startups And and I really like this conference in particular. My first one was about 10 years ago. I Was super nervous.  So I was absolutely terrified Fortunately, that's not what a conference is So I went over to the Ubuntu booth and they helped me get Ubuntu installed on my machine So that's kind of my my history with scale But we're not here to talk about that. We're here to talk about developer productivity So we're gonna start with an intro It's gonna start with kind of a mix of both like high-level concepts And then we're gonna get technical into technical solutions through the lens of a developer productivity team I know some folks here are familiar with with one or both of these tools We'll get into that but we're gonna start with some kind of 101 style content So first what is developer productivity? I am by no means an expert on the topic I actually think Hans doctor is he's written the developer productivity handbook. It's 91 pages It's quite a page turner and it really kind of divides developer productivity into two kind of different segments developer productivity engineering and developer productivity management and Developer productivity engineering is really optimizing machine processes such as a slow build on a laptop Where management is more optimizing people processes such as Joe's not doing a great job at work today How can we kind of figure figure that out? So for this talk, I'm gonna be spoke focusing a little bit more on the developer productivity engineering aspect of things and Through my work in dev tools. I've had the opportunity to have conversations with some pretty great engineering leaders and This is a quote that I've heard from from several engineering leaders at fortune 2000 companies and to me this is shocking. I would take this with a grain of salt I when I try to dig in and ask where does this come from that developers are only productive two to five percent of the time They I don't really know what where this comes from I it could be an angry person in C-suite or it could be a very kind of method Methodological like survey. I really don't know But what we do know and this is this is pretty well known is that enterprise developers spend 40 to 60 percent of their work day in their editor This is a pretty good number There's a lot of other things that developers spend their time doing whether it's sprint planning or doing documents or code reviews But this certain this data comes from the JetBrains survey as well as the slash data survey And something else the JetBrains survey found is that when the when a developer runs into an issue Maybe they're like waiting on a build or their laptop freezes up. They decide just to take a break So that's a lot of kind of time taken out of flow Again, I would take this data as well as a grain of salt I think it's really important in your organization to run your own surveys and Understand kind of where are the the bottlenecks on on your developer teams and there's plenty of good templates out there So now on to the type of productivity teams and the folks at DX This is getdx.com. They have a developer productivity platform They do a really good job segmenting different types of productivity enablement Teams and what I really want to focus on on this talk to to get even more specific is the developer tools teams These are typically developer infrastructure platform engineering Developer experience all these kind of sorts of of names So these are some common themes that I've gathered both from analysts as well as Practitioners on kind of what are the best practices when thinking about building a developer tool stack thinking about what tools Do I want to offer engineers that they could consume and opt-in to use and the theme that I see Very commonly is to adopt a product management mindset this means to really think about who your users are treat them as customers and a lot of product management concepts build a minimum viable product and and a lot of the the Recommendations to that I see is to to build internal tooling as opposed to buying a platform This is probably a combination of using several different vendors as part of your tooling but to really own that Rapper because you really want to own the developer experience and you also want to own and understand the developer workflow Another kind of fourth point that I see it's pretty important is to provide trading at enablement Team topologies is a really interesting Book on this and it essentially explains that you need to kind of have two separate efforts working in parallel One is platform Which is kind of maintaining the tools and treating it like a product and the second is enablement Which is more of a service which is going to different teams doing workshops Explaining what tools are and you have to have both of these things functioning it could be one team It could be two teams But these are kind of the four pillars that I've seen that makes it that they're really important when you're thinking about Building your developer tool stack and I put as a bonus to use open-source software We're an open-source conference and it's a lot more fun when you can submit a pull request to fix something versus having to wait on a on a product manager And so Today I'm talking about backstage encoder. These are two separate categories. There's the internal developer platform IDP and Cloud developer environments CDE and it's kind of a tongue twister You use your IDP to connect to your CDE and then you use an ID E. There's a lot of a lot of Lot of terms here So before we get into backstage and internal developer platforms, I wanted to explain the services problem Many folks here might be familiar with this, but it boils down to many software applications start very small They get larger difficult to manage at a large team They're ultimately split from monolith into microservices. This creates a lot of benefits It allows teams to contribute with each other. It allows these services to talk to each other with a common API And More services are created. They grow in size some get orphaned They no longer have owners and it becomes extremely difficult to manage and this visualization covers the scale and quantity of Services, but it also doesn't cover the complexity that comes with different dependencies talking to each other and hierarchy meaning This service sits on top of this one and it becomes when you're doing a request this giant chain down Services and it's very difficult to understand what's going on So now imagine one of these services stop reponding or you look need to upgrade a dependency And you don't really understand how it's going to impact the rest of the stack So this was the problem that Spotify was constantly running into And instead of building and testing code teams were spending more and more time looking just for the right information to get Started so the thing that they did first and foremost was they built a service catalog And a service catalog was one of the first things that Spotify built when they were creating backstage in 2020 and essentially splits these services into separate components Makes it pretty easy to track a specific Item in the component and you can see metadata inside there such as the team SLAs and a different hierarchy Independencies that these services rely on it's pretty easy to query You can read the metadata inside a specific service API documentation see I pipelines It's relatively easy to kind of track what's going on in in backstage So this is what backstage itself looks like it was originally launched by Spotify in March of 2020 is an engineering blog post in 2022 I believe they donated it to the cloud native computing foundation and It comes with a service catalog which I talked about comes with a pretty cool tool called the product the project scaffolder Include and then they also just have a plug-in ecosystem that really allows you to extend Backstage for your own uses So this is a scaffolder it's kind of like get fork on steroids meaning you kind of pick a template whether it's like react for example Give it some details and it'll fork your repo and then also set up an arbitrary number of steps as you go Another important thing to mention is that backstage is a framework It's not a platform meaning you'll need people with react experience when you're creating Plugins and extensions for this. It's kind of like a create react app for developer portals So what you're looking at here? It's called runway and it's American Airlines version of backstage They do a lot of presentations on it. It's really cool And you'll see that it's very differently skinned than what the Spotify backstage looks like Again just to emphasize that backstage is a framework not a platform. This is a documentation on how to add OIDC based sign-in into the application. It's not setting some yaml It's actually importing a react component and then placing it into your application And again backstage is a framework not a platform. This is the plug-in store on backstage Not only can you consume plug-ins from the community? But if you want to extend backstage the main way of doing so is through these plug-ins a big thing that I've seen talking to organizations that use backstage as they really promote something called inner source which is Engineers contributing up to the platform team to either add a plug-in for their specific team or make a change to An existing plug-in that they'd like to see So on to kind of how organizations are actually using backstage this comes from conversations I've had with with dozens of different backstage users in various stages and What I've seen for a good backstage deployment is it is a small focused core team that that manages this This typically involves one product person again Who's thinking who are our customers? What's the MVP? What's on our roadmap? What rabbit holes will we kind of save for for another day as well as an engineer with with react experience someone with infrastructure experience? That's kind of the three that you would need that could be one person if this person's a jack of all trades But really have seen that backstage teams are about six to ten people The the thing also that really makes a good backstage appointment is important services are actually being cataloged It's a service catalog, but if a critical application goes down, and it's not in backstage. It's probably not a fully mature backstage deployment at that point Third thing back plug-ins are added based on need again. This kind of goes back to the The product mindset you really have to survey and understand your engineers maybe if you came from a the engineering or it might be a little bit easier, but You really want to be able to add plug-ins that save time for developers Some plug-ins have seen work really well our infrastructure provisioning ones Maybe one that would give each developer a namespace for example or replace kind of a service catalog So I've seen people using backstage to request to do laptop and for example And then the fourth is it's kind of I should have added an asterisk here developers use it daily I think it should more just go to developers use it again If you create something and you don't have metrics on whether people are using it or not or you don't hear about it much It's probably not super successful Expedia has a really cool blog post on on this backstage blog talking about how their engineers use it and One of the metrics said that 4,000 developers use it for over 20 minutes a day So that's I thought that was pretty cool And on this kind of fourth point as well backstage has a discord and there's a channel in the discord called adoption And it's kind of for organizations who struggle kind of getting getting adoption on onto one of these projects On to a great backstage deployment though And this one is going to be things that are significantly more tricky and significantly fewer organizations have been able to reach this at least from the conversations I've had First this leadership gets it. I think this is really important American Airlines talks about this a lot They actually had an organization-wide okr around inner-source contributions and and being and making sure that everything's catalogued in this portal And again, I think if that's really to have leadership by and it makes these projects significantly more successful second is data-driven metrics The the third is true self-service provisioning So backstage can catalog with 30 exists out there But if you can't provide a golden path for developers and this is a very common term in platform engineering right like a golden path For when a developer needs to create a new service Then all you can kind of do is wrangle all the various services out there and try to consolidate it as opposed to creating a Pathforward for people to create new things using backstage And the fourth this seems to be a very popular metric for teams managing backstage, which is contributions This could either be people contributing to backstage plugins or people using backstage discover other areas in the the business that they can Contribute to outside of their kind of core function. I Broke this into stages to the first stage is installing customize and again I think this is a good opportunity to bring in this this product mindset Really to I think it's really easy to kind of get stuck in this phase and get the theming just right to line with a company brand guidelines or to create a plug-in for a very specific use case where I Think it's easy to go down rabbit holes before actually rolling it out to your users and using it The second is to catalog the status quo a lot of organizations talk to me about how many services They have cataloged in backstage and I actually don't think that's a good metric I think it's more kind of important to understand what like kind of Important services are cataloged as opposed to like the percentage of total services. It's someone's like we have 300 services in backstage again, if a critical production-facing app isn't there then It's probably not a complete Like it's snapshot of your service And then the third is a developers toolbox and this is essentially relying on the flexibility of backstage to be able to add different plugins and Tools into backstage to help developers with other parts of their workflow whether it's CI pipelines or Infrastructure provisioning or kind of keeping track of other tools in backstage again backstage is essentially the create react app for developer Portals so the platform team can really add anything in that that needs automation for developer productivity So this is data from an independent survey from slash data and it essentially covers the average time spent By software engineers in their day And backstage is a really great job at Covering many of these these are the things in the software development life cycle around planning maybe making sure it's something's deployed as being successfully monitored Making sure deployments run successfully Operating prod code as well as kind of having the proper security scans Visualized in backstage. However, you'll notice what's missing is where developers actually spend the majority of their time Developers spend the majority of their time in their editor whether it's setting up their dev tools writing code Or even managing code.  So that's where coder comes in it's an open-source platform for cloud developer environments It helps we see people using it for several use cases The first is faster onboarding meaning it takes on average about a week for developers to get their environment set up If it's completely automated away, it could go down to count to like an hour The second is consistent environments This is pretty important if you have a complex microservice mesh and if you're running code on your laptop You will have to modify or mock a lot of the data that you would typically be running in a data center on your laptop to get Things working when you're creating a remote environment. You're able to kind of give developers a full environment that has production parity The third is to be able to secure your source code And this is important for a lot of customers that run in Regulated environments where source code can't run on laptops or it has to be severely audited and laptops are bogged down making it very difficult For developers to get their work done. So what we see really is People will download and use our our project for kind of some combination of two or three of these things Coder has two main features the first is templates These are essentially blueprints for workspaces They're written in terraform or this is an open source conference. So open tofu and you can essentially Write any resource in terraform as a template So you could give each developer a pod or you can give each developer a VM and again this can run in in any Like on-prem or cloud environment The second is workspaces and these are individual environments for each developer This is the kind of coder UI on the left here and the developers able to pick a project and Set those those values and you are able to do things like vm and emacs as well So this is actually what the the coder workflow looks like I think the term environment gets conflated a bit this on the right is vs code And on the left is the coder UI and you'll notice in vs code If you're sitting close to the screen that It's a it's a Linux terminal and the file system is remote to that of my laptop So essentially i'm able to provision this remote container running Click the vs code desktop button and and get in and it already has my repo cloned and it also has my project setup So kind of back on to the the good and the great A good coder deployment is very similar to the kind of attributes of a good backstage deployment The first is a small focused core team. You don't necessarily need people with Software development experience, but you do need people with infrastructure experience So you'll need someone who knows terraform you'll need someone who understands cloud infrastructure And you'll also want someone with that kind of product management mindset Again, it could have be one of those two people, but it's someone thinking who are target users How do we want to use this platform? And and a good coder deployment is similar to a backstage deployment is great for spinning up a new Project and it also has patterns in place for people who want to bring a project that they're doing locally into coder For a for a faster or more secure experience Um, again for a great coder deployment. I think it's really important that the leadership gets it They understand the value props. It's significantly easier to kind of justify things like this if If there's buy-in from above Another good attribute is it's integrated with a golden path. So if you're trying to promote for example dev ops or or cloud development or An existing kind of pattern in in your organization with with backstage yourself service It's it's pretty great to be able to incorporate coder into this existing story And and the third point which i'll talk about a little bit is teams can bring their own dependencies. For example, I need to use python 3 java 11 and ruby for example Into coder and coder knows what to do with it and build it and we'll talk a little bit about how that's that's possible down the road So to break coder kind of into three different stages The first stage we typically see our users doing is these sandbox environments These are environments that come with a bunch of tools pre-baked in it Someone can click a button get a workspace and start tinkering with docker or react or terraform But it might not necessarily be ready for their actual day-to-day projects The second stage is is developer teams and this is essentially kind of Done in part with the sandbox environments. You deploy sandbox environments people say Hey, I'd love to use this for this project and then you kind of do this this enablement model to help get Coder kind of configured for that and and then the third kind of stage is this bring your own tools approach where Engineers can go into the platform. They can say I need these specific languages in it It'll be downloaded from a secure artifact store. It doesn't have to be downloaded from the the internet And developers can kind of develop that way So going back to this chart, we know now that Oh interesting We know now that coder backstage makes a great internal developer platform coder makes a great cde And now let me just go ahead and give it demo So to integrate coder kind of into your backstage golden path. We have an open source repo. It's just on I'll share the link after and like most things in backstage you have to do some Coding to get it imported in it's relatively easy. You add the package And add a few kind of lines in so here I have a deployment running and This is a relatively standard backstage deployment the only plugins that I have are the the coder plugins And you'll notice this looks like a pretty normal backstage plugin It has automatically detected though that this project is compatible with dev containers And I'll explain why it's a little bit important later And the the other thing our plugin adds is this coder workspaces panel on the right From here, I can create a workspace Takes me directly into the coder platform and what it does is it starts building a Project encoder based on the specs in my get repo I am I have one running here. This is the the same project And I can click one button via sco desktop. It'll open my desktop editor And create a tunnel over ssh into the workspace You'll notice it's automatically cloned a repo If I go into the terminal And I can make this a little bigger It has python installed if I needed node. I believe the image has node as well Oh, I think it has yarn Yep, it has yarn as well and I can start my project. So I'll go into the project yarn dev Flask run it's a python project And get in and start coding So I did not have to set up any tools on my local machine besides having vscode running And I'm able to connect remotely into this workspace and start working on a project Just to kind of prove this is a remote workspace. This is uh through the coder extension It's showing me that I'm connected peer to peer to the to the workspace And if I just run a uname as well, I'm in a linux environment as well Another thing coder offers is a A web based experience as well. So technically someone could connect through a chromebook or an ipad the chromebook experience is pretty good The ipad is it's all right And through the web browser the person can get the same vscode experience with their repo cloned And can be ready to go If I go back into backstage It has detected this workspace and if I wanted to create another one Perhaps for a second environment or I'm working on a new project I can just hit this here and it spins up another one of these cloud environments The other plugin I wanted to show is if you didn't want to use coder at all But you still kind of wanted this magical one click into a into a project Without something like this. It's typically a contributing md or some wiki where it's 10 to 30 steps on how to set up a project for an engineer It probably only works on one operating system meaning if you begin supporting Maybe you moved from intel to m1 It would be a whole new set of instructions if you moved if you had a developer using linux or windows It would be a totally different set of instructions So again, there are a lot of benefits to having this instant kind of containerized dev environment experience And if you didn't want to use coder We created a plugin as well that uses the dev container spec To talk a little bit more about the dev container spec I'll actually go into the repo There's a dev container folder here And in a dev container folder there's both a dev container dot json and a docker file The dev container spec is not something that we came up with it's not something backstage came up with It actually I believe was created by microsoft and they have a local extension for it as well and it's also in code spaces But it's essentially explaining kind of what tools I need for this project So to each developer doesn't have to run one of these It starts from the python space image It'll install the the editor And it's also installing two pip packages So this is quite a simple example, but you can imagine for more complex projects The dev container spec can still scale and install multiple languages multiple tools. It's essentially a docker file Dev container spec also has support for something called features. I don't believe I'm using any of those now But features can be used to install kind of sidecar processes and mount them into into coder Because it detected it has a dev container spec I have this open in vs code local button And what this does is it takes me again to vs code Gives me a pop-up And it's using the dev containers vs code extension to actually run the same exact kind of environment locally So let's see Let's close this out and do that again. Cool. So we can see now it's no longer using the coder extensions using a dev container extension But again, if I run uname It's a linux environment I have python installed And I can do my development this way as well So both of these plugins are open source. You can install them into your backstage deployment today And it's a great way to kind of add these extra steps to your workflow Really solves tools the problem of developer setup. Let's go back to slides Cool. So, yeah with that you can use backstage as an internal developer platform coder is a cloud developer environment And there's kind of a lot of things that these tools do differently But there's a lot of kind of concepts that they solve together Which are kind of around onboarding context switching and provisioning environments So with that, thank you. I think there's plenty of time for questions And the qr code takes you to our backstage plugins repo if you wanted to download or check out one of these one of these plugins Thank you. Thank you Ben for such a great presentation You definitely have time for questions. So if you have one, let me know and I will bring the mic to you No questions Anything else? No, cool. Thanks everyone. Thank you. Thanks everyone The presentation will be on the scale website soon as well as the replay on their youtube channel. Thank you. Bye. Bye Okay, all right. All right. Let's uh get started. So this session is about um Well, it's a little bit about uh cloud stuff a little bit about java and a little bit about serverless so Um, you know how the cloud is, right? It's like ooh super cool the cloud You can deploy your applications on there and it solves all your problems, right? Of course not. Um, so anyway, my name is Kevin Dubois. I'm a developer advocate at red hats. Uh, I'm based in belgium And I have We have another speaker that's going to be next is also from belgium, which is a pretty random Coincidence because we're a tiny country Um, anyway, so um, yeah, I talk a lot about cloud native development and Also about java. I happen to also be a java champion And uh, yeah, if you want to follow me and I'll share those at the end too That would be also really cool Anyway, so cloud computing, right? It's uh, it's nice because it helps you to uh respond more quickly to demand, right? You can just Create new environments pretty easily. You don't have to worry about, you know managing the hardware, which is convenient Uh, you get stuff like high availability and disaster recovery and resilience and Um, it basically allows you to grow your applications kind of in a manageable way, right? You don't need to create tickets and provision new machines, uh in your um in your data center and stuff like that And you know in theory at least, uh, you would use only the resources that you need, you know, maybe You might save some money. You might save the planet, but maybe not Um, anyway, so but if you want to create an application platform for the cloud Um, you need to build not just your application, right? You need some stuff around that application to make it You know manageable, so you need like some developer tools You'll need some way to automate the deployment with cicd. You'll need monitoring Logging, uh, you probably will need a container registry if you're working with containers um, you know, so We have to take that in into account too. So Of course, fortunately all these cloud providers are really nice. They provide a lot of these tools for us, right? So if you go to aws, you'll see a whole bunch of different services. This is just a subset Um, that'll allow you to you know, create really cool stuff With your application and you know, you can create databases and messaging systems and storage and all that good stuff Which is cool. And it's not just aws. Of course, it's Also, you know any kind of cloud provider that's out there, right? So azure Um google google cloud and yeah, I have to mention ibm cloud, right? I'm from red hat and they're kind of our corporate overlords. So They also have an ibm cloud Um, but you know the the the thing is with uh with these cloud environments is that What if you go in with one cloud provider, you know, build some really cool stuff and then um You want to switch to another provider? It could be for many reasons, right? It could be because there's regulatory changes like for example us in uh in europe Sometimes the eu says that you know, the data actually has to stay within the eu or here in the us Maybe uh, they'll say well data cannot leave the us or it needs to stay in a particular state or you know There's all sorts of weird changes Sometimes it uh politicians come up with for good reasons or for bad reasons. I'll leave that in the middle Um, but it could be you know, maybe there's some outages with the provider that you're with. Maybe they change their prices Uh, maybe the other vendor offers Better resources, you know, like maybe in this day of AI maybe, you know, some other vendor offers much better gpu's or maybe they're Um, you know a lot cheaper, you know, there can be many reasons a very typical one that I also see With uh with some customers is a new cio comes in and all of a sudden he has Uh his uh golfing friends from another company and he says yep We're going to switch from x2 to y2z or whatever Um, and then also kind of shadow it right so some developer teams They might be a little cheeky and they go and build their uh software on a different cloud and then you know You need to kind of converge or or not. So There can be many reasons why you might need to Uh work with different cloud providers, right? So you might have some workloads on a aws and with azure some on prem Because maybe you don't want to run it in the cloud and uh, you know, then comes the pain of lock-in, right? So uh, yeah, you've built everything with for example a aws or azure And um, of course they make it so that everything is of course a little bit different between the different Cloud providers making it a little bit of a pain to uh move and a little bit might be an understatement Of course, so you have to be careful when you go into the cloud and uh, I really think about, you know How do you build your uh your applications and what services do you do you need and which ones? You know, could you maybe replace so what is the up? What is the solution then right? Is it hybrid cloud? Is it multi cloud? Um Well, it could be you know, you could have you know some team working on aws some team working on azure And and but you'll see that the differences between the different cloud providers requires, you know expertise and each one of these different cloud providers Um, and that's you know, unless you're a very big organization and you're fine with that It's not so ideal, right? So it's like, uh, I want to go to the cloud but So fortunately the open source communities are are great and they've come up To rescue us if you will, right? So we've seen the advent of uh containers and kubernetes and then a whole Massive ecosystem built around that um, so I don't know if you've ever been uh gone to the cncf landscape and there's like a whole bunch of different services Um, but those services also include proprietary services So what I did I went to the cncf landscape and I actually filtered by only uh open source Uh community projects, uh, and then you end up with a much smaller list here, right? So there's still a fair amount of projects, which is the good news And that means that we can leverage these projects as open source community projects that can also be supported by a vendor But that means that we can use them on any cloud in the same way together with kubernetes and then I also Put one particular little project in there scupper, which is an interesting one That allows you to interconnect applications between different providers On a layer seven network So a little bit higher up so that you don't need to You know start figuring out how to build vpn connections or dedicated networks between different providers So scupper is kind of an interesting project, but that's a little bit of a sidebar So what about serverless? Um, so serverless Maybe I should begin with explaining what what serverless is, right?  It does mean that, you know, they're kind of abstracted away, right for the developer And the idea with serverless is that you auto scale based on the demand of that moment So if you have a lot of requests coming in you create a lot of resources And if those requests go down then you have less resources And the idea is that you get kind of build based on the exact usage of of what you're using so Why is serverless interesting? Well, this was a survey that the cncf did recently And they were asking what are the factors that are leading you to overspend and looks like 70% said over provisioning is a problem Of you know why they're overspending and then another one is sprawl such as resources not deactivated after use So they just you know sit there consuming Energy and you know costing money and then you see fluctuating consumption demands. So where you know you get Sometimes a lot of requests sometimes not, you know, it could be you know, because it's black friday or because you have you know, somebody calls Your service from a batch script or something Or just like poor planning and prediction on cloud consumption. So, you know, like But it's hard. It's hard to plan out how much your your applications are going to be used, right? So Those are actually all good reasons to consider serverless, so Of course, I have to put in some some graphs that makes me look fancier. Doesn't it? No But you can see here. This is an example of a traditional non serverless deployment and you see They have over provisioned for most of the time, you know, they have a certain amount of resources that can be containers or pods And let's say that they have 20 pods running to handle the traffic just in case that there's a big spike in traffic And then you see yeah, there's a spike and they can still barely handle it and then another spike And it actually is too much for what they've provisioned and of course, you know Some users are not going to be happy because they're not going to get to your service, right? So that's not so good. So even though most of the time they've over provisioned They're paying for what not what's not being used at some point. It's the opposite, right? They're not actually using or have enough resources so the difference with serverless is that your Your applications your resources your pods are going to scale exactly to the demand of that moment And scale up and down and you see like the third arrow there points to zero, right? So if nobody's using your resource if nobody's using your services in a particular moment, maybe at night nobody's Needing to go to your services. Well, then there's actually no resources being used. So that's kind of the the interesting part about serverless so Of course, I'm a big fan of java. So What about java and serverless because Java traditionally is not so fast to start up, right? So the idea with serverless is that we scale up and scale down based on the demand very quickly And then yeah, if I think of my java applications from, you know, five ten Of 15 years ago, they did not start in in milliseconds. That was uh, it was probably more like minutes And that's yeah, nobody's gonna wait for minutes, right for your for your service to come up Now fortunately, um, there's some uh, there's some Nice changes in uh in the java landscape, but here we can see an example of An application on the jvm that uh, that that's supposed to scale, right? So you see the blue line Which is basically The load that's going up and you know a new container a pod gets created Um, and the application might even start up and say that it's ready to receive requests, but the jvm is Something interesting because it