the split task in the meantime I can actually show you the Argo dashboard and where is the Argo dashboard? So if I refresh this page you'll see I have a distillium sort job that started did the split step now generating some numbers from zero to nine and the next step would be starting 10 pods so the same image I showed you that I was doing from job to job in Argo you can see like a nice little dashboard that is doing the same exact thing so it's running 10 different sort operation after they're all done I'm gonna do a merge that is gonna join all the 10 files back into a single file again same object, same exact thing but instead of doing multiple run jobs wait for it to finish, run job wait for it to finish I can just let Argo handle a workflow that can handle the entire thing as a single task so all of my sort is going on I'm gonna let this finish I have about 10 minutes so I'm gonna move on to the last bit of the talk and we can take a couple of questions while this finish I'm gonna show you that finish in a second so we did all that running so running batch workload on Kubernetes at scale we just did it at scale so this is done technically but let's talk about more things design so number one things about running batch workload on Kubernetes at scale is if you can manage to design the workload as a loosely coupled workload that works best anything that requires a lot of network communication between each other you will never reach the performance of something like HPC on a bare metal with infinity band that is talking over terabyte of network no matter how good your network gets it will always be slower than physical hardware talking directly over like wire, right? That next point is network is not unlimited so there is a lot of workload right now uses MPI that doesn't need to just because they can use MPI they usually just use MPI parallelization is your friends if you can create parallelized workload that doesn't depend on each other which is oftentimes possible some use cases not possible but most 90% plus use cases it is possible if you can do it Kubernetes is great because you can get parallelized and scale up your cluster to absurd amounts and limited communication between workers scales best so if you don't have to talk to each other you can just do their work that's fantastic optimize so this is one of the things we usually talk about you know what don't worry too much about like byte level optimization don't worry about shaving milliseconds of your work batch is one of the only places I will say do that because if you can say I save 5% of time by writing my algorithm better that is immediately 5% cost savings because you're no longer doing that 5% extra work in Kubernetes you can do the work and once the work is done you can scale down the cluster to nothing that means you're no longer paying for that resource and finally try to run jobs in co-location if you have a choice to run the workload in the same data center that is better because even if you had to talk over network now you are just talking on the same data center instead of going across scale Kubernetes scale down when not needed that's pretty obvious in Kubernetes especially in GKE we have an option to set up that I only one minimum of one node maximum of 100 when the work there is no work happening your cluster can scale down to one like just like barely nothing there is a limit on how much you can scale up to in GKE we can go up to 15k I think open source Kubernetes generated as about 5k which is a lot most people never need more than like 100 nodes but if you are one of those very few people that needs to spin up thousands of node for example a company called Fijiast use GKE to build their supercomputer and then needs about 50,000 nodes so one Kubernetes cluster wasn't enough they use 12 different GKE cluster to parallelize their workload to get 50,000 plus nodes at their peak and use spot VMs because you want to save money you don't want to and batch workload doesn't really care about running on VMs that has like four nines, five nines SLA node goes away it goes away you can restart the job that's fine monitor of course monitor cluster research usage but the next point is important because oftentimes people are like I'm gonna trace every call that happens in a batch workload every time you do any extra task that you're not supposed to you're using more resources unless you absolutely need to trace your every call I would recommend for batch workload don't do that because you build that application optimize it to the best capabilities and then you look at the cluster level monitoring for the information individual traces is not gonna give you much more limits so understanding individual limits for your Kubernetes cluster so Kubernetes has soft limit for both pod service node secrets and there's an upper limit that's defined but that doesn't mean you can do upper limit of every single one of them if you do the most part in a node that means you probably won't be able to do the most secrets in the same node you kind of have like an envelopes type system where the truth is if you're running all the things at the same cluster your upper limit is actually limited by other things for example HCD is a big bottleneck for Kubernetes it has a hard limit of about six gigs of total memory and then for individual type of objects if you have thousands of pods I think HCD has 800 megabyte limit on individual object type so if you have 10,000 pods each with a lot of metadata you might overrun your HCD limit on how many new pods you can create fault tolerance assume everything will fail that's just basically the fault tolerance guidance it will fail so you have to build your application in a way when it does fail you have enough checkpointing so they can restart back without having to do the entire work over again so as you're building an application batch workload you have to rethink rather than saying oh I am gonna write the best application I'm gonna write the best application that can fail but still do the job so last only one slide about GCP and GKE some of the things we are doing here I think that is gonna make running batch workload on Kubernetes a lot better is we have a node limit of up to 15,000 if you're one of those few people that needs it we have it manage from it yes with zero hassle we can just have manage from it yes collect all the information without you having to handle from it yes and scaling from it yes all that fun stuff and managed from it yes and Google Cloud uses the Monarch database that the same database we use for monitoring things like Gmail, YouTube, Google Maps which are fairly big systems compact placement so what this one talks about is in the same region you can ask your notes to be scheduled on the same zone if possible and if the zone goes down your cluster will be rescheduled within the same region on a different zone so usually when you do the same region your nodes get scheduled like round robin so one, two, three, one, two, three with compact placement you are asking Kubernetes GKE to put your nodes close to each other that way you can get the co-location benefit and the regional benefit at the same time time sharing GPU GPUs are expensive but if you're doing workload that requires GPU you can time share them and save a lot of money image streaming so if you're doing batch workload sometimes the images are really big I have seen up to 22 gigabytes of image sizes so with image streaming you can start an application without having to have the entire image downloaded so Docker images are usually layered so smart people at the team somehow figured out how we can start this container image without having to download the full image you can just start as it comes finally cost allocation if you are running a GKE cluster with multiple tenants you could have cost allocation that kind of you can do charge back and figure out which team use how much resources if you're having a shared instance you can do some cool things with that, thank you so much I'm Mofi Kursar the internet you can ask me questions and you can ask me questions for three minutes oh yeah, that also finished so while we're talking about it any questions? and it's for questions so I've got the mic and I'll bring it around I saw him first and then I'll come back to you and yeah, if we can finish it within the time we don't have any talk right after this I'm gonna be outside or hanging out here we can talk more about your batch workload if you have any I'm interested in learning more how people do their things question yeah, for the queues is there a way to configure priority and preemption and that kind of thing between the queues so it's being worked on so right now the idea is just like the best effort first in first out queue mechanism we're working on preemption right now it's on the roadmap questions, I think someone was here it's not centered to the queuing but when you mentioned the monitoring you mentioned that Prometheus endpoint I probably thought it wrong but I always thought that that was limited in the amount of data you could send it to it was like only for like edge cases but I might be wrong um, can you repeat that question again, sorry you mentioned that since these jobs are ephemeral you cannot scrape them you cannot have guarantees for scraping them so you were sending them to that endpoint oh, you're talking about the push gateway yeah, the push gateway yeah, so yeah, I mean the idea is when you're running batch workload you don't want to send all the metrics because again anything you do that's an API call that slows the thing down for the most part one of the key things I want to see on my batch workload is how much resources it's using currently because on programming languages you can check your garbage collector and other things just send that information because based on that it can resize your job limits later so other than that knowing how long the job ran it's not like a job's responsibility to send that information back out so I wouldn't care too much about that part yeah, any other questions? Giffy, you said you had questions at the end come on we have one minute, we can do it, let's go also the next talk in here is until 4.30 yeah, so we have sometimes we can go a little bit over, I think okay, um, sorry we know each other but we don't know each other well I don't know if you know like Bob's in my background was HPC workloads at the University of Michigan so I have a opinion yeah um, within Q are you looking or is the work being done on any additional schedulers? or no, I shouldn't say schedulers Q types so like yeah, first in, first out and then you have best effort but like those aren't necessarily what most people in scientific computing are expecting yeah, so you can handle that situation in two ways one is if the workload is that different that they shouldn't share the same Q you can have multiple Qs running on the same cluster you can have a cluster Q that is only dealing with GPU stuff on top of that, for the same type of workload for the most part, the best effort and the strict covers 80% of these cases and the other algorithms that we need to handle we're working on those okay, so I'm thinking like imagine you have 100 academics there is a Q or like several Qs that just several months worth of work are these schedulers or these Qs looking at packing as efficient as possible because I know like default Kubernetes scheduler not really good at that yeah, so Q is not a replacement for a scheduler it's a Q that utilizes the Kubernetes scheduler right plus on top of that, you still have auto scaling so if you all of a sudden find out your Q pressure has gone to a certain amount you could ask underlying infrastructure to scale up more so that way if you see that your Q size has grown to 100 we have knobs, it is not implemented yet it's because it's not implemented yet it's being implemented but if your Q size crosses a threshold we could tell the underlying infrastructure to spin up more resources how much thought or again I haven't dug into the Q or batch stuff oh, if you want to learn more about Q that's the link I didn't give continue asking the question so in terms of like data partitioning and keeping like QA and QB separate are you just using the namespaces are you using namespaces to partition people submitting different types of workloads an example could I using Q put like KIPA workloads in a namespace and be like these are safe or so Q doesn't really take care of your actual isolation on the namespace itself because anybody on that namespace will be able to see any workload on that namespace anybody who has permission to see things what you can do with Q the isolation is based on on the same namespace you can have five different local Q each of them concerned about different types of resources okay so it's yeah Q I don't think we undertake the responsibility of actually data isolation on the job level okay um I did something okay there you go I don't know what this is oh I think it kicks over to that automatically so we can see who's coming in next awesome any other question I'll be hanging out here for more question JJ has a question what about free BSD was the question well it's free I don't know what he wanted me to tell you um alright so yeah thank you so much for having me and thank you to the organizers for giving me the opportunity to talk to you so appreciate it we'll chat after everything is done yeah testing oh I can hear me that's good really it's time for us to get started so welcome everybody welcome everybody um I to the cloud native track or welcome back to the cloud native track we're gonna talk you've heard if you've been in the track for a while you've heard a lot of talks about running cloud native stuff on Linux um so we thought it was about time for you to hear a little bit about running cloud native on a different OS um free BSD um and this is gonna be presented by Karen Brunner um who I've known for quite a while as fuzzy KB um and I and she'll show us that including uh some demos sure thank you so welcome Karen thank you thank you Josh um how many people here have touched free BSD done anything oh almost everybody oh this is a lot more fun when nobody no not really um how many people so gonna I usually ask how many people run a Mac have Mac OS 10 and then I tell people that you're running free BSD sort of so so actually almost everyone in Silicon Valley has touched it in some respect or another um free BSD a lot of the for Mac OS 10 they took a lot of the free BSD user land and some of the the kernel and built it into Darwin OS 10 so I'm gonna talk about cloud native free BSD um and again most of you have touched free BSD so I don't maybe for the audience I will still continue um free BSD is of course an open source Unix like operating system um it dates back to uh rewrite of original Berkeley source distribution or Berkeley system distribution um they rewrote it to get rid of all the AT&T proprietary code because of licensing reasons um but that happened in 4.4 BSD light about just over about 30 years ago um so free BSD is built on that so are a lot of other other BSDs like open BSD and net BSD and works of those uh so why are we talking about cloud native BSD free BSD well it's kind of to probably talk about what cloud native is uh and this is one of those definitions that not as much as DevOps but one of the things where everybody's gonna describe it differently but it's basically higher level it's uh applications which were built to run on clouds which were built to run on you know uh cattle instead of pets you don't have these nice little servers that you know by name and they all have their little quirks and you can all say hi everything you're agnostic cloud agnostic you could scale you can do um a lot of things you can really do on hardware um so more specifically we're going to talk about one of the one of the most or at least most popular patterns is to use something like containers you know these little bits that you run on your servers and you can run lots of them and they all act like little servers uh you need to orchestrate these so you use something very often like Kubernetes there are other iterations uh there are proprietary you have AWS ECS their elastic container service which they use internally um OpenShift is Red Hats branded Kubernetes uh you have Nomad from HashiCorp which is similar but anyways there are a lot of uh we're gonna lean we're not gonna be too specific about which orchestrator we're gonna use but um so that's what it kind of means by cloud native 3DSD instead of running on Linux nodes we're not gonna talk about Windows nodes either instead of running on Windows instead of running on Linux you're gonna have uh how would you accomplish that having these little mini servers that are easy to bring up and tear down how would you do that on 3DSD so I'm gonna do a brief comparison to Linux um this makes we are at the uh you know Linux uh um exhibition uh event so I will at least throw some Linux in there um first thing is Linux is GNU Linux 3DSD is not GNU 3DSD um 3DSD is using the original BSD user land so PS options said all these things that you use on Linux which act in a certain way those are actually GNU user land options and they're very different especially if you have a Mac your Mac is using predominantly the the BSD user land so PS takes different options said doesn't work like you expect um a lot of these so you can install these easily on 3DSD you can install GNU-SAD which I end up having to do you can install GNU-MAKE GNU-MAKE which you pretty much have to do if you want to build anything but um this also gets into it's it's sure it's confusing to people who go back and forth maybe are not as familiar but if you run if you've been running on a Mac you're you're probably pretty familiar with those differences um let's go back to licensing um GNU is using or Linux is using the the GPL the GNU public license and this is kind of interesting it caused a lot of issues as far as embedding Linux and devices for a long time um Friest uses a BSD license this is extremely a very open license as long as you maintain the copyright you can pretty much do almost anything with the source code um can't necessarily merge it back into Friest for BSD if there are proprietary options but as far as using it for your own purposes it's it's it's very open so it always confused me a little why Linux became so much more popular when it had such a problematic uh licensing in some respects but anyways that's one difference they both have cute mascots um the BSD is a little cuter but they do both have cute mascots uh Linux uses systemd to manage orchestrate processes on a server um I'm not a big fan of systemd I don't know who you are so it may or may not be a feature that Friest does not have systemd system or Friest is still using the um the old RC I don't even remember what it stands for anymore so uh configuration um it's this is a little dated it doesn't scale very well their conversations about making it more manageable similar to systemd I'm sure some people would just want to port systemd to Friest I hope that does not happen but um so systemd is kind of overkill from my point of view and Friest's RC system is at this point underkill file systems so Linux at this point I think most distributions default to ext4 which has been around about 15 years now um it works okay uh Friest this always wasn't I haven't asked or scared but ZFS which came out of Solaris is first class citizen you can run it on Linux it is not a first class citizen you can even run root on ZFS it takes a little extra effort uh but it works um ZFS is it is very reliable scalable uh high performance and you have full features like snapshotting you can do you know copy on right so you grow your you can make copies of your little file systems and grow them which is really useful when you're running say a containerized option so you have a file system and the file system doesn't take up much space as long as you don't you know as long as you don't diverge too much from the original base image um I won't go into all the ZFS features I've I've forgotten them all but it's it makes stuff just kind of work in a much easier way instead of using something like LVM I don't even know what people still use different are people still using LVM and Linux anyways um so Linux kind of cool previous use super cool virtualization so hardware level virtualization you can use KVM on Linux the kind of analog on previous use Behive they both are using virtualization hooks in your CPU but it's just a different kernel implementation Behive is really nice I've used it it's great KVM I've used it pretty good but they look this you have a fully virtualized machine so we're getting closer to the cloud native bit um so obviously you have these servers and you have lots of cores now and you want to instead of having servers with names you want to run a lot of little discrete processes you probably want to keep them kind of in their own space so they don't eat up all your system resources so they don't you know write to parts of the file system you don't want them to write to so they don't hijack your network whatever so Linux does this with mostly with C groups control groups which runs in the kernel it you're able to assign processes to control groups so you can say this this control group you're limited process of limited to one CPU and maybe two megs around before the out of memory killer comes in namespaces are they're different but they're grouped in what we think of containers also uses namespaces so you can namespace your network and keep it partitioned from the actual system network previous day has jails and jails predate Linux C groups by a number of years I don't I think I want to say jails appeared in 1999 ish yeah C groups I think were merged into the Linux kernel in 2007 so jails work they were containers before there were Linux containers where there were Docker containers jails well I'll talk a little more about jails has its own section so I can't draw this is a AI rendering of the previous the Damon and jail anyway so previous jails are very very special they work originally they work instead of on a process level control group they work on a shrewt change route so your root system is not the system slash you move it to something else and then limit the process to access to that or to that part of the file system so I'm going to do a very quick demo to show you what that looks like maybe and I'm going to do this horrible thing and you're all going to judge me which is I understand okay I'm going to make a new directory called so let's say let's say I just want to maybe run you name in here so I will make this and I will copy into my oops I did that wrong I should not talk in type at this time okay so shrewt is really easy I run the process and give it my path and if I just give it a command it won't try to enter it'll just well you'd still give it a command but let's say I just want to have it run you name okay it gives me an error because the name is dynamically linked it wants it wants this yaw library so I'll just do this thing and then I'll copy again okay so we'll try this again share a library okay so I could go through this forever so I will instead do this um this is all part of the demo I promise um I can't do this because there is no bin of sage in my shrewted file system okay well jails are a little more complicated than that this is the basics but um you know I can do I can't really network out of them um not easily and they're not very isolated they're not very secure because I can do um if I say if I copy the ps very or uh binary into yeah so I have to mount the um mount the files which I did I write this I have this cheat sheet of commands to um cut and paste um and right now then if I try this again I can see every process on the server uh which is probably it works as long as I don't really care about security or as long as I'm not trying to do anything weird to the process table because it's probably not what we want so concept of jails came about uh in one sense to make this a little make jails much more much less leaky um so now let me and okay so they still they're still built on on shrewd um oh and I am demoing okay well I haven't demoed in jail so you saw the problem was statically linked you know these are dynamically linked binaries this would all be much easier if they were statically linked I wouldn't have to copy all these libraries over there are a lot of things like that it's uh you can do that um but if you want to just be really quick and well not quick but completely you can just kind of install all free BS or yeah all free BSD in in your files in your jail file system um I will not run this well I don't know what the network is going to do but uh in theory I could just do and I actually can't just do it um you know and I'd say this nice jail file system I would say my new jail well I actually already have something this would by default by default it goes on to the network you give it a mirror to go to and it will look and do the standard BSD install from it it will download your your base um as a tar file and I've done this so I will go to and I'll treat it like it's a an actual full server um and it's like the root file system on the server I have other junk in on the actual laptop root server but it's just a standard it's your standard file system um so I've also done some other stuff here uh chchchch I have I will show you so the jail stuff is built into the previous decolonial you just have to load the enable it um I have done this and I've actually created a jail this enables my jails it's going to clone use a clone interface which I won't get into but there's weird networking stuff um and I'm assigning it in the in the the loopback interface loopback one interface is going to be in the <IP_ADDRESS> subnet so now I have to actually configure I have my files I also have to tell the jails what it's going to look like as far as I give it a hostname I give it an address uh I give it the path to you know that I created my file system um I tell it to mount devfs we saw that earlier if you try to look at the process table or something you need the you need the file system and then persist which just tells it I want it to keep running um you can also do something where instead of keeping it running you can give it a single process to it and this is really great if you're using a web server or something where you're not going to go in and run a bunch of commands you just want the service to be running it's it's more efficient and it's also more secure so I will now start my jail yes um yes I had I had a demo with network collisions before but yeah I have the bridged I have it bridged I haven't shown the bridge um I have this nice bridge which I have other stuff on but it's yeah it routes through the um and I should have picked a shorter name but there we go oh yes I have to how did I do that I don't remember how I did that last time we also go back here uh da da da yeah I just need to create it and then give it the another difference I don't know if you notice this but most most Linux distributions moved from ifconfig to the IP suite um previously still uses ifconfig which confuses me less you have not been doing this for a long time okay there one okay now I'm trying to start my jail again yay I have a jail so uh I can look at the list of jails there's my jail there's the address I can now run something in my jail um do something like this hey hey that looks just like well except for the name the host name here that looks just like my uh my current system and this is also something very similar to my containers you're not running a separate kernel it's going to inherit the kernel that the system is on um same thing in a Linux container you're not running you can have a your container can come from an alpine base it can come from an Ubuntu base it can roll your own the kernel in there is not going to get run so I can also because this is a full install and ooh that's money so but it's still a file system and this is useful like you seen services um services kind of the analog to the system d control if you use Linux um it's it's jails are first class citizens and and 3dsd both they've been around that long but they're also really really useful um great great uh so it's yeah this tooling it's it's made to we saw bsd install you don't necessarily need to run it for a jail it's what you run when you install um it's directly to hardware or directly to a virtual machine but it's it's got I just pass the jail argument and it says hey I'm installing a jail um service service usually operates at the level of your laptop VM whatever whatever you're running but it takes the jail option and it knows that um well this is a jail start it knows that you're running you want it to run a process in a jail instead of so it's it makes it because it's still a file system though it makes it you just stop the jail and put some stuff on there unlike a docker container well you could still do that it's kind of ugly but where you you don't want to run app get inside your container you want to run it in your container when you build your docker container um but this makes upgrades easy this this system is set up to be a first class citizen um but I can also do stuff like stop the jail and if I wanted to I could just uh instead of you know pulling in a file from externally um I could run it but oh I didn't um just don't remember if I did this right oh yeah I got the uh results um it's actually you can limit you can limit the network um this is sharing the host network directly uh you can use something like vNet or some other virtual create a virtual interface which is going to complain and secure your network a lot more it's going to uh limit it to its own network and it's invisible the server will not be visible the server network um okay so that's a jail basic uh very simple um I can stop the jail and drop the file in there just directly from my host cpe file into jail file restart it it's it's there okay so what do we mean when we say making previously cloud data well it already has these containers these semi virtualized so processes so why do we want to do this so we'll talk about what the standard for a container is um these are all standard there's this cri container runtime interface one for network one for storage and this is a specification about the plugin if a plugin is going to call itself a run you know a cri compatible runtime it needs to run you know these functions we hate this way this needs to these methods need to exist you can run the you can create these in any language as long as the specification as long as they take certain arguments and behave in a certain way give you certain outputs so you have the runtime you have the network and the storage so the runtime network and storage are the names are fairly obvious but what do we mean when we talk about runtimes um so runtimes basically are run um when you tell it to run my container it creates the container uh the running processes um assigns whatever you know if I'm going to run a network uh if I want a network on my container it calls the container network interface plugin um storage is the same way okay so what's next well there's the open container initiative and these are standards about what container images should look like they hate um this is you know uh docker containers docker containers are kind of where docker is not really a actual implementation of the container runtime but anyways uh your images should behave in a certain way and it doesn't matter if they're linux images or they're windows images or they're previous images they just need to have certain files in certain ways and be packaged in certain ways so they can be packaged in certain ways so basically what happens is you have your runtime which implements the container runtime interface you have an OCI tool and then you have the actual container so runtime we're talking about things like again docker engine is not quite right uh it uses it's no longer standard in Kubernetes but they actually use containerd as a shim uh containerd which is a nice run you can run containerd um I think uh red hats openshift uses CRIO there's something like podman which is also out of redhat um there are a lot of them but containerd is the new standard for kubernetes so and kubernetes or containerd has been ported to previousd which is nice so you can actually just run containerd directly on previousd works so well you need your OCI tool so you can actually which knows how to manipulate your actual container objects um on linux you usually use something called runc run container uh so if you were using previousd jails you'd call it runj and runj is an existing project it was created by Samuel Karp um it's got other contributors now but it it implements the OCI specification so you pass it a certain um certain options and it behaves in a certain way and passes ends up creating your your object um so it's obviously instead of creating containers we're creating jails uh of course if you were running this on linux you could still have containerd this would go to runc and then you'd spew out some containers let me show you something first um so I was talking about runj and I'm going to do a lot of copy and pasting because you don't want to see me type um so I'm going to show you how runj actually works this is and I have container I hope I have containerd okay so we're going to use ctr which is the standard containerd command line interface um I've already done this step but I'll show you anyways so uh Samuel Karp has built an OCI compatible image for previousd and he kindly hosted it on a public AWS ECR last container registry ECR doesn't care if it's doesn't care what the artifact is so it doesn't care if it's a previousd image um it's a little cut and paste so I've already pulled this image oh and my network isn't working anyways so that's fine oh no I see what I made a typo yeah okay copy twice yay okay I did already pull so that's nothing new to pull and then yeah I did not make another okay so now if I were running this as a linux command if I were on a linux system and I wanted to run a container very much I could use CPR if I were running containerd and it would look very much like this um the biggest I already have this running um what call this okay so so the command line here run I'm going to run you know my thing run time I'm telling it specifically to use uh run day this is the path specification I'm just going to handle on the whole system the storage um CFS is again talked about lovely CFS it's going to manage our snapshots for us it's um similar to I think on linux it's using butterfs alright but we can use CFS we can do anything with it so including use it as our container snapshot um and then this is my image my previousd 13.1 release and I give it my name and my command so I'm in my container I do not have a hostname because I did not give my container a hostname but um and I can only see my processes because it was a nice lead on that for me um so I here I run it as as a uh with CPR on containerd I ran a container it was a jail but as far as containerd cares it was a container so okay so this brings us to another interesting thing the uh previousd is really one of the nice things about previousd is that it's it has gone out of its way and by it I mean the maintainers gone out of its way to uh play nice play really nice like be a nice host to technologies from other operating systems and so it had um pretty early support I'm pretty sure that I did an NTFS file system on it at some point read write and put files there and didn't corrupt it um I think it had a great EXT 3 driver I think I've tried this recently but I think the EXT 4 driver is read only because of journaling I don't know if you know the answer to that I keep looking keep looking at that for you but um a lot of this is nice it's it's one of the useful really nice useful things it did was make a kernel level linux emulator as long as you are running a compatible an architecture compatible binary it will linux binary it will run it for you um there are some caveats a lot of these are much nicer than they were and the last time I did this five eight years ten years about ten years ago um it's those basically linked libraries so you um there are now packages you can install that will install a couple different linux the distributions but will install the libraries for you the dynamic libraries you don't have to let those down yourself but it just you know binaries just run you load the you start the you load a kernel module you start it it's okay it gets you pass it a you know you try to exact a command that's a linux binary you can't find the library so this makes it possible to do some very interesting things um I could run a linux container on FreeBSD so how would I do that uh well go back to Runj um and I don't know why this works it just works uh and I've already pulled the image but I I will show you this but this is really small I forgot this is really small so um again using my CTR fan line and I've already pulled this image so it won't be an issue but I'm telling it same thing I did when I was pulling um the FreeBSD mostly image telling it to pull this image uh I'm doing something very similar I'm telling it to pull but I'm giving it I'm telling it specifically to pull a linux from for a linux platform image I didn't have to tell you that when I pulled the FreeBSD image because it's FreeBSD and so that matched but I'm on a FreeBSD system and I want to pull a linux image so I have to specifically tell it that and then I give it the um the actual image I'm pulling just the standard alpine linux container um so now I can do this really nice thing I can run a linux container directly on FreeBSD now to FreeBSD purist just run to else but I don't know this is really this is uh this is kind of fun yes wait so how is that running because it doesn't have a linux kernel to access it's the linux emulator the linux you later yep so I'm still using the runj runtime but because when it you know goes to execute the actual process it just the kernel says okay this is a linux finder but I know what to do with those so boom um I actually have linux uh so that's one I'm talking a little bit about the future that's one possible path but mostly um I'm gonna bump ahead a little bit talk about the future so Doug Robson has done also a lot of work and and Doug Robson Samuel they I mean they're doing this on their own time they're not paid to do this they're excited they want things to work Doug Robson I specifically asked him and he said as part of an answer my long-term goal is to make it possible to add FreeBSD nodes to a Kubernetes cluster as it turns out the control plane works just fine on FreeBSD already and he has a working FreeBSD only cluster in his lab so um you can you know we have Windows, Windows nodes in Kubernetes which are you know full-fledged first-class mostly uh nodes that you can spawn you can spawn containers on um why not FreeBSD which is much closer relative to Linux and that's the thing so we've seen you know Run Day gets a lot of the gets us most of the way there we just need some of this needs to be ported some of the the node processes whether it's the um the Keyblit or you know there are other projects to replace Keyblit. Keyblit is the little process that runs on your Kubernetes nodes which are you know your work courses where all your service containers run and um so it's what talks to the Kubernetes API which kind of sits between the Kubernetes API the control plane and your container run time uh so it's it makes it seems totally makes total sense why FreeBSD shouldn't be a it should be a first-class citizen in the cloud-native world um and that would be great and could then use I don't know what Windows it runs its little containers I don't know what they look like but on FreeBSD you could make jails and you know you'd understand that they don't behave exactly like Linux containers but hey you're running on FreeBSD you have these little discrete services servers that can run and you can they run processes and if one dies you know the orchestrator restart another one and it's a lot of fun and it's very useful um and as we've seen if you have Linux simulation mode turned on oh hey you could also schedule Linux containers on your FreeBSD system obviously there are some caveats if you're using some really low-layer Linux stuff but you know basically you can do it um so that's that's what excited me when I thought about it two years ago because I missed I've worked with FreeBSD a long time and I miss it um I was working a lot more in the Kubernetes world and why aren't we doing this why aren't these two things you know the Reese's Cup why isn't my peanut butter my chocolate and peanut butter why do I get the peanut butter on the chocolate because it's the we've seen the revolution that containerization and orchestration have made mostly on top of Linux just makes development it's changed the game hugely nobody almost nobody knew what Kubernetes was 5-10 years ago now everyone is running um and I say it just makes sense a lot of people are excited we made a huge amount of progress I haven't done anything except talk about it so I think that'll happen um and there's one I could unless people want to stop for questions there's one really oh yeah no I'll show you it anyways it's called the jail it's the full previous defile system so it's kind of big so you know Docker containers can be really small they can be really big but they can be really small so it's the same thing with you know we saw with this root example it's the same thing for jails you can have a very tiny image because if I go back to my you know all I had was the binaries but I also I copied my dynamic libraries but I can now do and I've already cheated and created this file um it actually may give me an error okay we'll do with that later um no and I need to edit this it may actually give me an error we'll see no it didn't like that okay um actually let me do this okay I will go back to my original this also has uh I don't actually want to do that I can use this this shrewd demo that I created earlier which we know just has um it's got the defile system on it but it just has a bunch of libraries and the psme binaries that's that can be my image um this worked run a jail off of that again the jail does not need based on shrewd I just need what I'm going to run it's this very similar uh looks the same on a Docker container um there is I think it's Doug Rapson has worked on an OCI jail spec I don't he has actually worked on um so he and Samuel Karp kind of came out of added different directions Doug really wants uh he wants podman to be a full pledged uh you know podman is basically docker without docker uh but not by docker uh and he wants it to be able a full fledged uh runtime he's gotten builda which is the docker build podman build he's gotten it working um so it would be nice to have it more general uh again people came at it from the container D direction um but it's yeah you can build these images he has some tooling to build the images um or if nothing else you can copy files into your shrewd file system and tar it up um yeah that's that's definitely part of you know that's standard tooling so people are definitely looking at that um oh I think I have it open do I have it open do I have it in the okay no I don't um I need to generate a list of my references but uh I will definitely point to I don't know if you're unmasked at all and you can okay um I will post those probably in the next hours um okay so anyways we'll make really small images um you can distribute them just like we saw you know with with run day with with Samuel carp's image it's just it was full-sized previous d but it was you know it was an artifact that was on uh awc's elastic container registry and I could pull it just like I pulled the docker image for linux so that's a that's a very this is just a snapshot of where we are um there may be people oh somebody else I read somebody else has a different kind of unrelated run time for containers that hooks into hashi corp hashi corp's no map uh container orchestrator people are I I think there are a lot of these corner cases which are not out in the public but these are two of the more kind of kubernetes approaching uh responses which is the interest that I came to it with um so I think I am a terrible prognosticator but I think by the end of the year we will have a pretty much especially Doug Grabson he's done a lot of work uh you know a full-fledged FreeBSD node um kubernetes um and it may go past that and whether I have no doubt that it will end up in the main kubernetes repo eventually uh and I think a lot of people are going to get really excited by that I'm excited by that um people who like I am I've worked with Linux I work with Linux now you miss FreeBSD that's great news um it's I think it's going to it will help bring FreeBSD back into uh a more mature uh part of the server of the server ecosystem uh which is you know where this is okay that's it um I'm on I have a Twitter handle I never go there so on mastodon I'm at buzzykb at hackyderm.io um and I do have a website where I post I posted some um tutorials on this I post a lot of random stuff so that's production with scissors dot run okay and do we have more questions um I know this might be a little bit beyond the scope of the talk but um like at work we were a really heavy BSD adopter even on desktop like we had uh like it was like a windows terminal server but it was with XRDP FreeBSD and it worked pretty well actually like we started early when things were just not there yet but it got way better and then we we moved to a point where I know you briefly mentioned it the beehive virtualization and it's awesome like we used it for like a Linux printer driver that I couldn't figure out how to get it working and like the Linux simulator and it worked great but like for graphics I was wondering is is there is there new is there news on GPU acceleration like particularly being able to share a GPU with a VM because I think that would be so amazing to have like for for like these enterprise providers that say like oh this is only supported on Linux even though it works on everything like our accounting system is like that and yeah so there are a couple pieces to that and I kind of know bits so it has to be oh should I repeat that or is it the mic pick it up okay so and I don't remember I may be mishmashing I think Michael Dexter is I think I'm not sure I don't want to miss title him but he's one of the leads on the beehive project and I remember asking about several months ago and I think it was about GPU support and I think he says they're working on GPU support I think if you go to beehive.org you can get transcripts of their meetings too and watch video if you want to okay yeah and I follow mutuals anyways so I don't know so the Linux emulator is I don't know its current status there are parts where it falls down so I don't know if you can just run the driver and it would work I have no idea what the status is I don't know I don't know it works really well but for some commercial applications from specialized domains I don't know but I think Microsoft Office would be awesome to have if there is a Linux version yeah you can and again beehive is hard work level virtualization so you can run Bihive is the same as KVM where it's managing at the current level, but I could actually show you I have. So there are a lot of jails that I use the command line, it's still a little cranky. There are lots of wrappers for jails that make it a lot easier to manage. Same thing for Bihive, and I'm using CVSD to manage my Bihive VMs. So I actually already have one, oh, an Ubuntu server running on my local, so I can, yeah, it's fully, it's a VM, it's like, if you go to AWS, EC2, you say I want a Linux VM, boom, it's what it is. Yeah, obviously it's heavier weight running a full VM, but I need to restart because CVSD does not like to go to sleep. Yeah, it's, Bihive is really, I actually created my first Kubernetes cluster on CVSD using Linux on Bihive. Oops, okay, okay, next, I will show you this once I restart. Any other questions? More questions? I have a question, since you can run like the Linux jail, could you run the, Kubernetes control pane in the Linux jail? So you're not technically running a Linux jail. You can run most Linux binaries within the jail. I, maybe, so there's still some network, there's still some virtual network stuff that needs to be worked out to make that stuff, make the specification look, get really full-fledged networking. Again, a lot of people are actively talking about it. I don't think we're there yet. I don't know how Doug grabs it, but this, okay, so I can log, this is a VM running locally on my laptop, it's a Bintu, you know, and it's completely separate from, you know, it's a truly, truly virtual machine as opposed to the semi-virtualization like containers in jail. I actually have a question. Yes. So, jails and containers are not the same thing? No. Has anybody explored trying to build a jail and ZFS native orchestration system? Probably, and I'm actually sure they have. I don't know what it is. No, there are definitely bits, people have done this, it's been, they've existed so long. Yeah, I think pot and no matter kind of similar as far as orchestrating jails, and CBSD will do jails too. Yeah, it will. What we think of as orchestration is a little different from the Kubernetes standpoint, but it will definitely help you automate managing it in large. So I just wanted to throw in there for you guys to think about is, you can do configuration management, there are things like Ansible and Salt, and there's no reason why you can't take what you know from Ansible and Salt and use something like HAProxy and Reactors and Salt and build your own version of all this stuff. I mean, you can tell it, build me a jail, configure it, install these packages, do all these things.  But I understand the need to put into the standard tooling and get into the community, but I'm just not involved in any of those communities, so for me, that's the route I'm taking with it. It all works fine. Yeah, I mean, there are so many ways to approach it, and organizations, people should do what works for them. There is no one right way. As I said, I'm coming at this from the more Kubernetes cloud native way, but these are all, that's not the only way. And some of these bits, you can take, pick and choose. You don't like, you can use the RunJ, you can use the Runtime without using Kubernetes, without using Orchestrator. These pieces, I am very pragmatic. I'm a site reliability engineer. Use what works for you, for your org. Any other questions? Thank you so much for coming, thanks for the questions. Sound okay? Oh, yeah. Testing. Thanks everybody for coming. We'll get started with this last session of the, not the last day, of the cloud native track at scale. Our speaker, he told me to introduce him as just Cotay, because that's what everybody calls him, but he is, he does have the name Michael, and he has asked that we hold questions to the end. When we get to that phase, I'll run a microphone out to you so that we can get that on the recording that we preserve after this. So thanks everybody, and Michael, or Cotay? Hello. Well, usually, I was realizing, usually I say there's only two people who call me Michael, and that's my mother and my wife, but I've realized also John there calls me Michael. So I'm going to start name checking him every time I get introduced, which would be great. Now thanks for coming during a dinner slot. I stopped by the Sheraton lounge. If anyone's hungry, I brought these for you. They are oats, so that may not work depending on what you do, but I'll leave them here if you get hungry. Well, yeah, so I want to go over, well, you're going to find out what I'm going to go over, but it's pretty much says right there. There's this notion of a platform or the application thing, whatever this stack of software is that your application developers use as their kind of primary interface with your infrastructure and your stuff. And well, not so much, I'm not really technical. I stopped programming in 2005, and now I make slides. So I'm much better at that. But I don't really want to give much of a technical overview, because I'm not capable of it. But what I can talk about is how people have been kind of the practices that they follow, the way they think about it, what are the lessons learned and the best practices for being successful with your platform, if you'll pardon the word strategy, your initiative. And before we get to that, I thought it would be good to talk about mayonnaise. But really, what I'm going to go over here is I think whenever you're talking about something like infrastructure or platform-y or whatever, I, having been an application developer, think it's always good to remember why you care about infrastructure.  I used to be an application person. But typically, in the case of a platform, the reason you have a platform or thing that runs applications is because you have software, you have application developers, and typically the reason you have applications is because people are using those applications to accomplish some task. I suppose you could have an application that no one uses, which would be academically fun, but not very useful. So I think as you think about how you get a platform in place, what it is, how it's successful, it's always good to have the use case, the goal, the reason, remember that guy who talked about the why's and stuff like that? Like the why that it exists. And that is when our mayonnaise friends come into play. Now, this is a story from a while ago, and it's a company that, back when I was at Pivotal, now VMware bought Pivotal, so I'll go between VMware and Pivotal. Although I'm pretty well trained, I'll probably say VMware. But we worked with this company a while back, which is why I know the story. And they're a food services company, which means they will ship food to you or deliver food or they will also run your big commercial kitchens, whether it's a campus or whatever. I don't think they would come into your home and run the kitchen, but they'll run a kitchen for you. And so, you know, let's consider, they're a business, right? There's three core things businesses want to do, among other stuff. They want to make more money, make more profit, and retain and get more customers, right? And generally the way that they do that is they focus on, you know, part of what they focus on is like optimizing costs, doing things as affordably or cheaply if you prefer as possible, not wasting, and then having good customer experience. So let's go all the way down to the kitchen here.  And they thought, we've got these kitchens and we want to have better customer experience and we want to optimize them. And currently what we're doing is we have these three ring binders of the recipes that these chefs use. And obviously we're doing some digital transformation so we need to put them on an iPad. So let's go get, and you know, that'll make things more efficient because paper recipes, you know, you lose them, you gotta ship the new recipes each week or whatever the period is. It's important that you sync up on the recipes because you want consistency of the food and also presumably down there at downtown or headquarters they figured out the most optimal affordable way to make recipes. You don't have this guy being like, well what if we add truffles to it? Like, you know, you're kind of limiting the expenses of the meals that you have. So it's very analogous actually to the way that a lot of IT is managed. You have a centralized governance that kind of gets sent out and enforced and you follow what's in the run book or the recipe book, so to speak. And so they set off the application developers and they gave them the mission. What you need to do is you've got to digitize these three ring binders. And now, normally what would happen is the developers would say like, sounds great and then they would do that and then they would give it to them like three to 18 months later depending on how good they were getting software out. But instead, and again, focus on like, this is the goal that we're shooting for, right? Instead these developers, because they were good and very focused on how they can make their software better rather than just doing what they were told, which now that I think of it is a great way to pitch a developer. Would you like to not do what you're told? And instead what they did is they, if you can imagine they got up at like, three, four a.m. in the morning for a couple of weeks and they would go to the kitchen and they would observe what people were doing, right? So right away you're like, well, that's crazy. But they go and the developer teams go there and they look at what's happening in the kitchen. And again, they have the goal, I mean, go all the way back up to the dry clean people. The goal is we want to make more money, we want to save, eliminate waste, save as much money as possible and we want customer experience, right? We want consistency in the food. And so they have this in mind, the developers and they say, that's great. You know, they're using paper recipes but we've noticed that there's something, all these people that spends in the kitchen, they spend a lot of time running around seeing what temperature the mayonnaise is and seeing what temperature the raw chicken is and they have the little thing on their sleeve and they like stick the thermometer, that's what it's called.  But they're spending a ton of time doing this, the kitchen staff, so the developers thought, well maybe computers could help with that. So what we'll do is instead of right now working on putting a recipe book on an iPad or a Samsung Galaxy Tab or whatever, what we'll do instead is we'll digitize that process because our theory, and this is the first thing is you're doing this kind of iterative, trying to discover the problem, come up with a theory of how you would solve it, you wanna run the experiment as a developer, all you can do is write code. I suppose as a developer you could not write code, but that's a little cheeky to be like, that's the best option to do. If you're not writing code, ultimately you're gonna be out of a job without this kind of a bummer, so you probably should come up with a reason to write code. So they went in and they, to stop rambling about this exciting story and get on to the rest of it, they digitized that experience of measuring the mandates, and sure enough they found by going through a few cycles, a few week cycles of tuning this and observing that it saved the kitchen staff a tremendous amount of time. So you already, if you've been listening to the whole DevOps lean and whatever stuff for the past 15 years or whatever, they're like, ooh, removing waste, success. So you've sped up the time that it takes to get that food made or to measure things and you free up this time to do other stuff to focus on better quality, following the recipes, not being so harried and running around, all that sort of stuff. I'm not really sure there's too much concern about kitchen staff having burnout and things like that, but it's very similar, like all the stuff you would think from the DevOps world about how you need to have a more humane approach, the developers are trying to give that to the kitchen staff. Now eventually they go on to digitize the recipe book and all of that stuff, but you can see that the point is the developers could go in there, they had the time to actually hang out and study people, right? They weren't like, well, we gotta spend, you know, sprint zero figuring out like this Kubernetes thing, but no, instead they went and figured out what their customers were doing and like how that was working and then they could go through every week and come up with a new theory and actually test it and see if it's helped solve the problem that they're, you could say customers, but that these happy people had. And that is the goal of everything that I'm about to talk about is like getting to that point where you can have a team of developers go in there and have that kind of experience, right? Purely focus on the problem that they're solving. And again, it links all the way back if people here care, all the way back up to the business side who wants to be like, what are all you computer people doing, right? Like are you helping us achieve the goals and sure enough, yes it does help them achieve the goal. And you know, I always like to note that the ultimate poor customer experience is just that mayonnaise and chicken aren't at the right temperature. And then I don't know if you've ever had the food poisoning, but it's not a fun customer experience to have at all really. Like I guess it's a good excuse not to go to work, which is always nice.  Like it seems like back when I was in Austin, every time it snowed, it was a kid's favorite time. No school to go to and ice as well, but they're not really into that where I live. So here's, let's get down. If we want to enable these teams here, let's identify one simple problem. Now, I neglected to get permission from a forester to use their actual chart, so you have to forgive this because I'm trying to be a good boy about things like that. But basically, if I round up a lot of studies and surveys over the years that go over how frequently are development teams releasing their software to production? So can they actually release their software every week? Can they come over the theory and get this loop in place? And depending on which survey you look at and how you interpret it and all that stuff, right? You can see that maybe, again, depending on which one you look at, once I get permission, I'll use the actual chart and then I won't have to disclaim stuff. But like, I would estimate somewhere between like 30 to 20% of teams out there can actually release their software a week or less, right? And I think, you know, I'm being very liberal in here, but well, or I don't know what the phrase is for that, but like I think probably more like 50 to 70% of people, it's a month or more, and I think probably 50% of the people is like a quarter if you're lucky, right? And so essentially think about that mandate's case. Like that's not gonna happen if it takes you like a month or a quarter to release your software.  So like this is essentially, again, so going down another layer all the way from the dry clean people to the developers and then right underneath that is like, so this is what we gotta give the developers. Like they can have all this other stuff, but if they can't release their software, it's not gonna work. So that is a large part as you'll see of what I think is kind of like the most important outcome of why you do a platform and what things will rotate around. And then just as an introduction, this is me. I actually, I've worked in many places. I worked with Matt Ray long ago. I found this, maybe you remember this. In 2005, this is like I said, this is the last time I did something innovative because you can see that I won the research development team innovation. Now that's FY05, so that was probably calendar year 2006. But I found this in my back attic, my back shed recently. And boy, I don't really even, that's a Galileo something or another. I was young enough at the time that I thought that was cool instead of being like, could you just give me the cash? But like, you know, it was nice. So I was a programmer a long time ago. I worked in M&A at Dell for a while on cloud and software. And I was an analyst at Red Monk a long time ago and another place. And now for about eight years or so, I think eight years, I worked at Pivotal at first. And then of course that's a part of VMware. And I've worked on the developer advocate team, even though I don't really talk to developers nor advocate for them. I more talk with managers and executive types and things like that about that kind of stuff. But yeah, if you wanna find out more, I don't know, I have a podcast that I do and stuff like that, you can go to cote.io, which I would appreciate you doing that because I have to pay like $35 a year for that domain name instead of like four or five. I don't know why, but you know, there it is. So first let's start with what is a platform? I'll try to keep this short because you can go on and on with the definitional thing and then you're just like, okay, but now what, right? But we'll jump into what a platform is.  We've had lots of names for it and lots of things that have been used for it. But I think in defining the platform, the other thing that I can do is kind of tell you why I think this discussion is relevant nowadays or relevant is coming up why you hear phrases like platform engineering and things like that.  And the reason it's pertinent nowadays is because, you know, we kind of go in cycles with the platform, right? Like we get really good at making like the best platform possible that everyone wants to use.  And the cycle goes over and over and over again, right? And, you know, you can go back further from Heroku, but Heroku like still kind of like best in class platform, everyone still loves it. Of course, the fatal flaw of Heroku is that it's too expensive. And, you know, especially I guess at this conference that's a big problem if something is too expensive. But that's kind of ultimately like what kind of pulled that back. In fact, that's a good theory now that I mentioned it, that maybe the problem why we keep remitting stuff is people stop, they don't want to pay for it, which is a problem if you're like money or need it.  And so that was another great platform. You see people even nowadays, of course, I'm biased because I talk with them, but you talk with people who still use Cloud Foundry and they're like, oh man, it's great. I deploy my applications, everything works really cool. Love that platform. But we're installing Kubernetes, so like memories. And indeed what seemed to happen sometime in here is like our friend Kubernetes came along and decided that like yep, it's time to restart everything. So right now, I think we're finally at the point with Kubernetes after four or so years where we have arrived at this, congratulations. So now you can stand up Kubernetes, you can go to your developers and you can be like, here's your blinking cursor, ready to go, right? Now, that doesn't really work for developers as numerous people go over. Like some developers enjoy not going to talk to the chefs and instead spending a lot of time figuring out what to do with the blinking cursor, but most of them would like to say, here's my application, let's get going, right? Like and just actually start using it. So this is why platform conversations I think are happening a lot now is because we're right here. Like we've kind of gotten rid of the past few cycles of platforms, those are no longer cool, no one wants those, too expensive. And now we're like at the very beginning of building out what this platform is gonna be on top of our new friend here, good old Kubernetes and other stuff. And so finally, what are the components of a platform? Well, this is from, there's a newer version of this diagram but I like this one better, so I'm gonna use it. But this is from a CNCF like working group of a working group to define what's in a platform. And I think this is pretty good, I like this because it means I can show you a vendor agnostic diagram of what a platform is. But I guarantee you if you go look at including our own at VMware, if you go look at any vendor's platform it pretty much looks like this, right? You've got your infrastructure under here, you've got this middle layer, not supposed to call a platform as a service, a platform. And then on top of here, this is kind of what's new and interesting nowadays is you actually have like very developer-centric tools and collaboration stuff, right? That hasn't necessarily been part of previous platform sort of revs that we've been going through. But you know, as you look through this stuff it's like yeah, I mean that's if you're running software you need all these things. Hence the thrill that we get to reinvent it all the time instead of just kind of using what was already there. But that's great if you're into like getting work because what you get to do is you can build this entire platform on top of this blinking cursor, right? Now, as you're out there building the platforms, right? You know, again, you wanna keep a focus on like who it's for and what the goals of it are, right? So this is back from 2018 from the ThoughtWorks people. You know, just as a side note, these ThoughtWorks people they're great at marketing. Every time they come up with something in my VMware Pivotal position I was like, ah, why didn't we write that like two years ago? Like we would probably do a pretty good job just like sending ThoughtWorks stuff to chat GPT and having them rewrite it and post it on ourselves. But they're fantastic at documenting things. And this one back from 2018 really, I think captures the essence of what a platform is nowadays. And remember, it's like very focused on developers, right? Like it's very, very focused on how do we make the application developers daily life and lives easier, right? That's what the focus of a platform is. Now I mentioned that there's this kind of new thing, like so you hear like platform engineering, right? And so this rev of platforms is, it has that much more closer focus on not only being for developers but thinking about what are the tools on top of the platform that developers use, right? Like what is, I mean basically like what's all that Atlassian stuff, right? That we can pile on top of there. And so that's where you encounter this phrase, let's see if my laser works. This phrase internal developer portal, right? Or IDP. Now I think this is one of the most boring names I've encountered in my entire career, like portal. I don't know about you but every time I hear the word portal I don't want to be involved. But whatever, that's what it's called. And it's actually like, again, if you, not again, but if you look at this diagram and you go back to this diagram, kind of the same thing, right? But you can sort of pick out how there's more developer centric stuff like your build tools and stuff like that. So in this rev of doing platforms, we're further trying to like pull in as many different tools as possible that are used to get the software out the door because it has that bigger focus on helping developers out. So that's as brief as I can possibly make a definition of platform stuff. Again, hopefully the notion that you get, I mean, especially if you need some references, I didn't mention this, but these are also platforms. So unless there's a lot of, well, judging by the look of some people, you probably recognize many of these things. But again, it's the stack of stuff above your infrastructure that you run your applications in, right? So that's what we're shooting for with a platform. Now, let's say you have a platform in place, right? Now, luckily, because we do have this like eternal reoccurrence of platforms, right? We actually have lots of knowledge and lessons about how to run them. So while we might delight in rewriting all the platforms and recoding them, we don't actually have to rewrite the culture or the process or the ways of using them. We've just got a better tool that we're swapping out and we can kind of reflect on like, at least I just kind of chose seven years arbitrarily. Well, actually, was it arbitrary? I forget, I guess that goes back to 20, I can't do math in public, that's very embarrassing. But back when I was making this up, it was basically about the height of the previous, of the platform as a service era, more or less before Kubernetes started coming in and just sort of bringing us back down to the infrastructure layer. So that's what I'm gonna do with the rest here is like, so you got your platform in place, like how are you gonna have it be successful, achieve the ends of having your developers be able to wake up at four a.m. in the morning and go watch chefs. I mean, if you're like operations people, like any excuse you get to wake developers up before 10 a.m. is probably delightful for you. And so that's what's gonna be motivating you to build these platforms here. So the first thing is, again, take on that perspective of like, a platform is to support application developers, right? So the first thing that you're gonna wanna do, straight from the sort of dev ops and lean sort of area of thinking is like, you wanna think, all right, so if the goal is to get application developer stuff out the door, like I don't wanna be involved in waking up that early in the morning and go watching people measure mayonnaise, they can do that. But what I can do is I can think about how can I speed up their release cycle, deploying software out to production. Now, what we learned from the dev ops world is that like, organizations are made up of silos and every single silo is delegated one part of the overall process and each of them, if you're lucky, locally optimized, but they don't really care about what happens in the rest of the cycle. Or to put it another way, like what really matters are all these arrows between all these boxes, right? Cause you can get the box to be really efficient, each of the different steps. But unless you can actually pay attention to not only the system as a whole, but those arrows and realize that the arrows are boxes. There's like a mind blower for you. Like, you know, you've gotta start doing that to think about the end to end process. And that's usually the first thing that I've seen over the years platform teams focus on is like the first thing we gotta do is figure out what it takes to get software out the door, right? And I forget, you know, I was just talking with someone a little while ago, right? And the issue with this, it's almost definitional. It's like, we've gone in and we've said the problem with getting software out the door and managing it is you have all these silos that don't talk with each other. So if I wanted to go talk with someone about that, no one exists that I would talk with, right? Like that's why you've always gotta go way up to an executive level who's just like, ah, I don't know, whatever, these people do that. Usually they're a lot more enlightened than I portray them. But that's what you end up focusing on. Somehow, as the platform team has to figure out getting this and figuring out like what it takes, what all the, you know, one of my more astute lean scholars out here can probably give me all the terminology that you would use for this stuff. I never really learned the difference between like lead time and cycle time, which is why I keep trying to say, release the software, deploy it. So, you know, you gotta map this stuff out. So what are, there's two easy ways that I see, I've seen people doing this. Like the first one is just like, you know, go through the exercise of what it would take to deploy one line of code. Maybe like, you know, we decided we shouldn't have drop shadows anymore on the buttons, so we're gonna remove that or put that into the style sheet, if that's what the kids call it nowadays. And then we need to deploy that so we don't have drop shadows on the buttons.  Like all the meetings you gotta have, like all the stuff you need to do, how long it would take in waiting and actually doing and you construct this, people call this all sorts of stuff, just the end to end process together. Now, and especially like clever way of doing this, I think it was James Waters I heard saying this, is like don't even deploy one line of code, just assume you're gonna redeploy your current version, right? And that means you've gotta start all over again as if that current version is like basically done and what's it gonna take to just get it into production? Like all the meetings you need to have, all the testing that you need to go through, all the checking of things, like how long does it even take? Which is actually kind of nice because it takes out your development time, which again we're targeting in a week, so you just load that week back in there, I guess, in your optimistic scenario. But what's fun about that, right, about this exercise, if you do it realistically, like the best, I would recommend if you're doing this, do this assuming that you're going to be doing this between the third week of November and like you're really, basically the third week of November is when you're gonna be like in the states at least to be different in Europe. But that's when you're gonna be code complete and then say like all right, you're done the Friday before everyone takes a week off. And then they're gonna be off for a week and then you're basically gonna have maybe two weeks before everyone disappears again. And so like schedule some meetings, see how that pans out, right? And essentially what you're gonna find, what you'll find by going through the SOT exercise are all the activities you need to do and that becomes once you draw that out, that's sort of like your map for things that you're gonna start improving, right, as the platform team. What can we do along not only like, people-wise and with process, but like how can we go in with tools that we have by changing the process around and optimize all those different things, right? So that's kind of your first go there. And then once you have that set in place, you've got kind of like a baseline that you can go back to and kind of keep tracking if you're improving things, right? So in the same way that the developers where they're focused on like experimenting on a weekly basis, trying some new theory of doing something and measuring if it worked or not, this is kind of like a thing you can use, among other stuff, to measure if you've actually progressed in all of your platform work. So then the second thing is, and this is again, as with all sort of like re-occurrences of stuff, this stuff was like always there, like I was there long ago when all this DevOps thing was happening and boy, all the ITEL people would get so upset and they would be like clearly, you didn't pay the 400 pound sterling to buy the nine editions of service delivery in ITEL or the nine parts of it because we wrote about that in volume six of service delivery. We've been doing this all along, right? Now, so obviously there's been this idea of product managing or thinking of developers as your customers, but that's an especially big emphasis in this current like gyration of doing platform stuff in platform engineering. And here's some of my friends at Mercedes-Benz here. Now, this is one of the people who, they're not running this group at least, I mean it's Mercedes, so they run everything, but they're actually running a platform as a service, a cloud foundry instance, but they're a great wealth of like what it takes to run a platform and anyone who's run a platform for a couple of years will say the same thing is that what you do, a huge shift you do is you're no longer doing service delivery, right? You're no longer delivering a set of infrastructure services, you're thinking about developers as your customers and your product managing the experience from them.  But if you don't have a product manager, that's like kind of like the next step that you go through is like, all right, if developers are my customers, what does it mean to have a customer? And you can go through all sorts of stuff, but in the software world, this is where having really only lived my life at vendors, it comes in very handy. I know what product managers do. I do a podcast with one of them. And like, product managing is a very, very well known practice that you go about. It's very well described. It's more or less easy to learn about. It's got its own weird little fun language and conferences and stuff like that. But the essence of product management, I mean really to boil it down, is you have a product manager and they know who the customers are and they also know like, let's say the capabilities of the technology you have, like what's possible. And there's lots of things product managers do, but the core thing they do is they say this week, we're gonna do this. And I know that sounds ridiculous, but I like to boil things down to simplicity or to be simple. And so the product manager is always thinking about, what's the next best thing that we can do to help developers out? Now that's in contrast, that's not like, what capacity do we need to achieve these, the kind of performance characteristics that we have and the cost metrics that we have, right? Like the product manager obviously cares about cost and performance and things like that, but their primary concern is like, is this gonna help the customer out? Not so much, is this the right way of going about stuff? So having a product manager, very, very important, right? That's a key role that's on a lot of these platform teams. Now, if you're kind of curious, like how to start doing product management for developers, for a platform, right? Like, again, product managers would be like, well of course this is what you do, but if you haven't really done product management, what you do, big shocker here, is you ask your customers what their problems are. And there's all sorts of ways you can go about doing that. I think this is a pretty good one. I help write up this kind of survey that some of our people have used with organizations and really, this is one, I know you can't click on it now, but you can get the slides later, but you can basically take this survey and just like stick it in a Google form or Office 365 form, whatever. And it's basically just a series of baseline questions to go out. And if you start sending this out, kind of pretty frequently at start, at the beginning so you can kind of measure your success, but you'll very quickly find out what developers need and what problems are, right?  Like here's the thing that we're gonna pick. And because the way the survey works, it's like on, I don't know what I'm about to say here because I'm a philosophy major, so I assume it makes sense, but you're doing it on a Likert scale, so you can go in and like sort from the top of the mountain to the bottom, ascending. And you can figure out like the most important things on going that you would want to do. That's the other thing, if you stop doing software development, you get marginally good at Excel, which is fun, ascending. I only know the word ascending because of Excel, because you have to know that. So you've got your, whatever you want to call it, value stream your end in process, the arrows are boxes, remember? And then the developers are customers, we got a product manager, Likert scale, whatever that means. So let's say you've built your platform. This is like the first thing that people encounter, and that is, and this especially happens if I talk with people all the time who have this. They're like, we built out the Kubernetes clusters. I don't know why I'm talking to you, VMware who wants to sell me managing Kubernetes, we have Kubernetes, we got five Kubernetes. And we built them for the developers because it's inevitable, it's the industry standard. Even though five to 10% of the industry works that it becomes a standard at that point, which is fine. You can tell them a little sore about Kubernetes, but I went through the full on grieving cycle back in the fall and I'm cool, it's fine. I just use humor to deal with things. But they stand this up, right? So you've got the blinking cursor and then they're like, Kote, although they might call me Michael because they don't really know me.  But so they stand it up and they're like, why are the developers not using it? In fact, they go off and they use their own thing. We built this great Kubernetes for them. And that is often because they haven't really thought about platform marketing, advocacy, and to a certain extent, consulting. So if the idea of product management is a crazy idea for infrastructure and operations people, try marketing. They're probably, I passed by someone in the expo hall. Who are they talking with? I forget, it was the Grafana people or whatever. And I just overheard them saying like, oh, no marketing pitch. And I was like, oh, come on, right?  What do you, like, whenever I hear someone saying no marketing pitch, what I hear them saying is don't be boring, right? Which hopefully I'm not being, only a few people have left, so I guess it's exceeding. This is not a marketing pitch except for a little part. But you gotta market your platform, right? And it's not only like you're gonna send out like your weekly email that's like, think of the trees before you print this if people still do that, like that kind of thing. But it's really, really like product marketing. Again, like us vendors would do, or really anyone, right? So what you're thinking about is how do I establish like, well, first you gotta have a name. That's incredibly important, right? And one of the things that's important about a name is that you've gotta come up with your own name, right? And this is, there's a colonel in the Air Force who told me once you're going down an elevator, he was like, oh yeah, yeah, you can't call it Agile, right?  And therefore, you know, it's like my hair's a bird, your argument's a valid kind of thing, right? And so you're gonna get in all these types of arguments because you're not following Agile to the T, like whether it's Scrum or XP or Crystal or I don't even know what people do nowadays. And so instead, what you wanna do is you wanna come up with your own name for something, right? So that you define what it is. So you've got a brand, you've got a name, and you can see you're starting to set up a bunch of principles, right? And actually, these organizations, they end up coming with like physical, like three-dimensional, like books that document not only the tools, but document their approach the way they think, you know, what a platform is, what the interactions are with them, right? And this is, two of these are for Mercedes here, right? But they spend a lot of time on marketing and brand and kind of building up like identity around it, right? In the same way that like, you know, again, back to the application developer world, like you can encounter someone who will say, you can encounter two different people who from the outside, they appear to be the same type of person. And one person would say, I'm a Java developer. And the other person would say, I'm a Spring developer. And these two people might as well be trying to kill each other, right? Like, these are very different like identities and sort of brands that they have with each other. I mean, I joke, they probably like each other a lot and they'll argue about like JVM optimization and other thrilling stuff. And I don't know, they're great. But, so you're gonna spend a lot of time marketing, right? And part of that, I mean, literally is this, is like, you should probably plan on a quarterly basis to have an internal conference. And like, literally that, literally, literally, so to speak, right? Like every three months, that's generally what a quarter is. You wanna plan out, and maybe it's like in various different regions that you're gonna travel around to, it's probably also online. But I've seen this over the years that they have a conference that people come to, right? And what you do with that conference, and this is where the marketing comes in, right? Because this marketing, it goes into not only bringing people developers to your platform, but it also is key that we'll get to eventually to scaling it out to your hundreds, thousands of developers. Because what marketing does, and I don't know, I'm not telling the thing to you, but by word of mouth, by one of your peers, someone who had your situation. So at these conferences, sure, you give a presentation about like, well, here's our platform, and it does this, and it's pretty cool, and whatever. But what you start building up is you have other application developers who have used your platform, and they go over their experience doing it. And they're the ones who become your, if you'll forgive the term, if you don't like it.  You're salespeople, right? They're the ones who are going out and establishing this is a good idea. Here's how to do it. And it's also like a great source of feedback for your product management, which is maybe a little too much deep into that. But, and then there's also obviously consulting that you'll end up doing. I mean, I won't go over that too much, but what I do see in the first year as though of platform teams is they actually allocate platform engineers or whatever you want to call them, people to go out to these initial development teams and work with them to understand the platform and kind of train people up for it. So then the next thing that I recommend, and here, I mean, I'm obviously biased because I work at a vendor, but whatever, here I am talking. You know, what I've noticed is that people who build a platform from the ground up, they often come and talk to all of us vendors like a year later and kind of like want to start with a pre-integrated, pre-assembled thing rather than building it on their own. And there are many, many options out there. There's only one good option. Obviously, I'll let you guess what that option is. But there are many options of a pre-integrated, pre-built platform rather than defining it all on your own. And based on what I've seen over the past like seven or eight years, right? Like it's a really good idea, whether it's for pay or a reference architecture you have to really just start with a pre-integrated platform stack. And instead of saying we will only buy is the wrong word, but we'll only get off of the shelf. We'll only get some pre-built thing if we need it. Instead, you should justify when you need to customize something, right? When you need to do your own customization and put your own components in. And the good news is the way platforms are nowadays, it's like, I would never use the word easy where computers are involved, but it's more possible than it used to be to swap components out and kind of put in what you want instead of what's in the pre-integrated thing. And so, and the reason that is is because, I mean, here's a list of issues that you can have, but like it's gonna be a lot of stuff to deal with, to get a platform in place, learn about it. And then the part that is the most annoying is that third week of November part where you're gonna uncover all these groups you never really wanted to know about that now you have to go work with, right? Like not even the networking people or the security people or like the audit or the compliance people, but just like all these various groups that you're gonna have to coordinate with. Again, think of this kind of paradox of the, I don't know, fixing the silo problem. Our problem was we've got all these different groups that don't talk with each other across silos. And by, as I just said, you don't talk with them. And so now you're gonna have to go find them and deal with them, right? So, and also you're gonna have to learn the technology. Anyways, there's a lot of work that you're gonna need to go through rather than just figuring out, building a platform on your own. And like I said, there are many options, but there's a super fantastic one if you're really interested in it. But, you know, really, I also wanna prove to the point all the different options that you have, right? They all kind of sort of implement the same sort of set of boxes that you have, right? So you can start with that reference diagram up there and kind of run through the, like, I don't know how many there are, but there's at least five, if not maybe 10, different pre-built things out there that you could use if you wanted to. So then next, I think there's about, there's a couple more things before I wrap up here, or by we, I mean me. And that is, and this one, this is something that I think is really important to go over with people, and hence, I go over it with people. And that is setting your expectations about the timeline for things, right? Now, we in, well, I'll speak for us application developers. Us application developers know that things are always, well, you could say that they're late, but I prefer to think of it as, you know, you just had the wrong date picked for delivery, right? Like my way of thinking is I'm never late for a meeting, someone just started it too early. Is really how things worked out. But similarly, if you're putting a platform in place, you're fixing up how you're doing software, what you're gonna find is it's gonna take a lot longer by years than you think it should, right?  Or like what I hear very frequently nowadays, there's this big, big re-interest in moving everything to public cloud despite, you know, the kind of like little bit of kickback about repatriation and stuff. But I talk with executives all the time, and they're like, oh yes, we have the target that by, the last one that I heard was great, is they said by March of 2024, we're gonna have moved 60% of our applications to the public cloud. And like, you know, I did a little bit of math in my head and I just, you know, changed the topic, because we were there to kind of learn about new customers we were gonna sell to, so I didn't want to insult them. But like the timeline is gonna take a very long time to achieve these big goals that you have. So you gotta really pull that back. And just as a representation of that, right? So I was lucky to be kind of in some meetings back in 2015 at the Home Depot, which normally I present to people in Europe, everyone here knows what that is. You get this, they call it a DIY store in Europe. Isn't that adorable, instead of a hardware store? But they started way back in 2015 of this goal of like, we gotta do software better, right? And they went through the process that I'll go through after this, but it's not, I mean, they were successful all throughout. But finally back in, it's actually 2022, not 2021. But if you go look at their, what I think it's like Q3 or whatever, some quarterly report they had, their CEO was directly talking about something. I was, talked with some people, not just me, I was just in the room about, and that is this pro business where the people who are contractors, again, it's great, you all know what that means, I don't have to explain it. People who are contractors, those are pros, and they finally have gotten their software in place where they've like, grown the revenue for some of the pro-contactors like by hundreds of thousands of dollars, right? And that's taken a long time, and that's just, it didn't take just that one application, but if you think about the scale that the Home Depot works at, they're putting a platform in place, it's gonna take years and years before you get like these gargantuan results that you're transforming all of the things over, right? So you've really gotta like set expectations that it's gonna be a lot longer than you think it would be, which salespeople I work with probably love that, right? Like it's gonna take a long time, so start small. So here's like the kind of timeline that I've noticed happening. So there's kind of two parts to it. One is let's just call it the first year or so, right? And the kind of initial phase. And this is the reason it takes a long time is because in true product management fashion, you kind of don't know what you're doing or what needs to be done. Now, all these diagrams I've shown, you kind of know what you're gonna end up with, but how it works in your organization, how you build it and fit it to what your needs are, you have to discover and figure out. So what you wanna do is you're gonna pick like one, maybe two application teams. You go out and you find an application and right around the same time when you're building out the platform, you find this application, these developers, and you're like, all right, we're gonna work for like three months and our goal is to get a release of your software out on our platform. And that's it. So whatever wacky business case you made when you close the deal or you got permission for this, it was like, oh, we're gonna have this return just to be like, that's great, let's put that over there. Just get the one application and focus on that. And it needs to be an application that's real, but not one that's gonna bring down the business if you mess it up. So like in the example of Home Depot, one of the initial ones they did is like the software that runs the custom paint desk. It's an important piece of software, but it's not gonna crater the company if it doesn't work. But you use that as a way, again, just like our mayonnaise people, of exploring and discovering and experimenting with what works. And that's also part of what's different kind of with a platform team and why you're product managing it is you're kind of learning and experimenting with that. Now, after three months, you get to six months and nine months, that's how math works. And then eventually you get up to a full calendar. You're adding one more team, then three more teams, and you're kind of like always going up more and more with teams. And eventually you get to this point where you use this practice. It was, I think it was, I forget which DevOps report, but in the last couple of years, they cited this idea of pairing and seeding, which is something that I've seen over the years platform teams and kind of use a lot is you have your initial team of that application developer team, right? And one of the things you strategically do is you figure out who's someone on that team who could talk with other people that they don't know. And we could even kind of help them and teach them doing things. And so that one of those people, you'll hopefully have one of those types of people on that initial team.  And you take that person, they're a seed, you take that person and you put them in the new team, right? And that way, again, think about what vendors love to do. We don't actually like to sell to you directly. That's, we actually do like that. But we also like y'all to sell to each other, right? And so you have the developer who is previously successful with using this new platform. So you kind of move them off into this new team, not only to like train and teach people how to do things, but also to like say like, it's great. It works to be a trusted person to endorse it. Now, what we advocate in the way that we do this stuff at VMware and in Tanzu and Pivotal, whatever you wanna call it, is we actually like advocate pairing up. So not only pair programming, but pairing product managers and pairing all sorts of stuff, which is, you know, some people like that and some people don't, but I found it's quite effective. There's all sorts of research papers that says it's good, but also you should probably floss every day and I don't. So nevermind that. Although I was told that in the Netherlands, they have these like flat toothpick things that have a ridge on them. And you're supposed to use those, you go in this way, straight, and in that way, and it helps your gums not bleed, which is a very Dutch sort of thing.  So I think it's actually once, but I could see them wanting to do it twice. And then your gums don't bleed. It's fantastic. What was I talking about? You pair people up, right? And then that's the magic of the pairing is that because you seeded these people over, you start pairing them and they transfer that knowledge. And again, I have a liberal arts degree, so I don't understand how math works, but there's some kind of effect of pairs. I think that's like compound interest or something. But you know, once you get enough pairs going and they go out and pair and they go out and pair, it's like that shampoo thing where I told a friend and they told two friends and two friends. And next thing you know, and that's why it's like the first year you're not gonna be doing much, but then I see this over and over again at large organizations. You reach this point after about 12 months where it's pretty easy to scale. I mean, not instantly, but you can see a path towards having lots and lots of teams out of your 25,000 developers or so using it. But you've gotta know going in that you're doing this and really engineer that you're gonna be following this pairing and seeding thing, which I'm not really talking about this here, but that's also, you know, a lot of the stuff that I'm talking about, if you think about it, is like the job of management, right? Like it's their responsibility to do that sort of thing. And so that's why getting the management people on board and kind of thinking about all these things is really important because they're often the ones who can look across silos and make decisions and make changes rather than just like, I don't know, I type into a keyboard. Like I can't change stuff unless I can type. Which isn't really how you change organizations. I mean, I guess you eventually click on a button, but you gotta have a lot of meetings. So the next thing, you know, just to kind of wrap that up, right? Like to give you an idea, you know, I mentioned the Home Depot and the anonymous restaurant, the managed people. But there's, I mean, if you wanna see like a very brief, it's only seven minutes long, but this is one of the people I know who runs, JPMC has been, JPMorgan Chase has been doing this long enough that not only do they have a platform team, they have like at least three different, if not more platforms, but then also they have the platform engineering team, but then they have a whole team that this guy runs, that's the platform advocate team, which, you know, makes sense for, again, I'll say it again, like 25,000 developers. But he's got a great talk going over lots more information and these are just two slides about the practices they've learned doing it, right? And I wanna show this one because like, it's great. I know this person and we work with him and you know, you like to see people that you know and work with be successful rather than not. That would be a terrible slide. This person tried everything I said and it didn't work. Let's go to dinner. And, but it's because like it actually is possible at this scale, at this sort of like normal of a company. Now you could argue JPMorgan Chase is not normal at all, they are getting custom servers built and things like that, but like at that scale, there's a lot of normalness, right, and I see this at all sorts of organizations, you know, particularly government organizations who do this who very quickly become successful at doing it. So it's definitely possible to build a platform and put these things in place. So with that, thanks for sticking around here. No one wanted a snack, but they're still here. If you go here, I have a page that lists kind of all my relevant platform stuff, including the slides that you have here. And like I do most everything through my newsletter nowadays because no one seems to read blogs. And like, you know, the Twitter thing is like not really panning out. And I mean the issue with Mastodon is like, now I gotta do this all over again, right? Like I had, you know, if my Twitter accounts from like 2006, so I've got lots of years and stuff and like, also there's no algorithm to promote. I guess what I'm saying is like, I just want people to click on my stuff. I don't really want to have a conversation, but if you like that, that's there as well. I'm available as my friend Andrew Schaefer likes to say, engage with my brand. And I'll be around if there's any questions. And with that, thanks. Chris, if you've got a question, please raise your hand and I'll run the mic out to you. I thought I was gonna get away, but. This was the best talk I didn't realize I was going to. So thank you. Oh great, I'll have to parse that as they say. But I appreciate that, it's very nice of you. I love how many like awesome reports and studies you have in your slides. I'm just curious, like, how do you find all of these? Oh. Like, you're mining gold here. I want to see where your gold mine is. Oh sure. Well, you should subscribe to my newsletter, first of all. But now, well, I was gonna say more seriously, but equally seriously. Yeah, well, I've been an analyst over the years, so you get access to stuff. And I think also there's two other things. Well, one, I spend a couple of hours each day reading stuff. I'm lucky. Well, yeah, probably at least an hour. I'm lucky in that respect.  Like, you gotta set yourself up there on the mountain to hear things. But the other thing, working at a vendor, right? So we get access to all sorts of things because we pay the analysts and we get the reports, we commission surveys and things like that. So that gets you on the radar for this stuff. But the other thing, speaking of the person who doesn't want a vendor pitch over in the Expo Hall, us vendors also love it if you go give us your email address and you can download all these reports, right? So we actually license all this type of stuff across all the different vendors. I mean, obviously you should start with VMware and maybe not look anywhere else, but if you wanted more, you could look elsewhere and go to all the various vendors, all the public clouds and you can find many, many, many of these reports, right? And, you know, I shouldn't tell you this, but you can just not, I use and really checks. But you should definitely, if you're clicking on the links I have, use your real email address. Cool, so yeah, that stuff is out there. You just have to kind of like know. Anybody else got a question, comment, or even a story? Michael, who are these people that have high cost dry cleaning bills? Oh, the executives, yes. You know, I used to work with them when I was at Dell for a couple of years. And they, at first they seemed crazy. People who wear the start shirts. And there was a guy to use the word again who literally drove a red Carrera. Did I say that right, a red Porsche? And I was like, boom, I've arrived. Like, I've met the ER dry cleaning executive person. He's a very, very nice person, very effective too. Like, so yeah, I mean, you could derive them as like the MBAs, the managers, the executives. But the way that I look at them, and I kind of alluded to this a little bit, like if I'm talking to that kind of crowd, I would say something more like the following is like this platform, this organization, that is think of yourself an executive as a programmer, and this is your application, right? You're responsible for architecting it, for developing it, for testing it, for deploying it, and caring for it ongoing, right? And so really like they're the ones who, if you think about your whole, if you're into systems thinking, which I barely know what I'm talking about there. But as I've demonstrated, never has stopped me before. But like, you know, if you think about the system of your organization, they're the architects and the managers of it. They're also the people who like make sure that you get paid once or twice a month and that the business is functioning. So very random, on the last slide, one of the bullets was cloud parties, and I was just curious. Which one is it? Number six. Number six. What exactly is a cloud party? You know, I don't know what that is, but it sounds like fun. Yeah. I think, you know, the presentation's only like eight minutes long, so maybe he defines what that is. But yeah, I'm not sure. I mean, my guess would be, is it else there? He might also mean like dealing with the various parties involved. Not fiestas, but the people involved who are doing cloud stuff. But I'm not really sure. Maybe he literally means like, yeah, we're gonna go have an event and make people think we're cool. There you go. That would be great. Depending on your geographic region, that may or may not be possible. One minute before seven, anybody got a last question, comment, observation? If not, why don't you? So you mentioned a number of times, you kept coming back to this idea that there's a goal and there are goals and lots of goals, and it's like, you said the word goal more times than like John would have said it. Was that actually a subtle call out to Goldrat, or is that? Oh, oh, no. It just happened that you used that term. No, no, but yeah, yeah. I mean, that is a, sure, yes. I mean, it wasn't a call out, but your sentiment is correct, right? Like, I mean, as I started with, right? Like, it's always good to like know why you're doing something. Which, you know, when I was a young developer as, I think Matt Ray is the only person in here who knew me when I was a young developer. Like, I was happy to have no goals and stay at work programming all day. But now, like, I don't have five kids, but I have three kids, and like, I like to have goals so that I know when I can stop. And like, I think that's equally good in this kind of situation, because otherwise, like, you know, you just, it's never gonna end. Which could be cool, but the dry cleaning people, they don't like that. They like goals. They gotta go pick up the dry cleaning. Okay, well thank you everybody for attending. If you haven't been to scale before, this Saturday evening does have a social event that we'll start next to the expo hall at 8 p.m. It's actually open now, but reserved for under 18 until then. And there'll be food, drinks, board games, and just chat with each other there. If you like me, enjoyed Michael's presentation, could you join me in giving him a round of applause? And then the Cloud Native Track will resume in here tomorrow, and the conference will close with the legendary keynote speaker Ken Thompson. So stick around for the whole thing. Thank you.