t of images tacos And then we will use uh label studio to tag those those images In order to indicate to to the model which object it needs to identify Uh, so so that's that's the first step the second step We're going to be using jupyter no books to fine tune the the yolo v8 model Basically just running the the training the training api of that ultra liquid for the ultra lyrics provides taking it from our data set That way we can we can specialize our model to find tacos, right? And lastly, we're just gonna serve a small application that you can input youtube video and it will process that that video find all the tacos in the video and ultimately mark them up in in the video So the tools that that we're using we we we mentioned some of the high level tools But these are the behind behind the scenes tools We're going to be leveraging docker and for containerization and Portability all all of our tools are going to be running inside Containers, which is the the way that kubernetes needs their services to in order to be deployed and use and then For the infrastructure managing the infrastructure. We're going to be using cdk8 Which is similar to cdk from aws. I don't know if you're familiar with that But that's basically just infrastructure as code. You can you can manage all your all of your infrastructure instead of using yml's co-formation Manifests and all that you can use code And cdk8 works pretty much in the similar in a similar way just specialized for for kubernetes manifest instead of aws And lastly, we're going to be using the helm. That's that's just a package manager for kubernetes Basically, it allows you to Bundle a bunch of tools into a single package that you later can install in in a single as a single unit in your cluster So yeah, the the setup that we have here, we have already Skipped ahead some some steps because just on the best interest of our time it is all of you all That you're going to be seeing here. It's it's already streamlined to a proper amount of time, but these processes take a lot of time because a lot of these images are on the geeks So downloading them and installing them into into your cluster. That's that's a Slow process. So keep in mind that whenever you you try this it's going to take a a lot more time That we're going to take here on on this presentation Uh, that that's also the reason why we already set up some some parts of the of the cluster and we didn't do the whole thing live So yeah basically the first step the first step for for our project was setting up a cluster and We needed a couple of tools for that since we the point of of this project was running The workflow in a private infrastructure Whether that be because you have privacy concerns or because you only want a like a quick lap kind of setup or something smaller That's that's several that's some of the reasons that why you would need to do this So so Some some of the components apart from two cases the low balancer and the gpo operator This those are core for running in your custom custom hardware If you run this project in a cloud provider You don't have to worry about those components because the cloud provider already takes care of them It's already set up you then you you basically just spin up your Your cluster and it already has access to to the gpu if the if the cloud provider allows that But we're gonna be setting those up manually in our case Uh now for for for these demo. I've already went ahead and set up the the gpu operator Which you can see it's a bunch of services and and the the ones that that we care are This lot over here at the bottom Especially the demon set that that's that's a demon set that's running inside the cluster that can't talk to To the hardware on the computer so we can make use of the gpu in order to either train or serve the The immature commission model Similarly I've also set up a load balancer, uh, which I Can yeah Load balancer. It's pretty simple It helps you manage traffic incoming to your cluster and It also tells uh, or allows Kubernetes to know when it needs to scale down or up or or Allocate more resources to your to your services in case in case it's needed, right? And as I mentioned if you're running in a cloud provider, you don't really need to do this But if you want to make a point of running into your own hardware hardware, you're gonna need to set it up So basically that's it and and yeah, this is part Some of the parts that we already Uh pre-recorded Uh, this this here that I'm playing. I'm just setting up the same the same environments that the same components that I just mentioned I'm first creating a kubernetes cluster Uh, and then I'm going to install first the the load balancer and then the the GPU operator and keep in mind this this is this video. I'm going to actually skip ahead Even if it's short Uh, it's about two minutes long. Uh, but this process takes about uh from 15 minutes to half an hour. So so Being mindful of that just because uh, the gpu operator sometimes takes takes a while to to spin up and recognize the hardware, but but essentially we are just going to to apply some of the of the Uh Of the files that we already have in the project. Yeah, actually I I missed that part. I'm sorry All of these that I'm going to show here. It's it's an uh in a public GitHub repository that we're going to share at the end So so please keep an eye out for for the link and that repository has also a read me Which is documenting all the processes setting this up From scratch and also it includes a branch which is called start from here Which you can check out and you know start from scratch Basically, you you already gonna have the the load balancer and the gpu operator But but basically you can use that that branch to bootstrap your your own uh lab Uh, so yeah, that that basically is just running a bunch of commands. There there are some some Workarounds as well because we are using kind kind is a development cluster for Kubernetes And it has some issues with with uh with hardware. So we have to find a workaround for that. That's that's the docker Modification that we're doing here. Basically, we're just modifying or creating a sim link inside some of the containers in the cluster In order for them to be able to see the the hardware Sorry, okay Yeah, I actually No, it's in the other step. Okay, so the next the next step once we have a That's basically a verbones Kubernetes cluster what what what I just showed you just setting up the cluster Now we need to add our tools that we're gonna need for our workflow And we do that using help charts as I mentioned earlier. That's just just a simple Not a simple A package manager for Kubernetes that allows you to bundle up different tools into a single unit Uh, so the two tools that we're going to be using our label studio and jupy off. I'm going to show you some code and I'm gonna Try to I didn't open the code. So Oh, sorry. Is that better? Yep, like that. Okay. Okay. So, yeah The first thing that you're going to notice if you follow or read me is that uh, when once you bootstrap your your Project, it already has an infrastructure as code folder and with some with some files Uh, these are basically the files that cdk8 Generates by default when you start when you start a project I just added a couple more more files to handle the infrastructure that I was just talking about the load balancer and the The gpu the gpu operator but Cdk8 in order to allow us to create things Has the concept of charts which are similar to to helm charts, but not quite the same the same thing with cdk8's Charts you can define your own constructs that are going to be applied to your cluster afterwards So basically you you code all of your infrastructure. You run the the synthesize Command and that generates a bunch of yml files that you need to apply to your quivernets cluster The the advantage of this is that it is way easier to manage. It's way it's way less cold and Effectively you can also, you know reduce the need for for the DevOps engineer, uh, because this allows developers to also Uh Step in into the the ops realm So because it it makes it more easy for them to manage and they are familiar with code. So so yeah That the thing this way works the thing The way cdk8's works is that we need to create one one chart where or multiple charts with the resources that we want to install into our Or cluster what I'm defining here is an ai chart which I I have added two helm charts, uh, which are label studio and and Jupiter hub and I want to install those into my my cluster All of these are are documented on their respective Home pages or the documentation pages uh jupyter hub has a helm chart, uh And it provides all the all the parameters that you can pass into the into the helm command Which are the same that you can you can tweak here on the under the each of these props for for each of these, uh Helm charts So how does that look? Once we have ready We have or Sorry There we go Once we have our code ready the only thing that we need to do is run the synthesize project the synthesize command I'm sorry that that's the thing that generates the yml files that kubernetes needs to know what it needs to run So basically, uh, we have two files here. Well three files two are from the previous, uh Infrastructure components that I was mentioning and the last one is is the one that I just that I just compile Which contains the label studio and and and jupyter hub and Once it create i mean it once it creates that you you saw that We also ran the cube ctl apply Command which is the the command that actually Installs it into your your kubernetes cluster now So if we want to see the state of of that cluster is going to be oh, sorry So, yeah, basically this this this namespace contains all all of the tools that I that I have installed into that chart that particular chart that I was showing and the the outcome of that is that Again, it spins up the the services for both jupyter hub and label studio, but it takes a while to to Become completely green So so be mindful of that But once that that's done and you can see that everything is running fine in your in your cluster You can go ahead. I'll actually know I I almost missed that You can go ahead And and check out under services both both of these help charts create a service that exposes That particular application to traffic outside the The cluster And you can check those in here. So basically for label studio, we have this this External ip Oh, where it is Oh, here it is. So This external ip which if I go into Into the browser Open a new tab and go into that You can see that we have our label studio So Instance running right and the first time that you come into into label studio. It's going to ask you to create a file a file a profile Since this this this is a local setup for the moment. It doesn't really matter what you input here But you're going to need to create a a new Oh, no, it's already created Yeah, yeah Come on There we go. So yeah once once in label studio what we want to do is create a new project Uh, which which is the the construct that we that label studio uses for for us to upload our data set So in this case, I'm just gonna do some Name it tacos. That's what we're going to be doing And and here once you have your project set up you can import your your data set You upload your files. I already have it ready here So you just How many images? Yeah, that's a small data set. That's not Yeah, that's around 40 pictures. So yeah, but the real data set that we train the model with That's about a thousand images so yeah So before you start tagging your your images, it's going to ask you to set up your project, which is which is obvious So since we're doing Yolo, we we want I mean we can set we can choose from several of the other Template there, but the the one that I'm using is is that simple square text. So it already comes with a template. I'll just need to to Replace the provider labels Create a new one you can create as many as you want for for the demo I'm just going to create a single one. So so I can We can see the proof of concept So once you save that, uh, did it save? Yeah, I think so You can go back into your your data set and start and start processing we already see that we have here or or Tacos labels down here, which is which is something that we want to use to tag Images, but in this first image, we don't have any taco so we're going to just keep it and go into the second one And just select this label here and then we can just you know Tag it like that submit submit the the frame And you know you get the idea you you have to do this a bunch of times until you no longer wish to work on what you're working on So yeah, uh, once you have done all of your your, um Oh, sorry, that's not it all of your data set You can you can export it and and the reason we chose label studio for these demo is because it allows us Uh to export it in multiple multiple formats Uh, but especially the jollo format, which is what what we needed to make it work with with jollo v8 So you can export that and should download a tip with with all of your images all of your label Data and all of your tag data as well the the coordinates in which The images are are tagged now Yeah, once once we are done with that we we can jump into into jupyter huff and fine tune all our our model So let me show you also how we do that. Uh, however, this is also a very time consuming Uh process so so we already have it.  Uh, it basically spins up and uh an isolated pot inside your cluster environment your cluster Your kubernetes cluster so so you can install all of your dependencies in there You can have multiple users having multiple, uh, you know Pots of their own, you know or having multiple users Access the same the same pot so they can work in a collaborative manner Uh, and and all that kind stuff. So yeah, we basically what we did here was just install the ultralytics Uh dependency And once that that was installed we just ran a small, uh You know, we just ran three epochs just as a proof of concept But we are running this in our cover net is cluster, uh in our local environment Uh, sorry Okay, uh, that's that's for the training part of the of the workflow But uh, once we have all that done we also may want to serve or or model or or make it available To some application or service or whatever, right? Uh, and the process to do that with kubernetes is is a little bit It's both straightforward, but that doesn't mean that is easy. All you need to do is is uh, Create a docker image and pull and polish it into into Registry whether that be private or public. That's up to you But but kubernetes needs those those those images in order to pull them and spin up new pots According to your your indications, right? so Our our application is basically just a wrapper around Let me just It's basically just a wrapper around the the jollo v8 api Jollo v8 provides a pretty neat api to either Keep training your your model or just just you know predicting or inferring stuff from from from images from Sorry from video or images And all all I had to do was just wrap that around an application that that the receive request Either to to start a session or to start predicting Uh, so And all of that is uh contained in a docker images docker image that we can see the definition here We're just building the application, which is fairly simple. You can you can look it up in the in the repository Uh, but the important part here is that we need to use the the ultralylics image Because that contains the the tools to interface with with the with the jollo v8 api Uh, you can do that with python, which is easier. We kind of experimented here and we did a Go application that kind of interfaces It's kind of it's kind of gimmicky, but it proves the point. So So that's the reason why why we wanted to do it that way Uh, so yeah, basically you you just need to use your docker image once you have it You can build it and polish it into your registry and Uh The way that we're gonna use it in our project is we're gonna add it into our same same chart You can add it into a different chart, uh, but i'm gonna use the same one And uh, and the first thing that we want to do is is set up all the registry secrets These are Kubernetes secrets that we need to create in order to tell kubernetes The credentials for the docker registry that we're gonna be using Otherwise, it's not gonna be able to to download those images So so we did do that We're using basically the same the same environment variables that docker logging Persist in your system. So if you've done a docker login in your machine You can use this code as it is and it should it should work for you as well So we we not only need those secrets We also need to tell kubernetes what to install. So the first the first component that we're gonna need to to do Or to specify is is a deployment And a kubernetes deployment is is basically i don't think i Yeah So a kubernetes deployment is is basically you setting up a bunch of specifications about your app Which image you want to run? the environment variables and you know Scaling rules all all that kind of stuff. You you can set that up in in in your development You have a pot spec here where you which you can modify the same way with That you would in a yml file But you can do do it directly in the code here so The deployment is is the application itself. It contains the the image that I was just showing you the docker file for But as I mentioned, we also need a service for for the application to be available for for outside from outside the the The cluster so i'm also adding Adding a new a new service and what is important here is that we need to tell the service which port to redirect traffic to So basically i'm using port 8080 in my application. I need to set up my target port On my service to be the same as my application. Otherwise It's not going to throw any errors, but you're going to be scratching your head You know wondering why it doesn't doesn't doesn't work. So so yeah Again if In the read me I go almost line for line explaining what what is what each of these lines do So if if we if you have doubts, we have questions at the end, but you can also take a look into the repository Right now i'm just going to compile it again Let's see And i'm going to apply again But only the the last file, which is the only one that I that I modify the the one for more for AI infrastructure I didn't modify the other ones, but it doesn't really matter as you can see as Right now where when i'm executing these you can see that CDk8 Is is capable of identifying if any resources have changed against your cluster and if it hasn't changed You won't apply any any other changes to that. It will just roll out the new changes to your to your to your chart So yeah, what what that did is that we should see a new pod in our In our cluster Here it is Which you you started like 20 seconds ago Which is the container for for our demo application So to Which is our fame tag finder i'm just gonna Pass in A youtube video I have The logs here which has already started running, but you could also monitor that Directly on your on your pot looking at the lockside. I don't have the Lens metrics installing my in my cluster so not able to see them But you can do a cube ctl as well in order to see that So yeah, if we check at the locks Hasn't detected anything anything yet, but what what this is doing is First it downloaded all the video from from from youtube and and that gets passed into the ultra lyrics api internally ultra lyrics They split up the the video in frames and analyze each frame individually to find The object of interest right so yeah, we have some some found, you know some hits here And it this is also something that that Takes a little bit of time It can be improved a lot if you pre-process your video So for instance instead of passing directly the view from from youtube We do some pre-processing and maybe modify it to say 20 frames per second it the the time the prediction time it How it becomes slower, you know, it's quicker But for now it's it's gonna take about Three or four minutes and and yeah You can see that it's running. Where is it? Yeah, it's running on the gpu directly and it's running from from our kubernetes cluster so Yeah, basically, that's it. Don't know if we want to add something else. Let's just give it some minutes to finish I uh, we can we can check out the result Time well, we are waiting. Do you guys have any questions or something? Okay, gotcha I'm more curious. Where do you find all these like you strung together a lot of different pieces It's just sort of trial and error or there's certain resources you Reached out to for how do we find all of these tools and yeah the tools and you know I you know show the contents of some of these files you've applied for the cluster, but yeah, yeah, yeah There's there's a repository for health charts, which you can You know look look up, but I was already familiar with both jupiter health and and label studio. So I was aware that they had uh health charts provided for for kubernetes. So I would say it's a mix of Experience and research because once we settle on the proof of concept that we wanted to make We also had to do some research to see if You know, which model was the one that fit best for for project and and if the the tools that we had already selected were the the right ones Uh, the kubernetes part is that's that's that's experience. You know, I've been working with with infrastructure for Over five years now and mostly doing kubernetes. So I when Actually, yeah, the thing that we didn't mention is syon came to me because he's working with a customer that does AI stuff There's actually a Like a personal assistant. They have a booth around but He wanted to simplify they they have some concerns. They need to run into uh into Custom hardware or a more local setup And we started looking from there, you know, what how we can accommodate that and make a proof of concept Researching the tools researching the methodology. As I mentioned, there it wasn't a Straight it was straightforward. It wasn't easy Because there were so so many workarounds that we needed to tweet out But but yeah, it was it was we did a bunch of research at the beginning We had a lot of experience, but ultimately we had to also, you know, try on error a bunch of stuff So I would say it's a combination of of all of those Is the model selection mostly just you have some pictures and you run through this process and then see if it Spits out what you expect or uh, oh, yeah, you mean the fine-tuning process, right? So so yeah, basically Uh, you know once you you can run this this is an iterative process So so you run your your training several times and The output from yolo is is you know these checkpoints this weight weight checkpoints, which It gives you the last the last iteration that you you Run and also the best the one that have the best results So you can pick up the the one with the best results and how do you know? Which one have the best results, right? So how do we know when to throw out some some results from your training or not? Uh, you have to monitor the I believe this is the May average precision median average precision I think that's does the name of this stat here. This should increase over over time Uh Until you reach your your desire accuracy So for this demo we just settled for around 75 to 80 percent accuracy So we train our model until we this this value here came close to to that to that value I don't know if that answers your question Yeah On the back Yeah, uh, well we I know that's kind of uh, you know like a quirky show choice that we made because we just wanted to experiment Uh, and actually we the only python that we use is is to is is this grip here? Which is which is just telling you, you know jollo v8 to we need to train again. So so that's that's it Uh, but our application or go application uses the cli actually the the jollo v8 cli jollo v8 Also has a cli which you can use on your local system if you don't want to set up this this kind of stuff Uh, so let me see if this Is yeah, so here here's the result, right? So it's not perfect as I mentioned. We are aiming around a 75 to 80 percent accuracy. Uh, it it Yeah It it gets confused sometimes but but overall it works and by the way have a narrows It's not a sponsor in any way But should shut out to them. I hope they're they're doing great So do we have more more questions? Yeah Yeah Yeah, yeah, you basically that's that's I believe I'm not not 100 sure but uh, that's a partnership kind of thing Ultralytics provides that part of the You know, they they handle it internally. So we we just pass the link and and they download the video on their own So I mean, yep, uh, that's like an api, right? Sorry, yeah So the the question was how do you get youtube to let you have videos?  Yeah, and also it I think they limit the rate of And the answer was that the yellow thing that you're using seems to have a Yeah, they deal or something they seem to Use some sort of api over here. I think it's this one here Yeah Do we have other questions? Is it is it question time? Yeah, we we're at the question period. We saw the tacos Thank you. Thank you for your talk The cdk8, that's not a substitute for kube cuddle At the command line because I saw you using for setting it up. So you still use kube cuddle To communicate and it's just adjusting You still use kube cuddle in yaml files I thought you said before it was a substitute for those but I guess It it you still use them, but it's controlling the help charts and it's controlling the back So, okay. Yeah. Do I have that right? Yeah. Yeah, exactly. So the question was that The if the cube ctl is still relevant While using cdk8 But yeah, yeah, it is because the output of cdk8 is still yml files that you need to To apply to your to your cluster, but if you use cdk8 alongside cdk Which the aws cdk? Those those two are interoperable You can invoke cdk8 charts from cdk and they will execute So you don't need a cube ctl a cube cuddle and on that on that scenario But for the rest of the world, we we just have to deal with it We had some more questions Yeah, I was just curious what gpu you have in there gpu that's uh Uh 30s. Yeah gtx 30 70 Oh, yeah, actually, yeah Yeah Yeah, yeah, we we we were looking into that, but we couldn't get one in in time. So so, uh We do we have you said you had a um like a walkthrough tutorial or something. Yeah. Yeah. Oh, actually, I haven't I haven't show you the last Sorry for this Say so, yeah, yeah, you can check out the repository over there and and you know give us a follow on arkas nexus There we go. Now we got a lot of phones up taken pictures any other questions Up over here. Yeah Yes Yep So my question is um, have you actually considered using operators to kind of automate some of the things that you just did today You could just like run them in the cluster and just feed that, you know models into them and get the results Run operators Kubernetes operators. Oh, no, actually. I haven't looked into that. But yeah, that makes it faster You can natively run them. Yeah Uh over here in the front I'm taking a long way around Getting my steps in So it looks like a gpu operator is doing a bunch of the heavy lifting for the invidia setup, right? Yep So you're on a single node box because you're just running on your workstation. Yeah. Yeah. Does it auto detect which nodes have Yep, invidia and Do you do something in your deployment to say which nodes to schedule to yep? That's a good point I can show you right over here. We have see this this We have some files for cluster definition. This is specific for kind But you would have to do something similar for either bare bones Kubernetes or any any other cluster provider, but uh, yeah, we we are telling me Here creating those files in in the amount mounting rather into the the containers. So so those Uh, are the ones that that allow the the nodes to talk to the to the gpu All right, big round of applause for erin and san Thank you I'm sure they'll be happy to talk to you for a few minutes at the front or in the hall or Yep. Thank you very much everyone on the way to dinner That's that's no No, so let me try this one here That's that's can you hear me? Can you hear me? Is the sound yeah, good Perfect Yeah Any scale first timers in here? one two For some more that I couldn't count fast enough What about um, so cal locals? Who who's traveled the farthest to be here? We got a texas what? France, germany. I think that beats texas Sorry texas I have a cousin who's french My grandparents met during world war two and my grandmother came back from france I sent a letter to a 20 year old address in france and got an answer back and went and visited my family in veer st paul Anybody coming to game night? two minute warning Okay, it's five o'clock on saturday in pasadena This is optimal approaches for real-time machine learning Warm welcome, please for haishim kinesh Thank you Switching microphones, I think can you hear me at the back? Perfect. Okay. Yeah, so thank you for joining This talk saturday Five five p.m.  I have been around for a long time Doing a lot of conferences first time i'm doing a given a talk at five p.m. Especially here, I guess in california anyway, so my name is haisham from france that I said and I uh I want to talk about you know, so the original text at the original title Sorry, it was not all this like, you know, shiny technical words. It was how to treat coronavirus fatigue and burnout By cycling to the office and you will see, you know, you will see how things Will lead us to that. So, I mean, you know the official thing is how to use Spark on kubernetes apache spark, which is another Foundation, we have a lot of things happening here around linux and you know linden linux foundation But the other foundation that manages a lot of software Apache one so how to run spark on kubernetes specifically for machine learning and When you want to run real-time machine learning Okay, so a few words about me. So i'm a open source project architect with insta cluster We have a boot. Please come visit us if you want to learn more about insta cluster. We are part of netapp So I was you know previous jobs in my life previous lives and I have had a lot of kind of data engineering slash a lot of Machine learning but from the platform and tooling side of things. I'm not a data science expert There are a few data science experts in the room If you kind of but don't challenge me about models and stuff like this I do understand but this is not my main topic is how to put all this stuff into production and build Toolings and platforms around this Okay, just a quick disclaimer here It's important that to kind of iterate. There is no contribution to any open source You know, this is like a project that we have been doing internally and with a broader community So it's not there is no Hopefully we will get there maybe to contribute to some of the open source project we're using Yeah, my vision of things is necessarily biased. I'm talking about these two technologies by the way Which has like, you know, I work for a company. We offer a managed services for spark and on kubernetes So I am I'll be biased and challenge me Please if you think there is there was something that is was too biased And the the main thing is this work is based on principle of open source Whatever open source is open data We use a lot of open data and the culture of knowledge sharing whether it's with researchers or engineers or volunteers of any kind And yeah human learning is a lifelong Work in progress It's my slogan. It's like, you know, everything we do can evolve and we keep learning The agenda for today is pretty packed So I'm trying to kind of, you know, go as fast as possible To cover everything and then again, I'll be around if you want to ask any question There will be no demos. The last one was a fiasco So I'll put everything out there if you want to, you know, have a look at the stuff we're doing We start just putting notebooks and we're putting things together. Yeah We're not mentioned the team, but our people from Australia a lot of people in France And people in guys and girls in the US working on this for now We wait to get until get all the things together So I didn't mention the team, but this is it takes a village to build something we're trying to build here Okay, so what is the the story here? After covet burnout Something I decided to do cycling and only cycling to the office. Okay So it took me Like an hour to cross Paris east west to to get to a net up office And I start looking and discovering all this stuff and open street map is really amazing And there is a vibrant community of of cycling You know like cyclists sorry behind behind a lot of things happening there But you know start cycling was new for me I started struggling a bit to get To get to work on like, you know, like having the same kind of ETA having the same time a loss of struggle to get there So I wanted to first thing first like avoid traffic With a couple of guys do it like we were cycling to the office and but all this stuff is static You know just just provide you go from point a and b and provide you for different kind of, you know, maybe two three Not recommendation, but roots or pads So so like, you know, how can I get there? Maybe by avoiding traffic because traffic affects cyclists cyclists and obviously pedestrian also So I went to I don't want to mention or name it I went to this app that we all use and they say maybe this is pretty amazing You know, like maybe if I use it I can avoid traffic and they end up in parking lots Yeah, there were a lot of accidents and closed roads So, yeah, so the guess was maybe yes, definitely. It's not the right app We can use other apps and stuff like this and I said maybe it's Paris is not the capital. I mean, it's not a city known for cycling So, uh, Trivia question here. What is the capital of cycling in Europe? Yeah, Amsterdam, great. So I said, maybe I can check what's happening there in Amsterdam But the result was the same and these guys, I mean, who girls they have exactly the same question What's wrong with the algorithm of people, you know, what what's wrong with it? Why it cannot show, you know, things that are smarter like, you know, than this And then if you look here and you all understand French Yes, you guessed already No, I'll help you with this. So all the paths or all whether recommendations are not based on Real time or real condition. They don't take in account anything. So, I mean I'm not doing any bashing or anything this up here. I mean, this this is the best that you can have actually So far if you want to go in urban in cities from one point to another So this has really like great features and you get a lot of information But I was expecting as a dead guy that this kind of things like, you know are easily Like, you know, I say like easily can find a solution for this Okay, we'll push the list the things a little bit, you know More than this coming from cobit. We have all this conscience around, you know, being healthy being more like, you know avoid Get diseases and disease and stuff like this. So Why you're cycling why doing effort? It's really bad. Like, you know, you can think about all higher concentration of pollution that arise near like, you know road and networks So us thinking how can I get there? How can we get there? And maybe try to get the least air pollution exposure routes, you know, so Have the least traffic Have, you know, let's be ambitious and maybe it's an utopia and have everything in one place that give us like different criteria or kind of Sorry, but based on different criteria And give us the best route From one point to another So unfortunately same problem. There was nothing out there There was he says some kind of, you know, work Mainly in academia that started mainly in europe Then there was it was taken there was a lot of work happening in asia I think it's a really a big issue In india and in china places like this, you know about air pollution, but you see that there was like a like At least people start thinking and sharing and like a lot of research articles around how we can adjust How we can predict how we can use, you know, at least machine learning at that point like graph normal network approaches to predict this kind of things and You you I mean you can see where I'm going here You can do this stuff only if you have real-time capabilities the Everything that happened on like, you know, like where you are traveling where you are cycling where you are walking to work Really that's going to happen. It's going to affect your route in terms of pollution and And exposure to put event So, yeah, we also kind of try to check in different areas where there was something equivalent But we didn't get you know, we didn't find anything Relevant to reuse at least as a open source okay, so Just one definition here at least in our context when we say real time So real time means that it's near real time Okay, we can discuss it here we we kind of you know Care more about Getting the right information rather than the speed of getting it So, you know seconds are tolerable like we are not going sub seconds to generate but so This is like the kind of target and real time Learning real time machine learning. Sorry in this case is just the application of machine learning The usual machine learning models and approaches To make decision in real time and based on like, you know Changing environment and based on Information that you gather there is an accident somewhere There is I don't know like, you know a peak in pollution because of traffic. There is an event There is a game tonight in LA by the way There will be probably a lot of effect and we want to capture this stuff We want to get all this in one place to you know, uh, get the the most relevant Prediction Okay So from uh thinking from whiteboarding to To building something so here is the kind of you know, open source stock again You can challenge these choices. There's probably 10s 10 others that you can choose And it will be legit We uh tried started with this building blocks like, you know They were common. There were a lot of help a community and we had a lot of skill set Around at least in the company. So we have contributors to this open source So we can tweak and twist the knobs If we want to kind of tackle challenging stuff So at the building block and this is what I'm talking about is You know Kubernetes everything runs on Kubernetes And there are some exceptions, but in this case everything runs on on Kubernetes and the key thing there, I mean at least spark and spark structure seeming Is the main this like the building block of our processing and our training And you know like the whole kind of Brain of the of the platform You could see things that uh, honestly, I've I have learned on the fly like, you know, like feature store I let at least I know worked in person aspect like this But an online feature feature store was pretty amazing, you know things like that act between a bridge between like, you know Um Data and machine learning there were a lot of key aspects that I learned But let's focus here on the kind of the the main thing, which is the infrastructure side Okay, um so What is the biggest challenges at least that we faced or Like like a lot of data data in a more broad or general Meaning data teams are facing when they try to transition So I try to tackle real much real time machine learning So I had a lot of inspiration and interesting discussion with ship She has been doing an amazing job and in this field She has an amazing website that you can read. She did a lot of interviews She went to a lot of tech companies in the us and china And she collected a lot of information around challenges when when we deal with real-time machine learning She has a of course, I will do some publicity for her an amazing book If you want to learn more about Designing machine learning system and there is a whole chapter about Real-time machine learning there's another mention also for to know our for practical ml ops And it is and I think another book coming so we kind of Have Had two different challenges the one from the machine learning or like say the data science side of things It's about modeling. It's about how do we extract features about what is the cutting edge model using You know like attention neural graph networks And stuff like this to train models using geo spatial time series You know and name it like a lot of stuff These are like I can say the blue stuff And the other side is the infrastructure or the tooling stuff And I I cannot agree more with chip and other experts on the field the problem The challenges are from the infrastructure side. I'm not saying I'm not minimizing Of course the the work that is happening from this side But if you look at it guys the problem is from this side here We don't have any blueprint. We don't have any kind of People sharing knowledge on how to stitch things together how to build things together It's it's still kind of new To to to have something common That is adopted and kind of stress tested at least in the community So yeah, these are the challenges and among them Since we don't have enough time We kind of focused on the scalability This was a key Long running. I mean never ending jobs collecting sensing information forever You have peaks you have whenever we have scalability sometimes, of course We are we have to think about cost. So we have to scale down or reduce the Like the the workers or you reduce all the infrastructure as soon as possible latency, of course is key I hear when I mentioned latency in the in the in this talk I I mean mainly latency about the processing. I will not mention latency about serving which is another You know challenging Area monitoring monitoring again is key I'll try to explain why and the distributed training And inference which I just said that I'm gonna kind of avoid for this talk Doug we try to cover here Okay, so In real time What is the most critical aspect is to have the the freshness Of the features where the features is this data the there as we said the serving latency the and the the possibility to have You know the accessibility of data and these pipelines Always running in a in a robust in a reliable way This is a kind of key definition or a building block. We want to keep in mind. We're building this work And Yeah, so spark and Kubernetes to the the streaming aspect of it We could manage string processing as I said training scalability and resource Efficiency which was key, especially when we start to dealing with deep learning stuff and gpus And we start burning a lot of money and you know companies say, you know, different companies who work for say like, you know, you have to be Be careful with your GPU bill at the end of the month Okay It's customary to ask question just to check that you are still alive, but it seems like, you know, you're doing okay But it's really legit. I want to ask you who is using spark on kubernetes Okay, no one That's a good transition in that case I will kind of go back to the genesis not not genesis of spark but Why we moved from spark spark from From the hadoupe ecosystem or at least yarn to kubernetes So Apache spark is the number one analytic analytics engine for big data and ai This is as objective as I can say The reasons are it has two green stars So far or still it's versatile It can do etl el t real-time machine learning bi That's really key. We don't want to choose just use one tool to do just real-time I mean you you you you can see what i'm going with this It's easy. This is subjective, but You're you're bringing someone from java background python sql. They can interact with spark The distributed aspect of all this big data is kind of hidden under the hood We have a lot of framework that you don't have to think about how your workloads are distributed How your python code even is distributed you just write code And it managed the distribute aspect for you And it's fast or at least it was fast This is what it has like start to be a gray area and we can discuss it But it was fast until like recent framework to do like massively Paralysable Read and write and processing. Okay, so this is spark. It was born in that ecosystem that era of hadoop You know hadoop is still around we don't cannot say Still like you know the king is dead the long live the king It's still hadoop is still still around and it's gonna It's gonna stick around so kubernetes is the new framework But I think you know as i'm saying hadoop will be still around so in spark Spark can do all this processing, but it has it needs a cluster what we call a cluster manager So cluster manager does all the orchestration of the jobs And all like just in a nutshell it knows where to put a processing unit in which worker worker is just a vm or Like a physical physical machine just to kind of simplify things here so It went through different resource managers all from the hadoop ecosystem era There was this standalone built built in like functionality, but really met it, but we had Mesos that has gone already and yarn which is still the most widely used And kubernetes is the like the most popular or the last cool kid in the block So we're really happy that this was a reality for spark users It took an amazing team and a team I mean a community of people to do this And I like to to say that from the apache Foundation side and from the linux foundation and the cncf So work together to make this like as a reality So like this is the history of of the spark on kubernetes spark on kubernetes. Sorry So it's kind of recent it went ga in 2021 okay, so there are still a lot of work to do And as for the last version we have like, you know, we still have a lot of major enhancement Happening in spark on kubernetes So just to give you a sense on what was the limitation of spark on yarn The key thing is I mean the key problems like you had this global spark version You can see like now now retrospectively. We know how it's doing. It's definitely good better on kubernetes, but This problem with isolation a limited I put a star there, but it was really limit limited docker image support and a lot of resource overhead And this is just like you give you an idea how yarn managers the yarn managers manages the cluster for spark As for kubernetes, of course, you know, you have this native Dockerization you have like simpler dependency management. This is really great when we we train machine learning model You have reliable execution.  I mean, it's a standard agnostic cloud infrastructure. Yeah, that's a really important point not not to to kind of miss it And one of the key things I have noticed that is this rich ecosystem for kubernetes that spark users like us You know can get, you know advantage of That's a that's a really cool thing There is two ways to run spark on kubernetes. That is the old-fashioned spark submit Okay, if you're familiar with this and the other one is we are lucky and there is a spark on kubernetes operator Really proud that we as you know, the company I belong to we contribute and when maintain this spark on kubernetes Covenant operator it was open source by google, but actually we can it's it's you can use it on any platform It's really open source and it's config like, you know, yaml style with sugar on top I say with config maps volume Affinities things and we have a concept that we created locally in our words called vngs virtual network and node group. Sorry so, yeah This is the spark operator and as I said like, you know, it's it's also a really kind of active project We are having a lot of support and I kind of you know insist on this. It's really important that If you are planning your journey or for at some point, you know, you want to test spark on kubernetes Please don't use spark submit you will hate it and because you know, it's bad Use a spark on operator operator and you will adopt it and yeah I am asking you like if you're starting this journey if you're considering Moving to to spark and spark especially spark on kubernetes if you are running kubernetes already It's easy just to deploy, you know, like a spark cluster and try try to start testing it And this is not just an engineering, you know to kind of migration to the next tool It's it's really kind of has some distinct advantages. I was showing you and If you look at it, this is a like this was from aws just the this this small kind of Cut or cost reduction for some of our customers It's represent millions of dollars per year. So it's it's really huge I mean in terms of cost in terms of management in terms of a workforce like an army of engineers you need to maintain Something like spark submit or spark on kubernetes So that's really kind of it's justified and the whole the whole industry is Going to I mean this this way Okay, just some challenges uh of running spark on kubernetes and offer solutions So as we said, we're trying to uh talk about monitoring scalability latency and model training This is the iceberg and don't be scared that there's no titanic in the story here We are kind of definitely in a situation where there is a lot of logs logs logs log and logs Everywhere. So I would recommend that you start thinking about this if again you Embrace the journey start thinking about your monitoring your login Monitoring before anything else like, you know, even if you are doing dev If you knew if you are doing experimentation Try to have this building blocks in place because as I was telling you there are events from spark Of course the driver the executor as you said, it's a cluster maybe sometimes hundred Thousands of nodes in some of the cases there are kubernetes logs and it's really hard to reconcile All this together in wall pace so again an amazing ecosystem and Fluent bit is something that really helped us a lot and Prometheus which has a built-in spark has a built-in Prometheus in it. So this is something that I could recommend again There is a consensus in the community about what to use but this is a building block and this kind of, you know ash where become more controllable more easy to to kind of, you know Monitor and log one of the key things here also is that You When you do this monitoring What we discovered is that we learned a lot of things That we were not expecting from me not monitoring We learned things about like, you know, or executors of over provisioning We learned also about like partitions issue, which is like a common disease with spark And a lot of things that we learned through our monitoring system So I really recommend that you kind of shift the paradigm It's not just like, you know to kind of firefight if things happen But it's also to help you to be more proactive Because the the complexity of such system you need to have like, you know, this This productivity if you have machine learning real-time machine learning jobs running in spark and cobalances Scalability so scalability there are three key factors here to consider Cluster sizing infrat choices and specs when I say like spec the kind of VMs or the kind of if we are EWS the type type of easy instance easy to instance you want to choose Dynamic allocation Try to say a few words about this and the shuffle data So shuffle is just like if in a nutshell here, if you are not familiar with spark Shuffle is all the data since we distribute the data on on each workers Shuffle is the data that we kind of transfer from one worker to another to you know through the network So this is something really important than the kind of symptomatic in spark So there's this three Three kind of building block sorry that we need to consider here. So for sizing it's important to Just there is no Automatic way to do this. It's just common sense and don't forget what you know, like best practices from From software engineering It's just a continuous and repeated exercise You know, you need to know at some point your data sources so the clusters that you need to put in place and the type of instances need to be like, you know Enhance your parallelism in in relation of the sources So parallelism means that you want to treat in parallel different events or different data coming and your resources your computer source needs to be like Kind of geared and configured to respond to this kind of Data you're having in it's streaming So streaming is cpu bounded cpu bounded in this case means that we spend a lot of Time doing cpu. It's not like an input output. We don't do a lot of reading and writing But you have to be careful in this case because we In our modeling we went to we we do a lot of stateful We do a lot of aggregation and stuff like this. So we have to choose resources that are You know, like cpu optimized, but we have to think about memory because we keep a lot of state, especially in streaming state event in memory Deep learning. So as I was telling you, we used a lot of gpu at least in the training phase You have to be careful also not to burn a lot of gpu on this. We have a mix of cpu and gpu on our Kubernetes we have was as I was telling you what we call virtual node groups We we create affinity and tint and it helps us to kind of make sure that only the The relevant workloads go to gpu's Yeah, and we actually just to finish this it's a mix of cpu and gpu and we make sure that we spend a lot of time on cpu rather than gpu's when we're doing with this kind of things Okay dynamic allocation. Maybe since we uh, we are not all kind of this is like a really advanced topic into Spark, but what it does just in a nutshell So it's kind of do an auto scaling like the same thing We have an auto scaler on on Kubernetes, but it's tend to do it in a smart way So for everything bot it works perfectly and it's really great feature But when it comes to Structure streaming it becomes a little bit tricky and it works really poorly for certain application So and uh, I mean you're familiar with you know this gira tickets and I have been watching this ticket I think for the last four years. It has been open I mean, we still have a vibrant and like, you know, a lot of committers But just just to show you the complexity We actually there is no agreement on this. How do we Dynamically allocate resources for streaming Structure streaming jobs So this is independently independent from Kubernetes. It's just a spark in spark problem So when you run spark on Kubernetes, it adds more it adds more complexity and this feature May cause a lot of issues So we have to be really careful with this when if you want to use dynamic allocation And don't trust it's there's no kind of autopilot in this I would say the the another thing is the the shuffle problem. It's which is like somehow consequence of the Problem we're having with the dynamic allocation, but not only so this external shuffle topic has been around for A long time and we have been waiting. You see that some of these future work External shuffle service. Hopefully it's gonna come soon So there are different open source framework that helps you to have this external shuffle service The store your shuffle data One from obviously multiple from AWS so you have one two three from aws One from Intel and another one from IBM and we have been testing Not not to be honest transparent not the last S3 csi driver and the high performance storage But the other one we have been playing and trying to reach out to the The creators and the the maintainers of the project and it's like a A choice a rational choice or educated and say choice here in this case And we are using the IBM spark s3 shuffle But it started from like, you know 3.2 spark like most recent spark version and it helps us to Kind of tackle this problem of the external shuffle whether for badge or for streaming it works perfectly latency again, it's it's This is just a small remark and I would say it's like for the young Generation or maybe just people who are starting in tech or want to learn a few things We used to learn I mean for those like, you know who lost her like me. Will you lose to learn latency? Numbers every programmer should know And it's kind of fascinating that a lot of young Young, I mean, you know, kind of they don't know anything about stuff like this So it's the basics or the fundamentals are really important So we may talk about cutting edge machine learning and like, you know, spark and kubernetes But at the end of the day, it comes through like, you know, knowing your basics And in this case, so latency We we did we spend a lot of time like, you know Doing wide boarding session and as I said, it's just the processing from a Kafka topic Spark stress and streaming to a Kafka topic. This is our kind of highway. This is our key Pattern and Yeah, kind of key things I would recommend to look out to watch out for is the spark configuration So whatever you learn before from your spark configuration kind of you need to tweak this and make sure You kind of master what you are doing Sub-second latency expectation are a challenge and I can guarantee you for this project with the amazing people I have been working with It's really hard to get even though like a lot of People out there say they can reach sub Sub-second latency This is something really hard to get And yeah, it's always it's always like a trade-off between using stateful stateful again It's like when I'm doing streaming and I want to keep a state like I'm doing the aggregation I want to count the number of red cars I get like every time I have a sensor on a highway Just to give you a sense of what's what we are doing and I'm counting the number of red cars that are going through the highway So I I need to keep the state. I do a lot of aggregation I need to keep a state with stateless is why when I don't care really, you know, what's happening It's just like which color which car has the which color has the car that I'm seeing It's just like red or black or whatever. So this is this again, like this is it has to do with design and The the the problem you want to tackle You want to solve Now this is a really important thing to take in consideration And things that we learned is use as much as simple computation Just say when you doing your data transformation or enrichment If you want to do like complex joins and stuff like this You won't succeed in getting anything latency, you know, reasonable latency So be aware of this, uh, like this is a lot of common sense, but it's important to kind of remind that these still Works like pretty good when you doing a dealing with these complex systems Yeah, so this pattern always use a message bus like, you know kafka I would say, you know before and after And in our case, yeah, it's it's kafka spark kafka And we have another pattern which is kafka spark and we have kassandra already this you always use this Rather than we've seen some configuration where like just like, you know Supposedly by removing all these layers or all this middleman We can increase the Improve the latency and reduce the latency, but it doesn't work This way by, you know, removing something like kafka. You lose a lot of functionalities in your in your platform Yeah, I mean rocks db is just a new Feature on on spark and it helps to store all the states Especially when we said like mentioned the stateful streaming pipelines and it helps to Keep all the states and we don't end up in this Gvm out of memory and other issues ML training this is really amazing I mean a lot of ML framework, especially if you are coming from like, you know, I would say the The the mono worker you you just rather running python on a mono machine or a single machine it's kind of Disturbing because a lot of this framework were designed for a single load environment Spark is another as I was describing. It's a cluster when you have a You know, like multiple machine you have to distribute and paralyze your workload So this is something you need to think about before you choose a framework And spark mlib, which is like a native machine learning library in spark is lagging behind Hopefully I'm not in the Apache Spark conference. I will probably get people screaming at me But yeah, yeah, I'm saying that but we should probably contribute to this But it was it's it's kind of, you know, not getting obsolete because I intentionally put this We have things like torch distributor, which is was included recently. There is still life out there But all the modern frameworks are not there So what we did is definitely use, you know, like the major kind of framework transfer flow keras and pytorch and Yeah, it's things like horror board horror board story transfer flow extended and there is this I mean, it's open source. But yeah at some extent There's no one from nvidia in the room No, okay. I'm safe So it's it's open source and it's pretty amazing tool and it's kind of unified for etl And machine learning, which is great. So you can really kind of, you know Using gpus, which is I was asking my friend recently like, you know, I don't understand how gpus can accelerate Things from etl and elt, but it works. It works pretty well So you have like with the spark rapids from nvidia. You have we have like a really nice framework that we used Intensively, it's just like and it's really hard to run outside nvidia, so We said like we are doing a lot of open source and we are kind of, you know, having this philosophy of sharing and you can run it on your any commodity Hardware so this is a bit, you know complicated I'd say at this point But yeah, we have using these frameworks and they are pretty easy and we kind of publish a lot of docker image Out there where we package everything together A lot of companies are using not our product, but at least our docker images to do a lot of stuff with especially the nvidia As the nvidia spark Accelerate rapids Okay, there is another thing. I mean, you know Sparkling component is great But for batch because anyway, even if we do in machine learning real-time machine learning at some point you will have to do Some batch and the default scheduler the default component schedulers kind of, you know, lacks a lot of What we call like specific scheduling strategies and So, yeah, it's kind of End up doing a lot of starvation for jobs And the deteriorate the the performance So the solution for this there are two major product a project story Volcano and Apache unicorn Volcano is in the scncf and it's growing and it's there is a like a vibrant community that is contributing to this And it's really good. It's as a custom scheduler. We can do like job priority preemption We can do a zone awareness because we can discuss this also like latency matters when you want to Locate your workloads in the same kind of data center You know, so we can do a lot of things around especially with volcano things like, you know, as I said tasks topology and job priority and gang scheduling Okay for the deployment, this is really something that it's it's kind of There is no we are and we have no Kind of guarantees yet. We didn't benchmark anything but like just as this mlops.org Deploy your model. Yes. Deploy a package your model as a docker image With all its dependencies and push it. We are using, you know, kubeflow and kserve to do this But this is not easy. You end up with huge docker images So we are still thinking about model optimization and model compression. They're really important And we are experimenting with on annex runtime, which is an open source from microsoft I don't think it belongs to any foundation But it's still maintained by microsoft just to manage all the dependencies and yeah So with all this We achieved A platform that I was showing and especially that Criteria around latency around scalability and having a monitor platform in order to You know build or tackle this real-time machine learning So I will try to conclude quickly. So yeah, try to convince you again if you are Starting your journey if you want to experiment at some point Spark on Kubernetes Is the thing to go to native native integration Yeah, you have all the best practices and best things that apply to in a kubernetes Apply to spark for free We kind of could achieve scalability latency and other things like full tolerance model training and serving with some caveat with the serving And integration with the ecosystem Okay tk key takeaways as I said use the operator if there are any contributors, please contribute Monitoring is really important Use the best practices from spark compatible with Kubernetes the rich ecosystem And contribute to the OSS There was a talk on Thursday saying that open source is not only code and open source I can confirm is not only code. So share everything you have there share your experiences Write a blog on medium and say like I struggle to do what this guy was telling for an hour It doesn't make any sense and like, you know Share your code ideas on github's Github sorry, share your challenges. This will help the community and this will help, you know grow and Try to solve real life problems because remember my problem was to Be healthy was to stop using my car. Maybe solve the planet if I can save the planet. Sorry And technology can today like, you know recommend the best Movie or the best food or like in real time The next food the next movie I have to watch but not like, you know, as I said Solving real life problem that maybe will affect all of us Then that's all for me keep cycling when you can that's all for me. All right. Are we ready for questions? Hi, uh, great talk On one of your slides, you had Kafka going to spark going to Kafka again with rocks to be in the middle Why not just use k tables or ksql? Why add spark if you're already in a Kafka infrastructure? so because I mean this uh We have this common pattern and I should have just to fight this so the reason is that we Kind of ride from topics that open data come in real time So we need a topic in Kafka and I guess you Why do we have to put spark streaming in the middle because we do a lot of complex transformation that We need something like spark that is distributed that is robust And we do a lot of parallel joins from different streams And then we'll ride back to another topic and this topic is used for feature Extraction did I answer your question? That's what that's what ksql is for I mean again, uh, this is something we should probably consider and remove Stuff like this, but we definitely needed at some point something that can scale and be highly I would say distributed to do this kind of things. I'll yeah I'll Definitely have a look at ksql Thank you more questions one up here Regarding monitoring the spark structure seeming job like there are two ways you can monitor like writing a listener Like async listener or a sync listener So, but even if you write a asynchronous listener, you still get the logs like only only the logs, but One of the struggles we are getting like how do you Get inside of the log you've got to go to prometheus or gaffanas. Have you tried that kind of stuff or so? Yeah, that's a good point actually we we use prometheus directly on the workers to collect the logs and we tend to Kind of you know reduce all the noise We target this is a learning process. We target few metrics that matters for us like, you know We make sure that you know Metrics a b are there and we kind of on the fly we do this process and we reduce We we kind of parse and reduce all the logs. We want to collect from the workers And we do this with prometheus Yes, yeah, yeah, we use these tools but we kind of there is some Intelligence there is some work to make sure we are not having everything And we need to parse it and then you know kind of process it We just collect from the source all the only the metrics that we are looking for This will kind of add some latency because this is a process. This is a thread running So it's awesome overhead, but it helps us to have like, you know, as you're saying like Collect the right information, especially when we do a streaming Did you find an optimal bicycle route? Not yet. So we're still working on this not yet It's a still struggle. So yeah, hopefully I'm still cycling but You know It's it's as I said, like sometimes it's more like human learning than machine learning Maybe I learn more how to avoid traffic than any system can help me too All right, great. Thank you round of applause. Thank you very much. Hello Good Perfect Yeah, hello, hello, hello, hello, hello. Yeah, it's okay, right? Just don't look at the wall over there Yeah, perfect How many of you this is your first talk of the day Nobody okay good because it's also the last one of the day We'll give it about one more minute Who traveled the farthest to be here today? Everybody's so Cal locals I didn't catch it Panama, were you in a talk earlier today? Okay Panama anybody beat Panama? All right, Panama it is that's the winner for this talk Okay, it's 615. This is the last talk of the day Saturday. This is scale 21 We have discovering business insights with open source machine learning Round of applause, please for Syria. Syria take it away. Thank you very much First of all, I would like to thank all of you for making it to the last talk of the day I feel very appreciate of you all that you guys make it to this talk and So to begin with today, I'm going to talk about discovering business insights with open source machine learning tools It's all about predicting customer behavior for your for the growth of your business One thing I want to say is in this age of LLM foundation models generative AI Whatever you call Most of the people are trying to solve their use case using LLMs Generative AI and all like large foundation models It's true that they are efficient They are effective in some way, but They come up with their own set of cost And that constitutes a whole talk in itself for some other time Today I'm taking a step back and exploring some simpler models simpler time series forecasting models Which can solve some useful use cases for businesses I would like to give a shout out to my colleague Hema Viradi who is a data scientist at red hat and Who who was the equal contributor to this project as well? To introduce myself i'm Surya Prakash Pathak a data scientist at red hat I'm based in Berkeley, California And you can reach out to me in my email github linkedin Whatever works happy to be Connect connecting with you all Yeah, so into today's agenda. I'll be talking about Why customer growth is so important for anything for business? And I'll go through some data sources time series models model insights And talk briefly about one more model monitoring tool And I'll wrap this up with the demo So this project was started actually The main goal of this project was to predict customer behavior Over time for the red hat internal products It's like predicting customer potential For different customers for different red hat products Although the data was private. I cannot share it over here But the model is open source And it's adaptive in nature So when I started this by talking about business insights and customers I can adapt this model to any other organization Let's say you have a github And you have a contributors to some projects And you want to track your contributors over time and you want to track you want to Identify potential contributors for your github projects You can also use this model. You can use this kind of framework For this particular talk, we have extracted data or generated data using open AI credentials Just for that That's all LLM we have used nothing more After that, we employed some time series forecasting model We got some forecasted data And using those forecasted data We classified our customer base Or for other use cases say Contributor base into High, medium and low Potential zones Now one of the reason why this predicting of customer potential is very very important is It gives you Advantage on which direction to drive your business through For example, let's talk in terms of github projects or contributors If you know What if you can predict your customer behavior over time It helps you to invest on your it helps you to manage your investment on your projects It helps you to manage your vacations as well So manage your resources manage your investment And vacations for your projects If you know what your contributor is when your contributor is active or when your contributor is not active Or if you can determine These are the high potential contributors and I should not lose them You can reach out to them give them offers or give them good services so that they be in your organization Or track low contributors seeking help You can give them help and suggestions so that they move from lower set of low potential zone to the high potential zones So in this way you can manage your business or any kind of projects So for time series modeling we are using arima To predict customer behavior over time Basically, how much a customer spend every month? We are predicting with respect to arima time series forecasting models So let's move on to the data sets that we have used so far. This is the simplest Ideal data set that we have used The data set is generated using llms So here we have different accounts. So one of the Important thing that you need to understand about this kind of data set is it's not one time series data set It's a multiple time series data sets. Say for example, you have 50 customers 50 contributors or 100 200 You have 200 number of time series and for each time series you need to make a predictions and out of that predictions You need to classify those customers based on their predicted data So here we have different account name account name one account name 11 Different dates in this case. It's varying with respect to every week You have the actual value you can correlate that with the contributor activity over time You can correlate this with the revenue over time And this is the predicted value from the arima time series forecasting models And this is the mape score Now what is the mape score mape? It's basically median or mean absolute percentage error It's the percentage error between the actual and the predicted value Now this is the good set of metrics for evaluating time series forecasting models If the mape value if the actual and the predicted values are equal then mape is like zero percent Very very accurate, which is an ideal case If the more the mape is the less accurate your model is Generally it is considered if it is from zero to 10 Percentage the mape is generally considered accurate If it's from 10 to 20 or 30, it's it's considered moderately accurate But if something above that it's generally considered A less of an accurate so you need to Change your model add up features do some data preprocessing and so on and again train your model and again predict it So that's how you Go into the cycle of like making your mape as accurate as possible So mape is a good Metrics to look into to check whether your model is working or not So we talked about the data set And now we let's talk about a time series forecasting models So here i'm highlighting two basic traditional time series forecasting models these which are open sourced And time series forecasting model are basically Those kind of models which takes up the data points from the past and based on that past trend it predicts the future You have arima, which is the set of statistical models That caters to the group of standard structure of time series data Arima it's it's based on the underlying correlations One interesting thing about arima is for every kind of time series data You need to define the hyper parameters to run the model Based on auto correlations and Based on auto correlations of the past values And moving averages of the past values So for profit which is similar to Somewhat similar to arima, but it's more of an additive model and tries to detect change point It's more like a curve fitting model. It's developed by meta. It's an open source model. You can use it So one good thing about profit is it's easy to use with default hyper parameters So you don't need to define hyper parameter at each run. It's automatic And it's So this is based on a personal experience I felt that profit works much better when your data is seasonal So if every season your data is like have some cycle of Cases then profit works pretty well arima You need to define your own hyper parameters, but it worked pretty well for us because we were dealing with Number of time series like for example 50 time series data 100 time series data so In order to run for all the time series data We chose arima Now I said I told you we have multiple number of accounts and multiple number of time series But for arima the catch is how to define hyper parameters for each one of them now to That is resolved so easily using the auto arima package with auto arima It helps you to determine the hyper parameter automatically based on the individual time series data So here in this curve you can see the training points given by the blue lines The orange point is the test set And the green line green dashed line at the left hand top corner is the forecasted points So what? How we run our model is we train our model And we test we keep some data for the test set And see through mape if our model working well or not So here in this plot we calculate the mape of 5.1 percentage which is pretty accurate for our use cases Now I'm again saying this mape is pretty accurate But this also depends upon the kind of use cases that you are in Sometimes your data is so sensitive that even a point one or point two Percentage difference makes a lot of difference in that case you need to have a very very accurate mape But in some cases some flexibility is fine So here we have the training set here we have the testing set and the forecasted set these green points Now based on these green points based on this forecasted data We classify the ultimate customer set into different zones high potential zones low potential zones and medium potential zones If we move on So generally if there is an increasing trend So one of the Ways with which we calculate trend is by calculating the slope of that line So if we if we consider slope as the trendsetter of that line Then for an increasing trend you generally get a higher amount of slope Now if your data points are not changing the value over time Then that is generally falls under the place of medium slope, which is constant slope And if your data points is decreasing over time Over the x-axis then those kind of data points are Then those kind of trend those kind of slopes are generally in the negative side and those are considered as low medium low growth slope So based on these three different classification of slopes We classified our data our customer base Into low growth medium growth and high growth candidates You can do the same thing for your github contributors list You can track your contributors activity over time And based on your contributor activity the forecasted data points Let's say you want to calculate your contributor activity for next month And next month for the four weeks you see an increasing trend then that means those particular customer or contributors Generally fall into the high contributing zone For the next month, that's the catch if I consider If I see any contribution activity with this constant over time Then those kind of contributors can be regarded as the medium Potential contributors. Let's see some of the examples over here This is the case of low potential customer or low potential contributors. We can assume that way as well So in this case we see an up and down phases for this customer or contributor Sometimes it is high. Sometimes it is low But if we forecast if we really see the forecasted green points over here This is the forecast for the next five weeks from the testing phase So which means that for the next five weeks what I'm seeing is this for this contributor They are contributing on the lower side, which means they are not contributing. They will not be contributing enough That's what I'm predicting it now And that's why I've segmented them in the lower category Now we should not make this kind of result as the ultimate truth But this kind of result can be a key indicator to inform the customer success team or those teams which is in In conversation with the customers or conversations with the cost contributors If we see these kind of indicators like for example in this account name 45 It's on the low category. We can inform customer success team that so and so customer or contributors in the lower category zone Is there any problem with them? they can then check with their customer or contributor and Talk about if there is any issue if it is not this can just be a false positive signal Similarly, if we go to the another example, which shows you the high category of customer This customer or this contributor shows an increasing trend over the period of time And for the next five weeks also it is it is seen as an increasing trend And this also calls for a key indicator or calls for further investigation from the customer success team Which means that if somebody is performing really well if some contributor is performing well if some Customer is spending well on your product. You can give them some offers. You can give them some perks So that they can stay there. They can perform much more better way so I think the main aim is to Keep the customer or contributor intact in your projects or intact in your business So these are the ways you which which you can track those contributors or customers And just start the conversation if there is any problem. So these kind of plot Can be a first hand key indication To check with them If there is anything Now I just said this is the forecast for the if I go back a little bit. This is the forecast for the next five weeks There will come a time after next five weeks You will again need to run that model again And after that five weeks again need to run that model so that It will so that you will constantly get the signal of which customers are in the lower zone Which customers are in the higher zone or contributors? So it's not a static type of model where you run once and everything is resolved But it's the it's it's something like a dynamic model where you run at every interval every month every quarter or every six months or a year And reanalyze your contribution. Yes, sir Yes Yes, sir. So when we run our model We check our evaluation metrics again Yes, and that brings me to the next point. So in in let's say if I need to run my model For 12 months in a year. What do I do? I need to check that evaluation metrics value at each point To make sure that my model is working well because what would happen is A concept of data drift or data might change at some point of time which might Have some effect on the model So because of these kind of changes, there might be some change. There might be some feature change There might be some feature missing So all this kind of thing if you are running a model at every instant of time This kind of model monitoring rules will help you to track your model progress So ml flow is an open source platform for managing ml lifecycle including model monitoring ml flow provides a variety of tools for monitor model monitoring You can you can track metrics and logs, whatever metrics you want to track You can log over there and it will it will it will be logged in automatically You can visualize model performances And you can also set up an alert if there is any change in the model performance like for example If you can also set up an alert for your mape values Like if you run it and if you if the mape score is above 10 percent Then just send send me an alert you can also set up those kind of thing with ml flow If we need to go to if so here's the ui So this is the ml flow which was deployed in open shift in our red hat So here's an simplest example of how can we track these kind of models? So this is the experiment customer growth model, which I have named And these are the two runs. So this is the two different runs I have made So this is the first run and here's the metrics the mape value that I have Paste it over here. Like this is what I've calculated. This is what we have got from the model running And if I go back if I go to another run, here's another matrix if which is like 5.3 So that in the in there are two things changed in between the two runs One is the mape score. It's just because it's not the it's not the fault of the model But it's just the I've changed the unit I've changed the unit from 0.5 to percentage. So generally the mape score is from 0 to 1 if the if it is there then 0.5 is Equivalent to 50 or like like 5 percent or something, right? So So just the unit is changed. So in this two cases and Say for example, if you have 10 different runs over here, how how should you visualize these things? You can also visualize by going to By creating some kind of chart. Let's say bar chart So if I add this so here in the first run I have this 5.34 percentage mape In the second run I had like 0.05 mape. So these are different values of mape consider That's for example, you have we have 10 accounts or like 100 runs you can also visualize if if if the mape has surpassed the threshold value or not So these are some of the cases where we can look into through ml flow or like Yeah, just things So now next thing I want to Go through is this repository, which is customer growth model. This is our project repository So this repository whatever results I have shown you today is the code is already over here The two main folders of this repositories are data and notebooks So in the data folder you have the raw data where Which was generated from using llm's And in the process data you have the prediction data set you have the label data set which So I can show you along the way. So if you go to notebook section So first notebook is growth model generate data And if I go over here This is the growth model generate data notebook So here you can see how you utilized open api key to generate a data set tabular data set for Working for this talk for this model So you have you need to supply the api k environment variable You need to give give it a prompt So the kind of prompt I gave is like make sure it's an account number and make sure the date is between January and july Just just random And then I made sure that a couple of accounts have increased amount over time Decreased amount over time and medium account over time so as to I can have some variety to experiment with So this is the kind of prompt I gave it to the model and So I'm looping through each account and making a request to this open ai credentials So with this request and there's some data preprocessing Steps that I go through and ultimately I'll get the data set like this You have the account name you have account one different dates varying weekly And this is the revenue So this is the data that I save in the data folder Once you generate this data set we move on to um model forecasting notebook In the model forecasting notebook you import all those packages for time series forecasting arima package which is from PMD arima And mean absolute percentage error. You already have this package over here. So You have this data set first now generally in a In a real-world situations you you might need to go through a lot of deep tap preprocessing phases Like I just want to give a shout out to the talk that happened earlier today about chaos and um tsunami data tsunami from kali They were talking about all those github data, which is so enormous that it's very difficult To at least like manage and make sure it's there, right? So that is some challenge, which is which we go through and I've skipped that step over here completely I've just shown you what happens after that So in this case to to come to this level of data, we need some data preprocessing We need some feature correlations or we need to understand the feature importance Which feature is important for this model? So we need to select all those things. Those are there So here is one of the Example where we create our own features a feature called tenure. So we didn't have this feature in our data set so how we created our this tenure is we we assume that tenure is something for which we Have a non-zero value If a customer is spending that month then that month is regarded as plus one tenure If the customer is not spending that much that month then that month is regarded as a Zero tenure. So that's how we calculated the tenure and it gets summed over The tables so it can act as a feature for our data set And then we go through so there are two stages over here first is the test train and test And evaluate so once we evaluate our model then since it's the time series We train our model into the whole data set And then forecast it so first phase is The actual value and the predicted value we We predicted it for different accounts account one every accounts are there. So there's like actual value predicted value Once we go through this we calculate the map and here I have highlighted. What's the accurate map? What's the unacceptable map and all? So this is the calculation part of the map So here we have actual predicted and made values and we now save all those results in the data folder again Now once we have those considerable map of like 5 percent, which is pretty accurate for our model And this 5 percent is like the median map that we have used we have We want to see how like for we can see the map at every line, right? How do you calculate the single score for all of them? How do you represent a map with a single score? So to get an average Value we calculate the median. So here we have the model forecasting Algorithm and then we forecasted the data for the future dates not for the current dates So once we have the account name data and the forecasted value Our next step is to take this data forward to segment the customers in two different zones And for segmentation we calculated the slopes of the forecasted value And that that is what we have done over here. So in this case, we have labeled the customer potentials based on different forecasted data So we have the accounts date and the forecasted values over here We calculate the slope here and based on each level of slope We level them and we level in such a way That we defined a threshold for each one of them So if we consider So if we consider different values of slope from minimum to maximum Minimum we take it as zero percentile Somewhere in the medium We take it as 58th percentile and the maximum we take it as a hundred percentile We classify slope in such a way that whichever lies for whichever account has slopes less than 30th percentile We take those accounts as low potential accounts And for those accounts which have the slope that value that lies above 75 percentile Then those kind of accounts are considered as high accounts. This is completely based on Our asm son you can have your own asm son sometimes You you might think that above 60 percentile is a high account So it's it's completely based on your asm son So you it's completely flexible and depending upon your use case scenario what you want you can define it your own threshold So giving the threshold I classified different accounts into high low and medium. So here you have the classification everything So once we have this classification results We can send this results to the customer success team and they can identify high low and medium and they can recheck their Customer recheck with their contributors recheck with their customers In the next case we Just a visualize and perspective we try to plot a couple of accounts Like for example, this is the high account over here as we as we saw earlier also like So this is like the increasing kind of slope The next is the Although it shows over the whole range. It shows somewhat of a constant kind of a slope But for the next five weeks it shows Lower so which means that is this low account for the next five weeks Now the third case over here we can see Although it is also similar to somewhat The above one but for the next five weeks it shows both minimum and maximum like it goes lower and it goes above So so overall for the next five weeks. It shows somewhat of a constant slope So if you take an average of all of them it falls somewhere in the middle It doesn't change. So we take this kind of account as some kind of medium account Where more or less there is a constant slope So once we have this visualizations once we have all these results We can log in the main values in the ml flow For ml flow we have to supply since it's deployed in open shift and we need to Have some credentials to in order to access them We have to supply the ml flow url and ml flow tracking token In the environment variable and once we have that we can Just log in our main values calculated main values over here 5.34 into the ml flow login And if I want to put this into a pipeline So at every round it will go through it once it runs this notebook It will automatically log in the main values in that ml flow segment So this is how the whole processes happens. This is how we can check the model. This is how we can Run the model and Run at every instant of time Having said that I'll go to go back to my slides now. So demo is done So one of the main key takeaways from this talk is It's not just for a business use case. It can be used for any kind of time series use case The main thing that is needed is time series Another thing that is needed is what do you want to achieve The main asking the questions like for example, you have the github projects. You have the github contributors What do you want to see? What do you want to track? For example, in this case, we can consider metrics like interaction frequency How often you interacted with the github member or github contributors Does that interaction interaction helped in the contributors growth or not? So these kind of features say it can be added later And these are some challenges in github. These are definitely some of the challenges in this kind of customer interactions because a lot of features are there which is very hard to extract like if you if you are if you are running an organization in github and you have to Converse with contributors at every now and then then Is there a method to record that kind of conversation as a metric in a table And add this as a feature set in this kind of model And if you can add those things if we can add those features as a feature set It's it's much. I think the map will be much much accurate in that case. So yeah Thank you. And here are my like I think this is the customer growth model github repo and the notebooks to the links to the notebooks are here You can definitely check them out. And if you have any time series, you can apply these models to your own time series And if you have an issue you can just open an issue in this repository and we can we can have a conversation over there as well Thank you very much. And Let me know if you have any questions. Thank you Do we have any questions Hi, thank you for the talk. Um I'm actually really really into time series. Sorry I'm I'm actually really into time series and stuff like that. So I do have some questions Uh, it's actually my expertise in it. Uh, when you compare arima to profit Uh, you mentioned that profit is sees is better at seasonal, but um Arima never had to taking account Seasonal because there's s aroma for it. Yes, exactly. Yeah, and then you also mentioned, um creating features like feature as in into A new feature to add into the time series. Yes Um Is that like aren't like so as far as I know it, um Regular arima doesn't take Features unless you're using arima x. Yeah, so so what you're saying is uh, you're the first question is You're saying that s arima does a seasonality, right? So one thing we are not, uh addressing seasonality over here is because our accounts is multiple so Let's say one account has a seasonal base Another account might not have a seasonal base. So in order to make sure Everything works for all of them like we are running it, right? We cannot run s arima for one and arima for another. So it's a bit tricky out there So just to make the matter simple we we Cut out the seasonal part, but um From the pmd arima package auto arima You can adjust the seasonality over there as well Like if you have a package pmd arima and you can call the function auto arima In the argument section, there's a seasonal part where you can On and off over there. That's the one another thing about uh, your question is What was that like Yeah feature, right? So that's a tricky part in the time series. So Again, um in the pmd arima package As an exogenous variable you can add up the features over there inside the argument itself But the condition over here is the if you are predicting for the future where you don't have any value for the feature or the The target variable It's it's it's a complicated thing, but Sometimes You don't have the values for the target variable, but you might have the future values for the feature variable Right You got it Yeah, so in that case while predicting it you can supply the feature values As an exogenous variable in the pmd arima package in the time series Yeah Yeah, I think I think it looks simple. It's it's like the one direct statement You just put the argument you just fill in the boxes But it most of the things would goes back and I think so Yep, please take a look. Yep Any other questions? All right, I think we had a pretty good grip for this late in the evening. Thank you. Yeah Great. Thank you a round of applause, please. Thank you very much