One two, I think it's it's 9 a.m. So I don't know if you should start but we are streaming so Everyone can watch the recording after that So welcome the big crowd the audience. Thank you very much for getting up so early on Saturday And also there is I think then wall she's having a presentation at the same time. So I understand that everyone is there So my name is Vashek Pavlin I work for redhead for six years and I would like to tell you something about what we are doing in a CoE with Jupiter hub on OpenShift and When I saw my talk on schedule I called it data exploration with Jupiter up on OpenShift and then I realized I'm not going to do any data exploration So I fixed it and I will call the talk enabling data exploration on With Jupiter hub on OpenShift so what I would like to talk about is How we deploy Jupiter hub? What are the what are components like the technicalities of the actual platform and tooling? And how we integrated with other parts of the of the platform So why do we explore data and why do we want to do whatever AI center of excellence? A smart person Clive Humby from UK said that the data is the new oil It is valuable as it is if you collect a lot of data, but it's it needs to be refined same way as the oil needs to be refined into plastics and and gas and and chemicals and whatever because without cleaning and transformation and stuff like that It's just pile of pile of bits, which you can't really make sense of in most cases because it's too much and and And it's it needs to be clean and stuff like that the gentlemen also came up with the Tesco Club car, which That might not sound as a lot But it is a great source of information about customers and what they buy and how they work. So he probably knows his stuff For being able to work with Jupiter hub on OpenShift there are some prerequisites So let me quickly go through that and that's one of them is OpenShift you need to have OpenShift running Which is kind of obvious if you want to deploy there What is OpenShift who doesn't know what is who never heard about OpenShift or doesn't know anything about OpenShift good But quickly for the for the audience of big audience on the on the YouTube It's an enterprise distribution of Kubernetes. It is built on top of Kubernetes. So there are all the It's a scalable container orchestrator So if you have anything to do with containers and you want to run them in production You want to use something like Kubernetes or OpenShift? and then it has all the basic all the basic concepts as Kubernetes so Things like pod services deployments persistent volumes, but it adds more it adds the the development workflow with builds and image streams and things Like that you can go to okd.io, which is a new new Place where to find information about the upstream version of OpenShift which is which was called OpenShift origin in the past The second thing that we we need for the work that we are doing with Jupyter Hub is some object storage You are probably familiar with AWS S3 We use Ceph and then Ceph implements the S3 API So you can use the same libraries like both of free in Python or the Hadoop S3 library in Spark To access your data in Ceph Sorry with the S3 API Luckily, I don't have to I didn't have to set up either OpenShift Nor nor Ceph Everything that I will use here is deployed on MOC mass open cloud If you saw Stevens or Sherard or whoever was talking yesterday about MOC We are working with them on something called OpenData Hub. So this is part of the OpenData Hub and We are deployed there So what are the tools that we will be using for the data exploration that we are not going to do? First and they're like the core part is Jupyter you can run Jupyter server Jupyter notebooks On your laptop and you can use that it looks like that basically So you have some it's split into cells and the cells can be either some kind of markdown or It can be code or it can be output of the code and then you as a user type something into your web browser, it's a web application that has a backend and You type something in a web browser and the commands the code is sent to the Jupyter kernel the kernels There are plenty of them. You can find them on github and there is Python, Scala, I even saw C sharp R is pretty popular and then it sends output back to the Jupyter UI and you see that in your web UI of Jupyter The good thing is that the the actual file is just a JSON So if you want to do something fancy with the content you can take the JSON and parse it and work with that Or you can view it in the in that Jupyter notebook UI and and and actually run code and stuff like that What builds on top of that is something called Jupyter hub and That basically the change is that Jupyter notebooks themselves are a single user you as a user Run them on your laptop and you can write the code and you can see see the stuff But if you want to provide that capability In a distributed way to like a team of people in your company or at school I think it's like for universities that might be super interesting or even high schools I give you want to start coding in Python you can provide these notebooks and you can provide the Jupyter hub to the to the Class and and they can just go in and they Jupyter hub will automatically when they log in it will spawn up Jupyter notebook for them and they can work with that and it spawns and manages those notes notebooks So that a user always gets to a his own persistent version of the notebooks that he worked was working with and The last part of the system would be apache spark. I assume that you are probably all familiar at least Slightly with the apache spark As the website says it's a unified analytics engine for large scale data processing So that means that if you want to process some data You want to do some cleaning and you want to do some model training and something you would use spark It provides API's and libraries like spark SQL and machine MLip machine learning library, which has implemented plenty of algorithms and it works in a cluster mode So you have master and workers and the workers do the work and master orchestrates them and we will use Jupyter notebook to connect to spark and do the processing in spark so that the notebook or the Jupyter server doesn't have to be dead beefy and and work with so much data So I have quick demo it is It is nothing fancy, but basically just walked through of how that works. How did you better have works? So this is I have open shift. I have Jupyter have deployed So I go to the URL that open should generate it for me and I will sign in with my open shift credentials there is quite a quite important because If if I don't I don't want to remember another credentials. I want to use something that already know I'll get back to that Later how that this solve in Jupyter have now we select from the list of images. I Want to use spark so I will select this spark image And these are basically just what do you would if you if you install? Jupyter notebook server on your local machine or your laptop These images basically represent your laptop and the install dependency So if you want to use spark and pie spark you would need some configuration and pie spark installed and Java installed on your laptop And the same way it works with this With this notebooks that the image contains all the dependencies that are needed So I have these two notebooks One I called Boto because I use Boto free library, which is a library that implements S3 API. So I Have my credentials in environment variables I'll show you how I got them there and I will I connect to some end point And so I can run this right So it installs the dependency if it's missing and I'll connect and then I can list buckets if you sort of Open data representation yesterday you saw Steven to create the his bucket here. So we are on the same Endpoint so I see his bucket and I created mine here. So He could see my data there And then I have this other one which I actually just downloaded from internet I looked for pie spark Jupyter notebook and I got to this repository Which someone created I don't know the gentleman And he has coupled so I just took the last one I think because I thought that that is going to be the coolest one Probably and I had to fix some stuff because he was writing it for Python 2 and we are running Python 3 But it was mostly just a syntax fixing And what it does is that it's again connects to it connects to S3 connects to spark see I have this Spark cluster URL in my environment and then it Downloads some data from them from the object storage, which I pre-uploaded there and then it does some decision tree Building and this is your tree tree decision tree training and Let me run that run all below and It validates whether the decision tree was a good one it uses data data set from KDD Cup, which was some Network intrusion detection competition so build a classifier for network intrusion detection and So that uses the same data set yeah So it's now connecting to spark and For the spark we can look here into open shift that it's running as Part of my namespace as part of the Jupiter have namespace. I have two workers And I have created a route so that we can look Into the We can't look into that because it doesn't have Me fix that quickly Okay, so let's not fix that So what do you would see here? Let me try different thing. Let me start fireworks. What happened? I'm sorry Yeah, so what do we see here is that we have two worker nodes Each each executor gets 20 gigabytes of RAM and eight cores Well eight courses together so each executor gets four and then that's where we are actually running the the notebook code So it downloaded the data and now it's it's processing the data splitting the CSV Into multiple and then it will be then it will be training the decision tree What do we have in open shift as I as I mentioned we have the Jupiter hub and we have now my own Jupiter server with Jupiter hub is routing to and we have the spark Spark cluster, so I'll Let it run it will take some time the training it takes like I don't know eight minutes So I'll go back to presentation and And then we can we can revisit that so about the architecture of Jupiter hub Basically the Entry point where you access to as a user is Jupiter proxy which then routes either to Jupiter hub API or that your server that you started and then there is something called spawner Which takes care of spawning those notebooks or those servers per user We use cube spawner as the name suggests it is a spawner that is working with Kubernetes And it generates the pod definition and submits it to open shift There is also a database which gets it's just a for tracking of users and started notebooks and things like that so that it doesn't And then the proxy routes and things like that so it doesn't disappear When there is some restart or something So how it works as you saw a user comes to Jupiter hub and Then it's redirected to some authentication there are multiple implementations of authentication for Jupiter hub you can have github authentication you can have Kerberos You can have a pre-generated set of users and password. So if you we when we were doing some demos We just generated 20 users and gave gave the attendees for the workshop those users and passwords Then when you when the when the server is spawned so the user requests the server to be spawned Jupiter hub generates Generates the artifacts for open shift and if there is if it finds that I want to start a spark notebook It also generates like a config map for spark operator. I will explain explain what spark operator is later but basically It takes care of the spark clusters so open she starts at my Jupiter server and Notifies the operator about the requested spark cluster and The spark cluster is started and then the user and then the user accesses His notebook and connects to spark and when he stops the server It also kills the cluster We use APB and simple playbook bundle You can learn about that from the documentation of open shift, but basically it's just a it's just a set of Artifacts for open shift about how to deploy each service and how a day should work together And you can have that in a catalog in open shift and nicely deployed it by three clicks or something like that So that APB source code can be found in the open data hub I owe and we have we have it built in quiet Under the organizational data hub so we can go there and you can download the image and deploy to your open shift and and try it tried it yourself So what is special about our Jupiter hub because this what you could do Basically, I built my work on top of work of Graham Dampleton who has I have linked at the end, but he is Jupiter hub Jupiter hub on open shift or Jupiter hub quick starts something like that And I just took that and build something on top of that and what are the differences basically are mainly these four things So image auto discovery single user profiles a formal clusters and publish and share What did me? What does it mean? So you saw that select box for the images that is automatically generated from the Notebooks that are built in open shift from the images that are built in open shift It is not very nice right now the user experience is not very good But I'm planning on improving that with some descriptions and then like install dependencies and things like that So it's it's it provides more information to the user But it's it's helpful if you you don't have to know you have to remember you just pick from the from the select books the single user profiles We quite quickly when we started to use Jupiter have we realized that every Sub team in our team and every image has to have different configuration like if you are working with Some parquet file that you download from object storage and you don't use spark because you just want to process it directly in the notebook you might need more more Memory for the notebook if you are working with spark, then you don't need that much memory But you need spark diploid if you are working with some specific object storage Endpoint or bucket you might need to have that in your environment variables. So we build this Library that basically is configured with a config map in open shift right now and you can Mix and match the images that are that are used and the user names and users with like what should happen when the user selects that image And how it should be configured You saw that I had that spark cluster inside inside my namespace and That works it's called spark operator and operators are a concept in communities and open shift that there is a service the operator Which listens on events and if you find some specific event it will react to it. So here user comes and says please operator fairy can I get can I get a spark and Says yes, sure you can get a spark and it deploys spark based on the configuration And when the user leaves and says I don't need a spark anymore. So it removes the config map or the custom resource It will delete the spark cluster again So we have that in the in the profiles that we say that if you select the spark image We want to instrument the spark operator about that Configuration of the spark image and say please deploy two workers And one master with these resource limits for us Yeah, and the last and the last bit is basically What we hit also quite early the workflow about Sorry, the workflow about how do you share your notebooks because if you have jupiter up and you want to I don't know I want to give share My notebook I have to download it I have to send it over email or push it to git and then he needs to download it and upload it to upload it to his Jupiter notebook to bless his Jupiter hub. Sorry Which is not very nice if I just want to show him a simple change in like line 24 I change this letter and now it works So I build a plugin for Jupiter hub where you click a button Where you click a button You give you give it some name No, you give it some name Yeah and You hit publish and you get a URL Which you can access and you get a nb viewer And we viewer is a tool that lets you view the notebooks without being able to execute anything So it's read only but it renders the notebook in a basically the same way as the Jupiter hub and this this URL you can share if it's it's public It's not behind the authentication so you can share it with anyone and then he can just view and he can also download the notebook So that that I think helped us a lot To speed up the speed of the process So how is our training going so we saw that we see that the decision tree classifier Got trained This is the decision tree so there is a lot of if and else statements and Now it's doing something else so I didn't really dig deep into this notebook I just wanted to show that basically with our deployment we can directly use spark and and The ML libraries without having to do many changes to the notebook that I found randomly on the internet So that the integration is it's really good So Yeah, so I just wanted to Go over a couple ideas that I have about like next steps for the for the Jupiter hub that we could do So we right now have this spark operator integrated But I've it seems that the dusk. I don't know if you heard about dusk. It's a Python based Distributed analytics engine or whatever you would call it Provides advanced parallelism for analytics and anything performance at scale for tools you love so it's basically seems like spark Implemented in Python supporting Python better than spark. Maybe So We are thinking about like adding that next to the spark operator having a dust operator Which would then spot one dust cluster if users wants that If you if you noticed in my notebook, I have I have these environment variables with with credentials They are not there automatically, but I would like to have them there automatically populated for users based on some secrets somewhere I Have to add them in the single user profiles config map so I would like to have that as a as an automated way how to get those credentials from some source of truth and and push them into the Server automatically so that user doesn't have to care about that I would like to work on github and github integration So you can have a button same as the published one I would like to push this to my git repo or I want I want to create a git repo for this notebook or something like that I've seen some attempts on the internet that people were doing that but it never really worked in a user-friendly way You saw that select box which was pretty ugly for the images So I'd like to make that more fancy more more user-friendly and make users Make it more useful for users And also jupyter hub exposes metrics So if you want to know how many requests how many users and things like that and maybe Build some alert think on top of that like my cluster is getting full because I have too many users using jupyter hub at the same time but we need to Enable I think they are enabled, but we I'm not sure what is exactly in there And we don't have a prometeus watching that so we need to set that up also for the jupyter hub apb and probably is extend the metrics because as we start using the spark and the connection between jupyter hub and spark we need to be able to map it together in the metrics and And that's basically It's basically everything I had these are some useful links. So this is the apb. This is the link for the For the single use profiles, which is a quite simple library just for that one use case Here are the open shift configuration for the jupyter hub, which is then used in the apb Spark operator a colleague from red analytics IO red analytics team in redhead was working on on the spark operator So that I just used it and it worked perfectly And this is where we came from with the jupyter hub the jupyter hub on open shift Which Graham them put Graham them put them put together and you can go there and you can try jupyter hub without all these Sparks and things like that just on open shift in a simplest in a simplest form Yeah, so that's that's basically it any questions Yes, sure Yeah, so the question is whether with the spark operator we get we have one shared cluster or if we have cluster for users Yeah, I didn't mention that so We are basically thinking whether we should deploy one big beef a cluster for spark and then let everyone connect to it But that has its issues and limitations like that you have to reserve that capacity on your open shift cluster like if you want to really have a Hundred users and you would you want to allow them all of them at the same time go to that cluster Then you need to have reserved hundreds of gigabytes of RAM for those for those workers or you can have ephemeral clusters So when the user comes and logs in and start the server It will start a spark for him and when he goes away, it will kill the spark cluster for for his spark spark cluster So we are doing the ephemeral clusters right now. So when the user comes he gets his own Fresh clean spark cluster with some resource limits, which are obviously tighter than if it would be one big cluster And you would be the only one there but We need to do some performance testing and and Get some more data about like how that actually works and if it's if it's useful for people Start items for what? The spark cluster it's quite fast Basically, I can I can probably show you so I'll call my cluster. I will stop my Jupiter server I Think it's well, it's basically a couple couple seconds or maybe maybe couple tens of seconds Why can I close that? Great, so once the Jupiter notebook Server disappears Which should be any seconds now But it has to wait for the timeout because there are no Shutdown scripts in that in that image. It's also one thing that we need to fix So when it goes down the the spark cluster will disappear as well, and then I can start again So in the meantime, we can take probably another question or two if there are any Yes Where's your data kept? Where is your data kept I have an HDFS cluster so we have that we have the safe cluster backing the backing the open shift and The safe the safe cluster is basically where we push and pull data from okay So I could just connect my head to cluster. Yeah It doesn't it doesn't really matter like what technology you choose for that So it's it's gone. So I'll just go here. I click start my server. I will pick this park and I'll go back here and you'll see that my Jupiter notebook is starting and basically immediately I got the two workers running and Master node takes a bit of time because it needs to connect to the workers and Figure it out. It depends on like if you are starting it for the first time There are there is some time that needs to That the images the container images takes to download on the note But if it's like the second start and the images are the same for all every user So once they are downloaded on the note, it is basically instantaneous Start I don't know what why the master takes so long now But I think that it's it's running. It's fine. It's just didn't update the UI. Yeah, so it was basically instantaneous start Yes The the notebooks plan spawner Does it scale up to multiple nodes or do you have to configure that for just on the Jupiter hub? Yeah, so this so the spawner the spawner is Not doing anything smart. It just Generates the pod definition and pushes it to open shift and open she schedules the pod so that basically means that as Is the it's up to the open she scheduler to schedule these so it would it would distribute them across the cluster It wouldn't put them on a single note Depending on the size and load on the cluster. I don't know the details of implementation of the open shift scheduler, but Yeah, it is it is it is based on the open shift scheduling so it would be distributed and The same same goes for the for the spark. It doesn't have and we could configure it in a way that it would have some affinity so like Get put my Jupiter hub close to the spark, but it doesn't really Bring anything because we are trying to we are trying to pull the data from the safe for S3 or something Not from not not sending it from the set or notebooks error. I Think there was some other question How mature is the spark operator?  I think it's like couple weeks old But honestly, there is not I mean there is missing and I still miss that I've already filed a couple feature requests I'm missing some configuration options like At the beginning there was there was no another way how to set limits Resource limits for the for the workers and for the master and stuff like that So that's there now and I have a couple more feature requests in queue for like I want to be able to force update the images And I want to be able to configure these values and stuff like that But generally the working of like spawning and killing the cluster It's it's working very well. I haven't I haven't had an issue with that The the guy who works on the spark operator. He actually built a library in Java I think it's he calls it JVM operators, which is like a library that you could use to build Another operators. So he's trying to get that very stable and then Spark operator would like to benefit from that Okay, I think we are out of time anyway. We have one more minute. Okay, so if there is there's a question No, let's hear it for our speaker. Thank you very much Okay, everyone for the next session. We have Andy Gaspardaric Software architect at Broadcom who will be talking to us about improving network latency and throughput with dynamic interrupt moderation As expected this is a I'm gonna talk with wide appeal open to turn us on All right, yeah, as expected this talk with wide appeal to many conference attendees something is Sometimes feels dry like the kernel or doesn't contain a container buzzword Might be problematic, but I'm really proud of this work. We did and really glad that we could That I could come here today and share a little bit with it with you on this and in particular I think this is a good fit for this track surprisingly, so Rather than thinking about calling it What it is which I can't read You know improving network latency with Jim we're gonna talk about auto tuning your network So and I thought I had a little little picture of our famous our favorite auto tuning artists there So what is dynamic interrupt moderation so for those that what this might be people probably Maybe maybe aren't familiar with how the packet how packets in the Linux kernel actually make their way from physical hardware into the kernel stack itself so The main idea here is that we're gonna tune the time between when the first frame arrives on the line and And when an interrupt pops and so there's a variety of reasons to do this We'll go into those in a little bit, but this is kind of the flow. So we have an interrupted pops We schedule a polling event and That polling event that ultimately reads reads the receive ring of the nick so in this Beautiful picture. We have frame zero to frame n and a head and a tail. So these are these are essentially considered to be the frames that have Not been read and pulled out of hardware yet and marked as complete So it's kind of a typical workflow if our arrow going from left to right indicates time moving on Each forward-facing or upward-facing arrow would signify an interrupt and the stack of five Rectangles indicates five frames that are read out of the ring buffer So in a fairly consistent flow of traffic that you have coming in this interrupt period As I'm waving my hands from this first one to this next arrow would represent The interrupt timing that we would have so we're going to get a little have a few frames come in Service them pop another interrupt more frames Etc. Etc. In a steady state. This looks pretty good So if we have a short interrupt time, of course This means that we have a really small number of frames process in each polling event now that can be good if your concern is Latency that can be bad if your concern is throughput because an interrupt is pretty expensive So if we think about doubling the interrupt period with the same traffic flow We have a situation like this where instead of just receiving five frames at each polling event. We would now receive 10 so this is a great case for a A great description of a workload where you want high throughput and Also the downside is high latency So as you might be surprised as you might not be surprised to find out This is not a particularly new problem. This is something that people have been dealing with for a long time so One of the first attempts to deal with this administrators. Yeah, it literally decades. I think I first probably came across an issue like this easily in the knots if we're calling this lap the previous decade that and Regularly would have to talk to customers when I was a red hat to try to figure out whether or not How they should tune their devices so the first attempt at really dealing with this was in one of Intel's 101 gig adapters they had a hardware feature called aim or adaptive interrupt moderation and This was actually the source of fairly What ultimately was a fairly long why someone was having a particular problem because they were primarily concerned with low latency Not with throughput and unfortunately at the time When we were only dealing with a single receive queue most people were concerned with throughput That was the big one of the big tests that was done So one of the things about aim is it was liked by some disabled by many Like many of the hardware features that have existed in the past. There's always a little bit of angst Hardware designer does it rolls out some software to configure it Maybe doesn't work exactly as everybody expects So then there's some significant frustration about, you know, why does this feature break my network? so Kind of the same story over and over works for a lot of folks, but the lack of flexibility that existed in hardware Was not good enough for some They always people always seem to default to thinking that software is more flexible and better And for many cases it is so at the time one of the interesting things is that We sat around the office and postulated whether or not it'd be good to have a user space demon that controlled this interrupt timing so At the time it was at the time this happened is when we're first starting to see some of the tune D profiling come out on various Linux distributions most of them red hat and fedora and You could you could tune your workstation for whether or your laptop for whether you were most concerned with high performance Or whether you were concerned with better battery life or I think they're at the time They're even some they're even some some networking Configurations that were available and many of these things twiddled bits and Intel's power management capabilities at the time And so we thought what if we did a parallel what if we came up with something that could that could really have Administrated the beginning of time say you know what this is a this is a workstation where latency is the most important thing So let's tune for that or what if we You know this was a file server where we cared most about moving bulk traffic on a regular basis so sort of sat around the office and pondered whether or not that'd be a good idea and And and I think ultimately when I look back on that now We've come to a realization that that was completely and totally the wrong strategy So we could say it was blind luck that we didn't implement that but realistically it was probably more about laziness than anything else and so Let's fast forward a few years and think about where we are now So Machine learning AI everywhere. I'm amazed I'm like sadly amazed by how much it's on our phones You know things automatically presuming a time of day that you want to do something based on where you're physically located Like and I don't know it's funny to me how impressed I am by just little tiny Simple simple things that you're probably never taught in any sort of CS or computer engineering program anywhere and so one of the first things that was that that came to mind is a talk that Tom Herbert gave and that's of in Montreal last year and He talked a little bit in his keynote about the impact of artificial intelligence and you can see I've got a screen grab of his Video on YouTube about this with the link here All very clickable for everybody right now and You know he talks about the fact that that machine learning and he's he's got a new a new company that I think Or machine learning will play into this one of the things he talked about is like will the latest congestion control algorithm TC TCP BBR be the last human written congestion algorithm that exists and It kind of struck me when I was thinking about this like How interesting it would be to think about that being the last one that's written and how through machine learning We could come up with better ways automatically. It's a little bit Skynetty a little bit scary, but at the same time I think the power that we have is massive compute power and their ability software's ability to Do the same thing over and over? Effectively that mouse is moving around Not good or is good for us the mouse is not good. So Coincidentally Mellon ox added support for what we're now calling dim in their main 2550 100 gigabit driver in 2016 and Bless you and the fact is we were I Was trolling around looking at their driver and wondered like now. What is this this operation here? It doesn't it doesn't really make sense. They're doing something on receive They're doing a little bit of data gathering it looks like and it's like they're kind of using it to make a decision later and that's exactly what they were doing so that they were Calculating how how many bytes were coming in they were counting the number of times an interrupt popped and They were using that data to come up with what they felt like was an optimal setting for their They receive interrupt timer. So if we go back here a second Remember our two pictures that we had so this one pretty steady state regular interrupts servicing a small chunk of packets at the time This one longer interrupt rates serving more bulk traffic. So it looked like they were trying to figure out a way to Know which time was the best based on the traffic that came in Not pictured in either of these slides is the fact that there's a different each one of these packets could be a different size Each one of these frames which also plays into it because again It's easy for us to think when we receive a packet and we just know that it's long it's easy to think that It's all the same that a 64-byte packet and a jumbo, you know 8k 9k frame is the same But realistically they all take a different amount of time to be on the wire because there are Discrete bit times required how to handle these things so so I thought I thought that was pretty interesting that that Melanox had that and We started looking at it. This is basically how it works. So in this slide credit to Algarboa from Melanox I gave a talk over this year on this Take a sample Compare that sample to previous runs previous iterations and then decide whether or not you want to make a change So when we dug into it it seemed pretty good. So the other cool thing and One of the things that we see as a kernel developer I'm okay with it One of the things we see a lot of talk at talks is you know escaping the constraints of the kernel You know people feel the kernel limits them and the dpdk is so much better or some other Thing is better for their application for their individual applications. I would 100% believe that one of the other things That's allowed us to do is by running this in a driver We escaped sort of the lock-in that the global e-tool API uses for configuring these interrupt timers So in the past when still today because there's interesting keep an e-tool pretty static if you configure interrupt timing It applies across all queues. We of course now live in a networking world where it isn't just a matter of a single queue receiving all these traffic all this traffic multiple Multiple scores are tasked with servicing this traffic, which is how we can get to 1500 and pretty soon 200 gig ethernet on a server So this allows us to escape some of those that kernel kernel lock-in So what we really found is that because it can operate independently We can also have different types of traffic being handled by different cores This is especially useful in a virtualization case where you might have an application That needs to be low latency that's running in a VM or you might have another application That's ultimately serving as a storage destination So having now all of a sudden we can have the best of both worlds We could run a net perf test and receive full utilization of of all of that of that core at max at pretty much maximum throughput and we could run a TCP RR test with that perf at the same time and C low latency because they're end up being serviced by different CPUs. So that was super cool So we'll talk a little bit about the algorithm It's not super amazing and the great thing about it is I'm here talking about an algorithm today. That's actually open source. It's in the kernel You can look at it. So spending a lot of time explaining how it works. It's not probably super valuable Because I know everyone loves reading kernel code. I know it's what that helps them sleep at night As well as it's the first thing they read in the morning so in a typical case we have five profiles that exist right now, so You can see what's critical at the top is the different timer settings So obviously down here on the far left. This would be the low latency case so we want the timer to pop really quickly and On down on the far side the timer of 256 microseconds would be the high throughput case so the reference to left and right is something that's baked into this algorithm and Everything starts down here at the low latency case I think that makes a lot of sense to start there rather than starting in the middle Because typically low latency is going to be small traffic. It's typically going to be quick sessions It's typically going to be a small number of bytes so it's default to that and the rate at which we sample and the rate at which we make changes quickly moves us Down the line to the right So this decision tree is really pretty simple We have our previous decision Either right or left we compare the samples that we've collected on every single packet We receive and every single interrupt we process and then we make a decision. Well is is Is this better or worse or the same as before if it's the same we park it Analogy that probably applies to everybody that drives If it's worse we go left in the case where we were previously going right and If it's better we go the opposite direction we go more to the right so The compare samples piece can also Can be tuned a little bit depending on your? Your workload or your speed, but really one of the coolest things about this is I've tested this across fast processors and slow processors and Super fast processors if we want to have three examples and what really works is this holds up across all of them so this is something that works in a small system, maybe even a 32-bit arm case and it's something that works on works well on the latest Intel Devices so all right, so I mentioned Intel and Melanox, but what about Broadcom? I mean they're the ones paying paying for me to come here and talk about this So of course this is the the big reason I'm here is that we found this to be interesting We ported it to our driver and we really liked what we saw in fact it was Other people confirmed they really liked what they saw so here's some super fun graphs so in the case of Default and adaptive coalescing in this first picture on the left You can see that basically the throughput was unaffected by the number of streams. This was a 25 gig nick That's why we're up there at the top so we can almost fully utilize that just with one core And certainly once we hit two cores or two streams. We're utilizing it a hundred percent and the graph the point of this here is to show that Even with the small hit that comes with cataloging this information We were pretty much right on my throughput. There's no hit there The graph on the right is a little more complicated to understand so I'll explain it a little bit The x-axis represents the number of streams in use and the y-axis represents the total CPU utilization So unsurprising with one stream. We're utilizing one core completely It's as the graph is kind of funny that there's a two and a half core example there That wasn't really what we did. We there's a dot there at two. It would be nice if that's how Whatever spreadsheet technology we're using a graph this would have chosen to put the lines at two, but anyway At two with the default coalescing settings that we have in our driver. We saw much higher CPU utilization because there wasn't the ability to adapt and have the interrupt timer move way out and On the case on the right lower is better. So adaptive is clearly winning as we scale up towards eight cores eight Eight cores being used for receive traffic. You see there's a 7.0 there. We're still not even we're barely over barely utilizing two and a half cores completely when you add all that up versus Close to probably four and a half four and three quarters With the default settings. So we feel like this is going to be a huge win in in the throughput case The other thing we did is we did some TCP our performance now this is a I Hesitate to show raw numbers here because every time I get a new system in with a new processor these numbers all change But we went ahead and put it in anyway So with our original static coalescing at the best rate we could do we could do about 20,000 transactions per second with adaptive or a little bit less So I'll talk about why there's a 4% reduction, but we were really happy with this to be honest the fact that we're paying attention to every interrupt paying attention to everybody that came in and doing computation on those not on every packet, but Statistically within a certain number of packets we would analyze whether or not we need to make a change the fact that that only caused us In this in this single stream test of a total of 4% hit We knew was going to be a real positive And at least for the the people we were going after for this and and they were they were quite pleased so We also confirmed that one receive ring can be optimized for low latency and another for high throughput This was really the case that I think was most interesting to me I think This flexibility just doesn't exist today in the Linux kernel So by adding this feature we were able to provide something that really other than Melanox No one else could do so I was really happy and and What we decided to do was rather than just take Melanox's code I'm completely added to our driver and that seems really weird in some ways I worked with Tau Gaboa Melanox, and we actually made a generic Layer in library now yesterday if you if you sat through One of the late afternoon talks there's a panel and said oh AI is not just about adding a library and thinking that like everything Just works magically I won't necessarily refute that but I will say that in this case That's one of the cool things about this is you can just add a library you add the right pro points within your driver You add a function call that can set this value in your hardware, and you can just use it and in fact After posting my first patch of stream I got several off-list emails about this people who are interested and One of them doesn't even work in does does happen to work for Broadcom, but not in my division So didn't know he was interested, but the BCM G net driver Used this right away, and in fact they also Adapted it and wanted to use it for transmit as well this is a great example in my view of the power of this because this is a Driver for an arm so see it's typically embedded in set-top boxes. So if you have used any pretty much Many of the triple play offerings from ISPs where you can plug a phone in and you can plug some ethernet in and has Wi-Fi built-in That's the type of application that this has and this Type of application for this and in their case they've got a wide array of traffic patterns You might have home use traffic that is you know streaming video And so you're gonna have large frames you're gonna want to want to make sure you're optimized for that but you're gonna have other flows that are very small or very short-term and As soon as this came out Florian Finnelli is the one that did this work. He was Excited about it because he'd been they'd been pondering the fact they saw such a huge difference in the way their systems performed When they would use different values so the fact that this could do it tune it for them without doing anything They were pretty pretty stoked about so more drivers to follow. I don't know. I've talked to folks at Intel. They have They have a little something in their driver that does something similar. They also have some hardware that has some fun fun features We additionally have actually amazingly lots of hardware IP blocks to try to handle this situation we have More than just a basic interrupt timer We've got several things that we don't completely expose because there's no API and part of the part of the challenge for this was actually Working Michael Chan and I working out which how we should how we should handle this and how we can how we can best You know give customers and more importantly administrators the opportunity to run this no longer are The theory should be when this is when this is working and this is in the distro and this is everywhere There should be zero support calls Again, that's the theory there should be zero support calls to anybody who who says oh my network's not performing in this low latency case Oh, it's not performing in this this bulk transfer case. This should be done They should eliminate those calls. So we have we have outsourced this this work to the machines So I want to share just a couple observations some some surprising some not For me, this is a fun thing to work on which at this stage in the game that working on the car as long as I have That's sometimes a little bit rare So one of the first things we came across is that programming hard work can be expensive And when I say expensive, we're still talking about milliseconds or microseconds But it can be and this is a common case across multiple hardware vendors In fact, we spent a lot of time tuning and understanding when the ideal point When when's the ideal point to sample? When's the ideal point to decide whether or not we should make a new decision? Because you can do it so frequently that you see a much greater than 4% reduction in your in your low latency tests and And this this expense when running on the same CPU as The traffic as the CPU receiving the traffic does is going to cause a small interruption in traffic so we talked about scheduling another CPUs and we decided that was That was a an experiment that we could look at for another time But but another thing to think about, you know the cost of doing these operations to hardware is never free So a good thing to remember The other thing We found is that we had a few benefits that it appeared sort of unexpectedly so when we were Doing some testing we are a typical test case where you have you know a whopping two devices involved And you're doing some transmit from one to another and you know in the case with almost anything You have an experimental group in a control group. So what we started with was using our adaptive interrupt moderation on on our test server Running an upstream kernel and we had another system just running an upstream kernel With our normal driver and we slam traffic at it and watch what happened And one of the things we found is that we were not getting the throughput that we expected And it was you're sort of scratching our head a little bit saying like well, you know, I would expect that that We can see that it's moving up to this higher profile. We added some debug FS support so we could see this in real time and It just wasn't happening as Efficiently as we thought it could And some of that was because I'd previously tested two systems back to back So we started doing this control group now what the performance wasn't worse It just wasn't as good as I thought it could be and what I realized is that if you're ascending system despite not having any transmit interrupt Moderation features enabled acts basically are classified as low latency traffic Acts are small they're coming all the time and the speed at which you receive an act Definitely determines how quickly you're gonna send out traffic again So we actually did some tuning in and so we we emulated what we thought the algorithms would have done and On the sender move the low latency back to the sender of bulk traffic Move moved just to a low latency profile and we actually saw improvements in CPU CPU utilization So that was kind of fun And I think to me this is one of the examples one of the things we can point out that Had I spent had myself or the other folks working on this spent a lot of time thinking about this ahead of time We probably would have come to this conclusion Maybe maybe not you never know might give ourselves too much credit, but the difference was that Just trying this enabled something newer and maybe more fun than we thought and was an improvement So I think this is a for me This is a thing I'm gonna continue to think about as a big win for AI showing us something that we didn't think of we could do before So the other big takeaway for me is that the kernel has a ton of configuration knobs a ton and so many of the folks that have worked on the kernel are Some of them no longer working on the kernel. They're doing the next the next most interesting thing that they think exists Or they're too busy, you know working on the next version of hardware Or whatever that I think there's a lot of low-hanging fruit out there for us to really examine Different kernel config options. I mean take for example, just the discussion I had about the the tune D profiles that exist well, why Do I need to Why do any of those need to exist? What is it? What would what would it take for us to figure out with any of these things what the ideal number of What the ideal settings are for highest performance what the ideal settings are for low battery life even take things like You know data plane technologies that are of interest to me right now whether they be BPF and XDP or or even DPDK Things like why do we have to guess at what it takes to be the proper number of packets that we batch anytime? Anytime we're doing reception we can improve packet performance by batching. Well, let's figure out how many that is Automatically, let's not figure out. Let's not spend four days with the person Recompiling and testing over and over again to try to figure it out. So that's that's my encouragement for all 11 of you that are here To go forward and figure out and think about whether or not areas you work in can be can be done automatically so also want to Leave a little time for questions, but I want to make sure to give a shout out to Gil Rocca Archie and Tao from Mellanox who came up with the initial Implementation of this the initial design and push the hitter driver and Rob Rice and Lee Reed and Michael Chan from Broadcom And then of course copyright holders saw images used in the presentation. So that's all I've got Questions, please say no No, that would have been I Given more time I would have loved to have auto-tuned the entire presentation Cool, well, thank you. Oh outside the lab Yes, absolutely. So the question was outside of a lab environment what sort of other applications have been tested and For me that's all I have done because we had a lot of this was motivated by a specific requirement from a potential customer and They had some workloads that they weren't able to emulate with net perf Pretty effectively and so they came up with this recipe and said, okay, you can do this and you can do this and you can do this And you can do this No touch, you know, you have a chance at winning and so that was a lot of what A lot of what I do in my job now is to figure out what it takes to do that And so we looked around and looked at different things And so they came to us with the TCP RR test with net perf and they came to us with some of the TCP stream tests and Some other specific things So aside from just a system-to-system test the other thing that this has been tested on pretty heavily from our own interest for another another reason Was actually syncing the traffic into a VM So a regular host that was pounding a VM with traffic But both the TCP RR and the TCP stream so the VM was the sync for the traffic And that's actually one of the interesting points to me is that's where this really shines Whether using Broadcom hardware Melanox hardware because the VM has zero good the VM might be running for IO may have zero control Over what's happening? So now what you've done is you've got a way where it doesn't matter what workload is being run on those VMs You you've given them a chance to to both be successful because typically both of those separate IP addresses separate streams They're going to hash to separate CPUs So they're going to be on unless you have really bad luck And then they're going to be received at different rates and that's that I think is the is a big strength And so I look forward to when These upstream kernel changes roll down into the main distros and are used In virtual in virtualized environments like that whether it's open stack or just other other places. I Think that's going to be key Yeah, so It was it landed in January in Dave Miller's tree. So that probably means for 16 So, yeah, so it's freely available in in everything shipping past that point Did you ever oh, yeah, absolutely So Yes, I can definitely say okay. I shouldn't say definitely. I Haven't been asked to so broadcast maintains our inversion and Collaborates on an ESX driver. I don't think I haven't been asked by anybody that maintains The question was related to whether virtualization environments ESX or KVM Etc. I can't say for sure. No one has asked me on the ESX driver team anything about this Which is typically a sign that it hasn't been implemented Not always but no one's asked on the In a KVM environment if you're running a new enough kernel, this is available So if you're running if you're if you're based kernel on your hypervisor is I'm just going to go ahead and make a blanket statement and say 417 although I think really 416 or 415 is probably right Like I said, it landed in Dave Miller's tree or this year and his tree is always it's a development tree So it's always you know one version ahead. So if I do like a get described I always have to add one to whatever is there because he keeps linus tags so Probably should have done that homework But yeah, and any I mean if you go out and run fedora with this right now or even probably I guess 1804 One two, it's probably got a new enough kernel that it's going to be there It keeps one it knows the last state and that's it Yeah, very very low overhead and that's one of the things that we really liked about it And why like it was kind of crazy how simple the how simple it was and how low overhead it was and how Small of an impact it had on I mean the impactful part is actually the couple millisecond delay hit that you take less than that but the Writing to the hardware if you have to make a change That's actually what we like we never had to worry about tuning I mean you're talking about one or two instructions that with a good compiler are probably going to slide right in With some other delay that you have in the network stack The the cost is always how frequently we wrote to hardware like I can tune that and watch it change Like if I write to hardware every hundred packets Throughput and latency suffer heavily because you're spending so much time writing out Obviously if you do it every million packets, it's less useful Especially since most flows aren't that long but but yeah, it's very lightweight. I mean I was shocked at how well it worked Like it doesn't and that's the thing too is it doesn't have to be complicated like we don't a lot of the base This layer is created in such a way that if you wanted to do a very a much more complicated Stateful inspection and keep track of your preemptively decide based on Something that's coming in that you should go one way or the other you could do it But this is such a great easy intro to start that minimal hit anything else Thank You Andy. We have a coffee break from now till 1120 and we will be resuming session then Okay folks for the first session after our break we have Share our Griffin senior principal engineer at Red Hat's AI center of excellence We'll be talking to us about building AI with Seth and open shift Thank you. Thanks everyone So Saturday morning, thanks for you know Not watching cartoons or Netflix or anything coming out here to see us speak today I'm gonna talk to you about what we've been doing in the AI center of excellence at Red Hat You guys have seen a number of us talk before about our strategies with AI and machine learning You've seen Vasek and Steven talk about Jupiter hub and Seth and spark as well I'm going to expand on that and introduce a concept of open-wisp where we have serverless actions So you'll see a little bit about how we chain all of those technologies together to get a nice machine learning pipeline First off to do some level setting and ground rules Everyone is familiar about the concept of machine learning and AI This is to just show an example of a typical workflow It starts off with of course Daniel mentioned this before the data in this case We have a data set live a library of data sets and from there we want to start to develop a model That's where a lot of the brains and the intelligence comes into Analyzing the data and understanding what you want to do trial and error From that developing of the model you then go into the exercise of tuning the model and training and using The data that you have to then train it Once you do that, of course, you know the big excitement is after you've trained it and developed your model You get to deploy it and actually see some data coming in so for today's session I'll walk through an example of all of those types of steps that would go through in a typical process So you can see how we can use tools in an open-shift world in order to achieve that workflow the first part of this as we mentioned before was the the Open-shift framework is as we start out. How do we do this in a containerized world? Well in this case, we're using open shift open shift as you all know by now is a container platform that's it uses certified kubernetes enterprise kubernetes and it also Allows you to do more hybrid cloud type of things. You may have some of your Infrastructure in Google AWS Azure you may also have it on-prem Open-shift gives you that ability to seamlessly manage all of that in one ecosystem Once we have our containerized world We need a place to store the data for this example. We'll be using Seth. You can also use other technologies The reasoning behind Seth is really because of the growing need to be able to Separate your data from where your performance and your compute actually happens And that was one of those architectures that came from some of the Amazon maybe a lot of you may be familiar with Amazon's Infrastructure you had a lot of the EMR type of scenarios We'd have elastic map reduce and a Hadoop ecosystem and then environment allows you to spin up a Hadoop cluster Process your data really quickly and then tear down the cluster and not have to worry about maintaining it Obviously if you tear down the cluster you have to make sure the storage is still there and Object stores like S3 and Seth came about we use Seth because it has S3 capabilities And we're able to leverage a lot of the technologies that are already built on top of S3 Of course, it's it has a restful gateway, which is nice to integrate with as well and it's a distributed system Some of the other software that I'll be using in this demonstration Will be spark you've heard a lot about spark in a lot of the AI talks and a lot of the container talks as well You know spark is a great engine for processing data allows you to do batch and streaming But it also runs on kubernetes kubernetes, and in this case we are using the rat analytics.io Work that they've done in order to move spark into a kubernetes framework with their of shinko and rat analytics spark engines that we'll be using Jupiter hub we've seen several several examples of that that allows us to have a multi user Console to manage Jupiter notebooks users can have many different notebooks You can also do many different users on there And that allows you to do your data science work It's designed for data science and research and a great tool to do that that'll be running on kubernetes open shift as well And then the last little piece here some of this may be a little bit new to you guys But once you actually do the data model work you want somewhere to deploy it in this case We'll be using open whisk open whisk is a serverless action. If you've ever used Amazon's Amazon AWS Lambda It's a similar concept where it allows you to focus more on the delivery of the real code and not worry about the architecture So I'll show a quick example of deploying open whisk and then taking that model and the code that uses the model and deploying That to open whisk as well so that you can have a nice rest API wrapper around execution of that model So to start off one of the first things you want to do is make sure you're collecting your data This is just a little bit of a diagram that shows you a typical workflow that you may have to actually Ingest your data internally at red hat. We have a lot of different systems that send us data This is consistent some of the examples. We do we retrieve data from git. We also have build logs CI logs All that information coming into us, but we also have a warehouse that houses IT data customer data customer feedback support tickets and we leverage all of that send that data into Seth and Then use the machine learning on top of stuff in order to process it We have a different bunch of different mechanisms for getting that data into Seth Some of it is through Jenkins. Some of it's through other types of workflow Managers and it just seamlessly moves that data into the object store I'll show some examples of that as well In order to start out with Seth the first thing you'll do is you'll actually configure it from an object storage perspective to Be ready to ingest your data I'm not going to go through the exercise of installing Seth I think some of you have seen that before but I do have a link on the slides if you want more information about Installing it. I'll just step into the part about Setting up a user and then actually after you have your user set up then just getting at getting your data loaded into the system So once you have set object storage installed You have to make sure the object gateway is installed and then once you have that you want to set up one of your users with s3 access and In this case, it's just a quick command that you might run where you would create your user in the Seth the Seth s3 environment and when you do that What you'll get back is an access key and a secret key That is just like as if you were working in any other s3 environment AWS or anything else and that access key in that secret key is the important part about letting you actually write and read data from the Seth environment So from there what I'm going to show really quickly is I actually have let's do a clear here and Hopefully you guys will be able to see this. Okay. I'll try it. Yeah, it looks pretty good. All right So in here What's that? Oh, whoa, that's weird. Yeah, that's seriously cleared it out. Oh That's weird. It's doing all kinds of funky stuff here Alright, can you see it? Okay, that's good So I'm gonna show really quickly is I have a bucket That I have let's see Oh, sorry about that in the Massachusetts open cloud and that bucket We just called it open data hub and since I'm using Seth instead of AWS Which you'll see here is I'm actually using the AWS command line interface and you would go through the typical steps of setting up and configuring the AWS command line interface Actually, I'll just tell you to show you a step real quick and Steven touched up on this before You would typically do an AWS configure where you take that secret key and that access key that showed you in the previous step And you would just populate that information I'm not gonna actually show my secret keys here, but you get the idea Yeah, I know it has awesome data in it though. Trust me So once I do that and have it all set up then I do an AWS LS you'll see here for my training Sub-directory I actually have no data in there So what I want to do really quickly is just upload some training data that I have here and I'm going to just run this command here What it's going to do is upload a tab separated value table called training data This is actually some data that we're using for doing sentiment analysis and you'll see that as I work through the example However, there's all kinds of different formats of data. I'm just using using tab separated values right here You could use snappy Compression with parquet. It can be JSON. It could be CSV files anything that is typically supported by a lot of the Hadoop ecosystem Now that I've uploaded that you'll see I actually have my training data. They're great. Awesome. Whoo. Yeah uploaded data now What? Well, here's where you start to analyze that data and a little bit of what Boschek and Steven has shown before You will then start to use Jupyter hub Jupyter hub is cool. It's very it allows us to Integrate with Seth and actually query the data and use tools like spark tensorflow scikit learn All of those other frameworks in order to process the data for this example I'm going to stick with spark and you'll see I'm actually accessing the data in Seth by using the s3n library the s3n libraries and jars in order to get access to it So a very simple concept there and what I'm going to do is show you really quickly What we've done as part of the work for the moc deployment So here I'm logged into an open-shift instance and this wouldn't necessarily be a data science role, but it'd be more of the DevOps, you know systems engineer type of role. We've actually created an APB that allows us to go through the service catalog And find Jupyter hub down here. So what I'm going to do is I'm going to create a quick project here And I'm just going to name it Jupyter hub and You'll see that project there and what I'm going to do is actually click on this servers broker Select my projects and it will start to deploy Jupyter hub now There's a couple of options that we have here the database memory Jupyter hub memory notebook memory I'm going to increase the notebook memory to 2 gig and then start to create this cool Now it's actually starting to deploy my Jupyter hub You'll see it's pending and then you'll start to see some really interesting things happen once the pods start kicking off here in a second So we have the Let's see There we go So the pod will start to initialize you'll see With Jupyter hub if you saw Vashik's example earlier today You'll see that there was a number of different Images that you could have we have a TensorFlow image. We have a scikit learn image We have a spark image and so this is actually building all of those images behind the scenes preparing Jupyter hub for you and also Creating a spark operator that spark operator once we actually start a notebook that has spark in it at that point Each individual user will have their own spark cluster that spins up behind the scenes So as that's running, I'm actually gonna shoot over to a different instance of open ship that I have Since this is still still running and getting started. I want to show you exactly what it looks like when you start to get Into Jupyter hub, so I'm gonna go back here in a second Once you put a hub actually starts up. In fact, I will Actually, I'm not gonna Just continue on my server once it starts up it gives you the option to select whichever image I'm gonna stick skip that step because I've already selected a spark image for the sake of time and In this park image, I've created a notebook called sentiment analysis training I would love to take the credit for all the intelligence on here But actually super did on our team was the the data scientists that helped us create this What this code is doing? I have no idea. All I just know is the end where it actually trains a model So I'm gonna actually run this really quickly and just kind of step through the code and show you a little Be the magic behind the scenes here first thing it's doing here is You do have the ability with Jupyter hub to add a few more libraries that may not be on the image that you're using in this case We do have spark and we do have Scikit-learn installed, but we do not have TensorFlow and keras and some of the other ones So what I've done here is I'm going through and installing those on the system and then it goes down a little bit farther and Starts to import some of those libraries Here's where a lot of the magic happens with Seth You'll see here. I'm actually instantiating a pie spark instance and what it's doing is actually leveraging the spark that's already on My Jupyter hub so if I go here to my Jupyter hub instance You will see that There is a spark cluster for sh Griffey. That's me. That's me So I've got my spark cluster and I am submitting my job to that But you'll also see there's some other people s-hules that Steven Huell's and then we have a couple of other people that are in the system here So what it's done here, I've actually used the S3n sorry, I said S3n, but it's actually S3a the newer one I've used S3a to actually connect to Seth in this case I'm reading a CSV file and it's just doing some basic printing some validation Hey, here's what the CSV file looks like farther down. I'm using a little bit of pandas And then shows you what some of the data looks like That data is stored in Seth, but now I'm reading it. You see some of the contents of it And then it does all the data science see stuff and then it shows some graphs. I like graphs So I said, oh that looks cool. Don't know what it tells me, but it looks cool And it continues down. We've got some more graphs here the interesting thing is When we're doing the sentiment analysis training we have to Understand whether the accuracy of this is pretty good You know, is this a good model that we're working towards or we're having trouble with the model and we need to refine it In this case, you know the example that I'm running through right here. We don't have much data in the system So I'm not gonna get too much into it But this will show you some some of the cool things we can do with the data We're showing a little word cloud where we're actually detecting what's being talked about in the training data But as it goes as it goes a little bit farther down, you'll see it starts to build out the models and that happens Somewhere around here where it's actually building the model. It's gonna run the model This is where going back to that diagram You would do some tuning and manipulation to make sure that the model is accurate and then at the bottom here Once it's actually done. It's almost there. It's at step 83. It's finished with that You'll see it's the accuracy printed out Accuracy of 69. Maybe that's good. Maybe that's not I would say it's probably not good. I'm not a data scientist But that'll tell you as you get more data in there as you train the model you can do some cool things there so now I've got a great model and I need to do something with it. What's the next step? Well in this environment what we would do is since we already have Seth We're actually going to store the model in Seth as well and that allows us to get access to the model from any number of places We can use open-wisk. We can use any basic Python code. We can use Java code Whatever we want and it's agnostic at that point so storing the data in the model in Seth what we're doing here is Pickling the tokenizers and the models and storing that there and the outcome of this if I go back to My s my Amazon CLI. I'm gonna do an LS on the model folder. Oh Yeah, sorry about that guys. Let's just shrink this and see if you can see a little bit better Does that work better? Is that good? Okay? Awesome All right now my screens all jacked up. Oh, let's see here. You still see that? Okay, cool. All right So what I'm gonna do here is I have a Do a list on this and you'll see I've got a number of folders. I have data sets metrics Models and training so now I'm gonna look at my models folder And I see that I have some sentiment data there and then I'm gonna go to the sentiment folder I'm sorry left out a slash And now you see okay great. I have my model. I have my dimensions. I have my tokenizer. Okay, so what do we do with that? So we go back to here Now we want to talk about How do we deploy the model and make it useful in this use case? I'm using open whisk which is a serverless action I mentioned that earlier But again, you can use any number of things if you wanted to use Argo if you wanted to use NIFI or any of those technologies as long as you can get access to the data and stuff. That's great Sometimes you may want to even cash the the model maybe put it into Some kind of caching layer that you can do that as well So the first thing I'm gonna do with the open whisk is I'm actually show you a quick and easy way to deploy it so I'm gonna go back to my Open shift here, and I will create a new project called sh griffy open whisk and You see that project has been created right there. So I'm gonna go back to the Command line. I'm just gonna copy this command really quickly, and I'll explain what it's gonna do and I Am just going to use the open shift command line interface to actually deploy this Recently the open whisk project that was done on top of open shift has moved into Incubator at Apache. So if you actually search for Open whisk with open shift on Apache, then you'll it'll come up what I'm doing right here Is I'm just taking that master template and I'm deploying open shift. So I'm deploying open whisk on top of that and Let's see so And what you will see here Is it instantly started creating a lot of different deployments you have engine X Strimsy if you're not familiar with Strimsy that is the The Kubernetes Kafka that's also been worked on from a lot of the redhead folks And it has couch DB and a number of other things What you see in the background here is it starts to spin up a bunch of different pods But again for the sake of time, I won't focus on waiting for this to be done I have another instance where this is already up and running Once I have the open whisk environment all set up Then you actually use the open whisk command line interface to take a look and see exactly How to deploy the system. So what I have here is I have some code That is actually going to consume that model and actually run a Sentiment analysis on top of that using the model Up here, you'll see some code where it says analyze sentiment This is actually where it will load the model and Start to feed in the text that I pass it and then I have some additional information here where it's just taking some command line arguments nothing fancy just a quick example to show you In order to deploy the open whisk The open whisk Python code that I just showed you you just run a couple of commands here. So I'm actually going to Create an action and I'll kind of step through exactly what this is doing In open whisk everything is just an action. So I showed you the Python code very simple code I don't want to have to worry about spinning up my own engine X my own web interface all of the REST API that has to go along with it So I can take that Python code and you'll see here what I'm doing is I have the name of the action called sentiment and Service and then I have main dot Python and I actually already have a docker image That has some of the model libraries that has TensorFlow has Keras already installed on that docker image So I'm going to do a quick deployment of that Boom, there you go. It's already deployed. So Once I want to actually test it because open whisk comes with a REST API I am going to use postman. Hopefully you guys can see that. Okay, man. That's a really big postman So I have two different postman commands here and All I'm doing in postman is a post With a to a REST endpoint in this REST endpoint. You see I'm now pointing to the sentiment service and I'm giving it some credentials. These are my open whisk credentials and the body of it is just some text that I want to Pass in that I want to send them an analysis on I'll send that along the way And down here I have an activation ID now if you saw you may not have seen it, but really quickly It's starting to build up if I go over here I'm going to go up to the open whisk project It spins up a pod automatically and there's my sentiment service running in open whisk and it's actually doing some Analysis right here. It's loading up some mod loading up the models You see it loaded the model and it's actually Using TensorFlow as well to start to process the data right as That's running. I can then pull and see well what's going on with the action So I'm going to take that activation ID. It's basically like an execution ID and replace that with Just the one that I copied here And I'm going to do a get when I do a get to open whisk to see what's going on with that service It'll all it says it's not exist until it's actually done here So I'll just keep sending it until it completes There we got a 200 there and now what you'll see is the result of that action There's a little bit of metadata that open whisk provides But then at the end of the day you see here my results sentiment equals positive Now the way that we've used this in red hat is we have a sentiment analysis service That sits out there in open whisk and as people are submitting their request They're actually calling to our rest API in open whisk to get results back We do a lot more than just this sentiment analysis that I've shown here We do entity detection as well so that we understand for the given text What are people talking about is there a good feeling about red hat great feeling about open shift? What do people think about devconf and it gives you a great way to analyze the data? But to bring it all back home Now we have an entire process and where we've built the model We we've actually uploaded the data built the model and deployed it so going back to What we had in the original slide It gives you the pipeline and what we've done here is for the storing of the data and the models We're using Seth To develop tune and train the models We're using a combination of spark and jupiter hub and then to deploy the model into the rest of the environment to actually make It usable is open whisk obviously for this use case there's a number of different users that you might have in the system where your Engineers your data engineers may be more focused on the Seth side And then your data scientists will be in the spark and the jupiter hub side and then for the dev ops and the data engineers as well They'll be involved in the open-wiz side So you'll have a number of different people involved in it, but that's that's teamwork, right? So that's all I have If you want more information We are starting to publish a lot of what we're doing inside the data hub to the open data hub project And just to let you know a lot of what I showed what I walked through here is part of the open data hub project So Stephen mentioned it before with the MOC and we've seen a lot of it as we've gone through the past couple of days So you'll see more and more information and more Code being submitted into that repo and of course if anyone wants to contribute, you know, feel free to join us We also have a lot of information about spark on Kubernetes and OpenShift you can see that in the rad analytics diet.io page You can also take a look at the now incubating open-wisk on on OpenShift project and contributing that way At the end of the day, you can always just contact me shgriffy at redhead.com any questions That was awesome live demo on the mass open cloud is coolest thing ever Of course, I did that for you you you So yeah, so we're running data hub on the mass open cloud Sherrard and I know many of you probably don't that the Infrastructure we're using right now on mass open cloud isn't necessarily ideal for Machine learning training models that kind of thing If you could if you could have whatever you wanted up there within sort of reason What what kind of architecture would you like to see what is it is it storage? Is it is it vector processing like what where do we need to go now? Yeah? I think that's an interesting one and The reason I'm so excited about the MOC is I actually would like to see the users tell us that I think We'll certainly be doing some performance evaluation on the workloads that are happening on the MOC I have a feeling that we're going to Explore more about Seth in the storage on Seth. There is going to be a Knowledge sharing that has to happen where we have to understand the best way to store this data I mentioned before you could have a tab separated value table or CSV file But there's other technologies that will allow you to process the data beta better more column They're formatted data like snappy compression with parquet things like that But from the physical hardware perspective was certainly want to move more into the GPU enablement FPGA type of enablement of the system So it'd be great if we start to leverage some of those technologies as well That will allow everyone to do work faster and I think that's going to be the nice blend of having the Right storage for your data, but then also having the right compute horsepower in order to make it happen Any other questions? My question is mostly around data acquisition We are currently using I mean in your example. We are using training data so what kind of Data normalization or cleansing Components can be used In the OpenShift environment good question The simplest answer to that spark is great for ETL We've used that internally at Red Hat for a number of our products and I think in this use case It's great as well the nice thing being again you've separated the storage from the compute so you can have spark You can do your cleansing you do your manipulation of the data And then what you would want from that is more of a workflow manager to Manage those spark jobs as it's flowing through the system Some of the other technologies that we are looking into is the ability to do hive on spark so you can take a hive job in Kubernetes but use spark as the framework to do that and Manipulate the data from there a lot of people are just more comfortable using using hive as opposed to spark And there's some other technologies that we're looking at but it gives you you know a good baseline to go off of So you mentioned open-wisc on Open-shift as kind of an alpha project. Yes, so what's missing there? Um What what major problems am I going to run into if I try to use it for something I depend on the major issues? I've seen honestly is how fast is being developed I'm actually excited the fact that it's moved to incubator because before then If I deploy it today versus next week things may break and there may be some inconsistencies there So I think incubator will give us a little bit of a better chance to version things off it has a lot of support for Multiple languages Java, Python, you know some of the other ones no J s I think the missing thing may be You know just putting it through the ringer of a real use case and see how it scales And then also we're working through the exercise of hooking up from Evious to a lot of these as well to make sure We can monitor the system So I think really it seems to be in a final Finally in more of a stable state. This has been the development on that's been going on for a year now So I think it's starting to level off and you get a little bit more stability there Okay. Thank you. Sure. Amazing. Thank you. Thank you while Seth principal architect at Red Hat and Yaret Hattuka Software developer working with Seth and they will be talking about adding smart disk failure prediction to Seth Take it away. All right. Hello everyone. My name is safe while Hey, I'm you read Okay, yep, so today we're gonna Talk about adding smart failure prediction to Seth. We're gonna start with a little bit of background What's that is why we're adding smart failure prediction. We'll talk about the journey We took in order to arrive at the architecture that we did the current status of the project And some next steps and then we'll talk a little bit about outreach II just an internship program That led to this whole project Maybe one of the most exciting thing about this project was how it brought together The open source communities along with the outreach you internship of course set open source community and We gained a new Industry participant in this project, which is really Excited about this project profit store Yeah, and this is how open source does it magic which is not trivial sure Is this better this way So probably all of you here know what Seth is but just a quick recap Seth is an object block and file Story system in all in a single cluster. It is Designed so all components scale horizontally With no single point of failure It is software defined so it can run on Commodity hardware And it is self-managing Wherever possible, which is really relevant for this project Because we wanted to make it even more possible to be self-healing And maybe again the most important part is that it is completely free and open source You can use it. You can see all of the code, which is It's awesome Thanks So the concept was to teach Seth how to collect health metrics from its devices and then to Passive to a pre-trained model that can predict whether a device is about to fail or not Or when it's about to fail It was important for us to keep the Design modular so we can either use an in-the-box model or to send all the collected data out to a Service on cloud make make it to the prediction and then get the Prediction back to the cluster and then with this estimation of what's If the device is about about to fail or or when we want to teach Seth how to respond to An imminent failure before it actually happened Which makes the data even more safe It brings us to Reliability it's a hard fact, but Devices eventually will fail Does anybody here ever experience data loss? Terrible losses of photos documents They still hurt On a personal level Yeah, it's it's it's not very fun to lose data for business businesses. It can be devastating and So we all Use redundancy in order to avoid the data loss We will replicate the data By using raid a razor coating so it's playing the numbers game just to Know how much we can Invest in that it can be really expensive to to be able to replicate data With lots of replicas. So whenever a device fails it Changes the statistics or the redundancy The redundancy gets worse and we will use rapid account and a window opens for even elevated risk for data loss Larger systems mean that we have more devices and it means that we have more Failures eventually and this is a sentence that used in fact itself Failure becomes the norm rather than the exception again hard truth, but True fact So if we can predict whenever a failure is about to happen we can act ahead of time and We can preemptively recover that device which Makes the cluster much more reliable Yeah And that brings us to the other part of the performance So it is natural that the cluster usage has a certain pattern of Picks and off peaks periods during the day In case we have a failure we have to respond with a recovery and The priority is very high. So it might happen on a peak hour, which is not ideal whatsoever So if we preemptively recover we can schedule it for A time which is off peak which is fantastic Yeah, and with stuff in particular recovery can be can have significant impact on performance Maybe say do you want to say a couple of words about that? This is sort of a I think in any system you're gonna have an extra cost that you pay in order to do the recovery It's a problem that we've struggled with in Seth's making that as little as possible But it's still something that you ideally want to do it in off-hour if you can So when we started this project We're sort of blank slate Our goal is just to make everything sort of work in step out of the box So when you install Seth as a standard user it would gather the metrics to do the prediction Do everything without having to install extra tools or external dependencies? So we would build in the data collection We would build a simple prediction model the expectation was we would start with something really simple It would be open source and we would hope that the community or would develop something better and it would improve over time But it was sort of a monolithic approach to the problem About halfway through the project A company came along called profit store that specializes in doing AI enabled Data center operations and in particular doing disk failure prediction And they had a south-based product that collects data metrics from the customers and has runs a very sophisticated prediction model that they claim like 97% accuracy and Has a dashboard very sophisticated and they were very interested in integrating with Seth so that Seth storage clusters could take advantage of their service And and reap the rewards and so they wanted to work with the community to figure out how to integrate this with us And their goal basically was to provide a free service to any set users so that they could share their day with data with profit store get Their predictions back or if they become a paying customer for example because they had a large cluster They wanted more accurate predictions and they could use that sort of fee for service as well So and they wanted to have both an on-premise option that would work in your data center or be able to use their SaaS service and so As we this conversation evolved with with private store realized that the two completely different models and ways of approaching it We're sort of colliding and we need to figure out how to how to approach the problem So what we eventually ended up with was a modular approach That allows you to swap out different components of the overall pipeline to enable to accommodate both models Self-contained and using a commercial service So it basically breaks down into three pieces and you have to collect the metrics on the devices You have to run some prediction on them and then once you know what the life expectancy of devices are then you have to You want to make Seth automatically respond and have some mitigation action and so we Built all these components. We want to build all these components into Seth so that the whole thing can work out of the box It can do the collection It can do a prediction and it can do the mitigation or if you want to use external tools because you're already scraping device metrics using Some other infrastructure Or if you want to use an external service like profit store, you can do that as well And by building sort of breaking it down into these three phases We allow you to swap in these different components and use it in whatever way makes sense so the first part of this is Gathering device metrics So what is smart? Smart stands for self-monitoring analysis and reporting technology It was It was started as an attempt to to give access to the devices health parameters and it So supplies a Very very simple prediction model whether the health The device of the health is okay or not It defines several attributes So for example, the device is temperature The number of the hours that the device is powered on number of bad sectors So on and so forth and each manufacturer Define their own thresholds for these for these attributes so in case An attribute crosses a certain threshold the device is considered to be failing This is a It's a very nice intention, but eventually It doesn't work many devices will fail without crossing certain thresholds or The It can be expected that the device will fail will fail, but it will not show it on the smart simple prediction So we decided to collect the health metrics ourselves And to analyze it But that came with a few challenges. So for example the interfaces of set up Sus and MVME they present different health metrics And of course it was the issue of the vendors again that They have their own So they don't have to include all of the attributes of smart. So if you buy Samsung device it will not necessarily have the exact same attributes such as IBM device or whatever and That that That actually Put some more overhead to normalize the data And also they have different scales. So They one vendor will decide on scale from zero to a hundred and another one would be set on a scale from zero to 255 so it's not really standardized We used the smart one tools smart control to collect the data, but If you guys had the chance to work with that, you know that the output is not ideal for machines So you can see how it looks First of all, it's really important to say that the smart one tools community is awesome They are super robust. They they give They support all the devices out there. They react quickly They're doing a really great job However, the output is aimed towards humans. So you can see there are all sorts of pretty tables in here but if you want to get this data and transfer it to some sort of a model It's pretty challenging There are out there all sorts of reference that will take this data and will process it and pass it to sort some sorts of Jason's but but the Thing is that they don't always Take care of all the cases What happens if there is a new device out there? It's not it's not ideal so We decided to do the right thing those of course staged idea and to contact smart one tools for A built-in Jason format Output So we prepared a patch that was part of Outreachies application process, which is amazing that It just froze you in the water and says, okay, get your hands dirty and make a patch for for that specific project So we contacted the maintainer of smart one tools Christian Franke Who does a great job in responding and helping us? We submitted a patch as You can guess this is a long-standing feature request for smart one tools Smart control in specifically And the good news is that it's about to it's expected to be released by the end of the year Which is just in time for the upcoming set not always release February 19 The second piece of this was that the way that Seth had been Designed and built up until now it can run on any hardware But it really didn't deal with the details of the underlying devices And so we would pay attention to where all the OSD's are and what hosts there are But it didn't have the internal tracking to map that to physical devices And so we had to add a bunch of infrastructure into Seth in order to maintain that metadata and last to store it So the first challenge was figuring out how to actually identify a physical device And it turns out that vendor model serial is a somewhat standard way to do that It's what the Udev and block ID libraries use and so we adopted that So we added this additional tracking into The Seth cluster manager so that all the demons are reporting which devices they're mapped to we have the sort of many Mapping between devices and demons so we know which devices depend on which and so on And we had added the ability to store a life expectancy property with those devices so that once we had a prediction We could tell the cluster what the life expectancy was and then it could respond as a result and then We adapted the initial prototypes from the start of the beginning of the project to add a new module to Seth that would First of all implement a command on the object storage demons Which are actually storing data to scrape the spark metrics with smart control and pass that back to the central cluster The there was a sort of a background operation that would scrape those on a daily basis and store them in a radar spool So that we had a recent history of all these metrics for all the devices And we had sort of a self-contained metrics scraping and collection framework The question that we frequently got in this project from other people was why don't we rely on other tools? There are you know Prometheus plug-ins for example that scrape metrics and there are all sorts of external tools to do this And then and sort of the balance that we're trying to reach is to be able to make sure that every user Who's using Seth can have something that works out of the box without having to install extra separate stuff But we also because we adopted this modular approach We still leave the door open so that if you already have an external scraping framework You can still use that and have some other Infrastructure that's doing scraping prediction and it can still feed back into Seth to tell Seth what the life expectancy of those devices are And then the same Seth sort of automated management logic can say oh I know this device is going to fail and and do the right thing as a result Which brings us to sort of the second phase of the architecture, which is the the failure prediction So today we have two approaches and two ways to address this problem profit store contributed a pre-tained prediction model To the upstream open-source project.  I actually don't understand the data science at all So it's a comparatively simple model, but it works it runs inside the Seth manager demon And so you can sort of have an out-of-the-box cluster Analyzing the metrics and doing predictions based on that There's also the ability to enable a feature where it will call out to an external SAS API and some service Either hosted in your data center in the cloud That will feed the metrics to an external service and get a prediction result back and then store that in the cluster So you have both the sort of external cloud model or the the the online model And again profit stores goal is to have sort of a free service that you can use Or you could pay them to get their very accurate predictions to do it And because we built this around sort of a SAS API There's an opportunity for other people to implement that same API or one similar So that you're not a profit store wouldn't be your only choice. You could implement your own External prediction service as well The natural sort of question that we ask is how can we build a better model? So we have the initial model that that profits are donated. That's that's relatively simple as I said But the goal in all of this is to build you know the most accurate prediction model So we can have the best data reliability that we can and there are sort of two key pieces to this And the first is that we really need the disk failure data So a lot of the academic papers have been published about Disc failures and predicting them and they tend to rely with private data sets that the researchers You know got from Yahoo or Google or whatever from their big data centers And they do their analysis and they publish a paper, but the data isn't public Backblaze is a Cloud backup company that is very generous and that they publish all of their failure data And they have a huge fleet of hard drives that they use And So that's that's really the only public data set that's out there And the challenge with sort of both of these is that the breadth of the device models is limited by what those particular cloud vendors Or back place happens to buy If you look at sort of the enterprise world where you have companies like EMC and that app or whatever deploying their devices They're of course gathering all the metrics for the devices that they buy for their customers But again that data set is private and so although those particular vendors might have failure prediction that they build in There's nothing really for the rest of us So the bottom line is that we need we need failure data and more data in order to build build a better model I'm the other thing that's interesting is that There's an opportunity to use metrics that aren't actually necessarily from the device To enhance the quality of the prediction so profit source model for example looks not just at the device metrics that they get from smart But also like things like the server load the network Traffic and how many processes are in the system all this other stuff that they scrape about the cluster and the systems that are actually consuming The devices and they use that to generate a more accurate prediction of when things when things are going to fail So there's a question of which which metrics are the important ones and are there even other opportunities that that we haven't thought about And this led us to come up with this concept that what we really want is an open source public data set of Disc failure data so that researchers and the open source community can build build a more accurate prediction model And the question is what can we do to help help make this happen? That the concept that we came up with is a sass like service not unlike what profits are doing But one that's run for the community in a open and transparent way where you have Systems that are basically sharing their device health metrics with this public data set service So they're publishing their smart metrics and in response. They get a disc failure prediction. So it's sort of using Providing an accurate. Hopefully failure prediction as a care in order to motivate people to share their data People are obviously and naturally skeptical of any instance where you're sort of sharing data about your internal systems So it's very important that you make the system transparent and protect the privacy So for example uniquely identifying devices With randomly generating IDs and hosts instead of having without having any identifying information like IP addresses logged and so forth There's sort of a question around whether you want would want to share your serial number That's sort of the only really identifying information within the device metadata And it's a trade-off because if you do have that you can identify things like you know Bad batches of devices that come off the manufacturing line might correlate with failures But people might be more paranoid about knowing that they might have that particular batch So hopefully the goal is to motivate people to share as much information as possible because you get a more accurate more accurate prediction And then the result would be that you would accumulate this big database and build generate this data set and then share that with Academic resources and the open-source community and people who are trying to build better failure models So you sort of bypass the problem that you have Currently where there just is no good data to train these things against one of the key challenges with this that we've identified is identifying the failure events because when you're training a model you need to know what the signal is that you're actually Trying to predict when did the device fail after all the hell of these metrics are not and part of that is a definition is Arriving at what the definition of that actual device failure is because that might vary between different users So is it when the device is completely offline and it won't even spin up Is it when you have too many read errors and you finally decide that you're not going to use it anymore Is it when you get a single read error and you decide not to use it? Different environments have sort of different thresholds for when they say I'm going to stop using that device and it's it's no longer acceptable The other thing is that when a device fails in the real world do you imagine in the wild Somebody might be running any system whether it's stuff or something else a device fails There's some action that they're going to take you know They're going to their their rate array might use a spare. They might replace it is they might just leave it failed in place And lots of different software stacks and human interventions might be involved in that And it's I think it's unrealistic to say to require that those users like take an additional step of notifying this Cloud service that they're sharing their data with that. Oh, by the way, I decided that this device failed Because it's if it's not automatic Then they're not gonna they're not gonna share that information. So it's hard for this Public service to sort of get that signal know when when the device actually failed And the other thing that you you have to be careful with is if your failure prediction is working really well Then the devices won't actually fail right? You'll take them out of service before they actually crash and burn And then you have to be careful about making sure that that's not Polluting the model as it continues to be refined and trained So one idea of how to how to deal with this is to how to infer failures Is to associate the device with the hosts that contain them? So typically you have a server that has multiple hard disks in it And as long as you have a unique identifier for the server then the servers can see that there are some number of devices that are associated with a particular host And over time you're gonna be receiving a stream of metric updates for all those devices But then after a failure presumably just that one device you're gonna stop getting reports for it So the idea is to basically infer that a device failed if you continue receiving reports for all the other devices in the system But that one particular device failed And perhaps you only do that if you see signs that the device is likely to fail And then it goes away Then you can sort of assume that it went away because it actually failed or because it was taken out of service And I think the real question here is a data science question Is it is there a sufficient signal using that kind of inference in order to train an accurate failure prediction model? and Probably the more like specific task that would need to be taken would be to validate that type of approach for your You're inferring those those data points from an existing data set using like the big back plays data set Which actually does have failure events because they took that time to actually say this device failed But to ignore that Try to infer failures using this method and then see if you can still train an accurate model in order to validate whether This in fact would work. So that's a that's a question for a data scientist that hopefully someone will pick up so that brings us to the Sorry That brings us to the third phase Okay, thanks. Yeah, I don't know what happened So we had the metrics collection we had the prediction model And now we we have the response phase So it's it's a question. What what do we do when when a disk is about to fail? So the question is how much time we have left If we have enough time, maybe we can just let the self operator know that one of the devices is about to fail and then they can Do whatever they decide But if we don't have enough time we would like the cluster to Automatically try to self-heal like we mentioned that stuff is Self-managing and can be self-healing as well So what we do is we will mark these OSDs out of the cluster and we will Migrate that data to other devices Yeah, we also divert the workload away from these devices so So Yeah, we will not we're not causing more harm All of these So we define the thresholds to these Actions so they're all configurable. I think that the default now is in case a device is about to fail less than two weeks from now We just automatically take action But other than that I've left life expectancy is greater than two weeks The set self operator Can can decide what to do that? Yeah, and then there's a question After the device is successfully off-loaded What why do we do that should should we drive it to failure To prove the model was right or wrong Yeah Some open questions So currently we have we have merged The part in the portions of the device management and Matrix collections and the respond they automate automatic response We still have under review the Huge pull request of a profit store Hopefully it will be merged soon And we're targeting this feature for February 19th next Next release of self And in the future we wish to see an open SAS service that And open that say just like sage mentioned And to have an improved free and open source prediction model, so this is a call for academia for professionals for everyone who is Interested in this story in this project Everybody can help So this project Happened in collaboration with our three-tier organization like we mentioned earlier Which offers paid internships with that promotes diversity in free and open source software There are many communities which are participating this is just a partial List as you can see so the way it works that is that applicants Look at the projects that are participating They pick a project that they're very passionate about They contact the mentors they see Which contributions can be made to this to that project? and again, this is hands-on so it's not just Studying theoretical stuff. It's just it's real and many times That there is a certain Well, maybe it threshold needs to be the right word here That prevents people to contribute to open source because it can be intimidating so we with this organization you get the support of mentors which is fantastic the contribution does not have to be just for developers you can contribute For the commentations bug fixes marketing so There are a lot of ways to contribute Yeah, and then you you fill out an application form I think you made many projects Happen many many contributions to open source And that's that's a win-win for everybody because the interns get experience the interns get again the support that is not trivial and also mentors Potential mentors if you have any project that you think would be good for newbies in open source I encourage you to do that Because Eventually it makes the project happen The internships run twice a year a year sorry next one starts December and Applications start they open September 10th. You can see the website. I I really encourage you to go and check it out Tell everyone about it on a personal note I'll share my experience with the project so So picking the project I was passionate about I was making the contradiction with the Sage guidance and then once it starts a Very common Syndrome for interns is to have the imposter syndrome so There is You're all starting to realize oh my god Maybe I'm not the right person to do that That the code base is huge. Where do I write the next line of code? You kind of freak out you have some fear and doubt but then Yeah, we'll know that but we tend to forget nobody knows everything we all learn all the time And it's just It's good to remember it and Seth's community is is is great. They're Sincerely happy to help. They don't criticize you. They know that it's okay. Nobody knows it and nobody Not everybody knows the project and at the very first stages Sage Told me that there isn't such a thing as a silly question Yeah, just think about it sometimes We are afraid to ask so Yeah, so first of all I would like to thank our treatise organizers say sharp Marina and Karen and all the team Who are were very attentive? Responded to everything super quickly and really wanted to help all of the mentors that Took part in this project and of course sage huge. Thank you for all your patience and All your will to help Thank you so the challenges that With just just a quick cap for the project. So we still have smart controls upstream We hope that We hope that we're gonna see that release around the end of the year We still have some changes to the architecture So we're done Yeah, and we still don't have the data to to build our own model hopefully We'll have some open data soon And the outcomes that we had from the project is we do have a modular approach for the metrics collection The prediction model and the response We have a new participant in the staff community profit store for a very Thrilled about this project Yeah Sorry And that's it Thanks to the smart modules upstream Christian Frank who has been great to profit store for contributing And outreach you of course first setting up the internship. So we have a Kind of out of time, but if you have any questions Please please find us and ask So the last time I did do this in production was a couple of years ago But one of the things that I found was that the data supplied by a solid-state storage Was you know both completely inconsistent between different vendors And often extremely sparse Has that improved at all in the last couple of years? It's It's quite complex. I mean That we're focusing on collecting the data well we're collecting the data from Potentially all the devices, but what went at the beginning we were focusing on set up for hard drive so did they do oh Sorry So I'm not sure if it's if it has improved Yeah, I think mostly we're relying on smart control to scrape everything because they've been pretty thorough And I think that as far as what data if we're actually getting out and the ability to actually predict on that is sort of the Self-contained prediction problem at least that's the way I've been doing it And so we're starting with something and the hope is that The situation will improve over time. I was wondering how you evaluate the Accuracy of the profit store predictions The truth is that we haven't yet They've provided a model and we don't we haven't had the time to evaluate it yet. So We're focusing just on that on the system integration problem first I guess I had a similar question. So about the model, right? It says like it's 95 percent 97 percent accurate But it doesn't tell a lot about like a more better metric would be what's the false positive rate? And what's the false negative rate because if it's a false positive then there's a bunch of things that you guys do to make sure That there are no problems, right? So how good is the model at in terms of that? And how much do you care about the false positives? Yeah, that's exactly right. There was a talk that I saw at vault last year that Defined it. I'm forgetting the technical term, but it's a two-dimensional matrix or essentially you have the probability of positive or negative false negatives That's the number that they've given us and again, we haven't actually analyzed it from that data science perspective. So Unclear. Okay. Thank you, Cajun. Yeah. Yeah, what we have one more question So basically if I if I were to try something out like this Would I need to set up an agent to collect these logs and push it somewhere? The you're talking about the open data set Yeah, or yeah, so they did I think they're they're sort of two goals One is to make it work for staff out of the box and if we have a public data set target You could just turn it on But the expectation is that not all storage is set obviously of lots of other storage in your data center So we don't want to build an agent that you just install any host and turn it on and point it at upstream And they would just it would share share that data. Thank you. You are it. Thank you sage We have lunch served at the ziscan lounge machine learning application, so if we want to develop an application we would like to use some programming language and If we take a look for example developer survey results that are published each year by a stack overflow we can see that Python is becoming more and more popular programming language and To me it's not an accident Python has a wide range of of libraries for data processing and as That was stated in the previous thoughts data becoming more popular more interesting then actual code so Python is really nice to how to how to Analyze your data and come up with some some outcomes Okay, even if we take a look at predictions Python is predicted to be rising more and more when it comes to its popularity and As I stated there is wide range of libraries that can be used So we will use Python as a programming language. We will use tensorflow as a library for machine learning We will use flask Flask as an API server and this API server will run on top of whiskey server That is J unicorn and we will use pandas for data frames Okay, so these are all the Requirements that we have and in Python world you create something that is called requirements TXT file and in these requirements TXT file Okay, okay, so sorry for that so Okay, okay, so you have these requirements TXT file and In these requirements TXT file you state all your requirements for application So that was tensorflow pandas flask and J unicorn in our case You take some time you develop your application and you have a good repository of of your machine learning application So the next step is basically to deploy your application For that we will use Some some platform highly scalable So we can use for example open shift and we provide Directly git and use feature that is called source to image and we directly deploy our application to open shift so now we are running our application and As time passes our application is becoming more and more popular and after eight months We have community and our application is serving user requests Okay, so After eight months you decide that you would like to Do some improvements in your application so you code and you open a pull request and in this pull request you change some behavior in your application and Suddenly your CI fails Okay, so what do you do you spend some time? debugging your application and After some time you realize, okay, there was some change In a library that you are using so the error is not directly caused by by changes you introduced in your application But we're other in the library that you are using so what is this behavior change? So previously you used pandas in version all 21 all and in this version you had some some data that you analyzed and You expected that values that are not stated in a in a data frames are summarized this way so if you have ten and None ten plus none is is ten if you have none and none it's none and 30 plus none it's 30 in pandas data framing this version okay, then you did something like mean and The mean was computed like 10 plus 30 divided by 2 because we had two values and none was omitted because it's not an under So the result was 20 with new version of pandas. You suddenly have different behavior, so You have exactly same data, but now when you summarize them You suddenly don't get none plus none is none, but none plus none is zero and that also means that it affects your Mean so now we have ten plus zero plus 30 divided by three Okay, so basically by chain Basically by the change that was introduced in pandas In new version your application is suddenly not behaving as you would expect and you were lucky because your test Uncovered this misbehavior The problem is that when you developed your application, you didn't state it exactly what library is you expect in both versions and these libraries change over time, so Even thought your application is not changing these Libraries are changing and you didn't state it like okay. These are the libraries that I'm compatible with So now let's do it differently. So instead of having requirements the xd file where we state names of libraries We can have requirements in file that's where we state names of our libraries And then we let generate requirements the xd file so now we have Generated requirements the xd file with all the libraries in specific versions and if you As you can see these versions are flipping down So they are pinned down to a specific version and it's not only of our Requirements or our dependencies we direct to use that is TensorFlow pandas flask and pandas I guess but it's also There are stated also dependencies of dependencies of these libraries that we are using that is somehow Goods because even if there is change in these transitive dependencies We have still pinned down to a specific version and we don't no longer rely on a resolution that is time-dependent This approach is kind of old school. It uses peep tools and its command peep compile Now the recommended way to pin down your dependencies in a python application is to use peep and And peep and does exactly the same thing. So it locked things down all your or your Dependencies including the transitive ones Okay, so we fixed our application We pinned down pinned down dependencies to specific versions and we pushed our code into Give you a repository and suddenly we have another warning We found a potential security vulnerability in your application or in your dependencies as you can see by pinning down dependencies We can just by checking versions or what you've stated in these requirements the xt file provide information like okay, this Will be installed and it's no more time-dependent on during installation. So you have specific requirements stated and Just by checking Which which versions are installed we can directly Say what's wrong? Okay, so we exclude the given vulnerable version from from our requirements the xt file and we push it again And suddenly our CI is read again so it's probably not that easy to find the right software stack and In in some cases you you have something like okay There are things like you are not using this or you you you should not use this But there is nothing like okay What should I use instead and that's something that thought is trying to approach okay, so If we have these dependencies that are stated in requirements in file and recommends the xt file like flipping down We are probably losing some information. So we have We have lost some some information. So let's take a look at what information If we check our direct dependencies, we have tensorflow pandas flask j unicorn, but these dependencies have also dependencies So if we plot it as a dependency graph, we see something like this. So tensorflow is depends on gas to store and and so on and so on and Then for example tensorboard that is that is dependency of tensorflow relies on numpy markdown and other Other libraries. So this is our dependency graph that is resolved at the given point of time okay So now let's let's say that for some reason we would like to to change version of tensorflow What does it mean? That means that by changing tensorflow? We are basically changing all its transitive dependencies. So we are changing also tensorboard possibly if tensorboard is not requirement of tensorflow that that was the case of previous versions of tensorflow So if we take a look closer, we can see that for example numpy and verzoic are also affected by change of tensorflow and They can resolve into different versions Given the version range specification of tensorflow That also means that pandas and flask is somehow affected because they are They are using numpy or verzoic in specific versions So if we change tensorflow indirectly, we can change pandas flask and also other transitive dependencies of these These applications of these libraries and suddenly we are changing most of the libraries in our application stack This is an example of another Step that was resolved and we change version from tensorflow from version 110 to By changing this we basically introduced two or three new libraries in our application stack. There were also removed libraries and And there could be also change in versions. So it's not that straightforward to just Change some version or just to change some some Library in your application stack Okay, now let's suppose that we would like to change something in not directly That's not something that we directly use in our application So let's suppose we would like to change numpy and if you change numpy We are again in the same scenario that we are affecting directly or indirectly other libraries and their Dependencies so this is something that thought is trying to approach and thought is trying to resolve these dependencies on on server side and Besides the resolution resolution algorithm. It also tries to Incorporate other information based on experience based on observations based on monitor repositories and with this information guide basically the Resolution to to not the latest Latest version of some specific libraries, but to the Back to the best one the best or the optimal one. So based on knowledge we have and we aggregate Okay, so let's say we again have our requirements for application and we have tensorflow pandas class j unicorn and we explicitly state that we Want tensorflow at least in version 1.5 because of some feature now Todd will resolve to a software stack and it will Basically give you information like what is wrong and why should you use some libraries in some specific versions? The design of thought is that it's actually a team member if we have bots that are communicating with with back-end and these bots monitor monitor Your your repositories and if there is something wrong if they learn from from observations and proactively open open pool requests to to To repositories Okay, so now how to find the best software stack Let's suppose that we have An application that uses one library and this library has no transitive dependencies And we would like to come up with the best possible software stack So when we have something like best software stack, we need to have some scoring function So we need to have some mechanism how to say how good the given software stack is so Let's suppose we have this function and if we plot results of this function for some specific versions of our Library we see that library It's called simply that we are using in version one point all our scores Zero point eighty six for example and in other versions other values we can and interpolate polynomially these values and We can see basically Some kind of trend in in this data. By the way, if we have enough Data to the past we are basically doing machine learning because we can predict based on previous patterns that were found Future so for example if some library releases too early the major release and then tries to fix all the All the bugs in in for example patch releases or a minor releases respecting the same there then You can see this in in data Okay, n