I'm going to talk today about at least two of them, which I'll post in a month, I don't know exactly when, but probably within a month, maybe a week. I'll call you a little sooner if I weren't here, but that's how that goes. We thought it would be a little sooner than now, but it wouldn't be a conflict, but that's not how it worked out. So how many people here have used it or haven't installed it in some way? So maybe a tenth of the people here, once I could use it, and a third, or maybe a half, but I'm going to give more background and less detail, because a lot of people here have not spent much time with it. So are you talking a little bit about what AT-Cluster is? What can you do with AT-Cluster? You know, what you can do with what you project? How people use it? What kind of customers we have? How many of you have them? And then talk about what release two, release one, what the current release can do, and release two, which is the one we'll be out very soon. We'll do. And talking about YouTube for this, I'll put that slide in. And I'll have time anyway, so I'll be out of time anyway. So the idea of putting together a group of computers at AT-Cluster basically means a network machine acting as one. So taking the right of service continually, or more or less continually, even when one system component fails, when the whole machine fails, and this fails, whatever, when one machine goes down and the other takes over, it's work. This follows things like IP address takeover, things like that. It's not primarily designed for kind of clustering. It's not primarily designed for high performance. Although you can't get performance improvements out of it, that's not its primary purpose. One thing I want to say, there is no such thing as 100% availability. It does not exist. People who talk about, I want 100% availability. What they really say is, I want to spend an infinite amount of money and effort and time to get something less than 100% availability. Because, you know, it takes days to occur. There's a lot of diminishing returns of lives here. The more you spend, the more you have to spend to get the next improvement. It's like calculus. You know, you can cut the distance of half each time. Maybe we like the example where they got the mathematician and the engineer. He said the mathematician has the other half of the distance. You can go towards this pretty girl at the other end, but you only cut the half of the distance of half each time. He says, huh, I'm not going to do that. You set the engineer down on the same couch as the other end. He means he starts half of the distance. And he asks, what do you do that for the mathematician? What do you do that for? You never get there. He says, but sooner or later we close it up for all practical purposes. So, you can get H.A. when you're close enough for all practical purposes. You really can't get 100%. Just like the mathematical example here.  That's the main thing. It can make your actions very short. It's designed to recover from single failure, not multiple failures. Although it often does, it covers multiple failures. It's a lot like a magician's trick. And in a lot of magician's tricks, sometimes you look at him and say, wow, how did he do that? Sometimes he excites and says, that's a cheat trick. That's not so good. And the thing is true about H.A. Sometimes you'll think it's wonderful magic. And other times you think that can be better. But H.A. is designed to improve. It's one of the few things you can do by 90%. What that says is you're still left with tips. So, of course, as you try to improve it more and more and more, it gets more and more and more complex. And eventually your complexity becomes your entity. And it becomes so complex you can no longer maintain it. So one of the things we try to keep in mind as we design things is complexity is the entity of reliability. We've really worked the engineer out single points of failure. It will cause the whole system to stop delivering service. And good H.A. design will eliminate single points of failure. That's sort of obvious. But how does H.A. work? If you have a single point of failure, the obvious thing is to provide a redundant component of some kind that does the same thing you can then use to do what the failed component used to do, when it worked. So we've managed the redundancy to improve availability. Sort of like you have an init process, it runs over the whole cluster. And I say super cluster, super cluster running in on steroids. That's kind of the way to think of it. And he thought of init with respawn as, oh, that respawns my candy and that works really well. Good to be trying to respawn the network. You'll discover that respawning network doesn't work too good. But this is analogous to saying you have a respawn on your services that if they die or they stop working, oh, they just get restarted. That's what really H.A. does. You have nodes on death of nodes on loss of productivity outside world for services that aren't working. Now, I know this has never happened to you guys, but, you know, sometimes databases, like, they're running but they aren't doing anything. Apache is running, but it's mainly, you can tell it's mainly consuming memory of nodes as well. You have a lot of redundant things you have to then manage. One of those things is redundant communications. And one of the most important things is your nodes in the cluster can't communicate with each other. They cannot make any decision about what to do. They can't talk to each other. All else is, it's kind of hopeless.  So external communication is also typically essential to the provision of services as well. And also, communication to the outside world is accomplished more through routing tricks, BGP or OSPF routing to get that done. And for that piece, it's helpful to have an expert in that around to manage those things. So I'm going to talk less about that and more about the other aspects. You also need redundant access to your data. And you can do that either through replication or for sharing or for making it somebody else's problem. So if you're ready to tell your guide to the galaxy, it's like covering it with somebody else's problem field. It's still there, but no one can... You have replicated access. You have data over here, you can share the same piece of data and either one can access it or you can do backend storage of some kind where somebody else has to manage the availability of that service but you can't or don't, or don't want to, whatever, and they manage the availability in their own way. Now, making it about your problem. Now, okay, thank you. Thank you, guys. This is a verb. Now, that's the next question. The two of you need to raise your hand to get it. Always the same guys. Thank you. The interesting useful question is why are social abuse systems high enough to save everybody who wants more and everybody doesn't have it very quickly? The commodity hardware costs which are very important. Whatever it is, you can buy HP if you really want. But you can buy it. Cooper is hardly one with the point if you build it for white box parts and you're set... I built it for 800 euros. No single point of failure. The cost can be eliminated here. The cost can be taken care of here in a way that people, both people, do not appreciate or understand. However, the second item here, you cannot give away complexity. Complexity is like latency in networks. You can always solve a bandwidth problem by throwing more money at it. Solving latency. How many people here have actually solved latency problems in networks? It's much harder. Complexity is like latency. It's very hard to get rid of. And complexity is the enemy of reliability. And so, once you solve the cost problem, pretty spinny stuff with... Well, the real reason I want to do this is because it makes my head hurt. But we try and manage that as well and I think we do a pretty good job of it. This is a general open source chart about HA... about open source in general but about HA in particular. Initially, people have charged for HA systems kind of in $100,000 to a million dollar range to have an HA system. And I'm just going to show you an HA file that's over for $800,000. I guarantee you that. But so if you take these costs and go over the down and down and as a person who gets the data, I get more developers. And this is in fact what's been happening is we've decreased the cost and there were people using it that's gone up significantly. We have people using it for things like we have high availability bad readers out there, man. That turns out there's a really good reason for that. But they wouldn't just pay a quarter of a million dollars to make their bad readers high availability although maybe in this particular application it should have because they wouldn't get it. We have, you know, I got locked out of a there's these storage unit things in the U.S. you go and get storage somewhere and I tried to get out, I clutched my number to get out and it was lucky out. And I finally found out why is the Windows machine had crashed and it was taking the next 10 minutes to reboot and I was stuck and couldn't not get out. I don't know, I don't want to sell an HA system cheap to these guys.  You know, I think, you don't know what's wrong, these won't let you out. So a little about the Lettings A.J. project I guess one more thing here about this is developers go up here too and the whole thing about open source software is it works once you reach critical max and the project becomes self-sustaining. I hate that part of the analogy of a nuclear explosion and open source software but it's very similar. You have to reach critical mass so that the reaction becomes self-sustaining. So what I have to hear in my project why I've done after the low end stuff is because the volume goes up very rapidly as the costs go down. The project, not as old as being the largest associated community we have thousands of people we have about 1,000, 1,500 people in the middle of this moment and now around 10,000 clusters in the world today. The core piece of the Lettings A.J. is what we call heartbeat a lot more than that. It's one of those things, what's your name? So I think it shades the name. This is a production about Lettings A.J. and I and it runs like a division that's called Lettings A.J. also runs on 3BSD servers and I think it's written on the openBSD in the past but I'm not sure what the correct status on that is. Lettings A.J. is shared with every major length distribution Damian, Gen2, Set1 SUSE and so on, all except for the project distribution they have their own solution to that we're hoping to change that in the future and we think that might happen but a lot of people use it, that's the point People use it for all kinds of things these are for low end users web servers, database servers bad readers, custom applications like bad readers firewalls, retail point of sale solutions one of our big things I gave ourselves this with a solution that how many people hear from Germany? You know, Karstads, right? Karstads, RedLittics, A.J. in the back office of all of their retail, all the retail stores, the store I'm called and authentication, files are there's branches, everything, all kinds of things people use it for. I didn't know of those users for so far that's come up as SAP SAP has an interesting architecture so we'll work with it some of the people that use this imagine you use this Karstads I mentioned we're very in regularization I like that story because guys sent me an email and he said we were there and then my sister administrator made a mistake and he crashed the machine now I know you guys they don't make mistakes like this guy's did but he made a mistake and he just fell through the machine and he said, and it worked it worked for the east Steve said he's making interviews with people in New England not New York, where they had the power outage and he was still up but I've been always playing credit how many people have seen this the thing about architects designing a swing you've seen that cartoon it shows them getting infinitely more complex and all the customer one was a tire swing well let's say they release one it's like a tire swing you can get on it and you can just get your H.A. and swing to your heart's content as long as what you want it was a tire swing and that's why SAV did something a little more than a tire swing one of these three layer swings or something like that you saw those pictures but it does lots of things it knows when those fell and fell over some services it can use various communication methods and it can also detect a lot of conductivity the outside world can be connected lots of sand, accountability accountability, passive most people actually want most diplomats are two known clusters because people understand two known clusters it's like there's only two possibilities either they're both up or only one's up they can do chances or none of them you also know what to do there too either everything's running on one machine or you have things running on both machines potentially or if you have an active passive you have everything running on one machine or everything running on one machine so the choices are even simpler but people like those things because complexity is indeed the enemy of reliability so even though release two does a lot more things a lot of people are going to use it for two known clusters because they know how they work and occasionally you find that people who design these systems are not as smart as the people who manage them and then you run into problems but if you stick with the lowest number of denominators they'll work for you that's not related to linking J that's just related to good advice so it has some some sort of simple kind of administration tools and if you want to monitor resources like when your web server is working or not and release one you have to do something like mod or monitor one of those tools to monitor that and it can cause a failure in the trigger when you say J but you can do SNPNs but it's kind of limited compared to what some people want the most commonly requested thing is this ignorant of the need for an experimental tool to monitor resources like your web server to see if it's really working so when release two does that add a lot more things it does have built-in resource monitoring which I put first because that is the most commonly requested feature so that you don't have to write another tool and continue your tool the average person by the way gets release one up in about a half day we don't know anything about AHA don't get the software downloaded it's all in the machine if they read the manuals and more funds around they didn't get it working without reading the documentation which appears to be the paradigm that most people use but I'll give it a hand I'm not going to give people to read the manuals so we've actually spent a lot of effort putting messages in that says this isn't going to work because you can't read the manual because Unix administrators read the system logs Windows administrators assume there's nothing meaningful in the system logs that they've ever read them therefore we support much larger clusters up to well we've been currently testing with about eight nodes I don't know what the limit is it's probably the limit of how complex you want to set your configuration and so on we have a lot more sophisticated dependency model where you can say this resource depends on that one but they don't have to run on the same machine that's the kind of thing you need to do for SAP SAP does not, for example if it's anything about the network to another component it goes down, you have to restart both components and even though they're running on different machines and that's the kind of thing you can't do with release one it has a lot richer set of constraint support and so on and the complexity at the end we rely on does that sound more complex than a phrase? we are still just as concerned as we ever were about it so we're adding as little complexity as we can but you don't get something from that and the resource configuration stuff is all XML based which simplifies writing and doing some sort of work and other auxiliary tools because it's a standard format like it or not some of these things here that in CVS and the process working on that will occur probably by June and things like a configuration monitoring that will happen soon to that we're going to work with GFS which is the next talk about so we can integrate and work together with GFS multi-state master slave resources most resources is like I didn't stop, we're starting there are some resources that aren't there are more templates than that they're in stock, they're in slave models or they're in master models so we're going to be handling some others most other EJ systems handle that by a combination of weird clues you have to kind of know how to make it work we're trying to handle it a little more directly here to start with it's not going to have ideas and models like we just learned that unfortunately there's something that's just really simple in 200-plus years that we have to think about how to do it right in release 2 that we haven't so release 2 credits I'm going to give a special credit to Andrew Beacock who wrote by the far the most complex piece that released 2 of this CRM and CRV which we'll get to in a minute some other people who work for NCSA in the US and for IBM in China Lars we're asking for you if you know what you're going to use the architecture and he's been a PHB a 49er boss for Andrew Beacock and I've done various things mostly as little as I can that's the whole paradigm of the idea is you get a project together and you get other people to do it the release 1 architecture has various processes and so on basically everything here starts a master control process and the processes that actually start the network are separate and there are security and simplicity reasons why we do that you can have a client our APIs and so on here and release 2 has more processes and so on than that in this process most of what was on the other charges in this party communication comes to the layer but I can talk a little more about the release 2 architecture because it's I think far more interesting there's plus a resource manager that is basically the policy is in charge of policy it has 2 pieces of P policy engine and T transition engine which are part of it which I did show up at several times this is like all the blocks that you could open it has a membership layer which does consensus membership which has a lot of nice guarantees of consensus fully connected consensus membership a local resource manager which basically does the job of starting it's like a dominant it starts and stops things locally for the zero the cluster information base which is a set of data consistent that doesn't say database because that would be like too sophisticated we're not that sophisticated it's much simpler than that it stands for shoot the mother and the head shoot the other than the head so it's basically an engine that's designed to terminate machines that have left the cluster without permission so it's not like it's in a high school room without a note going to the bathroom without having a note you leave our room we shoot you so these are the basic components that go into the community heartbeat is now its primary purpose the old heartbeat code is its communications layer it's a reliable multicast and things like that it's always like that and these all the green arrows and our clients of heartbeat all the blue arrows are a we have a lot of clients over time relationships here and that's what most of the arrows are and people just go in the heartbeat and the blue ones go elsewhere see I thought the green code was the most important that's what I wrote can you tell us something about these consensus cluster members that sounds fascinating the consensus cluster membership guarantees that who is in the communications is that it breaks and as a result one thing becomes more than one thing because you get divisions in the communication that can happen I know you've never had it happen where you had a switchboard go out but it does happen and so you can imagine again maybe you get divided in half depending on how you have your switches and stuff set up and the kind of bugs you have and maybe you screwed up your firewall rules you guys don't do that either what my customers do so if you become our customer then you can screw it up too so the idea is maybe your communication only four nodes here can communicate with themselves but what that guarantees is that everyone is in that four node they all have agreement that they're all in the membership and then they can all communicate bi-directionally with each other so it's a sort of transient closure on the connectivity matrix combined with selecting a click or a collection of those machines that can do that and if you think about that that's probably an NP-complete problem therefore we do that by heuristics since we don't have a solved P versus NP for those of you who don't know what P versus NP is it's a major major computer science problem it's it would make you famous infinitely famous for generations if you would have solved that problem so so basically what we do is we have a heuristic set of machines that can communicate with each other and it guarantees that all of them in there all agree that they're members guarantees about who to communicate with who and who all is in the membership and everyone that is in that membership agrees that they're in that membership and when you have you ever try to get a bunch of people to agree this is like that except that they're stiffer than the people but not as not as often so is it actually there are a lot of results if you haven't looked at the theoretical results for clusters there's hardly anything you can prove you can do in fact you can prove you can't do most things in a perfect way in an optimal way however it's like the other thing you do a good enough job for all practical purposes and that's what we do here does that help with that question I'm doing pretty well so we have what we call resource objects instead of just resources they're kind of a more abstract resource object thing but no it's not a C plus plus it's still in C we wanted it to be reliable and actually that's honestly the C plus plus versus C was about memory allocation one of the goals of an HA system is think of it this way we want to run without stop for a hundred years how many bytes of memory do you get there zero how many people see any C plus plus program we're running for a year with zero bytes for every leak and it's not that you can't it's just that you use all these other libraries that you do and it's easy to use them but it's hard not to use them and they aren't always as good a quality as your code so we made the choice to use C instead of C plus plus largely it's a memory decrease honestly memory resources which is something like a web server it's a different flavor of the resources basically one of the things you can do is give it an script for your service you can tell us oh that's a resource and we just know how to deal with it by itself we won't know how to monitor we're not going to do websites with it you can do heartbeat style which is release one style or these open cluster framework style ones they're all pretty similar to each other so there are slightly variations on them and basically that means like starting to stop into service or you can have something which is called a resource incarnation I want to have 10 copies of this writing I want to have for example if you use a GFS of global files you want to have a copy of the mount on every machine resource groups I get to talk about co-location of linear ordering of states but basically the mouse do basically a clone of the kind of things you can do with release one I'll just say that I don't know what the states are in a minute multi state resources which I talked a little bit about the master state resources they're really useful for replication because DRVD is a system that replicates data from one machine to another it basically acts as a block device which replicates the data over the land at the same time and you can either be running DRVD or not and if you're running it it can either be in master mode or in slain mode so that's three states, not just two and most HAA software deals with that badly and since we think DRVD is really important we try to do a little better job with that maybe something they don't do and something that doesn't work it's just that that isn't obvious and it's not obvious that the most probable cause of an HAA system to fail is to properly design, administer, and set up is human error what the software is going to do and that means the more your mental model that the software has matches the mental model of the human being the more likely they are to do the right thing now when I did a system, I like to get these calls at 2 in the morning nothing ever happened during the day where's she at I actually, the first ones were I came in in the morning and said this hasn't worked since 2 in the morning and I didn't call you I could have gotten up at 2 fixed and gone back to sleep and then told my boss I'm going to come in late today but now instead I came in on time and everyone's breathing down my neck I'd rather get up at 2 in the morning um basic dependencies is the kind of thing you can do in release 2 you can have already dependencies that is to say this service has to start before that service you can also have co-location dependencies by co-location, I mean must run on the same machine as this other machine for example, if you have an attached web server it's probably a good idea to co-locate it with the IP address that the web server is trying to serve if you don't you'll be disappointed and someone on the mailing list will be happy to straighten you out so you've got to start before or start after and must be co-located with and this one is actually here for SAP, cannot be co-located with there's something in SAP that has to be run on a different machine for this other piece or it doesn't work it's not the same as SAP, but SAP comes to mind so instead of having this linear list we have a dependency A right before B right before C right before D and they're all on the same machine this is more of a direct to a separate graph more flexibility in how you set it up and more ways to screw it up so that's the only reason why a lot of people want to use resource groups because they're simple and it's hard to screw up I love things that I cannot screw up you can also have mandatory constraints and never getting this sort of preferential constraints a mandatory constraint you can tell a resource that has to run on one of these machines that has a fiber channel attachment to the disk that you want to run and maybe every machine doesn't have that like in the case of PRPD it's two-way mirroring so it has to run on this machine or on that machine so it's better to run on one of those too and the fault, by the way is that a given resource can run nowhere so if you get to tell it it won't be able to run anywhere you can also have preferential constraints that say well I would really rather it didn't run on the same machine as this but you can have no choice for example like all the other machines are down it will run it somewhere else basically trying to satisfy the mandatory constraints first and the preferential constraints second and you can provide ratings kinds of things that says well I prefer for this to be run on this machine but I prefer even more for it not to be run on this machine kind of thing you know you can deal with those kind of things in the appropriate fashion and this is more complex this is the kind of thing you use when you need to use it a lot of people in the center are very very happy with the tires we work a lot of the leaks do stop and satisfy their main needs the point about this is this is the kind of thing that makes a difference between our system which works pretty well and a lot of people are really happy with the one that we're really competing with any commercial A&A system out there that's where we're at it is comfortable to pretty much any commercial A&A system out there and that would be important not because everybody cares about that but because people like to run the same software everywhere and if we can't solve all of our companies problems they're going to want to get a solution they can and produce that same solution everywhere so it comes us out of all the places where we can do the simple thinking because they don't want to learn too fast oh this is useful for managing what do you suppose I meant that's useful oh managing these things there that's what I meant resource incarnation that's the example of where you can have more than one or something running and that's useful for managing low-valency clusters just like it says here low-valency clusters for example where you want to have a low balance for a front which is right behind our machine so there's a bunch of behind it and behind that a database server well all these different web servers you want to run a hundred copies maybe or expand across the machine so you give it a low-valency cluster we want the end of them to be slave servers it's useful for dealing with cluster files which we can actually have the same files on more than one machine simultaneously and not crash the data it's useful for certain kinds of IP alias techniques called cluster IP alias where more than one machine responds correctly to the same IP address you have to set up your arms like that but that's something you can deal with resource groups like I mentioned before they're kind of a shorthand from providing ordering and co-location dependencies all at once as long as you don't care that you don't mind starting them all one after the other in a linear sequence as I mentioned here before each resource object in the group is basically declared to have a linear start after order relationship and each resource in the group is declared to have co-location dependencies which basically means they all run on the same machine and they all run in the order you have in the group that's much simpler to describe this is the easy way of converting resource groups into lilies too and honestly what a lot of people are going to need most of the time for most of the things they do and it has the advantage of it does what you want in all part of a linear it does what you want master study resources as I mentioned I've talked about this a couple of times but the main thing to note is it's ideal replication resources where you're replicating data from this machine to this machine either that it's running the replicator or it's running a master order some kind of replication can actually do by an actual replication that's not an issue here that's another kind of resource but not everyone can some want to know which is the master which is the slide you can have associate arbitrage now we're getting into this a little more more esoterica honestly those can have arbitrary attributes associated with an example you can say these nodes have a fiber channel connection you can say fc equals yes and then you can say color resource this resource must run on a node that has fc equals yes and so that's the way I'm specifying them that they attribute or property to the resource and then having the resources run up the ones that have those properties as opposed to just an extended specific list of them and that's often the hand here or you can say for example release greater than 4.3 or whatever as it says there's types of integer string and version version is of course being strings that compare like integers but not 1.2.3 is greater than that 1.2.2 and less than 1.3 are those kind of things the weird things that people use in version and you can do the kind of comparison operators and you can also say I want to run on one that has this has to be defined to be something or not defined or I want to run on a node that's co-located with another resource that's not co-located and these are preferential constraints so you can say that this can get a waning out of this which then says that I would really like it if you would run on this but if you can't that's alright too if you get into the larger clusters in a complex environment you get into things which you have to want to spend the time to test what you're doing here you have to have a good reason to do these in my view now not that there aren't good reasons to do it but the majority of cases don't need this 90% of the H8 cases in the world don't need this because the more complex you get in here the less likely you are to get a rock even beings will like that I don't understand about me anyway I'm not into other kinds of constraints each constraint is associated with the difference before I said and as my way to reach a node it might run on and then if things are not to be true then the waning that's given there is applied and the waning can basically be a positive or negative integer and the waning and all the things can fly are added up and then try to figure out roughly where you want it to run there's no attempt here that's really hard the integer linear programming problem that we probably would need most of the resources to possibly compute where we ought to be running the resources and that was something we decided that probably wasn't good for release 2.0.0 tongue in cheek guys it may not be good ever because it's the kind of thing one of the things you want here in a reliable system is you want to know what it's going to do because you want to be pretty confident you've tested that situation because an HA system which isn't work, isn't tested isn't HA you may get one probably won't all the time so those are the kind of things you do I want to talk a little bit about security here which is a slide where my hobby orgs is to talk about regarding clustering and a clustering is a computer whose back plane is the internet how many people find that right good, this is good you may think you have a secure clustering network there are two kinds you're probably mistaken now you will be mistaken later okay and the reason why that's difficult is security is not often well understood by admins and the person who sets it up they've done just fine when they set it up but they've turned it up to somebody else who wants to do better things left the company, former company whatever and the people left behind doing it they don't understand it as well and then you have your hardware installers to go in the half-room and plug cables together and maybe they cross-coded to two networks which shouldn't have been maybe somebody set up a route on one of the Windows machines and you had thrown it on the Windows machine set up a route I guess deliberately hardware installers don't fully understand it and that's very, very abnormal most very crazy to do it in fact from trust and staff and staff turned over a big issue where I mentioned the person installed it and understands it the combination of viruses and peer-to-peer technology are frightening instead of having one that sends out spam how about one that comes and makes all the Windows machines and your company send out spam how about one instead that opens up an outgoing firewall hole that's certainly going to happen the question is when so security is a very big concern and remember the reason why it's a big concern for clusters and clusters the RAC plane is really the internet now you can argue it's a secure network but I already made reasons why I have to select that suspect now honestly I like crossover cables they're reasonably secure I love this stuff good H&A software should be designed should be designed to assume insecure networks good H&A administrators will assume that the software doesn't work and they'll put it on so what they think could be secure networks so each of you needs to mutually distrust the other to have a good installation but if you look at clustering software the vast majority of it kind of assumes of course it's really important and you're going to have a secure network I know you are, really, truly and so this is something to be cautious of a lot of software does not do that in the way I would consider a reasonable right so actually I got it through the whole talk this is amazing maybe I should have had that extra little caffeine before it came in so this is kind of the appeal we kind of try to do in these two people use this for all all kinds of solutions I really, truly do have an H&A file to burn my house and then my MP3s play all the time so I have my priorities if you wonder and that's actually how I test it when my wife was out of the house and so I started playing music on this machine and I went on to this other machine and I was playing music and this one was using this client and this one was using this other client and this one was in the fastness and it had 4 or 5 songs you cannot get these songs synchronized so it kind of sounds a little hot and you know I failed the machine back and forth and I was just happy and just really glad my wife was so what I'm trying to say is this technology, indeed can be afforded don't misunderstand that you don't have an application for this that you didn't think you had these things work particularly with technologies like this for biosystem replication in DRVD that's a key for what makes it inexpensive I replicate from one ID to another ID now on the other hand what that means with the IBM sales that Deputy U.S. says buy a higher channel we know buy everything the IBM salesman tells you to buy but because that's what keeps me employed and they make money I keep my job so I'm happy on questions yes the question is about how that relates to single point of failure right so he's asking here about setting up something and he mentioned that you feel like you said it up for me the point of the thing is you do want in fact to have no single point of failure actually in the case of DRVD if you had a second with heartbeat certainly it would have known the two sides were back up and you can heartbeat over as many different interfaces as you want apparently it's only limited by your budget and complexity but two is a minimum recommendation for heartbeat pass you mentioned earlier on even though I was talking too fast that it was very very important that inter cluster communication be replicated because if it's not then you get the result you got because the two sides don't know what's going on and each of them thinks it does and they don't so that's a bad thing and that's also another reason why people use numbers as well in this environment further questions oh sorry I shouldn't walk into that life I can't see go ahead he's asking what kind of heart beats are currently available on let's say J and you can heartbeat over UDP multicast UDP broadcast serial you can set a funny kind of heart beats called pink heart beats where it appears that your router is a cluster member but the normal ones are UDP multicast UDP broadcast and serial the heart beats the only communication in release 2 is the same as the communication in release 1 we actually have a pink mechanism that won't come that we do for this drive in release 1 eventually we'll get adapted to release 2 after we figure out how to do it right it's not going to be something that's going to happen because honestly we have a lot of things to do and we need to think about we have a really nice one that works very well and we need to figure out how to adapt it to release 2 and it's much more complex further questions before the rest of the guys leave here over here you you're right about testing you can build one of these things tell your manager you can test it so what he said was if you don't test it it isn't HA that's the short answer right now that's not something we're prepared to deal with let's talk about it it's really one of as much of the LVD issues as well as the HA issue he's asking about backup slave nodes for the LVD and I don't know the answer to that question really I haven't talked to the LVD for a while we've talked about various things but I don't know the current status there's another question up here up the aisle yes that's actually something to support it release 2 we have to monitor the application but that's supporting release 2 yes you have a question here we support every configuration support we support configurations that they don't support ours is an open community project which a lot of people have been open over a number of years and historically it has to be an available outside of the organization we're working with them and they're looking seriously at release 2 because they're like we and with regard to whether that will happen or not you know like they say about the outbreak it's over to the family you see but right now we have some positive discussion with them and that all looks pretty good right now but you know it's not for me to say it's for the LVD yes where's the question let's look at the audience when they ask that question okay alright I need new glasses I think go ahead with your right hand we'll take the IP address with a new active note what about the MAC address is it supported to take the truth is that IP address takeover is just a resource to us we can take over any kind of resource for which someone has written that resource agent if you write a MAC address resource agent it'll work for you we don't currently supply one we know people who have done that but if you want to supply one we're always looking for patches okay you can't do that raise your hand if you raise your hand while putting your coat on you're asking the question what's the question over here we got you asked a minute ago go ahead do you have something do we have special techniques for long business clusters in release 1 we have absolutely nothing for that in release 2 we're thinking about it again we want to get 2.0 that zero out and stable and we're going to start looking at profit we haven't tried to address the path that is something we have had in the back of our minds when we were designing release 2 we have some good ideas to work but it's not ready yes up there at the top do you have integration do you have integration with re-imaging systems if you start with 5 machines doing handling one service another 5 machines doing another service and all 5 in this service break down you need to rebuild 2 other machines and over that service if you have a if you can write a resource agent that does what you need to do I'm sorry I mean the question is it's a script if you write it a knit script that does what you want it to do then that in this script will work just fine for us yeah you can code a situation like which says oh dear I have no nodes left in that that service that's a possibility it's not something people have asked about honestly but the thing that happens is time goes on people use it more and more they deal more and more different things with it I'd be interested to hear if you want to do that there are probably some difficulties like everything else probably some difficulties don't come but I can't think of any theoretical reason why you would one wouldn't be able to do that pretty soon when? release 2.0.0 will be within the next month I would imagine we'll have a fully stable remember HA is about paranoia you're not carrying out the paranoia which you should be right not that we're trying to produce crap that's not it it's about paranoia it's about making sure that other people we're going to try to have the 2.0.0 imagined by people would be happy to run on any system no matter how fertile you are but you have to get the 2.0.0 release in order to get that point there's no way to get there without going through a 2.0.0 release for the girls you can go now


Let's look at the variation of the planetary boundary layer over the course of a typical day. We'll start midday when the sun is out and solar heating of the surface is causing buoyant air parcels to rise until the virtual potential temperature matches that of the overlying air. These air parcels have momentum and they overshoot the level of neutral buoyancy. In the process, they entrain air from the free troposphere. Clouds form in this layer. The rising air parcels collide with the air above them and rub against the air around them, producing a whole range of different eddy sizes in mixing. These large buoyant eddies cycle in tens of minutes, mixing the air. As the sun sets later in the day, there is less solar energy to power the convection that stirs the mixed layer. And the boundary layer collapses, leaving behind a residual layer that contains the mixed layer air that was left over. Emissions from the surface keep pouring into the boundary layer, but the boundary layer height is much lower. With less turbulence in the residual layer, the air can speed up. The faster moving air above the slower moving air in the boundary layer causes a shear to develop between the two air masses, and sporadically, turbulence is generated when the shear breaks down, mixing air and increasing the boundary layer height. At sunrise, solar heating again begins to warm the surface, and the warm parcels rise up, and training residual layer air, until eventually the mixed layer reaches its maximum height again.


I have not watched this video yet. I saw it on Reddit, but I didn't watch. I thought that I'd save this for stream So apparently this youtuber is going to film himself getting a job at McDonald's, but in actuality He's not really working at McDonald's. He's larping as a McDonald's worker for a day or so And he's going to essentially from my understanding of the description of this video drive home the point that it's not that bad to work At McDonald's they respect their workers and also there's a little bit of a problem This video is sponsored by McDonald's UK Mmm, so a little bit of a conflict of interest. So the youtuber in question here is Only white folks. He is a British person. I don't know if that sounded more Australian I shouldn't try with the British accent, but I don't know Like if this is really a political youtuber per se seems like he's you know just a standard Regular youtuber like vlogs and shit surprising my brother on his 19th birthday. What is it? Okay?  So I don't think that like he's Explicitly political, but like it or not this video is going to be political if you're going to drive home a certain point now look I Haven't watched it yet So if it is the case that you know, he explains how his experience When it comes to this PR stunt for McDonald's, I guess, you know Whitewash their treatment of workers if he explains that that's not like representative of the average McDonald's UK worker Okay, I Guess that's better But I think we all know where this is probably gonna go right these gonna do propaganda and make it seem as if McDonald's You know, it's all hunky-dory treats as a they treat their employees well now. I don't know about the UK I'm sure it's the same there same company, but in the United States McDonald's workers are treated like complete shit They get their hours cut so they can't you know qualify for health care That's not a problem the UK obviously, but they don't have unions. They're underpaid overworked They have no benefits essentially. I think that management does have more benefits But still like McDonald's workers are exploited and they're treated like shit by management by customers So Let's see him apparently whitewash McDonald's employees treatment Welcome to the blog now. I've heard that you can apply for a job at McDonald's using Alexa You know how much I love my food and you know how much I love my tech as well So I thought why not try and apply and get a job at McDonald's Alexa. Help me get a job at McDonald's Here's the skill McDonald's apply through by paradox incorporated from McDonald's Hi, what country do you want to work in? United Kingdom Okay, just from the beginning This is an advertisement for McDonald's like it's sponsored by McDonald's, but he's like going in Right off the bat saying oh, did you know that you can apply for McDonald's using your Alexa app? Why is his watch pixelated? I didn't notice that Yeah, that is weird Okay, we're not I'm not gonna watch every second of it. Let's get to the McDonald's portion, okay He puts on the uniform. He's officially larping and first of all, I think that this doesn't get set enough but Employers should not be able to control every aspect of our lives I mean think about how much control employers have they control everything they control the clothes that you wear They control the way that you speak when I worked at subway for example I would get reprimanded by management because instead of saying hi welcome to subway I would say hi. How can I help you today? Nope, Mike?  I'm I'm greeting the customer.   So we have democracy when it comes to our political system Kind of right, but nobody's even thinking about how we should have democracy in the workplace But that's besides the point Okay, this is so much bigger than I thought it was going to be You need to put it here Oh, hand it Okay, you're going to hand it now Is that good? Does that got my hair?  Like so this is already not accurate And they do that in a very British voice, I'm assuming He's fine, he's challenging everybody Okay, this right here, just pure propaganda You have a camera in a worker's face and asking him how do you like working at McDonald's? I mean, what do you think he's going to say? Do you want him to be honest? Of course you don't What do you want him to say? Whenever the district manager would come into, you know, a subway when I worked there Or, you know, any job that I had Do you think that we would be honest and upfront?  What, like a thing? I don't know what, like a thing like this Awesome, right, I really want to make something, what should we make? What should we do? Oh, I never...which one can I make? Oh, you know what? Let's make a big one, let's make a big one Okay, this is... I'm going to be very slow I've got no stars, right? Why have you got four stars? What have you done for four stars?  Like, this is going to be broadcasted to Like how many views? Almost 600,000 views So you know, they want to make it look good So they're going to make sure that he's not overwhelmed But you know, a regular training day Probably doesn't look like this Okay, this is five minutes and 18 seconds So we cannot watch all of it Let's go ahead and jump a little bit We're going to have to try some chips, right? Yeah?  Like you already have to put on a fake happy face When you're at work already, right?  Maybe I am Maybe I'm just too cynical But if somebody came into my job When I'm trying to help customers And I have like a camera in my face I couldn't tolerate that It's hard to keep your cool sometimes Well, I shouldn't say never I mostly held my composure on customers But it's hard when you're dehumanized Like Especially if you have to interact Directly with customers Like again, I worked at Subway So like you'd be like making their sandwiches And these motherfuckers are ruthless So you'd put olives on their sandwich Like okay, can I have some green peppers You'd put green peppers They're like Like holy fuck Where did that come from?   And if you like Maybe See that reluctance? Back away, don't make them do it Like she probably wanted to be nice But you're making her do it Like, mm-hmm I don't like this So this is part of the propaganda I'm not sure if this is witting or unwitting I'm sure that they gave him Like a little bit of a script to follow Oh, talk about the skills you have I don't know I don't know I don't know I don't know I don't know I don't know I don't know I don't know I don't know I don't know I don't know I don't know Oh, talk about the skills that you learn The management skills The coordination The time management Like that's probably what he has to touch on At some point during the video Per the rules with the sponsorship Although, I don't know the criteria to be honest But like you can just see the propaganda All over Like it just reeks of propaganda And really the takeaway at the end Is what I want to What I want to see Like what does he say at the end? So, how do you think I did? Quite good I did pretty well I did pretty well How much of an a-hole was your subway manager, Mike?  Isn't this a wonderful employer? What a great experience What are you learning?  Like, how many subscribers does he have? He has 1 million subscribers So if a... Where are we at? Like, 380,000 subscribers If they pay me $300 for a really short ad This dude's getting some... Like, thousands of dollars Thousands of dollars, for sure That's the impression that you made From working as a crew And then coming up to this window Okay, the salary That is not a good thing I was on salary before as a manager When I worked at Blockbuster I'm dating myself But salary means that You are going to get Fucked When you're no longer hourly And they realize they could take advantage of you To the extreme So I went from working like You know, 35 hours a week To 40 hours a week I was full time And then jumping to salary My pay was increased by like Maybe 5% each month And I was working like 60 70 hours a week I was working... There's one week Where I almost had a nervous breakdown Where I worked two 17-hour shifts in a row Because everyone had called out And I was the only one available So it was like me 17 hours a row With like, I think one or two other employees Two days in a row So I went home... And I was commuting So I basically went home to sleep And then came right back Could have just... It really should have just like Spent the night in my car But like, when you're on salary They bleed you They treat you like a robot So her being on salary Like it sounds good, right? Because you have this reliable income You always have a certain amount And if your hours are cut Well, that doesn't affect your income Except they're going to take advantage of you For sure So sales bonuses I didn't get sales bonuses But if we did make sales quotas I would get more labor So I'd be able to give employees more hours Which helped me to do less work overall But I did get some bonuses Although it wasn't until The store closed down And I got a bonus of...  I'm good how are you? I'm very good I've had an amazing Amazing afternoon Working here In McDonald's today How do you think I got on? Pretty good I was nervous at first I was a little bit nervous Yeah But I mean With different roles I saw that anyone was confident Which is great And this is what we do Exactly And you are The people Supervisors Yes You're the one that actually hires Yes I mean I'm Supporting The clients And just teaching them Actually So do you think I could potentially work here Or develop into another role? Yes Would you hire me?  I think I did pretty well I'll see you guys next time With another video Bye I think I'll pass on the subbing What do the comments say? Hi guys Had a great time working With the fantastic team At McDonald's UK It's so easy to apply For a job there And it was really inspiring To hear from the staff About how they developed In their career By working there I hope you guys enjoyed the video And let me know What you think 18 minute ad YouTube is really pushing it Right?


Looks like both Vaughn Miller and Mark Andrews are telling the Baltimore Ravens, cut the check! So YouTube team keep it clean, what's going on? It's Aang Graven here with another video and in this video, money! Mark Andrews, who has been making his way around? I mean, the guy was just in a pro bowl doing this thing and then Super Bowl week, he has been making his rounds on different podcasts and whatnot and he was recently on the Vaughn cast which is Vaughn Miller doing this thing. And there's a clip that has been floating around where not only Vaughn Miller but Mark Andrews as well they said that Lamar Jackson deserves not the same, not around the same amount but Lamar Jackson deserves more money than Packers quarterback, well current Packers quarterback because that could change but more money than Aaron Rodgers. Let's listen. He does more than Aaron Rodgers. Oh my god, he does everything. He throws, he passes, he does everything man. And they got Aaron Rodgers is getting 50 million a year but you're speeding facts man. He's like you said like everything he says is true man. He deserves it. He deserves more man.  Every time he steps on that field he's the one man. He's the guy. There's no one like him you know so he's a very special guy and I know he's gonna get paid and I just hope that you know the deal gets done because I know the Ravens want to get the deal done and I you know I think that you know you know he wants to be Raven as well so hopefully both parties can get together to get that done. You know it is though. So before we listen to that clip I told you all the what and what the what was was that Vaughn Miller and Mark Andrews both said Lamar Jackson deserve more money than Aaron Rodgers but after listening to the clip you got to hear the why. The reasons why the breakdown, the simple breakdown too which I appreciate it on why they feel like Lamar deserves more money than AR 12 and they put it so straightforward and so simple because he does more and however you personally feel about Lamar Jackson whatever you feel like he does well whatever you feel like he doesn't do so well whatever you feel like he could improve on whatever you feel like he does a phenomenal job of already. How do you feel about Lamar Jackson? One thing that cannot be argued is that he is asked to do so much more and then when you really think it like really really think about it and you look at all 32 teams in the league and see what their quarterbacks are asked to do versus what Lamar Jackson is asked to do as the starting quarterback for the Baltimore Ravens. You could certainly make the case that he is asked to do more than pretty much every other quarterback and when you think about that like let's let's just break it down and in simple terms like they did as well he's obviously starting quarterback for the Baltimore Ravens so what do you have to do as a starting quarterback?  Now when you look at the the playmakers or the receivers or the Baltimore Ravens it is it's not a job that a lot of other quarterbacks would be envious of when it comes to Lamar Jackson they wouldn't be envious of his supporting cast so Lamar Jackson has because really when you sit there and you go year by year by year by year by year and you look at the supporting cast that the Baltimore Ravens have provided to Lamar Jackson it's like wow wow some of the names let me just think of them off the top of my head there's been Willie Snead uh there's been Seth Roberts there's been Des Bryant who was at a league for two three years there's been Sammy Watkins twice twice um there's been Hollywood Brown no that was a nice one there's been Rashad Bateman and he's still coming along I know he's dealt with injuries the past two years but he's still coming along there's been Miles Boykin there's been Tyler Wallace there's been James Proshay there's been Devin Duvenay um but when when you think about it man you look around at other supporting cast and you look versus the Ravens you're like oh okay oh that's that's what they was asking Lamar to work with and then the type of offense too Greg Roman's offense hey running offense amazing passing offense left a lot to be desired throughout the year started off hot started off amazing but well you know the rest of the story uh Ravens didn't do enough to capitalize and and build off of what they started in 2019 but there's a whole another conversation for a whole another day so again that's Lamar Jackson as the Ravens starting quarterback that's what he's asked to do and that's who he's been asked to do it with oh and I forgot Deshaun Jackson 35 year old Deshaun Jackson who end up getting signed to the Baltimore Ravens because he went and talked about them in a positive way on the podcast so that's what the Ravens have been doing now he's also asked to be Baltimore Ravens leading rusher he's asked to be their top rusher of their running attack every single year so not only do you have to be the top passer you also have to be the top rusher as well yeah that's a lot and it'd be one thing if like his rush is in all the rushing yards that he got it came off of him just scrambling or something it came off of broken down pass plays or something like that but it doesn't and not to say that it's so bad or what not because obviously Lamar Jackson he'd be making stuff happen but he's asked to make stuff happen and he does it for both the passing game and the running game so when you when your job when your job asks you hey can you take on task a okay cool i'll do it that's that's my job description i'll take care of that hey can you um can you take on task b as well can you can you take on that too normally we don't really ask people who take on task a to task to take on task b but can you do that as well you say oh uh okay sure i got you but am i going to be compensated for it are you gonna take care of me with it and then they'll be like oh well think about it we'll talk we'll talk but in the meantime take care of task a and take care of task b thanks got it bye so that's what the Baltimore Ravens have asked Lamar Jackson to do so initially when you hear somebody say or maybe not even initially but when you hear oh Von Miller or Mark Andrews they say Lamar deserves more than Aaron Rodgers some people could say what are you crazy said maybe some people that be like hey buddy i don't think so Lamar Jackson don't deserve that simply let them know hey look at what Lamar Jackson does and again like i said earlier with who he does it with take uber for example say for instance uber reached out to you and they said hey we want to hire you we will provide you with the car all you got to do pick up customers take them to their destination you're like okay cool it's a job it's uber i'm looking for work got it all right so you know some other uber drivers who are getting paid certain amounts and whatnot but they they got the Cadillacs and whatnot they got the BMWs they got the Mercedes they even got the Tesla's they got all the the fancy cars and whatnot you like all right oh that's nice i like that i see a big dog hey i i like it all right uber what you're hooking me up with and uber comes to you and they say here you know what we got you we want you to do the same job they're doing but we are going to give you a 1991 Honda Accord with no AC but we want you to do the same job they're doing we expect you to get the same results that those guys those other uber drivers are getting with that 1991 Honda Accord with no AC how would you handle that that is with what it is with Lamar Jackson that's what it is he's asked to do the same thing that a lot of these other quarterbacks are doing and and more but he's been provided less so why should he not be compensated more why should he not get top dollar why should he not get his bread i can't think of any reasons why


Hello everyone. Thank you all for attending my talk today. I'm Hima Virabi and I'm working as a software engineer here in the AI ops team and the AI center of excellence at Red Hat. In today's talk I mainly want to go over what exactly is DevOps from a data science perspective. What are some of the GitOps principles that could actually be applied. For a machine learning model development lifecycle, a quick demo on an overview of a machine learning use case and how we can integrate GitOps into it and finally leaving some time for any Q&A that you may have. With that, I also want to mention I have a couple of poll questions that I've created. And I think Hima and Eric would probably be dropping it in. So please do take some time to answer those poll questions. I would definitely love to hear what kind of audience is sort of interested in this kind of topic. And yeah, also drop any other questions that you may have. And I'll try to get them answered by the end of this presentation. So with that, let's go ahead and get started. So what exactly is DevOps? DevOps is part of the agile manifesto and one of the principles which I believe is highly relevant is stated as so. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Now in the case of machine learning examples, it would basically be continuous delivery of valuable insights from data. So how exactly is this different from your traditional DevOps approach? So when you look at your traditional software development lifecycle, we know that it comprises of a series of steps. So for example, you start off with creating your source code for your software or application. You go ahead and start building out this software or application. You go ahead and create your integration testing to make sure your application is meeting all the requirements. Finally, once you're happy with these tests, go ahead and deploy it into a production environment. And lastly, you have monitoring in place to observe the performance of this software in your production environment. And typically, all of this takes place in a couple of minutes. So meaning that this entire end to end sort of lifecycle is pretty rapid and it's pretty fast. Now, this is not entirely the case when you consider a machine learning development lifecycle, for example. So machine learning model deployments are quite different in nature as compared to your software deployments. A recent survey conducted by Algorithma actually found that 55% of companies have not even been able to deploy their machine learning models. And those which managed to have these machine learning models deployed, it was seen that very few of them around 15% managed to have it within a week. Some of them or vastly a majority of them around 50% of them managed to have it within a time frame of one week to three months. And then you had a small portion who were able to do it within more than three months and some even taking a couple of years. So clearly we can see that there is some sort of discrepancy between the fast release of software applications versus your machine learning deployment. So why is this so? Now, when you look at machine learning model deployment process, the core component is basically training your machine learning model. So when you're training this machine learning model, one of the challenges we see is that the code that you have for your model training purposes is independent from the data itself. And also the machine learning model is the resulting artifact that comes out of this model training process. So in your code of your machine learning model, it's basically where you define all the algorithms that you're looking into for training your machine learning model. So this could be deep neural networks, for instance, it could be clustering algorithms, it could be regression based models that you're looking into. And this is where you're actually defining the architecture of your model, along with configuring various parameters required to train these machine learning models. The data is of course a very important aspect for your machine learning model. You need to validate your data, you need to clean it up, and also ensure that you're able to shuffle it accordingly and extract all the relevant features that are needed for your machine learning model training. And finally, you'll have the machine learning model created, ultimately out of this. And we basically look at all its outputs, we look at the predictions of these models. We try to understand the performance, the accuracy, and we look at the key metrics for looking at the observing the performance of this model. And once you're happy with it, we go ahead and try to get it deployed into production. So as you can see, the machine learning code is only a small part of this entire solution. There are multiple components involved and acting sort of dependently throughout this process. So often, we also see various teams who are involved in this entire machine learning model development process, right. So for instance, you have your data engineers who are looking at consolidating data from various sources, making it easily available and accessible for your data scientists. You have a data scientist who are mainly concerned with extracting this data, building the machine learning model for it. And in RT, we use a tool called Jupyter Hub or Jupyter Notebooks for actually writing up the code for this machine learning model. And once we have this machine learning model available by the data scientist, they typically work with DevOps engineers or they work with other developers on the team to have this machine learning model running into your production or open shift platforms, for example. Now, having worked with data scientists myself and in fact being involved in machine learning projects myself, I have often seen that data scientists are now sort of also be responsible for managing the deployments of their models as well. So it's equally important that the data scientists are also very familiar with the DevOps principles, with the GitOps principles, so that it makes it easier for them to understand this end-to-end process. So with that comes the importance of GitOps, right. So like I said, in our team, we have multiple data scientists who are collaborating, who are working together to get the code for your machine learning models. And hence we need a central location to sort of consolidate all of these models and data for which we use GitHub or GitLab. And this basically allows you to track and modify any of the changes that you might need, similar to how your software applications need to be version controlled in Git. We also would want the machine learning models to also be version controlled and tracked inside Git. And this also allows us to have external contributors, so they can submit their PRs, somebody who wants to sort of improve the performance of the model. They can just create a pull request with the repository with their changes and we have our continuous integration pipelines in place. So this can be like a Jenkins pipeline, which is basically checking whether your code successfully goes through. And if so, it basically containerizes your entire application, creates a container image out of it and you can then push this into your query repository. And then comes the aspect of continuous deployment. So you have your application manifests again, which are residing within Git and our team uses Argo CD as the deployment tool to have these applications deployed and having them running within OpenShift platform. So essentially you see that GitOps is also highly relevant for a data science perspective. So now that we know this entire GitOps principles and how it's sort of applicable, let's look at the entire workflow of how this looks like from an end to end machine learning model deployment process. So as you can see, first of all, we have the data enthusiasts who are basically looking at the data, trying to figure out some insights from it, performing all of their analysis and creating the machine learning model. And like I said, we use something called as Jupyter Hub as the primary tool here. You'll also see this in the demo, which I have in the next couple of minutes. So once they have this sort of created and developed, they basically push all of their code into Git. And once you have that into Git is where we want to now have these containerized images pushed into Quake, for which we have what we call as tecton pipeline. So this tecton pipeline is nothing but an open source end to end framework basically, which allows you to create your CI CD systems, allowing developers to build a test and deploy these applications across various cloud platforms. And we work with our talk team, which basically is the AI DevSecOps teams within our organization, and they primarily work a lot with setting up this kind of continuous integration and continuous deployment pipelines. So we were able to get their help to have this configured for our use case. So essentially what this tecton pipeline does is for all the changes that we submit and we push it into your GitHub repository, you can associate your changes with the new tag. Just like how you have new releases and new tags for software development, we do this, follow the same process. We make all our changes associated with the tag. And when you create this new tag inside your GitHub repository, it triggers this tecton pipeline, right? So what the pipeline does is it triggers that there's a new tag being created for your repository. It creates the container for your entire machine learning application code, and it pushes this container image into Quake.  So that's the pipeline in basically in the configuration we specify, which is the Quake repository you want this image to go into, and there are certain other configurations that we would need to set up inside the repository. And once that's done, it sort of pushes it on its own. It takes a couple of minutes to run through all the checks and then makes that image available now inside Quake. And this image can now be used inside your production, which is the third and final step. So you can use this to sort of expose your machine learning model. Let's say you want to expose it as a web service. You want to sort of publish all your predictions of your machine learning model into some kind of web application. So you basically can do all of that and have that final deployment sort of customized for your use case. And like I said, in our team, we use Argo CD for sort of managing these deployment manifests. And ultimately, let's say your goal is to have some kind of dashboard or some kind of insights that you want to share about what this model is doing with your stakeholders. And you could also have that exposed as well as part of your entire pipeline. So this is basically the overall workflow. And in the demo, I mean, you want to focus particularly on step one and step two, basically seeing what is it like for a data scientist to interact with this sort of CI pipeline. So now that we know this entire workflow, you can sort of get an idea that most of the customers sort of questions or requirements from an end to end machine learning perspective is sort of being addressed over here. Starting with how do we have our code and how we have our simulation and deployments to production. And our solution to this is we're looking at a container based CI CD approach. How do we want to look at maintaining and managing all these different configurations. Our approach to that is solved with our end to end get off sort of free work that we have set up. And of course, how do we do all of our data processing. We're using all the open source AIML stacks, which are available. Like I mentioned, Jupiter, Jupiter hub or Jupiter notebooks is one of the primary open source tools we use to do all of our data running and data processing. And finally, where do we do all your machine learning model training and deployment and pushing it finally into production. And for this is where we actually use the open data hub platform, which has CI CD enabled with it. And if you had the chance to sort of attend the keynote today, you would have heard about this open data hub and this operate first sort of initiative. If not, we also had a prior talk, I believe by Michael and Tom, who sort of gave a walkthrough of what exactly is the open data hub platform. So before I sort of dive into the demo aspect of it, I will be using one component of open data hubs. So for those who may not have heard about it. Open data hub is an end to end AI machine learning platform available here at Red Hat. And it basically integrates a collection of multiple open source projects, which can be used as from a data scientist perspective, which can be used as a data engineer, which can be used from a monitoring perspective. So basically integrates all these bunch of tools. And some of the tools, like I said, our Jupiter hub, our go CD, we have monitoring tools like for me to use in the fauna. We have visualization tools like Apache super set and various other open source tools available. So if you're interested to learn more about this, I would encourage you to go to open data hub.io. And we also have a community over there available or looking for any contributions as well. And if you're simply just interested to sort of play around with this, I would definitely suggest to have a look at that. So with that, I think I'll go ahead and move on to the demo. So let me go ahead and quickly switch screens a bit. So this is the Jupiter hub instance. We have publicly available as part of our operate first initiative. So you don't need to be connected to VPN or anything of that sort it's publicly available. I'll also try to drop it in chat in case you guys are interested later on to sort of see where this is. So when you go to this URL, you see this button to sign in with open shift. So you can go ahead and click over here. We have this you basically this entire Jupiter hub instance is hosted on the mass open cloud cluster. So that's where it's currently hosted at. So this is the login for that. You can go ahead and click on Google account. And I'll go ahead and log in with my red hat. And once that's done, Jupiter hub actually throws this spawner UI option for you.  So for each of our projects, we have actually created a customized notebook image for every project and we've made them available here. And apart from that, we also have some default notebook images that you can play around with. If you're interested in playing around with TensorFlow. If you're interested to play around with spark. These are some notebook images that are supported. For this demo, I'll go ahead and just spawn a simple default. A minimal notebook. And once that's done, you go ahead and click on start. And then I'll take a few minutes to just get the PVC attached for your particular server and once that's done, you should be able to be successfully into the Jupiter hub UI. So while that's doing taking time to spin up, I want to go through this particular machine learning use case that I'm showing in my demo today. So this is the repository. I created a fork of this. This is the upstream repository within our teams organization. I've just taken a fork of it for the purpose of this demo. So what we're essentially doing in this project is we are trying to identify if we can apply AI and machine learning for identifying flaky tests or predicting any test failures that we see in our continuous integration testing pipelines. So the ultimate goal of this is basically to alleviate developers from manual time sort of debugging why exactly did a certain test fail. So we just want to understand the CI pipeline a little bit more in depth and see if we can come up with some kind of intelligent solution, allowing developers more insights into what the CI sort of tests look like. And the data set that we're actually looking at is the test grid data set. So test grid data set is publicly made available by Google cloud platform. So they've made this test grid data available that we're actually looking at, which basically gives us a combination of tests that have run on different open ship platforms or on different red hat platforms. Along with the timestamps of when these tests were run, whether the tests passed with the tests failed, or would be flaky in nature. So that's essentially what this project is trying to do. It's still in a very early sort of exploratory phase right now, but I thought it would be interesting project to look at from a data science perspective. So let's see if we can have it up. So yes, the server came up.  I've already done that over here.  So you can see I have already set up my OCP repo and I have my local for people as well. So let's go ahead and look at some of the notebooks. So as I mentioned, we're looking at the test grid data set. So I have this notebook here called test grid clustering.ipind. So that's the extension of your notebooks when you create this code within Jupyter Hub and typically it's solely based on piping. So most of the packages and everything would be dependent on piping. So in this notebook, what we're doing is basically going over the test grid data set and try to see if we could find some interesting features out of it. And like I said, we have a bunch of piping packages that we are sort of using throughout this notebook. So we start go ahead and start importing them and we define these in our FIP files as well, which reminds me to install all of these dependencies so that the notebook runs without any issues and any dependency issues. So while that's installing all of these packages, I'll just go over what the notebook is doing. So as you can see here, we have the test grid data set, right? So we have tests, each individual tests on this column on the left side. And here we have all the timestamps associated for every test and all the different runs executed for each of these tests. If the tests failed, if the tests were successful, if they were having any sort of fake nature, all of these are sort of indicated in this test grid data set. So we are basically trying to look at these different grids that we obtain from test grid. We're trying to see if there are any duplicates such grids based on different platforms where we're collecting this data from. Just simple answering simple questions like are there any subgroups within these tests, which tests do all the sort of groups share in common and just sort of looking at this data from an analytical point of view. And then having all of these sort of key metrics that we want to track like what was the test case, test coverage percentage, what was the total number of tests and so on. And how can we also compare these tests if there was any common sort of behavior among them. So that that's more of the analytical steps that we're doing within these notebook. And we're also trying to sort of plot it in a more visualized appealing manner. And, and the main part is basically what are the features that we want to look at right. So, for example, I want to look at what is my test pass rate. I want to look at how many failures we've been having. What is the sort of trend that you see in your failures. Is there like a failure streak commonly observed for your tests. So what is the frequency of these kind of tests. So these are some sort of features that we came up with or key metrics that we thought would be relevant for our analysis of these tests. And hence we consolidate them into this neat data frame, which has the six key features that we want to further build our machine learning model on top of. So now that we have these features ready. And what exactly we want to sort of observe, we can now visualize visualize these data points that you have. So, like I said here we have like six dimensions but of course if you're looking at a 2D plot, you need to sort of cut down your dimension so that it's suitable so from a six dimensions we go into two dimensions. So we have this Python function or this basically a technique called the PCA technique. Don't want to go into too much technical details, but it's basically a method where you can cut down these dimensions so you basically say that I want to cut it down to two dimensions. And then you go ahead and plot your data points accordingly. So the purpose of this plot was to actually sort of see if, as you can see these kind of points are scattered right so you can see that some seem to be sort of clustered in one group, some seem to be outliers, and so on. So this sort of was a motivation for us to go ahead and train a clustering model that so it gives us labels on what are the different type of clusters based on the features we've defined and how each of these individual data points get grouped into these clusters. So for that we have this clustering model or DB scan. So we've basically invoked to train this DB scan clustering model and then finally give us the output of this clustering model. So it's basically just an array. So for every data points it's telling us which cluster does it belong to and so on. And finally, we want to save all of this cluster model. So you can store it in like a pickle file you can store it in a job lift file, depending upon your use case. And once we do that, I'll go ahead and run all of these cells above. So yeah, once you do that, you can basically load this stored pickle file or job lift file, whatever it is, and we can check to see if it's the same. Yes, we see that it's resulted in the same as what we've seen over here. So now let me go ahead and just change this name just for the purpose of the demo. So I'm just going to load it again. You can do it here as well. So go ahead, store it with a different name, and then load this as a different name. So now you should be able to see this new pickled model file available. So now I want to actually push these changes, right? So you can see that there's this new model file available. I do not want to modify my notebook, so I'm going to ignore that particular change that happened. And I just want to add this new pickle file. I'm going to see this is my updated trained model. And now you should see my latest comment over here on top. And now I'm just going to push this to my branch. I think there's all those. All right. So this basically is pushing all my changes into my repository. And if we go here, just do a quick refresh, you should see this new commit being pushed. We should see our new train model over here. This test because updated train model, right? So awesome. Now how do you want to trigger your text on pipeline, right? So like I said, there are these tags associated that can be configured in your repository. If we go to the repository here, you would see a bunch of tags. So let's create a new tag version <IP_ADDRESS>. Now I'm going to push this with this new tag. So when you go to your repository, do a quick refresh. I see this new tag has been created in my repository. Now the most important thing for setting up the CI pipeline is we have this YAML file where you're actually setting up this entire CI pipeline, which the top team were happy enough to tell us how to do it. So they have also documentation as well on how you can set this up. So the main important things that you need to provide here is basically details on where you want to push this image, right? Of your container image of your application. So we are using Quay in our case. I am using my own personal account in Quay. I have a project created within Quay where in which I want to sort of push all my images too. So this is what the Quay image registry would look like. I have this CI analysis project created for my project and these are the various tags associated with it, right? So for every tag that happens, it basically gets pushed and the image once the pipeline runs successfully, it gets pushed into this repository over here. And we also have a tecton dashboard. So like I said, all of these are being triggered with the tecton pipeline. Since you saw that I pushed it as a new image, as a new tag, it got triggered and this is the latest pipeline run that's been running with the tag release run. And if we go into more detail, you'll get to see what are all the tasks associated, what's the status of these tasks and the steps within these tasks. And it's not necessarily that all the tasks would pass, some of them would also get skipped. So like you can see here, four of them were skipped and they were not relevant for our particular tag release. And finally, this is the last step that's currently running. And once it's successful, so basically this is the final part where it's actually pushing it into Quay. So once this is completely green and once all of these steps have completed, we should eventually see this new image tag being created in our repository here. So those are pretty much the steps that we need to do to configure the CRI pipeline. And this is the repository the thought team uses and they have a great documentation on how this entire architecture is set up for them and how you can use it for your purposes. And that's about it. I think in terms of my demo, while this is sort of taking few more seconds, I think few more minutes to just run. I just want to go back and conclude by sort of saying that there are many other tools out there which sort of give the same kind of results. It doesn't have to be tacked on. It doesn't have to be, it doesn't have to be any of these sort of tools that we have used in our team. But there are also some great tools that we also want to explore more in the future, such as Cubeflow Pipelines or Ellyra, which is basically nothing but a JupyterLab extension. And like how you saw in the Jupyter Hub, the Ellyra basically allows you to create like a drag and drop pipeline for your entire machine learning model development and you can allocate resources for every single step in your machine learning deployment process. So this is something new that we want to definitely try out and sort of integrate, making it easier for data scientists also to get hands on into setting up these sort of pipelines and workflows for their use cases. So with that, thank you so much. If you have any questions, feel free to have them answered now. I think I'll have a quick look so that I can stop sharing. Okay, one question I please. How many ML containers are typically generated by the pipeline? One with, I think I lost that question. There we go. One with all the ML or is it broken out into different model containers. So in this certain use case, it's just one machine learning container that's being sort of created by the pipeline. But you could also definitely break it down as you know you have a project which has multiple sort of models that you're trying to train accordingly. I would definitely say it's left to sort of your use case and your customization that you can do. In this example, it's definitely just one simple container that we are looking at. We also have another question, William Henry. How many ML containers are typically generated by the pipeline? One with all the ML or is it broken out into different model containers? Yes, yes, I just, I think I just answered that question. I think the other question was how do you handle data versioning and hyperparameter tuning in your architecture using Kubeflow as part of it in some other way. So yes, so hyperparameter tuning is definitely something we have right now not really invested in a certain tool per se. In the past we've used this open source tool called MLflow. So MLflow was pretty useful for us where you can sort of, it gives you like a UI kind of interface where you can actually plug and drop your hyperparameters and tune your model. But like I mentioned, we now have these Kubeflow pipelines and we now have the project Ilyra, which I think would definitely be a good tool to sort of invest for this kind of hyperparameter tuning use case. But yeah, as of now, we are not really looking too much into that.


Good evening everyone. I am Dr. Pooja, JR2 from Naira Medical College and Hospital, Sassaram. Today I am going to present a case on chondromixoid fibromas. Introduction. Chondromixoid fibromas are extremely rare, benign, cartilaginous neoplasm that account for less than 1% of all bone tumors, most commonly diagnosed before 30 years of age, mostly during second and third decades. There is no sex predilection in case of CMF. Now coming to my case, an eight-year-old male came with the complaint of pain and swelling in later aspect of distal one-third of right leg which was present for five months and was progressive in nature. Exid was performed and these are the radiological findings, multi-loculated eucent lesion which was eccentrically located in the meta-diaphysis of distal left tibia having narrow zone of transition and thin slerotic rim was noted. This was the lesion having narrow zone of transition and thin slerotic rim was noted. Next CT was performed. Here we can see moderately sized lesion of size approximately 3 into 1.5 centimeter which is multi-loculated radiolucent lesion is eccentrically located in the meta-diaphysis of the distal left tibia having narrow zone of transition and thin slerotic rim likely benign neoplasm. Next MRI was performed and a cortical based lesion of size 3 into 1.5 into 4 centimeter was noted with erosion of lateral cortex involving the distal tibial metaphysis and diaphysis not involving the growth plate which appeared hypo intense on T1. The lesion appeared hyper intense on T2 with her hypo intense rim this was the hypo intense rim which correspond to slerotic border and this represent the benign nature of the lesion. There is no surrounding edema hemorrhage necrosis or fluid level noted within the lesion and no significant extra oscis soft tissue involvement was seen. Next the lesion appeared hyper intense on T2 and ster. Post-contrast study showed heterogeneous enhancement in the lesion. Now coming to differential diagnosis the diagnosis was made benign neoplastic lesion of the bone and most likely differential diagnosis diagnosis made was chondromixoid fibroma and non ossifying fibroma. Then USG guided FNAC was performed on the patient and the smear confirmed it as chondromixoid fibroma this is the FNAC report of the patient. Coming to discussion most chondromixoid fibromas are located in the metafysal region of the long bones approximately in 60% of cases and may extend to the epifysal line. The classical site is the upper one-third of the tibia which accounts for 25% of all cases with the small tubular bone of the foot the distal femur and the pelvis or sacrum being other relatively common locations. These are the typical sites and these are less typical sites. Radiological features of chondromixoid fibroma mostly tibia is involved in 50% of cases and the lesion is metafysal it is a geographical reason which is eccentric oval or rod it causes endosteal scalloping it is expensile have soap bubble appearance and rarely calcification can be seen in matrix in few cases. Now coming to MR findings on T1 weighted image it is iso intense to skeletal muscle and internal hyper intense pokai can be seen in 37% of cases on T2 weighted imaging two patterns can be seen first may have peripheral band of intermediate signal with central hyper intensity second diffusely and homogeneously hyper intense on T1 weighted contrast there is peripheral nodular enhancement which correspond to peripheral band of intermediate signal on T2 MR another contrast enhancement can be seen in the form of diffuse enhancement. Now coming to DD of chondromixoid fibroma it includes aneurysmal bone cyst giant cell tumor. Giant cell tumor is well defined and have non-cellerotic margin in most cases non ossifying fibroma and chondroblastoma which occurs in younger age group and metaphyseal fibrous defect. This is the case of aneurysmal bone cyst in a patient which is sharply defined leucine bone lesion with thin wall cavities and on MR we can see fluid fluid level which is characteristic of aneurysmal bone cyst. Next DD is giant cell tumor this is the APN lateral x-ray of the knee which demonstrate leucine lesion located eccentrically within the distal lateral femur abelting the articulate surface and later it was confirmed as giant cell tumor pathologically also on MR a well-defined lesion in the distal metaphysis and was seen which extent which was which extend to the epiphysis of the right femur and was hypo intense on T1 and hyper intense on T2 and ster and was and show enhancement on post-contrast study there was small cystic lesion small cystic lesion within the lesion which was also demonstrated on T2 weighted images. Next DD is non ossifying fibroma this is the ankle x-ray which demonstrate a well circumscribed eccentrically sense in the diaphysis or metaphysis of the tibia the margins are slightly slerotic there was no periosteal reaction and it doesn't about the growth plate it was a case of NUF. MRI demonstrate an eccentric well-defined lesion composed of soft tissue which was a hypo intense on T1 and T2 both but shows a post-contrast enhancement it was a case of non ossifying fibroma. Now coming to summary condom x-ray fibroma is one among the many benign lytic lesions of the bone potential sites are proximal tibia and foot although the name suggests that it is a conduit lesion but calcification are usually not seen it resembles non ossifying fibroma. Now summarizing all thing well-defined osteolytic bone tumors and tumor like lesions have a plethora of differentials in different age groups. For simplicity a wide widely used mnemonic for lytic bone lesion is fog machines which include fibrous dysplasia, fibrocortical defect, osteoblastoma, giant cell tumour or geodes, metastasis myeloma and urizomal bone cysts, condom x-ray fibroma, condoblastoma, bronze tumor in case of hyperparathyroidism, osteomyelitis, infarction, non ossifying fibroma in condoma, eosinophilic granuloma and simple bone cyst. Now we can differentiate these lesions on the basis of location and age. In less than 30 years in less than 30 years of patient we can consider the lesion as condoblastoma or infection if it is located in epifysal region. If it is located in metafysal region we can consider condom x-ray fibroma in condoma, abc, simple bone cyst, non ossifying fibroma and if it is located in diaphysis we can consider fibrous dysplasia eosinophilic granuloma and if the patient is above 30 years of age and the lesion is in epifysal region then we can consider giant cell tumor, infexin and geode and if it is in the metaphysis we can consider in condoma, condosarcoma and if the lesion is in diaphysis we can consider it as metastasis myeloma or lymphoma. Thank you.


This research examined the use of silver nanoparticles stabilized by exopolysaccharides for the degradation of two common textile dyes, methyl orange and congo red. The results showed that these particles are effective at facilitating the degradation of both dyes, indicating their potential as a green and cost-effective method for treating contaminated water. This article was authored by Chinoshan Magam Saravanan, Regendron Rajesh, Thanum Gam Kaviersan, and others.