I don't drink milk always, because I drink, you know, the almond milk. Yeah, that's good, better. I don't like the milk always, you know, even with the protein shakes. Yeah. I mean, like, all I'm saying is, like, milk is suffering, and it's not halal. Like, I mean, yeah, yeah, yeah, like, in the egg industry, is the egg industry is always suffering, because these egg laying hens, they're suffering in these farms, in these barns, you never see it, you never see it with your eyes, and then they go get slaughtered. I don't think there's a dozen care about feelings or anything. They just sell them, that's it. Get money, that's it. If you think of your religion as trying its best to make the animal not suffer, you think of your religion, right?  This is what they're trying to do. It doesn't exist in this reality. It's not during, but in the, you know. They're trying to find the best way to take the animal's life, because it's about compassion, okay? Then what's more compassionate about it than just not leaving the animal as it is? And then? Leaving him in the sanctuary, leave him and be.  What's more compassionate than that? What for? Well, why would you care if an animal suffers? I don't want him to suffer, I want him. So why would you care? I want to eat him, that's it.   We can get tasty chicken vegan, you know? It's made of soybeans, it's made of plants. In Mac. Yeah, look. Vegan chicken? You can get... You're eating animal's food, yeah? How was that? You're eating animal's food, yeah? No, no, no, why, why? Because I eat plants? Yeah. I'd rather eat the plants than the animal. Vegan chicken, yeah. Yeah. So are you against animal cruelty? No, of course I am. Okay, then why are you against animal cruelty? You were against animal cruelty? Yeah, but we eat. You eat the animals, so... In the halal way, we're probably... Yeah, we just talked about halal. Yeah, we talked about halal.  You see, you guys electrocute them and stuff. So where do you live? We're from Boreng. Where are you eating here? Where are you eating here? Everywhere. Halal Nando's. Halal Nando's? Yeah, you also have meat liquor over here. They have halal chicken. Halal chicken, okay. So have you seen halal slaughterhouses here in the UK? No. So how do you know there's sauce that's being slaughtered with that pain? Well... You're assuming? I'm saying it's halal, so we eat. Yeah. It's their fault if it's not. They're negligent then. Well, wouldn't you want to know? Of course. So you can't guarantee it to me? If I can't guarantee it and I can prove it, I could... It could cost them millions because they claim it's halal. They're saying it's halal? But they're... Yeah, but you don't even know what halal guidelines are here. And... Halal, halal, halal, halal. They're raised in farms. It's a misrepresentation of halal. These animals are raised in big factory farms. Halal, halal chickens. You say there's no suffering in a factory farm? Of course there's suffering in there. That's how they make a living. And they cram them into trucks and they send them to a halal slaughterhouse to be hung upside down, have their throat slashed open and bleed out.  They don't feel any of that.  They get electrically bathed or they... Yeah, we don't do that. We don't do that. That's not halal. Do you think they stun halal animals here? They stun halal animals here. We don't stun animals here. No stunning... There is. In the UK, they do stun... What do you think they do in the UK? No, because if you're eating halal lamb here, they stun three quarters of those lambs. There's only one quarter of the lambs that aren't stunned here. They have a different halal guidelines here in the UK. Well, it's their problem, you know. It's their problem because halal is a guide lamb. But essentially, I can just get a bit of human flesh, put halal stamp on there. You'll trust me and then you're eating a human too. It's your problem. It's not ours. It's their problem and it's illegal. Halal guidelines come from the bonds. It doesn't come from the country itself. It's illegal to do that. It's our religion that says that. Yeah, but I think when you think of halal, which is what we just discussed before, like I'm not a Muslim, but I looked into it a little bit because I'm an animal rights activist. I believe that animals deserve rights. So I've seen orthodox... And I literally saw a video of a guy putting his hands up a cow.  He's artificially inseminating the cow for dairy.  Yeah, we don't do that. You don't eat any dairy. No cheese? No, of course we eat dairy. We don't do that. When you pay someone for dairy, you're funding this industry, so you have some responsibility here. These industries don't exist without you guys paying for it.  If the calf's a boy, they kill the calf, they shoot the calf on the farm. I don't know if that's halal. They shoot the calf on the farm and then... So how about to shoot... Well, okay, well if you're...  What I'm saying is, if you're a Muslim, you can be a Muslim and a vegan. God doesn't say... You can. Yes, you can. You can. You can. It doesn't say in the Quran, you must eat animals. Yeah, but... It's not haram to be vegan.  Nice stuff. Peace, my brother. Take care, take care. Did you take this information? You don't have to eat meat to be a Muslim. You don't have to. It's not mandated. Did you know what I say? I am not. It's not mandatory for you. Some people prefer not to eat meat because they have cholesterol issues. Okay, well here's a good one. If meat contains cholesterol and saturated fat, it gives us heart disease, the number one killer of human beings. Why would God... Why would God, Allah, so Almighty Intelligent God, create animals to eat, for humans to eat, that suffer, feel pain and also contain cholesterol, saturated fat, which gives human beings the number one killer? Well, don't plant human... No, no, no, I'm asking you a question. We can go to plants if you think they feel pain. I think that's ridiculous. Yeah, yeah, okay, go ahead. Use your smile in because you think a carrot feels the same as a lamb. It's ridiculous. That's why there's no halal slaughter of carrots, is there? Yeah. Is there halal slaughter of carrots? No, no, no, you're going somewhere else. Okay, well let's stick here, stick here. Heart disease, number one killer of human beings on Earth. It is, indeed. Okay. You do not exercise if you don't do anything. No, no, no, you don't. You can be exercising and eating bacon and get heart disease. Our ancestors and your ancestors ate plants. Yeah, of course they did. Just plants. They had a very high-fibre diet.  They didn't just eat plants. But they also had a lifespan that was very low. They didn't have medicine and technology and they were dying really young. So I don't know if you want to do what the ancestors did. But if God created animals with products for you to eat, right, why is heart disease a number one killer of humans? I can, you know, I'm going to take you 180 degrees away from this. God also created Adam and Eve, not Adam and Steve, but that's okay. But this is... Isn't it okay to eat meat? I'm not asking you... No, it's not, it's not. It's completely illogical.  I mean...  You just have to add cholesterol to your diet. It raises your blood serum cholesterol and clogs your arteries and kills us. It's the number one killer of human beings. Yeah, because it's all artificial now, isn't it? Yeah, and all I'm saying, right? All hormones, added sugar... No, no, no, it's got nothing to do with that. Get grass-fed organic states. It's got cholesterol in it. Even fruits have preservatives on them now. It's got nothing to do with... Heart disease doesn't have anything to do with preservatives. It's cholesterol with saturated fat. Cancer is diseases. Cholesterol and... Cancer doesn't kill as many people as heart disease does in the world. Heart disease is the number one killer. Heart disease is the number one killer of human beings. I am with you. I am with you. Lots of people have it in their genes. So why the hell would animals...  Most of those people are not educated. They don't move. They don't exercise. They don't do anything. Exercise is nothing to do with it. You just feel it at ten times. But my friend, I'm telling you, exercise has got nothing to do with your arteries clogging.  He nearly died of a heart attack. He was 50 and he was fit as anything. Does your arteries can clog whether you move your body or not? Yeah, but do you see their diet? Where did they eat those bodybuilders? He's a healthy guy. Yeah, what did he eat? Healthy, but he was eating steak. He was eating bacon.  I don't know. I don't know, just probably a normal standard diet where you're eating... Their diet is not standard. He's built for their... He's bodybuilders. He wasn't a bodybuilder. He wasn't a bodybuilder. He wasn't a bodybuilder. He was a fitness instructor. Yeah, that was great. We'll flash him up on the... Six meals a day and... They eat a lot. All of them contain meats. So are you saying all these people are dying of heart disease are eating six meals of meat a day? No, they're not all. I'm not saying all. I'm saying there's definitely other lifestyle factors, but the biggest factor is your blood cholesterol being raised by satiric fat and dietary cholesterol, which are in animal products. So why not promote, let's say, heart disease instead of... Well, I promote animal rights, OK? I care about... I'm just asking you for logic, right? Now, you can look outside of your religion for logic. OK, now, humans are... If you look in your religion, it's against animal cruelty. I mean, if you feed, if you even just give water to a dog, that kid, you know, there's something about heaven and doing that. OK, you're doing good for animals, OK? Now, your religion is trying its best to make sure animals don't suffer. OK, slash a really, really sharp knife. Don't let them get scared. OK, why would we mass breed and put them in sheds and kill them en masse just so you can have a five-minute meal if your religion cares about... Businesses. No, I want to talk about you. I want to talk about you. In our countries, we don't farm them and buses them. Where do you live? I live in Bahrain. So, you live there now? So, you're on holiday, basically? Where? You're on holiday here? No, I'm studying here. You're studying here? So, right now, in this country here, you're buying the meat and dairy from these farms that I'm showing?  Yeah, yeah, yeah. I never knew this. Yeah. For example, fat fish, I mean fish fat. Omega-3s, but you can get Omega-3s from the same place the fish get it from. Where? The algae. And they make algae-based Omega-3 supplements. So, you don't have to murder the fish. You don't take in... I'm not a fan of greens. What do you mean? Like, not a fan of green food in general. Well, you don't have to necessarily eat green food. I recommend eating some green food or you're going to be unhealthy, but... Yeah, I know. I eat, like, a lot of bread, a lot of beans, a lot of legumes, a lot of mock meats. It's full of carbs. Carbs aren't bad. Your brain runs on... Starch turns into fat. Same thing. Yes. If you eat too much, too many calories, you'll gain fat, but I think carbohydrates have four calories per gramme and fat has nine calories per gramme and fat's already fat, so it's already going to store as fat. But carbs don't fill you up. They just last less than a year. Well, fat, no, no, I think whole food carbohydrates, like beans and legumes and whole grains are really satiating. I'd say if you go into Subway, right, you can get a vegan sub with meatless meatballs, right?  These are all the swap-outs, you know? There's ways you can eat if you want to be unhealthy, if you want to have a pizza with your friends that don't involve exploiting and killing animals. That's all I'm trying to...  Unless you've slaughtered the animal yourself. Even then, I'd still say, even if you slaughtered the animal yourself, it's completely needless. You don't need to. You don't need to cut an animal's head off to sustain yourself now. We've got technology, we've got access to food and water. True. You know? The Quran was written, I don't know when the Quran was written, a long time ago, right? Now we have access to food. There's no survival. You don't have to live off the herd of lamb. You know, obviously they thought killing animals was wrong. Otherwise, they wouldn't be trying to find this amazing way of doing it where they're not scared and not suffering and all these things. So they think your religion even says that animals have enough moral value not to get them scared and make them suffer or cause them pain. Yeah. So they think they have moral value. So why not just avoid eating them?  He's responsible for stabbing you but he wouldn't have stabbed you unless I paid him. Okay?  We're both responsible. I'm responsible for paying him to stab you and he's responsible for stabbing.  Because without you guys, without all of us to say the whole of Leeds stopped buying chicken, what are they gonna do with the chicken farms here? So if you told him to stab me. No, I paid him. Yeah. Like I paid him to stab me. As a... So he could steal my shoe and sell it. So if I go to the supermarket and buy that shoe, I've supported you. He's not the shoe. Yeah, I bought the shoe. No, no, no. I've just paid him to stab you so I can eat you. Oh, he could eat me. Oh yeah. Well, that's what you do onto a chicken. You pay someone to stab him so you can eat him. I do agree with you to stab him. Like if you, like look, you didn't, let's just say, let's just use a horrible example here.  I'm aware, I'm aware. I'm educated a little bit. Like I'm not saying I know the Quran through and through, but I'm saying that I know that you can be vegan as a Muslim. I know that, you know, there's this principle of animal cruelty somewhere in there that you believe. We're against it. Yeah, you're against animal cruelty. And the Quranic... And against killing, as well. Against killing. And any way of... Where is the most suffering and killing happening on Earth? To what? To animals or something? Animals and human. It's animals. Yeah. One to three trillion marine animals and something like 74 billion land animals are being executed every single year. And in the Middle East. In the entire world. Okay. But where, mostly in the world? The entire world. Like, so it depends on how big the country is, obviously. It's just by population. But in the entire world, it's one to three trillion marine animals and about 74 billion land animals. Yes, see, for example, I'm against killing endangered species. Why? Because... Because you care about their species. Yeah, okay. I don't care about their species. I care about the individual. The individual, yeah. The individual being. Okay, yeah. It's like I don't care about a race of people. I care about the individuals within that race. I care about all human beings and the individuals. They all deserve the same rights. Yeah, I am. So when you see, like, a rhinoceros and a pig, I think they both have equal moral value, even if the rhinoceros is endangered. Because I could be like... But why don't you focus more on that? Because, you know... We could. Your resources are scarce, you know. Species, extension, animal agriculture, they're both linked.  But you... I do. I reach 7.5 million people a month on my Facebook. Yeah, yeah, yeah. But that's just me alone. There's a huge movement of us. I understand, but is it possible?  People already care about endangered species. I don't have to convince you shooting an elephant is wrong. Do you care about a chicken? In some countries, yeah, I do care about a chicken. But you turn them into poo and put them in the toilet? See, I have birds. I have a cockatoo and a macaw. Yeah, so they're no... I care about birds. You care about birds. Yeah, another species. But when you eat a chicken, you're not caring about that chicken. Well, it's kind of dead, so how can I really care about that? You paid for them to die? I didn't pay for it to die. I just bought it from the supermarket. Yeah, but you know what I'm saying. Supply and demand chain.   I've been doing this for a long time, my brother. Like, this argument here with clothes and... Do you buy your clothes from...? Well, even against, like, an Islam, although most people don't follow this rule, but we don't wear animal leather or animal fur. Yeah, yeah, that's good. Well, I guess that's... Would you equate making fabric to executing trillions of animals here? Would you say that's morally on par, the same thing? No.  To make a car. Just to make some clothes.  Yeah.  I've been, you know, ivory trading. So, we don't wear fur, wool, leather, down, silk, anything made from an animal. The point is, the point we had a good conversation, I enjoyed the conversation with you guys. Like, look, I think you're good people. Like, I don't think you're bad people. We respect your opinion. Yeah, your opinion, yeah. I mean, look... And I do agree with you on some things. Yeah, like, look, it's pretty basic. Look, animals are being tortured and killed. They don't need to be. You can eat plants and you can be healthy and thrive, you know, and we don't have to mass-holocaust trillions of sentient beings for a five-minute sandwich that you'll forget about tomorrow. You know? I think it's a more compassionate way to live. So, obviously, you know, it was Hamilton, yeah? Yeah. So, his dog, Roscoff, you know... I don't know his dog, but I know who he is. Yeah, he said his dog was vegan now. Yeah, yeah. Dogs can be plant-based. If they formulate the diet correctly. If they think a dog consented or, you know, is that his vegan? It doesn't matter. As long as a dog has a good well-being... You choose. Well, let me put this to you, then, because this is a good thought experiment here.  Would you be okay with that? No. Why not? You're okay with feeding other animals to a dog? Chickens, pigs, cows, fish are all fed to dogs and cats. Is that okay? I'd say no. Okay, so... Dogs don't exist in the wild. They're domesticated animals. They're used to. Oh, yeah, but I didn't care what happened a million years ago. I care what's happening now. They're domesticated animals, so we should feed them ethically. When they're in your moral... When they're...  So if a dog can be healthy as a plant-based, you know, dogs are omnivores. They eat plants and meat. So if you can formulate plants to contain all the nutrients that meat contains, you can take the fiber out, you can add more protein, you can add supplements in there. Why not? Dog has good well-being. Vegan dogs can live long. So you know why I'm smiling? Why? I don't want to say that you've convinced me on some points, but I do agree with you on some points. I think about this. When people say, oh, cats... Oh, my God, the guy was feeding his cat vegan, right? Yeah.  I got that point. Speciesism is cool. Yeah, you're right on that point. So you know like racism, obviously, is like discriminating between races. Speciesism is discriminating between cats, pigs, dogs, fish, rhinoceros. As you said, endangered species, that's speciesism as well. I care about the tiger, I don't care about the pig, or, you know, most people don't. If there is just one last white grain or last... We do not focus on that. This is coming from, like, I know what you're saying. Like, it's something sentiment... Okay, we'll have... I'm the last of my particular race, okay? And then there's you, let's just say there's a lot more of your race. Yeah. Now, me and you are standing in the room. Who's got more moral value? Who should we execute? It's obviously... Should we just use it based on the fact that I'm the last of my race, that I get to survive or... No, but, I mean... You know what I'm saying? Like, it doesn't carry moral weight. Yeah, I get it. Yeah. But we have to... You have to go, you have to... You know what, mate? Here you go. This has got some information on there. Challenge 22. Nice talking to you. Nice talking to you. Thank you. And just please... We'll blow your faces. What we had here was a productive conversation, and this can help so many people. You know how many Muslims follow me now and, like, we have these good conversations and I think they're important to have. Yes. Especially when it comes to... This is an important issue. Like, no one really talks about the animal rights issue. We're talking about politics and all this other stuff at the moment. The animals too often get forgotten, you know what I mean? And they don't have their voice. They can't talk like us. They're even some people from some countries. Yeah, exactly. There's some horrible things happening to humans as well.  You know? Hmm. I'm going to pack up now. Have a good one. Nice meeting you guys. Take care. OK, so we're packing up the screens now. We had a good day, good conversations. I'm going to leave a link for stands for change guide down below. So if anyone wants to do something like this, look, I don't need a group of people to do this. I only need me in the stands and that's it. So if anyone wants to do it, I'll leave a little guide and you can find out what batteries they use, what TVs they use, stands. You can get these signs printed. I've got all the templates down below. The brochures that I hand out to people, they'll all be in the same link. So check it out. You can start a little stands for change in your area. Get these stands, get a table, hand out some food if you want. Do it all in your style. It's not an organisation. It's just a little guide to help you do this and I think it's really effective. So happy activism.


Hi. Thank you everyone for coming. My name is Mike Tujeron. I work at Lithium Technologies. And I'm Ilan Rabinovich. I'm the director community at Datadog. So we're going to talk a little bit about monitoring OpenSack at Lithium with Datadog. You can kind of go through that. So we use OpenSack in a variety of ways at Lithium.  We're a SaaS, so a lot of communities for Fortune 500 and enterprise companies. The service is running on OpenSack. We got Redis, Java, MySQL, Node, Elastic Search Katandra, and a whole bunch of other things in smaller usage. We run a lot of our infrastructure services on there, Kubernetes. On there with Docker, our Chef, DNS. We run Console and other kind of infrastructure services. All of our development environments are running on there. A lot of teams use it for R&D. It's OpenSack. It's a cloud. So random stuff gets to run on there, which is kind of nice for the developers. But it kind of makes monitoring a bit of a challenge. So a little bit about what our stack looks like. We run in two regions. In the EU, we're running Kilo. In the U.S., we're running Ice House. Kind of a vanilla installation under the newer services, mostly separate from Ice House just because Kilo was a new introduction for us. We have about 60 hypervisors. Probably 80% of those are in the U.S. We're running about 1,000 instances across the two different clusters. 10 terabytes of RAM. We're using Contrail for our SDN, which is really nice. Does add some monitoring challenges that we'll talk about later. Within Cinder, we're using both SF and SolidFire for provisioned IOPS. This is all managed primarily by a team of three engineers. There's five of us on the team, but about half of us are dedicated to managing this OpenSack cluster. Monitoring. So we have tools out of the box. You have your horizon. I'm assuming everybody's seeing the horizon dashboard. Not very useful. It's quick snapshot, sure, but you don't get anything cross-tenant easily available. We're multi-region, even worse, because then you got to look at the tenants across the regions and so on. And it's a single view. Each team, all those different teams, want to look at OpenSack, but horizon just gives you that one view. It also doesn't give you time-based metrics. Graphs over time are super important to us. You also have the Nuva APIs. You also have Keystone, Cinder, etc., all the Python APIs, really. And you can roll your own monitoring. But I mean, I'm sure you're all busy enough. The rest of your job, rather than writing it, you don't want to write your own monitoring solution if there's stuff already out there out of the box that you can use. But we have used these. We've used both of those before we switched the Datadog integration.  And then we'd have to visualize it somewhere else, yeah. So the way I look at it is, you know, a lot of this graph here, this is from what, the 60s? Yeah, it's a telescope from NASA. Revolutionary at the time. You look at it now and you go, what the heck is that? And that's the way I kind of look at rolling your own, especially making API calls on itself. It's just, it just doesn't work well and it doesn't scale. As you add in multiple stacks, multiple regions, new tenants, new whatever, it's complicated. So we went with Datadog. We were already using Datadog for our application metrics. So on all of our VMs, you know, we're running, I think we have 2,500 servers of some form, some flavor, public cloud, private cloud, bare metal, VMs on bare metal in Datadog already. So being able to pull in our OpenSec metrics was really powerful. But one of the most important things up there are those top two bullet points. Users want graphs and dashboards. Nobody wants to see raw numbers. They want it to look nice and pretty. How tall, comparisons and time series. It is way more important to see how I'm doing today compared to last week for some teams. From an admin perspective, perhaps not so much, but from a finance perspective, maybe you do care what's our rate of growth of utilization of our resources so that we can do budget planning for the future. We also want Datadog easy to implement, just quick, yum or apt install. Real super easy to configure with all the standard provisioning stuff, Ansible, puppet chef, salt, you name it. And the open source aspect of it's really important to us. We worked very closely with Datadog when they first were working on their OpenSec integration. And they were taking one approach to look at the data. And we needed another approach to look at it too. We needed more than what they were looking at because of our custom use cases. And so we were able to take their code and put in PRs to get to it. And we were also able to basically have a second integration that basically extended the first one that they wrote that gave us all of the metrics that we needed. So it was really nice to be able to customize the monitoring solution but still push it all back up into their SAS solution up in the web to get those dashboards and graphs. Yeah. So to give a quick overview of what Datadog is and what we do, we're a SAS-based infrastructure and application monitoring solution, as Mike was saying. The agent is open source. So that's all the bits that run in your open stack environments or on your VMs written in Python. You're always happy to accept contributions but also feedback on how we can improve it. But that's all up there, you know, BSD licensed in GitHub. But really what we look at is time series data. So that's metrics and events. And we'll talk about the difference between the two in a little bit. We were looking at about a trillion data points a day at this point across our customer base. So we're working at quite large scale with folks that are running open stack, folks that are running in the public cloud on various other solutions. We like to help you build insightful dashboards, not just graphs that you kind of scratch your head about what's going on, help you democratize that so folks across your organization can build their own dashboards, and then bring in things like intelligent alerting. So as your environments are changing constantly, you're in the cloud, whether you're doing auto-scaling or some other programmatic dynamic infrastructure, you can't really stick with threshold-based alerting. And so we bring in things like outlier detection and soon more algorithmic style approaches to alerting into dashboarding that will help you there. I'll also mention that we're hiring. If you want to work on integrating with various open source technologies, stop by the booth. We'd love to chat with you. So we're talking about, we're here talking about monitoring in open stack, but it's good to take a second and just talk about why monitoring is important. You wouldn't drive with your headlights off in the rain. Why would you deploy anything in production without monitoring? We actually would tell you to go, you might have heard of test-driven development. I would suggest that you take it to the next level and consider monitoring-driven development. Start writing those monitors in your staging environments or in your development environments, even on your local machines. Not because I want you to send us more metrics, but because when you get to production, it's really important to know what normal is. Collecting that data is going to be cheap, whether it's in your development environments or in your production environments. I mean, we're here at a cloud conference. We know that we've commoditized a lot of the things that we interact with in terms of storage and compute. So yes, it's a lot of metrics, but you're going to find a place to keep it.  The thing is though, not having that data when you need it can be really expensive. How many of you guys run postmortems in your organizations when you have incidents? It's cool. So most of the folks in the audience, and how many of you have done a postmortem where the only thing you can say to your boss is, I don't know what happened, but I promise I'll add monitoring for the next time. People are sort of shy raising their hands on that one, but I guarantee you've all done it, and that's really what it comes down to.  Hopefully you'll be able to get ahead of it and have a proactive response rather than a reactive response, but you're going to get some bugs on your windshield here and it's going to know why and be able to move around it. So we say instrument all the things and that's going to be collecting data from your OpenStack environments, collecting data from your underlying hardware, collecting data from the services that are running in your OpenStack VMs, all of the stuff that powers OpenStack, whether it be RabbitMQ or MySQL or all the other bits that are involved there, it all fits together and it all tells a story and even if you only look at it 20% of that time, that 20% of the time that you look at it, you really, really want it. So as we're talking about though, let's talk about the operational complexity in our cloud environments and sort of how things are starting to change. So operational complexity, we define that as number of things that you want to measure and the velocity at which things are changing in your environment. So just on average took some numbers from some of our customers and what we're seeing around. Folks are collecting about 30 metrics per node or per instance from OpenStack. They're getting about 100 metrics, for instance from the operating system, about 50 metrics from any custom applications, whether they be off the shelf things like Redis or MySQL or I have you, some of them more, some of them less or something they've written in-house. Just to add to that, that 30 metrics, that's just kind of monitoring pure OpenStack. Once we started adding in solid fire, we got to monitor that. You add in 50 metrics from there, we got a monitor contrail, you add in another 50 metrics from there and so it grows a lot beyond just monitoring OpenStack. So in there I'm mostly speaking about Nova, yeah. So you start to multiply that out. How many folks are doing things with containers here on top of OpenStack? Okay, good set of hands. So that's really, you're now taking that 150 and you're really multiplying that by the number of containers you're running per node. So let's do some math. Let's say you had 100 instances, on average we're seeing people run about four containers per instance. That number is increasing over time but that's again, that's the median of what we've seen. So that's about 400 containers. Again, this is all simple combinatorial math. You do take 180 metrics per host, that's the 150 custom metrics that we had plus the 30 or so from OpenStack on the underlying instance. That's about 630 metrics a host. Again, not so bad, but you start multiplying that out and that's 63,000 metrics that you're looking at right there and hopefully you're collecting those at somewhere in the second to sub minute level granularity. In the world where things are changing all the time, you really don't want to be looking at five minute averages or weak averages or hour averages. This RRD methodology that we've kind of come to accept from our cacti's and our MRTG's over the years really don't work in a world where things are changing by the second. So that's a lot to look at. It's a bit of metrics overload and what we'll talk about today is a bit of how lithium has managed to bring all these together to put together some insights into their environment as well as some monitoring theory from Datadot, what we call monitoring 101 and just how to interact with your metrics and your data. So we've talked about the number of things that we have to monitor. Let's talk about the velocity, right? We did a study back in October on doctor usage and cloud usage and what we're finding is that while VMs are now lasting about 12 days, so it's not the years people aren't treating their VMs as pets like they might have their servers. They're not naming them after Greek gods or after planets and lovingly speaking of them anymore. They're turning pretty regularly and in many cases, if you're in a dynamic environment, it's likely less than 12 even. On the container side, we're seeing about three days and that's decreasing all the time as well. So these things are changing all the time, right? Things that we used to see, the host half-lives that we used to see, if they were hours before, they're minutes now. If they were days before, they're hours now.  All of these things are changing much more rapidly as we move into sort of everything being API driven. Kind of makes you wonder, I'll change this before it gives somebody a seizure, but this is the kind of like the response people have as they're trying to figure out what do I do with all of this data and all of this change in my environment? So the first challenge that people run into and the type of things that they need in this type of environment is more dynamic management of their monitoring systems, right? If your goal is to detect normal and normal changes all the time, you can't codify this in a Nagios config, right?  Like your chef runs or your puppet runs or your Ansible runs, whatever it might be, are not gonna converge fast enough in an auto-scaling world where things are changing all the time. This guy's hilarious, you should follow him on Twitter if you haven't yet. Honest update, he's, it's honest status updates. But yeah, I mean, we've got so much stuff going on here, these things are moving, you don't know where they are, you can't keep it in your head. And so you need to find a way to do it in a more dynamic manner. The other thing we need is a way to look at data from all of these different places. You know, Mike was mentioning, we talked about Nova a little bit, he mentioned that they're using solid fire. You know, you might be looking at Seth, you probably have some hardware underneath all of this because I know we're talking about the cloud but I'll have to run on something. So you probably have a pile of Cisco or Juniper, which is somewhere that's powering all this, maybe some cumulus gear. You got the Rabin MQ's, Cassandra, you're probably using Contril for your networks. And so all of this stuff is important to bring together into a single place to look at it. And then I know we're all in OpenStack then, but you're probably in some public clouds. So you know, I've got Amazon EC2 up there, but maybe you're in GCE, maybe you're in Rackspace, maybe you're in any of the other cloud providers. And so being able to look across your infrastructure, your applications across all of these different providers in a single set of dashboards and alerts is a bit important. And then finally we were talking earlier about some of these ideas of programmatic alerting. So this is an example of outlier detection from Lithium's environment. This is one of their MySQL clusters. And what you'll see here is, you know, we're doing things like rather than telling you, you know, rather than setting thresholds, what we've done is say, identify the guys that are different from the rest of the cluster.  Another lesson is tooling needs to really look at things from a service-centric point of view. Stop looking at things from a host-centric point of view, right, the host is dead. We've got some amazing abstractions on top of our hosts and on top of our infrastructure with OpenStack and with other tools in the world. Stop trying to measure where things are. This is a model of the Potolomaic, you know, the Potolomaic model of the solar system, right? Earth's at the center, sun and all other planets are revolving around us. Look at all the crazy lines that one has to go through as they're trying to figure out where the other planets are in relation to yourself. It's sort of like monitoring, trying to figure out where your services are and how your customers are interacting with them, starting at the host rather than coming service-side in. Right, this is much simpler. This is the modern model we look at today. So, yeah, as we start to look at this, I guess the question is, where do our normal monitoring tools fill in? You know, as Mike was saying earlier, they start to feel a little bit of like a relic from the past and something we might want to reconsider, both our strategies and the tools that we're using. So, with all of that data that we just talked about and the views at it, it's really important to decide how to show what it means to you and to others. For us, visualization is key. You may have seen this at the Tokyo Summit. I didn't re-grab the dashboard, but for a quick snapshot, this is a quick, easy way to see how our stack is running. You know, it's a reduction of information, but it has comparisons and you can see some overtime type information. You have, you know, some colorization for this in a good state or not. And you're basically distilling that information overload down into a piece of data that somebody can glance at and know if something's working or not. And another important thing about visualization is it's not you that cares about this.  Your XYZ team cares. Your client block cares because they're running their services on here. It's not just you managing your open stack cluster. So your dashboards, even ones that are kind of private to your team perhaps, you still want to make them publicly accessible. So if somebody's going through to kind of see how the stack is going, they have their view, but giving them the information for the other views as well is really empowering for them. Plus they don't bug you asking you questions. Hey, how much of this do I have going on? The comparisons of now versus last time also, super important. You know, we have the number of instance changed since, well, yesterday or last week. So, you know, hey, we're running 27 more. You can start to see the growth of how the stack is going. So that goes back to your perspective matters. You know, let's call A, your admin, you know, open stack admins is the people making sure the stack is up and running, hardware's working. But then you got B, who are your operators? They're the ones that are saying, oh, maybe some user management, some tenants, some quotas, you know, all that kind of stuff. How is, you know, I'm managing this open stack application, not necessarily managing the hardware and infrastructure that manage the open stack infrastructure, and then you have the users that are just like, hey, I got an open highway, I'm in the cloud, woo, you know, I just wanna go. Everybody looks at data differently. You know, admins, you gotta carry about your standard system metrics, of course, your disk, your CPU, but you know how to manage that already on your hypervisors or on the different things. You know, there's a lot of guides out there for how to manage RabbitMQ. But what's not always obvious is you wanna be able to store your data in a way that you can compare hypervisor to hypervisor. Not just what's the CPU of this hypervisor versus the CPU of this hypervisor, it's what's the CPU of the hypervisor, but this one is running, you know, 20, 16 gig instances, and this one's running 24 gig instances. How's the performance matching? Is this one's CPU the same as this CPU usage on the hypervisor, but it's not running as heavy of instances on there. That type of monitoring is crucial for admins to know, are you distributing your instances properly? And so tagging comes into play really important there. A lot of standard tools, it's very difficult to tag. Data dog's really nice because we can tag the hypervisors, not just for the class of machine of the hypervisor, but the instances running inside those hypervisors, that kind of tagging can go along with it. Yeah, so instance of flavor. So as an operator, you're carrying a lot more about your quotas, your usage, how is this team or this client utilizing the stack and is their growth gonna go outside of what I have available for them? Are they able to launch new instances of this size or do they only have this size and where am I sticking them and avoid that? From a developer perspective, they care a lot more about is it up and running? Do I got myself, if I'm having application problems, I wanna dial back into B and A to kind of troubleshoot that down because I don't know how many of you run into it, but when there's an application problem, sometimes it's, oh, it's the cloud that's broken, not the application. No, that happens. Let's go to the next one here. So some important concepts to remember when you're monitoring open stack is you are no longer monitoring an infrastructure stack.  It's actually a services, microservices application. There are n number of components and services to open stack, but it's not as simple as just saying, oh, Nova, that's one service. Well, no, because it has databases, it interacts with Rabbit, it has the control planes and all of this stuff tied together. You're talking dozens of services that need to be monitored. It's kind of a mind shift for a lot of people on ops teams that are used to managing infrastructure and coming to the cloud. It's very important to remember. With that, it's very difficult, you gotta be able to monitor how those things are working well together. Your servers may be responding fine, but the application is not. And the other thing to remember is you're gonna have more than one network that's going on there. You may have your network that runs the hypervisors and then maybe a network that does your control plane and then, oh, hey, you got Neutron, that's running your network inside open stack, but maybe Neutron's running contrail behind the scenes and all of a sudden, you've got four different networks you're having to monitor. Any one of those screws up, it may not be visible until you look deeper in the stack and see how it's actually impacting things. So that's why I kind of take the Wikipedia, think about sharing, human share everything. Now, open stack kind of is all tied together and shares everything. So with that, people always talk about, oh, you can't compare apples and oranges. In this particular case, you're making a fruit salad.  That's just me, I like watermelon and fruit salads, right? You may have a perfectly fine fruit salad and everybody loves it, but there's no watermelon, so something's not right with it. It's not there. The whole thing, you still got your fruit salad, it's still up and running, but your stack may not be. So here's an example of some graphs that we have. Top left there, it's the current workload in open stack. So what you can see there is there are jobs that are going into the open stack queue. So in this case, I believe it was to launching instances. The next one over, well, how many messages are in my RabbitMQ queue? Then you go into, well, what's my publish rate of messages going in there? Well, that kind of looks okay, right? But then all of a sudden you see, I have this unacknowledged rate inside Rabbit. So what this means is somewhere along the line, somebody tried to launch an instance and it didn't start up. In this particular case, it wasn't Nova. Oh, it wasn't Rabbit, it went all the way down that it was, was it Cinder? No, it was SolidFire. And in this particular case, it's because the tenant didn't have permission to attach a Cinder volume using SolidFire with this particular IOP level. To a user, they don't care, they're looking to go, gosh, right at my instance and boot up, open stack is broken, right? Or Nova is broken. But in reality, it was something very much under the scenes. And so this is one of the complexities with open stack, of being able to find charts that show with this. We talk about outliers that are really important. Yeah, that's an outlier there for your RabbitMQ unacknowledged rate, but that went away. And suddenly, oh, do I really want to get blipped and paged on a 0.15 average rate of unacknowledged messages? Probably don't want to get woken up for that. So it doesn't matter for you, but your users, they care. They don't have their instance. And they hate the answer that, oh no, it wasn't this, it was way down here. We had an instance, not series of instances, we had a problem where elastic search would just lock up and crash. Oh, crud, you just can't send any searches, can't index anything, it just straight up broken. What again here you'll see is we were doing checks on the average current workload, right? Not very big blips there, some stuff's happening in open stack, no big deal, right? Well, you look at the TCP retry or retransmits, they're a small blip, but you know, shoot, four of them in what is that, a five minute window? It's TCP, that's fine. We're not gonna worry about it. It's not a high enough outlier for long enough that we care. But then when you look at the sum of the TCP retry or retransmits for all the VMs running in that stack, our numbers start to get kind of higher. It's like, oh, okay, well, you know, but we weren't looking at that. We were looking at perhaps just for those elastic search servers.  So what was happening here was instances would be launched or destroyed of a larger number that would then screw up the flow in contrail, and we were running an older version of contrail, by the way, this bug has been fixed. It was fixed a while ago, but we didn't upgrade. And so that flow would then tie up the network. Packets would be dropped, need to be retry, and we still don't know why elastic search didn't recover from it, but it still got screwed. The problem here is that we would be looking at individual graphs, and everything looked fine. Who cares about one TCP retransmit in a five minute window? But taking all this together is, you know, what's there? This took us weeks to resolve, by the way. Not a fun one. So, you know, kind of, you know, Mike's given us some great examples of sort of troubles that he's had in the Lithium environment and how good monitoring tooling, you know, I hope you choose Datadog, but if you choose something else, you know, the point is good monitoring and good visibility into your infrastructure is important. But we've talked about all these metrics, and the question is, what are these things, which of these things do you alert on? Which of these things do you just keep around for, you know, as resources for your investigations, right? We talked about how you wanna monitor everything so that you have it when it comes down to that postmortem time or to that incident response. And so we wrote this guide, monitoring 101. It's up on our blog. I'm gonna give you the TLDR edition for your Reddit fans out there. The links are in the slides. You can, when we post them up on the, we'll post them up on the summit site at some point. You can check them out. But we'll dive into this real quick. So as you're trying to figure out how to avoid that pager fatigue and what's important to you for alerting, we encourage you to break your metrics down into three, into one of three categories. The first being work metrics, resource metrics, and events. And we'll go into defining a little bit about those, I mean, in a second. But think of your application or your OpenStack environment as a bit of a factory. You know, you're outputting, I think this is a textile factory of some sort. I'm gonna go with cars, because that's a little bit more, we all know what those are. But let's say, so you've got a bit of a, you've got a factory and you're trying to figure out what you're doing here. Well, so work metrics are gonna be the actual output. They're the things that your customer is consuming. So if you're an OpenStack provider, it's likely the number of VMs you're letting folks spin up and shut down or the number of things going on, that's your throughput. In the car factory example, it's gonna be the number of cars coming off that assembly line. Success versus error rates are gonna be things like, how many of those cars show up without all their hubcaps? Or how many of them are ready to go to the show floor and be sold to your customers? And then performance is gonna be how long it takes you to make one individual unit of those cars. And you put those all together, and these are the symptoms. These are the things that your customers care about. They're the things that your customers are paying you for. Resource metrics on their hand are all the parts that go into making those cars, right? It's this pile of tires and how many of those do you have available to put on the next set of cars? It's how many more VMs, it's the capacity in your environment. How many more VMs could you be spinning up and down in OpenStack? How many more, what is your utilization looking like? And then the last one being events, right? So let's say, again, we're a car factory here in this example. Let's say your CEO got on stage and promised some car that doesn't exist yet and that is gonna make 350,000 of them sold them in a week, right? That's an event that now changed your queue. You now have way more things to produce in a very short amount of time. You can put that on your graph, overlay it and understand why your metrics went one way or another. Maybe you change the formula of your car, or in this case, of how your team assembles the car in the assembly line to get more resources out of it or it slowed you down. These are the types of things that you're gonna pull in there around all of the metrics that you're collecting to figure out why things are going one way or another in regards to the changes you made. So in your OpenStack environment, maybe you've upgraded to the latest release on the first day it came out, or maybe you've made some sort of a change in your environment. You're gonna wanna overlay those on top of your graphs and figure out what's going on. So we'll take some examples from Nova. In your case, customers are consuming from you. Again, the number of VMs that are running at a given time, being able to start, being able to create the number of VMs that you're letting them spin up or shut down at a given time. Things like IO performance, things like network performance. On the other side, the resources are gonna be things like, are the hypervisors up and available and usable? How many VCPUs do you have ready to go so you can keep track of utilization? RAM and disk, things like that. And on events, again, these are gonna be the upgrades, things like we talked about before, code deployments, those might be code deployments in your VMs or to your VMs. They might be auto scaling events in your environment, instance migrations and creations, adding and removing nodes, all of that. And these are, again, the things that are gonna provide you context as to why the utilization on your cluster is higher or lower. So we're talking about events and the idea of overlaying. So here, for example, we have a graph on IO usage in the cluster. And the idea of, when I say overlaying, those are those pink bars that run over this. These are events in our environment. We've added or removed nodes from the cluster and you can see how that's impacted IO. So when do we wake our teams up? Nobody likes to be woken up at three in the morning. So what we encourage you to do is page on those work metrics, page on the things that your customers are consuming from you.  Those are the things that your customers in the open stack environment care about. If it's an API running on top of all of this, it's when your customers can't make those API calls. These are the things that make you money. They're the things that customers are coming to you for. What you'll do is then use those resource metrics, those events, and other work metrics from your environment to find out why the situation is occurring. I realize I'm at an operators conference and this joke probably follows a little flat here. But your CEO probably doesn't page you in the middle night to say, everything's working fine, but CPU usage is kind of high. What they call you to say is, customer tried to spin up a VM, customer tried to make an API call. It didn't work. Why is this happening? Fix it now. And so it's important to think about when we say alert. Alert doesn't necessarily mean page. It could be email. It could also just be create a record in some sort of a time series, as we were talking about earlier. You want these things to be, if it's going to wake you up, make sure that it's important. You have to act on it now. And you want those alerts to include, what do you do about it? Who do you call next if you get stuck? Every one of these should be actionable. The minute your team gets so many alerts that they're ignoring their pager, it's called pager fatigue, and it's not a good situation. I used to work with teams when I was an open-site operator or an infrastructure operator at Viola. And I'd see their pager duty queue, and I'd see 10,000 alerts in one week to their on-call. And I'd say, when do you sleep? And they say, oh, I looked at it, it looked fine. I went back to sleep. That's not a good situation.  You want to fix that. So avoid cryptic alerts like this. Like, don't tell people Nova Node 1 is down, or DB server 1 is down. Nobody knows what to do with that at 3 in the morning. They've swapped all of that information out to disk. They're now dreaming about something. They're talking to their family. They're having a good time. What you want to do is, what do you do with this? Nobody knows. You've got to go look in some wiki page. So you want these things to basically say exactly who to call, what the business impact is, even link from your alerts to the wiki page or include the contents of that wiki page directly in the alert so that you know what to do next. But yeah, you're going to get these alerts. And when you do, the flow that you'll take is you're going to look at your work metrics, see which one changed, what the symptoms are that your customers are experiencing. Hopefully these are leading indicators rather than trailing indicators. And you're responding before they've paged you, but from a customer ticket or what have you. But you're going to use those, look at your resources, look at the events, and diagnose. If you can't find it within the service that you build, every other team in your environment that you depend on has some sort of a work metric. So maybe if your work metric is the number of VMs you're creating and shutting down, and all of a sudden there's some issue there, maybe below that, the next work metric is something coming out of contrail in terms of packets coming in and out of the environment. As no, but that's your resource. As contrail, that's your work. And so each person all the way down the stack has a work metric, they have a thing they provide. And it's true in your application stacks as well. Maybe the work metric that your customers notice, again, is those API calls, the work metric that you as an application developer might notice would be maybe something from your DBA team. How many SQL query latency or something of that effect. Every team has one. And so figure out what your work metrics are, figure out who your resources are, graph that out, and that's how you know what to alert on. And so that brings us to kind of what's missing. Some of these are complaints on my part, but some of them are things that you need to think about how it applies to your stack. What Alan was talking about here with the different services and teams with those metrics, tagging is important so that you can aggregate and overlay to each other to say, oh, this did this so I can look at the graphs for this and see this, kind of see the whole thing together. The problem with this is it's very difficult to see how open stack as a whole across those dozens of services is actually performing at a glance. It's very difficult because you still have to dig in to each one to figure out what's going on if something isn't working. So that's one of the problems I see right now with kind of open stack as a whole with monitoring. Grabbing this stuff isn't gonna help us if there's so much there. So it's hard to say when those different services, you're almost like outlier here, outlier here, outlier here, now I care about a problem. So that's one of the challenges right now with monitoring open stack, that correlation of disparate metrics.  Or vice versa, all of a sudden your hypervisor is doing something weird, maybe okay for your hypervisor, but now it's causing your applications to perform poorly. So you gotta be able to correlate those kind of things together. Sorry, one last thing on that last one. Real quick. Service degradation, when services don't perform as well as they should. You gotta know for your stack what's important. If this service isn't working right, it may not bring down the whole stack, but it's an acceptable risk. So with your monitoring and your alerting, you gotta decide for yourself what matters. If you can't shut down instances for your private cloud, does that really hurt you compared to not being able to start instances? So these are just some of the things that it's a judgment call for your stack, which makes it really hard because it creates these missing pieces in the monitoring that's not easy to solve. I haven't come up with a good way for us to do it at Lithium. Then if you've got great ideas, let's share them to the community so that everybody can kind of solve this problem. So I got that one. So anyways, we're at about 12.45 now, which is sort of the end of the session, but we've got a break here, happy to hang around and chat with you guys a little bit about how we're interacting with OpenStack at Datadog, how Lithium's using it in their environments and some of the monitoring challenges they've encountered or solutions that they've come up with that would be helpful to you. Also feel free to stop by the Datadog booth. I'd love to give you a demo of what we're doing with OpenStack and talk with you about some of the roles that we're hiring for this week at the summit. So thanks.


Nipendra, Nipendra Kharaz is our next speaker. And we have Nipendra here. Hey, Nipendra, how are you? Hey, all good, Karan. Thanks for having me. Good morning, everyone. Glad to have you here. Thank you. So Nipendra, you're gonna give us understanding of GitOps with Argo CD, right? Correct, correct. That's amazing. I mean, as I mentioned before, Argo CD is one of the most popular GitOps, Cloud native GitOps tools out there. So great. So Nipendra is the principal consultant at Cloud. Cloud, you got technologies. And most importantly, he is one of the, are you one of the, you've created the course on Kubernetes introduction, if I, on edX, right? Yes, on edX is taken by more than 200,000 users, right. And you know what? I am one of them. So great stuff. We are on the community in Nipendra and fantastic to have you here on this show. And I think we can get started. Sure. Thank you. You can share the screen. Yeah. Hope it's live. Okay. All right. This is Nipendra and yours. Yeah, thank you. Thank you, Karan. So this is Nipendra and I'm founder of Cloud. So as I promised that we're gonna do some hands-on, so I'll jump directly to that. So on our website, you can follow along with me if you want to kind of do the hands-on along with me. So I'm going to go to my website and we click on the hands-on labs there. And then we have multiple hands-on labs which we can try. So we're going to look at the CHD with Jenkins and Argo CD. That's what we have promised to finish today. I have only triggered the lab a bit earlier, but if you are on the page right now, you need to click the button of lab setup that takes around two minutes to come up. But once the lab has come up, you can follow along with me. So first we'll look at the demo of CHD with Jenkins and Argo CD and then we'll talk about some additional concept of Argo, like rollouts and so on. Okay, so let's kind of go ahead and do that.  Depend, is it possible to share the link if possible? Yep, you can share the link here. I don't know, how do I post it here? Okay, is it open link or did you do? Open link is open link. You just go to our website, we'll find the link and then you can try it out. Yeah, you can just. Very cool. Okay, great. So what is GitOps? As the name suggests, right? Git plus ops, right? So you're doing something with Git using which you can do some operations, right? So as we know with Git, we kind of store all the commits and we know who did what and when, right? The same thing we're going to kind of bring in with operations now. You basically kind of say, you put your applications, it's configuration in a Git and then somehow it gets picked up by some tooling and which deploys the apps in your environment, right? And that's what GitOps is, Git plus operations at a high level. It is declarative and all those things which you can just kind of read through it. Argo CD is one of the tool which kind of implements this GitOps on Kubernetes. So we are going to have our YML files. We are going to have our Helm charts or whatever in a Git repo which Argo CD can pick it up and then deploy it on the environment there, right? Whatever Kubernetes setup you have. So this is what we're going to do today as a complete workflow. So there are two source code repos which we're going to play with. One is a source code repo of your source code which is your actual app code, whatever you have. And second, Git repo which contains your Helm chart, right? So as a developer, we are going to commit this thing to our code to the source code repo. From there, we'll have some kind of a Jenkins or I mean in this case, Jenkins running which kind of pull your source code and if there's some change, it's going to pick that change. Build an image. Once the image get built, that image get pushed to the registry, whichever you would have, in this case Docker Hub. And once we have done the image push to registry, we are going to update our Helm chart using which we kind of deploy our apps with the Helm thing and so on, right? So Argo CD is going to pick up that Helm chart change because we would have updated our image version and so on there because the change on the Git repo which Argo is watching, it will pick up that change and deploy it on the Kubernetes cluster what we have. So we divide this thing in two different parts. We'll first combine this A and B step in which we have a Git repo which contains our Helm chart and we're going to have this Argo CD which is going to deploy this app for us, right? So we have, I will say that, right? I triggered the Argo CD setup earlier that should be up by now. So we can check that up, that Argo CD is up or not. So Argo is up and running here. So what you have also on the top right of the page where you can find the link how you can access the Argo CD. So I click on this and I'm on the Argo CD UI. I can then log in with the admin user, the default user which I'm going to use and now to find out the password of the Argo CD we are going to run this command and this command gives me the password of Argo CD. So I just pick up this up and go to the Argo CD UI and sign in. This is going to allow me to sign in to the Argo UI. Now I'm going to basically fork a repo which contains our Helm chart. So if I go to this Helm chart, this contains the Helm chart using which we're going to deploy a sample RLCP app which contains a Python frontend and MongoDB backend there. This is like contains, as you say, that a chart configuration here. I'm going to fork this up under my username here. This has been forked. Now I'm going to go to the Argo UI and deploy this particular app. So now to deploy this particular app with Argo UI we'll click on this new app. We'll give the name of our app whichever you want to give. Project default is a project for the Argo CD so we can just keep a default one. I'll keep the sync policy as automatic which means that if there's a change in the forked repo which we are going to use, Argo should sync it up automatically.  Prune resources which means if you've basically changed something in the, some YAML file in the Helm chart or something like or your whatever, Helm chart or whatever YAML you have been using then prune those resources because those files have been removed but for now we just check that up. Self-heal means that you can, if somebody kind of modified the objects in Kubernetes by mistake or something like that, self-heal it. So that's all I can kind of talk about it because of the time constraint. Then we'll give our source code repository which we are going to use to deploy the app. So we'll click on this Helm chart and say that like I want to have this particular the GitHub which contains our Helm charts and in which I will say, my Helm chart is on the top, so this is dot here which saying that look at the dot here. Now, next step is where would you all deploy this particular applications so we'll deploy on the same Kubernetes cluster for now but we could configure to deploy on some other cluster as well. So I'll go and deploy in the local cluster, give a name space, name as app. What I can do also is I can create here, auto create name space. This name space would be created on the fly as I'm deploying the app. So I'll keep the values as it is and I'll click the Git button here and now this is going to look at the Git repo what I have and start deploying my application. So as you can see, something is happening behind the scene. It figured out there is a Helm chart for front end which is the Python based app and there's a backend MongoDB app and it's deployed the respective services, deployments, pods and so on and that's what we are seeing here. We can kind of go back and check that out. Is it happening or not?  So we have one replica of the backend and three replicas for the front end. So let that happen. I would now go ahead and deploy the ingress using which I want to access that app. So I'm going to trigger this particular ingress configuration and now our ingress should be up and if I go up again, there is going to be a UI or the ingress link which I can click on this and this is going to show me the UI for our applications. Hopefully it works as expected. Yeah, so as you can see about app is now we can put down some information here if we need to but otherwise we can skip them. If I refresh, we can see that the information is going to come from different replicas of my environment. So as you can see this changing here, so that's all good. Our app has now been deployed. So we have done kind of first part of it here where we have configured an app using the Orgo CD. So we have a Helm chart which has been deployed by the Orgo CD. Let me just trigger the Jenkins as we kind of speak here because you will take some time as well. So Jenkins is going to now get installed as we just talk here. So Orgo CD can pick up in this case we have used Helm chart but I could have used a YML file or maybe a customized thing or whatever. You pick it up.  Now the next step what you're going to do is we are going to kind of perform the CI stuff now in which we would basically going over our Git repo. Let me show you the Git repo for that. There's a Git repo here which I open it up. This Git repo contains the source code which is required to the front end source code which I want to deploy. So we go to fork this repo. This has been forked now and here we have this front end RSVP.py file and there is this template in this we have profiled HTML file which basically shows you this kind of a thing. So we'll change something here and we'll see how it kind of different change in the actual deployment. So we'll update our source code something in this particular front end and we'll see that change eventually get deployed by the ROCD behind the scene. So what we have done is we have forked that repo under our anchorage username. So that's what we have done now. Let's go back and check if our Jenkins spots are up and running. So Jenkins is kind of coming up so we need to kind of give it some time before it comes up. So what we can, yeah, so we can talk about the workflow here. So what we'll do is we are going to configure, I mean we already got the source code repo. Now in the Jenkins, we are going to configure that polling part. So we'll poll every one minute of our repo and once we pull it up, we are going to kind of build that image. Now how do we build the image and how do we push it? For that, we are going to use Jenkins files and I'm pretty sure you're aware of Jenkins file. That's the kind of a pipeline script which you would be building up. So let me kind of quickly talk about that pipeline as the Jenkins setting it up. So what we have in this Jenkins pipeline is we have two steps here. One is a build stage or step or stage, whatever you say, so build stage and the deploy stage here. So what we're going to do is we're going to create a dynamic Jenkins slave. As we do the code commit, a dynamic Jenkins slave is going to come up which kind of runs a dint Docker in Docker which would perform two steps here. One is build step in which we basically get our source code repo and build an image and then once that image been built, we are going to push that image into our Docker hub accounts. Of course, we need to kind of have a credential so we'll configure that as well. Similarly, then what will do the deploy stages? We are going to whatever the new image we have built that image would have a new image tag. So what we'll do is we are going to go to our, the Helm CI CD Git repo and here we are going to update the values.yml files. Assume that you are aware of the Helm chart working. So we're going to update this values.yml file on the fly and that way we would change our Helm chart which would be picked up by Argos CD2 to kind of take the step forward. So let's see if our Jenkins up and running or not. Jenkins up and running. So let's go ahead and look at the Jenkins UI now. This is going to bring you the Jenkins UI. So again, the username here is admin. From the password of Jenkins, I can just come down here and figure out the password of Jenkins. This is my Jenkins password. So I'm on my Jenkins. Now what we'll do first thing is we are going to configure credentials for our GitHub and Docker hub so that we can push the Docker image and update a source code rep. So we'll go to the managed Jenkins here. We'll go to managed credentials. Then we go to global here and then create two credentials here. One for Docker hub, one for GitHub. So I'm just going to say for nQuery is my username for my GitHub. The password I'm going to use with my personal token here. So I'm just going to copy that token here. That's going to be my thing for my GitHub. I'll give the ID as GitHub here. Save it. Second thing I'm going to configure for Docker hub. So I'm just going to give a name again, whatever the name I have for Docker hub. Give the credentials of mine. Give an ID as Docker hub, which we are referring in our Jenkins pipeline file. So this is being done. Now I'll come to dashboard here. So we have configured our credentials to GitHub and Docker hub, right? So we have done that part. And we have also forked our repos. This is the forked repo. Now on this forked repo, we are going to build this JenCon talk, create this new file called Jenkins files. I'm just going to go to the Git repo here, add a file, say new file, give a name to Jenkins file, copy the content which I have here. And now I'm going to change the values here because I'm going to modify it according to my username. So I believe you have to make three changes here at the line number 53. We'll specify under which username. So basically whatever username you used to fork it, just give that name here. I give my incorrect. I'll give my email ID. One more change, I believe, recall, let me just check where the changes. Yeah, so here is the Docker hub change. So these changes you make, so like we are saying that under which account you have to send data. So I'll just kind of go ahead and commit this particular file. So now I have added the Jenkins file in our source code repo. So I'll go back to the Jenkins and say create new item, give a name to it, the item name. It is a pipeline stuff. Say, okay, again something I can specify here. And I won't trigger this whole SCM every one minute. So I'm just going to take that, what do you call it? Say the schedule here. This is kind of say, do it every one minute. This is what it is for every one minute. Then I'm going to the pipeline script. I'm going to get it from my SCM, which is Git. So I specify Git as my SCM. And now here I'm going to give my source code repo URL. So I'll copy that stuff here. This should be done. And this will pick up the Jenkins file on the root of it. That's true. So I'm just going to save the file here. And that's it. So what I have done is, I have now configured the Jenkins pipeline so that it can trigger as soon as I make a change. So let's go ahead and make a change here. So it's triggering up, but I'm going to cancel the trigger for now because I don't want it to take a lot of time as I would change it. So I'm not going to go ahead and change my source code here. So let's go ahead and change the UIs, I mean my applications UI change. So I'll go to the templates here. I'll go to profile.html. And I'm going to change here. So I'm going to enter your name. So this code added the file here. Enter your name. So I'll just go to say here, enter your, so capital I'm putting here, enter your name. And just I'm going to put a definition here as just for as we are here. So enter your email. I just add definition here, nothing else. I'm just going to save it here. So now as I'm going to save the file here, what should happen effectively is this should pick the change. So basically my Jenkins pipeline should pick up the change. I think it will do it every one minute. So as you can see that, I think did it pick it up or not sure. It's happening now. So as you can see now, it is kind of put them to change. And it's going to now create an on the fly Jenkins slave. It is going to perform that step which we have mentioned there. And it's going to perform that build and deploy step here. So as that's happening, so as you can see it's kind of got the slave now. So let it happen. Let's wait for a while. And this is our UI as you can see, it just says enter name, enter email for now. So hopefully demo guards will be with us to kind of do all the stuff on time. Any questions on the screen? Basically if you have any questions, please put them and if current get put them on the screen here, I can see that. I can see those questions if there are any between because going to kind of pull up some images and pick up to the steps. So we just wait for these things to happen. Any questions? Meanwhile, so go, go. There was one question from Shubham but Sagar answered that already. But maybe it would be nice that this is building. Can we maintain, so the question is this, can we maintain dependency between apps deployment like A, B, like app B needs app A to be up and running. So yeah, as you can see here in our chart, what we have is already like that, right? Like here we're deploying a backend front end, right? This is a Helm chart thing, right? So of course this is with the Helm chart, but of course in the Argo, you have Argo workflows or your workflows. So there you can make dependencies in a better way. But even, so those are steps or those are kind of a different thing. For example, you want plus it's, for example, if you're configuring Argo CD to kind of first deploy the cluster on AWS, then perform some steps, right? Something like that. So those are the workflows which you can do with Argo CD. So I want to do workflows that thing for that, which can help you. But here I'm talking about in terms of packaging. So because I require the backend with front end, so I'm maintaining the dependency with the Helm charts here. All that answer your question. Yeah, so now as you can see that, it has happened now we are kind of pulling an image of Python and then it's going to perform the steps like it will take some time to get the, so we are kind of building our MSC. If you kind of look at what would be happening here, we have a Docker file in our source code and that Docker file is being, I mean, so that requires Python to be pushed in, pulled in first. So we are getting that particular stuff here. So if I kind of go back here, use this Docker file is here in our source code. We're performing these steps right now in our pipeline step. So once this particular image would be, so we are performing the steps to build an image for my application. And that's what's happening now. So now image has been built. Now it has been pushed to the Docker hub account, what I have. And once the image get pushed, you will see that it is going to change the values of YML as the push would finish. So now that has been done. Now I'm kind of cloning the Helm chart repo. And in that, we would basically, I mean, there is the YQ tool here. And then it's kind of done those changes. So whatever we changed required in the particular Helm chart is succeed. So basically if we kind of go back to our thing here, our pipeline succeeded. And now if I kind of go back to here, you can see that it would be changed here, right? 10 seconds ago. And what change we have done is, we would have basically changed the image repository and the tag. You are using what we have changed there. And now what should happen is Argo CD should be able to figure that change. So it kind of do it every three minutes. So other than waiting for that, I'm just going to first refresh this page here. So say that there's nothing is not running here. I'm going to now sync it manually. But if we wait for three minutes, it would sync and then it will continue. But I'll sync it right now. And I'm going to hit this cool button here, cool things. I'm going to hit the synchronize here. Now what's going to happen is going to figure out, it is seeing that there's a change. And as you can see, the new parts are coming from our front ends. And as those parts are going to come up, we would see our application eventually would be changed to the newer version. So we'll see something different would come here. Here it says enter name, but then it would now say enter your name and colon definition kind of whatever we have done there. So let's wait for the Argo thing to happen here. So it's kind of pulling the new image. And once the new image will get pulled in, so we can verify that with our UI stuff here. So we can again do this. As you can see it's getting created and this would be kind of pulling that image which we just pushed to the Docker hook. And as the image get pulled in, we would basically see the change to the new deploy. Now as you can see it's been changing here and we can see a similar change in our UI as well. So let's just wait for that to happen. And if I refresh the page, you will see that. We should see that in like as you can see, right? The new changes come here. So this is how Argo can basically help you. So basically you do a code commit on your Git repo, which kind of the, which kind of finish the entire cycle, the CICD and everything. Okay, so this is kind of first part of the presentation which I want to kind of cover. Again, it's like went very fast, but as I said that right, you have the entire stuff at your hand, you can go ahead and practice and do share your feedback with us or on social media, whatever you can think of. So this is the first part of it. Let's quickly go and figure out some additional stuff what Argo can do for you. So again, this is going to be primarily a very overview kind of stuff. So we have done a complete workshop in the Kubernetes community Bangalore earlier this year. So if you're interested to kind of find out what this is in detail, you can kind of go back and search for the, basically KCD Bangalore 21 videos and you can do the stuff or you can do one more details about it, right? Okay, so let's go talk about Argo rollouts. So if you talk about in Kubernetes terms, in Kubernetes you can typically do with deployment object, recreate a rolling updates on it. Kubernetes by default does not support blue, green, Canadian, process delivery, what you want to do in the advanced stages once you kind of figure out the basic commodity stuff, right? So for that purpose, Argo has come up with Argo rollout. So just to kind of take a step back. So there is an Argo prods. So Argo prods is the kind of a collection of tools. Argo series is one of them. Argo rollout second, then we have Argo workflows and Argo events. So there are multiple projects under Argo prods. So don't get confused, but these are event projects. So Argo CD we talked about is the Argo rollouts, which kind of can work along with Argo CD. And I think I used individually also, but I may be wrong there. So Argo rollouts is going to help us implement these three different strategy of deployments, right? So it's a drop in replacement for the deployment object. So now if I'm using Argo rollouts, I'm not going to use the deployment of it of Kubernetes. So we kind of have a YML file, which creates a rollout object. And with that rollout object, we build this, we kind of have the strategies there. So no deployment object from our site, we'll be building a rollout kind here with rollout count here. We are going to replace that, we're going to replace that deployment object. So now you can see here, this is a kind rollout. And to have this rollout thing, we have two services here, which are called a canary service and stable service using which we kind of manage our strategies of blue green and rolling out basically the canary thing. The way it works is, so we have this rollout object in which we define strategies, what you want to have. Then we have traffic routers using which we can route traffic like 50% here, 30% here and so on, something like that. And then we have this metrics provider using which we can monitor the performance of the, let's say newer version. So I'm moving to the newer version, I want to see how it's performing for that purpose. I can do this collection of matrices and then we can see what the result is that and then we can continue our canary updates and so on there. Let's look at some diagrams here. So what we have here, I said that, right? Rollouts, that is the beam done with the two different version of services. So here we have a stable service and desired service like a preview one. This is what we want to be there. So at the stable state where everything is all good, you have only one revision and both the services point to the same version of that. Then if I want to move to, let's say, blue to green, so I'll deploy a green version of the app and point only the desired or the previous service to the green app. And once I'm kind of comfortable with the green app is what I want, right? Then I'm going to even point that stable version to the same green version and then get rid of this revision one. So this is how the rollout can point to these two different services and help us achieve the blue green. So like, as you can see there's an ingress which we can have in front of our application and then we have the two services which can be preview on the stable one which point to different replica sets and whatever we saw earlier. So ingress would point to preview and the stable one simultaneously depending on the percentage value and so on and then we can move there. That's the blue strategy here. Similarly for Canary, then we can define the weights, right? For example, I want to kind of first, so canary simply means that, okay, when I want to go to the newer version, I want to see how does the newer one performs, correct? If it performs better, I want to continue. In blue green, you just move to blue out to green, correct? But in Canary, you want to kind of move from one to the version over time. That's what we have here. So here we can define the weightage, how you want to kind of configure that. And now again, like as you can see here, we're kind of saying that, okay, send 75% effect to the app V2 and 35% here. This is what canary does. So again, canary, again, we have two services there. So in the basic canary, the canary is done with respect to the number of ports you have. Like for example, if I'm doing the, if I want to move the 25% of the newer version, so I'll get only one part of the newer revision. And now because of three parts are here, 70% of the traffic will go here, 20% will go there, correct? And then I can move to the other way and then finish by moving all the parts of the newer version. This is the basic canary one. But then if I am going to implement along with some kind of an ingress-based thing, right? So I can configure my ingress, be it engine X, be it traffic, be it HTO one or so on. I can pick up any one of these and using which also I can consider the percentage of my traffic, not by the pod count, but the percentage what I want, right? For example, here I have stable and the desired version. I can, though I have four pods here, then I can say 95% traffic, go here, 5% go here and so on. So I'm not bound by the number of pods what I have, but because of this ingress-based canary, I can control the traffic flow as per my wish. Okay, so I'll quickly move away from here. Then the next is the process delivery in which basically what we're trying to achieve is, I want to make sure that when I'm kind of upgrading my app, I would basically do some kind of analysis, right? Or some kind of what you call, say, see that how does the new version performing and I want to go to the new version and so on. So for example, if I move to the other version and if my success rate, this is kind of the template which is there, if I run the template and figure out, okay, if my success is beyond the threshold, then I'm going to move my app to the new version. So this is called process delivery in which you go to the new version by some kind of a logic, which may be metrics from from Atheist or maybe a simple job, whatever you want to do, you just check for the condition and move on. Okay, last thing I want to talk about the workflows here. So workflows are simple, which is going to be like the steps you want to perform. Like for example, if you take an ML job, right? In the ML job, you'll kind of get the raw data. Then you kind of clean it up. Then you kind of run trainings on that and so on. This is how the workflows, you can define these workflows as a container. Like for example, here is a workflow in which I kind of start, I run the hello workflow, then I run hello two and two being parallel, correct? Other workflow, I want to kind of do this A, B and C in parallel and then it'll go to D. So I can kind of build this kind of dependencies with the Argo workflows. Yeah, I think that's it. I think we should be on time. Karan can correct me if I'm a bit late or earlier. Yes, we are right on time here. Great. Any questions? Thanks, Nipin, for walking us through this amazing journey of Argo and Jenkins and how these two tools could be used in tandem. So a few questions here from, yeah, can you explain what is Jenkins files? For Jenkins file is a file using which you decide how the CI should happen, right? As I said, right? See, okay, let me kind of go back here. This particular, if you look at here, see, whenever I'm a code change, right? As I'm a code change, I want to perform some step, right? I want to run test cases, right? Against my source code, whether they are done correctly or not, right? Then I want to build an image in this case. Once the image has been built, I want to push to the Docker hub. So there are two ways of doing it. One is you basically write those steps one by one in your Jenkins. So when you are building a Jenkins job, so if I talk about a other job, this is a pipeline thing that you put up, but I pick up a freestyle project, for example, I just give a name here, freestyle project. In this project, you need to perform all these steps one by one. So as you kind of say, build step, right? Execute shell. So here maybe you can write, okay, clone the source code, right? Then the other step can be, okay, build my Docker image build. So either you can write these steps one by one or you can basically put them in a kind of instructions, right? Those are called Jenkins file. So Jenkins files kind of live along with the source code. So you don't need to kind of write those steps manually in your Jenkins job. You define those steps what you want to have. Like in this case, we are saying, I have a build stage in which I want to kind of build a Docker, build an image from my Docker file and push it to the Docker hub, correct? Deploy a second step. So rather than somebody is modifying these steps writing one by one, you define them upfront so that you don't have to kind of deal with the Jenkins every time. You define in your source code repo, it goes as a source code repo and in whichever Jenkins environment you would be, you will come up and then it would start. Like you see that, right? I got a fresh Jenkins because I had Jenkins files, it could perform all that, just one by one. All right, very cool. Thanks, thanks a lot Nipendra for explaining this and I think we are at the time I look at it for this session. Thanks a lot for walking us through this great journey of Argo CD and Jenkins and I'm really happy to see Jenkins UI after a very long time. Yeah, I'd last remember my memory was like 2012 when I was working with, I was one of the Jenkins team but I was in a different team but Jenkins felt, hey, my Jenkins build is running very slow. Like back in 2012. So yeah, good to see Jenkins after a long time. Thanks Nipendra, thanks again, thanks a lot for joining in and helping us with this great session.