Okay, right, so we're going to get started. I guess everybody can hear me okay, the microphone seems to be working. So hello, it's nice to be here in Boston. My first time here personally, it's a beautiful city. So my name is Sean Handley, I work for a company over in the UK called Data Centered. We run an open stack based public cloud. I personally have been working with OpenStack for about three years, and I mostly come from a web applications background. So my way of talking to OpenStack is all API driven. Hi everyone, my name is Tobias and I work for a company called CD Network in Sweden. We are also a public cloud based upon OpenStack. I've been working with OpenStack for about three years now, and as Sean here, I've also mostly been working with the APIs, and my main focus has been developing our cloud management system that we are using for managing our cloud. So we are here today to talk about how it is to run a public cloud using OpenStack. So as we said, both we at CD Network and Data Center are using OpenStack as the base for our public cloud business. And today we will cover what life is like behind the scenes, just doing that, running a public cloud with OpenStack. And just to have it said, this is not a sales pitch at all, yes, you know that. We just want to make OpenStack better for us as a public cloud provider. Yeah, indeed. I mean, ultimately, it's kind of a scratch your niche sort of thing. So we want to focus on how to simplify life for ourselves and for other public cloud providers, because OpenStack is a wonderful toolkit, but we found that there are some gaps in it that make our lives busier than we'd like to be. We're obviously very lazy people, you know, we don't like to be too busy if we can help it. So what we want to do is start conversations, get ideas from the community about other features that may be missing from OpenStack as it stands and how we can fix that, ultimately, as a community together. So it's a bit late in the day. I'm sure everybody's starting to feel a little bit tired, so we're going to do some stretching exercises. So by show of hands, hands up if you use OpenStack on a weekly basis or a daily basis. So almost everybody, cool. Okay, hands up if you operate an OpenStack cloud. Okay, mostly the same people. Hands up still if it's public cloud. Okay, great. This is just the kind of room that we wanted. Any upstream developers here? Okay, good. Excellent. These are the people we need. Cool. Okay, so the outline for this presentation is that we will talk to you about very important stuff for about 30 minutes and then we finish off by doing some questions. And yeah, last year, me and Sean met at Summit in Barcelona. That was the first time we met. And it became pretty clear, pretty quickly, that we were doing about the same job on a daily basis with plugging the gaps as we express it, what we feel are missing in OpenStack for us as a public cloud provider. So that could be for performance reasons. It could be missing features or just things that don't work the way that we want them to work. That would be good for us. So this is how the world looks like that we created, right? Yeah, so we have two layers here. We have the sort of, well, bluish layer here. This is OpenStack as you will recognize it. And then we have this red layer of magical proprietary code on top of it that gives us the abilities that we need to run a public cloud business out of this. And without this magical layer, we'd actually have quite an impossible task of running OpenStack as a public cloud business because it just, when we first started getting into this about three years ago, it just wasn't all there. So that's how I started working with OpenStack. I got drawn into figuring out how we can get this up and running as quickly as possible. So rather than actually get involved with contributing code directly to OpenStack, I was like, hey, I know Ruby. I'm gonna write some Rails applications and do all of this stuff. So it's worth mentioning that when our users interact with our public clouds, they're not necessarily just doing it through the magical proprietary code stuff. They do actually have access to the APIs as well, and that's all exposed and modified. But for things like user signup and some aspects of billing and quota assignment, we just had to have this magical proprietary code layer. And this is where we live. This is where we spend most of our lives.  We know because we've read it. It doesn't benefit from the love and care of being out in the open source community. There could be bugs in it for a long time before even we realize it. If we get hit by a bus, that's bad news, because suddenly a large percentage of people who know this code will disappear. So it's not really an ideal situation for us to be in. So before we start talking about how we wanna change things, we wanna just do a little bit of information on how we think running a public cloud might differ from running a private cloud when it comes to OpenStack. Yeah, it goes without saying that it's a difference between running a private cloud and a public cloud, and a different challenge. So yeah, what difference make it necessary for us to create this? Do you say crappy code? Yeah, that's one word. Propetarily, at least, yeah. So do you have the...  Do you have these in the U.S., self-service gas stations? Yeah, you have, good, good, good. So in theory, everything should just work automatically, right? That it's like the same in the public cloud. Users should just be able to come along, sign up, consume some resources, hopefully a lot of resources, and pay the bill, hopefully as well. And the reason for that is because it scales, right? Yeah, it's the only way it couldn't work. Oh, exactly. So to be able to keep up with the virtual demands that comes along with this, we carefully need to add physical capacity along the way. So due to that, we care a lot about quota, usage tracking, resource management, and those kind of things. And yeah, it's probably one of the most, yeah. It's things that we could care about the most. It's something that worries us, I guess, making sure we have enough quota. Exactly. So we also get these weird and unpredictable workloads. We don't know what people are gonna be using our cloud for because it's public, and if they're paying, then as long as they're doing nothing illegal, they can do what they like. So we don't know what's gonna be running on the VMs. We don't know when there are gonna be spikes in demand for us, or how big those spikes are gonna be necessarily. We do have relationships with some customers where we have some idea of what they might be doing, but generally every day is a surprise for us. And this is my least favorite part, really. We have to stop people signing up and abusing the service.  And it's tricky, really, figuring out what's actually going on the cloud and what people are using it for. We can do some things in the signup process to reduce the risk. There are organizations who you can send IP addresses and emails to, and they'll give you some kind of risk score. But that doesn't always work, and it's hard just keeping tabs on what people might be doing because they might start out legit, and then suddenly start trying to sneak in some spam along the way. So we need to stop this happening for good court management, and also just, because we build people once a month. We don't want them coming in and using the system for a month, and then suddenly they're not paying us anything because that card's broken. We get quite a lot of these every week, and it's really annoying. I still don't know how to fix this, but it's something that takes up too much of, certainly, my life having to buy YouTube ads. Yeah, pretty much, yeah. Yeah. So before we get into the things that we think that we are missing in OpenStack, it's worth mentioning some good stuff about it as well. Since we use it to, yeah, it's our bread. So, yeah, OpenStack is mostly a very good platform to run a public cloud with. It's very easy, actually, and that is for several reasons. It's a very large toolkit of very cool cloud features, as you said, very large toolkit, and yeah, it makes it possible to us to run a cloud that evolves together with the users and their needs, and don't fall behind and become obsolete in the end. So, transparency. This is a great thing if you can get to the code and the code is broken, then you can fix it. Half the time, if we realize something's broken, it turns out it's already fixed. It's just in a newer release than what we happen to be running. Personally, I think Garrett is a little bit off-putting for a lot of people, but ultimately people, people start using it, they get used to how you contribute through it, and once they realize that it can be done, okay, it's not GitHub, but it works, it's pretty much fine, and if people are technically competent, they can get in and get their hands dirty and start fixing things. What's the next thing we're gonna talk about? Oh, yeah, DevOps tools. So, this is the kind of thing that makes our DevOps team happy because there are so many deployment options and good automation tools for this. You can deploy OpenStack with Puppet, Ansible, SaltStack, Docker, Packer, all kinds of other things, more things than I care to really pay attention to. There are a million different ways to deploy code these days. I think we jumped away. No, no, no, it's all good. Yeah, OpenAPIs, that makes it possible for customers to manage the resources and from any applications that they want to, even if they have a proprietary code in their system. Magical proprietary. That's the tutorial. And there's no vendor locking. You can easily choose from any cloud out there with the same deployment code that you have to where you want to put your application. So, for that, there's a lot of good tools like Shade and Clouds.yaml, those kind of stuff. It's very good for that. So, we're gonna kind of get to the reason that we wanted to have this talk now really and that we wanna figure out, well, what is missing? What do we think isn't an OpenStack but should be and how can we get it there? So, for me personally, this is the really obvious one and this is how I started getting involved with OpenStack. So, has anybody ever seen the film, The Hobbit? Yeah, a few nodding heads. So, we recognize this door, this magical door in the side of the mountain. So, it's not something people consider too much when they're running private clouds but if you're running a public cloud, there is no door in OpenStack. People can't just walk in and start getting involved because they don't have any credentials. So, if they have no credentials, they can do nothing. So, I was first hired to effectively build a door. It's set on top of Keystone. It had its own Keystone admin account where it could create users and then people would arrive on a page, put in their credit card details that would go via Stripe. We'd tick it off and say, yeah, okay, these are valid payment details and then they would be able to get into the system and do wonderful things with OpenStack. So, I don't see why we couldn't do this in Horizon and Keystone. I don't know if anybody's tried in the past. I've asked and no one's really given me much of an answer but I don't see why we couldn't have some kind of sign-up flow mechanism where you could kind of pull out to things that you wanna tick in the journey through to saying yes or no when somebody signs up online. So, maybe you wanna integrate with Stripe in some way. You wanna integrate with MaxMind for fraud prevention, whatever other checks you wanna have along the way but basically I don't see why they shouldn't be a way for people to arrive in the middle of the night and sign up on OpenStack without somebody answering the phone or responding to an email when they asked to get in. So Tobias, do you wanna talk about some, is this the quota stuff? This is the domain level stuff. So, one thing that we think would be really good for us is to have better domain level support. Policies at domain level so that we can let users do all the admin stuff at the domain level. For example, project management, create products, delete products, domain-specific IP pools. It could be maybe authenticating to all the different, all the products that you have inside your domain at once. It would probably save a lot of load for Keystone. Which will reduce a lot of the proprietary code as well that we have today. A lot of the code is deleting products and that kind of stuff. Also, quota management at the domain level. It would be much easier for us if we could just hand out some amount of, yeah, quota to one domain, one user, who can split that between the products he decides to create. Today we do a lot of, yeah, quota management behind the scenes for users. And we don't want to do that. We don't want to do other fun stuff instead. Yes, a lot of interesting problems. Yeah. We have been to a few sessions together here and we also realized that some of this is coming in, I think it is in the next upcoming release of Queens in October. Yeah, Queens is introducing hierarchical quota support in Keystone. That's going to make life much simpler for us. Yeah, it will for sure. And we also have the orphaned resources. Today, if you delete a project, all the resources that are behind that project will still be allocated in the system. So this is something that we saw with medical proprietary code today. And yeah, a couple of scripts will be needed for that. And that's nothing we want to do either. And it's also something that has been discussed in Keystone as well. So hopefully we will see some results there as well for us. This picture of Oliver is because these are orphaned resources, by the way. So when you delete a project, suddenly all of the cloud resources that belong to that project still exist. But you've got to clean it up because it's taking up capacity on your hypervisors and so on. I want to complain about Cilometer. It does look like the telemetry project is going in a better direction. And I'm assured that it's much more scalable these days.  What are we, Nick, around the metaca level release? For Cilometer? Yeah, where are we? Kilo, okay. So we've got this backed by Mongo, which was what we were advised to do several years ago. Pretty decently sized clusters. The ones that we're running data-centered, we've got eight CPUs, 32 gig of RAM, SSD. And the ones over at CityCloud, I think are... It's a little bit bigger. I think it's 24 cores and 128 gigs of RAM. So these are sizable machines. They have some good horsepower to them. And good disks. Yeah, so any guesses how long it would take to return if you query it for one day's worth of public cloud usage data? Any guess? No, nobody. Well, that's good because we don't know either. He's a little bit of pseudo ruby code of me just getting samples from Cilometer. And this is across everything. This is instances, volumes, images, neutron stuff, everything. Just querying it for a slice of two hours worth took 25 seconds. Single instance? Single instance usage? Usage for a single instance? No, for everything. For any event that's happened and been recorded by Cilometer for two hours, 25 seconds. For four hours, we're up to 44 seconds. And for eight hours, the gateway timed out. So we have to do some horrible things in just taking chunks of data and then aggregating data off that and then we can actually turn that into something we can put into a billing system. Big problem with this is we can't give it to customers because it's just too slow. And if too many people use it, then the whole thing grinds to a halt. It's just, it's never gonna work. Hopefully we can fix this. I mean, we're paying close attention to what's happening in the telemetry projects. And this is the kind of thing that our bosses really want us to get right because if we can't build people, we don't get paid and people get sad when it happens. The customer does really likes when we bill them right as well. Yeah, they do. They don't like mistakes. So yeah, hopefully this is all getting better. I'd like to see this exportable in some kind of common formats that you could feed into invoicing and accountancy programs. I'm not sure if there are any standards, particularly for that, because I haven't started looking into it in anger, but it would be nice to just feed that into, say, your financial force or whatever it is that the business has decided would be good for billing people with. To buy it? Yeah. As I said earlier, customers would like to consume any cloud anywhere. And they would, yeah, it should be easy to switch between clouds. And today what we see is that different clouds needs different parameters put them into the authentication part. So when you're authenticating to Keystone, some clouds, you will have to put in your product ID, your user ID, your domain ID, and for in some other clouds, also running OpenStack, they don't accept those kind of things. You need to specify project name instead, and or tenant ID and product ID to be able to authenticate correctly. Maybe all of these clouds are doing it wrong, I don't know, but it creates a hassle for the customers when they are trying to consume resources in different clouds. So what we would like to have said with that is that some kind of more generic structure for authenticating to Keystone would be awesome. Actually, it makes the world easier for our customers, at least. And yeah, security. With this slide, I don't want to say that OpenStack isn't secure, that's not about it. The security team is doing a great job with the vulnerability management team. And yeah, it's good. And so, but we are missing a couple of features here. And one of those are two-factor authentication, together with Keystone, that would be great to have, and also a decent VPN solution. Yeah, because of the one we have right now, VPN as a service is left to die. And interesting session tomorrow about that. And the one we have, the existing one has also been a hassle for customers to work with. And a lot of customers really like to have a VPN solution onboarding into the cloud, and also when they are using multiple clouds and communicating between them. A good audit trail would also be nice to have, logging whatever happens from API point of view with your, inside your tenant would be great. And also better separation of inside the logs so that you can easily separate out all the logs for one tenant. And yeah, maybe even expose those logs to the customer, if it's not, if it doesn't contain too much sensitive data. Okay. So how do we close the gap? How do we fix these problems? So the answer is fairly straightforward actually. In the last three, four months, about four months ago, the public cloud working group was established. We're still figuring out what our agendas are and what work we wanna get done and so on. But the formation of this is just bringing awareness to the community of things that OpenStack needs to have to better support public cloud providers. So... Yeah, we're looking for more involvement. We are, yeah, this is kind of a call to arms really for anybody else who's in a similar situation to us and doesn't wanna keep duplicating workload and living in the magical proprietary code layer forever more. We want you guys to get involved, tell us what you think is missing if we've missed something that's important to you. And hopefully we can have some discussions about people actually developing solutions to solving these problems really. Sorry, I think I just talked through your slide. Oh, it's okay. You did it well. It may not be obvious, because we're that good, obviously. We didn't practice this at all. Tobias lives in Sweden and I don't. So sorry if we're a little bit rough around the edges today. So yeah, we wanna take our proprietary code and start porting those features into existing OpenStack projects. I'm not suggesting we form a new project because I think there are way too many OpenStack projects anyway, but bringing things into the core projects like Keystone and Horizon and Nova and Neutron and stuff like this. So there is a little bit of a problem that we found so far in that most companies offering OpenStack public cloud are small to medium enterprises. They might not be able to sort of get the money aside for paying for a full-time upstream developer to build features that suit public cloud providers. So I don't know, there might be a few ways around this. Maybe people could give 20% time to one developer and they could spend one week a month doing this sort of thing, or maybe one day a week, depending on how they prefer to work. Upstream Friday. Upstream Friday, absolutely. Pizza and beer. That's the way I get most things working when it comes to changing things. So yeah, if you work in a company that's interested in doing this and talk to the right people and hopefully many hands can make light work of these changes. Cool, so we're 25 minutes in. We're a little bit faster than we planned, but we're up to question time. So if anybody has questions, ideally come to a microphone, but if you're too far away, just shout it out and I'll repeat it for the video camera. Yes. So for the first problem you just mentioned, I think it's about send up, right? So you mentioned there is no way for OpenStack as a public cloud for customer. So we have an internal product named StackTask so that you can, external customer can send up and after you send up, you have the project manager role and you can invite more project members into your project. And we are going to open source it in the next couple of weeks, I think. And we are using it in our production environment. Ah, sorry. We are a public cloud based in New Zealand. Yeah, we're using OpenStack. So, and as for the COTA policy stuff, I think it's happening, but it's very slow. So don't expect it to happen very soon. For example, in GANs, for now, I'm still a cloud member of GANs. So for GANs, GANs is still using hard-code COTA, not like NOAA, Cinder, and Neutron. So you can expect it's still very slow to get a global COTA policy. And as for VPN, as a service, for now we have one developer working on that. So it shouldn't be dying. Okay, good to hear. What was the name of the project you're open sourcing? For now, internal we call it StackTask, but it's going to be released with another name. We will send an email in the OpenStack dial, mail list, and it could be operator's mail list as well. Excellent, cool. And as for Celerometer, we are running into the same problem. It's very, very slow. But as for billing, if you want to just use the metrics from Celerometer for billing, I assume you will have another billing system to get the metric samples from Celerometer. For example, per hour or for two hours. So it should be fine, but if you want to use Celerometer for alarm, another option is just deploy another Celerometer, but only set a worry short TTL if you are using Mongo. Just set it as a worry TTL so that after you got the alarm, the metrics will be deleted by Mongo automatically in a very short time. And as for billing, we also have a project named distil. It has been open sourced yet, and it can help you worry easy to map to your backend ERP. I think if you are using OpenStack as a public cloud, I assume you have an ERP system to manage your customer and billing invoice, all the stuff. So for our project is named distil. In distil, you can create your ERP as a backend so that you can map in, you can export your invoice quotations and even credits to your customer because I assume you have already got some TKs asked. I want to see all the invoice history by API, and I want to see my current balance of my credits. How much am I paying this week? That sounds really useful. I think there's a fishbowl session right after this and it's downstairs, I think in room 103. 102, I think it is. If you can come down and talk to us and tell everybody about that and we can write that stuff down in an etherpad that sounds really useful. Yeah, yeah. More questions? One more. You guys were asking about the public cloud working group and what to do next. By the way, disclaimer, I work on the same company of that guy so I appreciate that we're taking too much time at the microphone here. The only thing I would like to say is today I went out for lunch with the guys that run the internet cloud and also VH and just realized how much overlap and how much time we're spending exactly on the same problems. Exactly. So a good start would be, let's just get the people that are running this public cloud together. I can write a little list and say, hey, if we are all fixing the multi-tenancy problem on Trove or the workflow for signing up, creating something that allows people to define a workflow there, maybe we can share this precious development resource that we have and say, okay, we're focusing the two people on this one, the rest of the people on this one and spread the load a little bit. Absolutely. I think that overlap is not very productive. So that would be a good start. The work group doesn't need four months to define what it will do. We have a little list of things that need action right now and a few developers that each one of us can contribute. So maybe that's a good place to start. Definitely. Cool, thank you for that. Thank you. More questions? Yeah, go for it. We're good for another 10 minutes, according to my watch. Okay. So I'm just kind of inspired by watching the VIO presentation, VMware Integrated OpenStack and wondering why, why they would do it. And the reason they seem to be doing it is just for simplicity. What they do is they go in there with a sort of take some of the chaos of OpenStack, if I may use that term, and they frame it off and suddenly make it a lot more approachable. And then I couldn't help but think of Linux itself and how the more technically difficult the problem was, the more Linux seemed to attend to it during its developmental years in the 90s and the knots. And the more user friendly side of it, the gooey side of it always seemed to stumble. And I wondered, I don't think they're any more technically difficult. There's probably something about the community, the economy of the system that sort of leads to this strange development where the really tough parts in the core, you know, the symmetric multiprocessing and so on are dealt with exceptionally well. And then outside on the edge, you're left with a gooey that, you know, half the time you can't comprehend. And I wonder if we're not seeing the same, you know, when you're looking at laying out a bit of a future for a public OpenStack, if we're not seeing the same kind of pattern development, if we are, what do you think is behind it? So you're, just to try and understand that in its depth, you're saying that OpenStack behind the scenes is very complicated, but the interface that we put in front of it is not good enough, so. Well, yeah, and I know I could get nailed to that, probably. But yeah, sure. It's fine, I mean, a lot of people don't like Horizon, for instance, if that's what you're referring to, but. For example, we created our own control panel, totally, because Horizon wasn't suitable for our customers, too complex for them. And yeah, we had to create our own. Yeah, but just the install and the update, I mean, if somebody is a relative newcomer to it all, and sort of over the last year is, you know, tried to expose and understand and so on, what I found was it just, it reminded me really of circa 1995 trying to install Linux when we had Linux Clubs. And it was, and I always marveled at how Linux was so brilliant in one way. I mean, Microsoft was on the run and the majority of web servers today, you know, Microsoft lost that market because the system just works better. And this technically is more difficult in many ways than just facilitating a front end that would allow more people to get into it more quickly and maybe move it forward more effectively. It'd be good to discuss what we think that would look like, how we could improve on what we already have. I mean, personally, I don't think Horizon's too bad, but I guess that comes with a lot of time and domain knowledge and I'd probably take it a little bit for granted. I'm sure it puts a lot of people off interacting with it. But yeah, it'd be good to define what a good user interface would look like. Absolutely. But still kind of tricky because OpenStack does a lot of stuff behind the scenes and you have to try and plug it all together. So, no, it'd be good to discuss it some more anyway. We'll talk about that in the fishbowl session. Any more questions, please? Good afternoon, I'm Antonio. I work in America Mobile. And we're in the process of building a public cloud. We have built others based on being where an open virtualization and so on. But what we face with those products is that we have to wait a lot to get new features and every time it's very expensive, right? And now, going into extending the cloud with the features missing with OpenStack, which was an option, CloudStack, still they're an option as well. We are entangled in the discussion of which distribution should we use? Because at least in Latin America, there are not so many developers participating in the community and the skills are expensive and are hard to find with. So, first question would be, so that was a bad one, sorry. So, is there any recommendation for a public cloud OpenStack distribution? And the second would be, do you have your own developer teams? Do you maintain it by your own or do you depending on one after the party? Do you not go first? We use the one that comes upstream. We have early use REO as well. I guess it depends on what you are looking for. If you're looking for getting features quickly, for example, you should probably look for using Vanilla. Basically, that's the biggest difference, I think. Time to market of new features. That's my experience of it, at least. Yeah, we run Ubuntu and we just get, we use the puppet, not libraries, puppet modules for deploying OpenStack. So we just take whatever release we wanna deploy and when we deploy it that way and then we don't get anything sort of vendor specific beyond that. And if it breaks, we have a team of, how many people Nick now? Eight, seven, we have seven DevOps and developer technical type people who can get their hands dirty when it comes to OpenStack. And yeah, some days we have to just really dive into the code and scratch our heads and we have been in positions in the past where it's hurt us because it's been broken. But we've always been able to fix it. I mean, that's the great thing about being able to get to the source code. And so we'll put a fix in place and then we'll try and get that upstream. So that's the way we prefer working because we love details and we love being able to control everything. And that's worked pretty well for us so far. More questions, two more minutes, right? Two more minutes, no? Okay, so we've got the link here for the public cloud working group. If anyone wants to get involved, we are on Free Node IRC and the OpenStack public cloud channel. These are our nicknames if you wanna send us messages or anything. Yeah, as we said, we're having a session shortly in one or two. Yep, downstairs. And on Thursday, there is another public cloud working group session. Can't remember where or when that is, but it is on the summit schedule. I think it's around lunch somewhere, cool. So if you wanna have a conversation with us, do come and talk to us afterwards and thank you very much for coming along and listening. Thank you.


I'm Kunal Sen, the director of UNE Wider, and I'm one of the three editors of the book. The core policy challenge that we have right now in the Estastro-Libu-Delupengol Agenda is inequality. There's been a lot of work on inequality from very different scholarly traditions, but what we don't really know much about is inequality across generations, and that's social mobility, and that's exactly what this book is about. So far the focus has been on making sure that households cross the poverty line. The focus on social mobility goes beyond this because we are lifting the perspective to also ask the question about what happens to those who move above the poverty line. Are these just marginal changes or are there more substantive changes? What I discovered to my horror was that while many people had moved out of poverty, they just moved that much above the poverty line. Why not further up? For us, the editors and the contributors to this book, this is not enough. We're also interested in understanding and trying to measure and document the underpinnings of more substantive progress. Why is social mobility so low in developing countries? I asked young people in high school or college, what do you want to be when you grow up? Young people could not see things like becoming a hedge fund investor, an ace athlete or a professor. All they see are the low paid careers that exist within and around communities like theirs. At the same time, there may be many impediments that are driven by social structure, by inequality across different social groups, by gender, by race, by ethnicity. But if you see the work that's happened in social mobility, we tend to see very little work on developing countries. This book is going to be important both for future research and for future policy about social mobility and inequality reduction. UNEWIDER has had the history of setting the agenda on critical issues in research and policy. This book follows in the same tradition. We want to inspire research that can essentially help us understand this complex issue.


This is Stockinmer 10646. We are here at Summit Automotive in Fond du Lac, Wisconsin. You're new and used ram and heavy-duty truck headquarters. Today we are checking out this super clean 2017 ram 2500 crew cab short box. This truck has the 6.7 liter Cummins diesel. It has been fully safety and inspected by our service shop. Has a fresh oil and filter change. All the fluids have been checked and topped off. And this truck is 100% ready to go. Brilliant black crystal pearl is the color. We shoot all of our videos in 1080p, 60 frames per second. So if you have HD capabilities on your computer tablet or smartphone device, turn them on right now because it is definitely your best way to ensure the quality and condition of the vehicle before seeing it in person. And if you want to check out all the photos on this truck in the upper right-hand part of your screen as they link right to our website, click that and check us out there. It's going to come with a 20 inch painted alloy rims. Part of the Bighorn black package. Has Falcon Wild Peak LT27565 R20 tires with right around half the tread left on them. The frame and underbody is an exceptionally nice clean condition office. Didn't see any dents or dings on that front fender. You get the blacked out headlight bezels. The painted front bumper part of the Bighorn black package. Didn't see any dents or dings on that. And the hood is in really nice shape as well. I didn't see any dents or dings on that. Passner side front fender is in nice shape too. And the passenger side rim no stuffs or scrapes on that. Note that you also get the blacked out Cummins and Ram 2500 logo. Part of this Bighorn black package. As you go down this side of the truck you can see just how clean that body is, how reflective and mirror-like that paint is. We take these HD videos so if you are far away or even if you're close by and just cannot make the trip down but you're still interested in purchasing the truck, you can see the truck, hear the truck and have confidence in the vehicle that you're looking at before you even get here. So when you do get here there's absolutely no surprises. And that is why we offer the Summit Auto Fast Pass option in the upper right hand part of your screen. It links right to our website where you can apply for financing, get pre-approved, see a payment code, even appraise a trade off from the company on home. Has all the remaining factory exhausts, the frame and underbody is an exceptionally nice clean condition. All the lower rockers look really good and the step bars have been powder coated there. So that's pretty cool. Back rim no scuffs or scrapes on that. And the back tires have just about as much tread as the front tires. Maybe just a hair less but very very close to the front tires. As we come around to the back of the vehicle the rear bumper is in excellent condition. I didn't see any dents or dings on that. The tailgate is in excellent condition as well. Notice the blacked out 4x4 and Ram logo part of the Big Horn Black package. It has a LineX spray and bed liner. I consider LineX probably the best bed liner you can buy. Has the LED bed lighting and I didn't see any major dents or dings in the box. Tailgate shuts nice and solidly and that is where your backup camera and locking tailgate is. As you go down this side of the truck just as clean as that passenger side, see just how reflective and mirror like that paint is. And for full disclosure this back rim no scuffs or scrapes on that. Down the rest of the side of the truck very very clean. Didn't see any dents, didn't see any dings, didn't see any paint imperfections. Very very nice. Step bars look good. Has the heated fold-out tow mirrors. They fold out like so and they also power fold in which I'll show you in just a second here. And now it's like showing both sides so that you know that both sides are working good. Inside the Big Horn package gives you the gray cloth interior. There are no rips, there are no tears on these seats. Both these seats are heated. You do get a power driver seat, lumbar, factory floor mats throughout, power pedals, audio controls in the back of the steering wheel and auto headlamps. Power windows, power locks and power mirrors and you can see that door trim is in nice condition. As we hop inside the truck here you can see that this one has 44,658 miles. You get a digital speedometer outside temp and compass and a clock on there as well. You can change all those corners to read whatever you'd like. Has the heated leather wrapped steering wheel cruise controls on the right, Bluetooth and information center controls on the left and then you do get audio controls on the back of the steering wheel. It comes with the six speed automatic transmission with the optional tap shift, the 8.4 3C radio, AM, FM and Sirius XM radio capabilities. This is also where your backup camera shows up and you can do your heated seats and heated steering wheel there as well. You can also do your climate controls right here too. Notice you do get dual climate controls and if you don't like using the buttons up there you can do the buttons and audio controls down here. The other cool thing about this radio is if you wanted to add navigation to this radio you definitely could do that and we can do it right here at our shop. It's a $595 option to get that added to the radio so talk to your sales professional for more details. Turned out four wheel drive, factory brake controller, heated seats and heated steering wheel buttons, stability control, tow haul, factory exhaust brake and your backup parking sensors you can turn those off and on. 115 volt, 150 watt plug-in. You do get a cell phone holder and two cup holders there. Passenger side seat and floor mat are in excellent condition. I didn't see any rips or tears on that. The headliner is in really nice clean condition. I don't think this truck's ever been smoked in. Smells very clean inside. You do get map lights, power side rear window and your home link buttons for your garage door security systems and lighting systems and finally get 911 and assist options in the mirror. Those are features of the vehicle that you never have to pay for. We'll take a quick look at the back seats and then we will check out under the hood. The back seats are just as clean as the front seats. No rips or tears back here.  Power slide in rear window with the built-in rear defrost. Very very nice condition back here. These seats fold up for extra storage. You also get a load floor here. These are nice if you got pets. You can put both sides down. They got a nice flat place to lay or if you got muddy boots or tools you can put them on there and you're not going to get your seats all dirty. There's also storage underneath there. Also if you're hungry you can put both sides down. You got a nice flat place to put your shotguns. You have removable bins for easy cleaning. You can put ice and drinks in there.  And the back door is nice shape and you get the child safety locks on the back doors. Low rockers of the doors look good and we'll take a quick look under the hood. I would personally like to thank you for checking out the video today and hopefully from this HD video you will be able to tell just how clean this truck is all the way around inside out. Under the hood we have the 6.7 liter Cummins diesel. Engine bay is very clean. Runs very smooth. This truck has been fully safe and inspected by our service shop as a fresh oil and filter change. All the fluids have been checked and topped off. And this truck is 80% ready to go. I would highly recommend it from a quality and condition standpoint. And to see more pictures of this truck or one of our other 450 new and used cars, trucks, SUVs, minivans, Wranglers, half-tons, three-quarter-tons, one-tons, you name it we got to go to our website www.summitauto.com full pictures and descriptions of every single vehicle from two locations all at summitauto.com. If you'd like to check out more HD videos you can go to youtube.com slash summitauto. Remember to like, subscribe and share on this video and all the videos that you see there. In fact in a second you will see a link to subscribe to our YouTube channel on your left. A link to more Ram 2500 truck videos like this one on your right. If you have not been to our website on the bottom link to this vehicle on our website. Click those check us out and we really look forward to helping you with this super clean 2017 Ram 2500 crew cab short box big horn black with the Cummins diesel. Thanks again.


Safety and efficacy are important is how we translate things from animal studies and from Petri dish studies to caring for our patients and improving the lives of our patients. This whole family, so to speak, is important and rubbing up against the people I rubbed up against in fellowship and then that we see at the meetings on a regular basis is important. Having a collaborative scientific event is very important too because for these translational steps you have to have big populations of people and so we're all going to have to work together in order to answer a lot of these questions. We need collaboration between sites. We need large numbers of in the hundreds to two hundreds to three hundreds to answer some of these questions and the key to getting that data to getting that information is collaboration between sites and so it's always fun to to see my mentors again to still learn from them but then also to strike up ways to collaborate too.


Everybody welcome back to another premier pro tutorial this one. I'm gonna show you how to do basic zooms in about 60 seconds I'm gonna show you the different types of zooms as well. What am I talking about? I got this guy vacuuming here start the timer and Presto we just zoom in on him just like that. How did I do it? Well? Let's start from scratch Okay, there we go. I got some footage drag and drop it in it'll create a sequence now I'm just gonna go ahead and stop it at about five seconds or six seconds So it's a little slower a little bit shorter. Okay. Good stuff Now I want the zoom to start at one second right there So this I've got the playhead at one second and now I'm going to go to my effect controls panel And I'm gonna go ahead and click on this scale stopwatch very important And then I'm gonna drop that little triangle down as well Just so you can see what I've got going on here And then let's say by three seconds is when I want it to be fully zoomed in so at the three seconds So this is a two-second zoom and of course you can go ahead and do it as you see fit I'm just gonna click the step forward. So I'm on three seconds. There we go. And then presto I'm gonna zoom it in like this and it's gonna go up to about let's go to 200% so we've doubled our zoom Okay, here's what it looks like right now. I'm just gonna go back to the beginning here Let's go back and hit spacebar zooms in That might be all you need Just a heads up if you want it to be a little bit more nonlinear if you want it to speed up and slow down kind of like you've got a Like a ramp on it a speed ramp you can click on these little key frames here under scale You'll see I've dropped the triangle down now right click on it and you'll see here that you can click on ease out and Then on this one you can ease in so you're easing out of the first one I know it sounds a little backwards, but this is so it comes out It's coming out of the first frame and it's going in to the next key frame And if you do that and you look at the little graph here You'll see that the velocity goes up and then it goes down on the zoom both are great ways to do it That's all you need to know guys. Thanks for watching ton more stuff coming up. Stay tuned


Hi, everyone. Yeah, so good morning. We're excited to start our sharing today, and I hope everyone is settled down and make yourself comfortable. So we're excited to start our sharing today on how we co-locate Hadoop Yarn jobs with Kubernetes in order to save massive costs on big data. My name is Irvin, and with me today, is my teammate, Hailean. Yeah. Yeah, and we're both platform engineers working in the engineering infrastructure team at Sharpie. So before we start, I just want to give a quick introduction of our company. So we are from Sharpie, an e-commerce platform that operates across multiple markets across the world. Today, we are the leading e-commerce platform in Southeast Asia, Taiwan, and Brazil. We are also the number one shopping app in these markets by average monthly active users, as well as total time spent in app. We continue to grow and scale so we can build on our strong brand recognition across the region. So why did I just share all of this with you guys today? So as an e-commerce platform, I cannot understate the importance of robust and scalable infrastructure to support our fast growth and large user base across the world. We have completely embraced Kubernetes within our company, running hundreds of thousands of clusters, sorry, hundreds of thousands of ports across tens of thousands of nodes within hundreds of clusters on more than 10 data centers spanning across the globe. As we support the growth of the business, we expect these numbers to keep increasing in the months and years ahead. So now that introductions are out of the way, let me dive right into the problem that we want to share with you guys today. So one of the largest and most important days at Shopee are what we call campaign days. We run and operate several large campaigns every year, including the 9-9 Super Shopping Day in September, as well as the 11-11 Big Sale in November. As you might expect, supporting an e-commerce platform poses several unique and difficult challenges for us. In order to support millions of active users on our platform, we need huge amounts of compute resources. Large numbers of users tend to log on to Shopee at the same time, since many of our larger markets are in the same few time zones around the region. During low volume periods at night, we also experience rather deep trials in which large amounts of our resources remain idle and underutilized. So to make things even worse, during large campaigns like mentioned and on the previous slide, we also have even higher peaks as what we call what is known as campaign spikes. So during campaigns, the peak CPU utilization can rise up to 10 times as compared to non-campaign days. This is also because the campaigns in our larger markets also tend to start at around the same time. As such, campaign days are one of the most important days of the year for the business. In order to support such a disproportionate increase in user traffic, we need to ensure that our clusters are adequately provisioned with sufficient buffer capacity. To summarize the challenges that we faced, you can see that firstly, as an e-commerce platform, our user traffic patterns are very bursty, since our users are all around the same time zones. On campaign days, the peaks are significantly larger than normal, which makes things even more difficult. And lastly, our users are extremely sensitive to latencies, especially during the most important campaign days like mentioned. When put together, this makes capacity planning rather challenging for us as a company and within our engineering infrastructure organization. So here lies the main problem.  But at the same time, we also found that other departments in our company, such as big data teams, they are also facing rather tight resource crunch from rapidly increasing demand from their users. So is there a way for us to reconcile these two problems together? It'll be a good idea that during periods of low utilization, we could run some low priority workloads that have weak latency requirements, especially if they have predictable usage patterns. This way, we can evict them whenever we urgently need resources to handle large spikes in user traffic suddenly. For example, we can run some low priority batch jobs, big data queries, current jobs or even some machine learning training tasks whenever resources are idle. So let's take a moment to think, how can we do this with what we have in Kubernetes currently? So let me just share a potential solution over here. So using the horizontal pod autoscaler as well as a priority class that you can configure on your pods, we might be able to spot these requirements and let me show you how. Let's assume we manage to schedule some batch jobs represented in yellow on the right. And these have a lower priority class than the prod services represented in green on the left. So when resources are idle, batch jobs would be able to exist peacefully with the prod services since there should be ample amounts of idle CPU resources available to schedule the batch jobs. So let's say there is now a sudden increase, sudden spike in users opening the app at the same time. And the CPU usage has also increased dramatically. Using HPA, we can automatically scale up the number of prod service instance, creating more pods that need to be scheduled. So if there are not enough CPU resources to schedule the prod service, the Kubernetes schedule will start to evict the lower priority batch job pods. At this point, the prod service pods have now successfully been scheduled and scaled up and we can now handle the sudden increase in user traffic. And let's say it's now nighttime and our users are sleeping, the HPA will automatically scale down those pods and since there are now lots of idle resources, any previously evicted batch jobs can now be scheduled again successfully by the Kubernetes scheduler. So did we use this solution in the end? As you might guess, I wouldn't have presented our actual solution so early on at the start of our presentation. It's actually a lot more complicated than that. So firstly, depending on eviction to free up resources is too slow. As you might have seen earlier that user traffic can increase super quickly, especially when a campaign starts. Next, if we wish to support non-Kubernetes workloads like Hadoop and Presto, this is currently not very feasible. If we have to reprovision the entire node, then it is both slow and risky since we have to keep draining nodes very frequently. And lastly, frequent eviction of batch jobs may result in actually result in wasted CPU utilization which does not contribute any real value to the company. As such, we want to present our solution today that we have painstakingly worked on for the past one year. We'll explain how we manage to co-locate low priority Hadoop yarn jobs alongside critical Kubernetes ports in a safe but yet scalable manner. We allow Hadoop jobs to reclaim allocated but underutilized resources in real time while ensuring stability and the performance of critical ports to handle rapid spikes in user traffic. So how did we do this? Let me welcome my teammate, Hylin, who will share more about how we managed to do so. Thank you. Okay, so thank you, Irving. So let's see how to achieve the co-location. So actually, in my opinion, it could be divided into two parts. The first part is the allocation. So literally, it defines how to allocate the prod service at batch service in the cluster. And what the amount of prod and batch services should be scheduled for every node. So to schedule clearly, we need to first monitor all nodes and calculate the estimation on resources. So after that, we are able to allow to do some scheduling stuff according to the strategies. And then the second important part is the allocation, is the isolation, sorry. So to achieve this, we have suppression and eviction. So suppression here means to suppress the batch services in real time and in case of any spike from the prod services. So therefore, it is used as a protection strategy to protect the prod service in real time. And the eviction, it avoids diversion, causing by a long time of suppression because of the burst in prod service. And it enables us the risk scheduling to guarantee the stability of the batch service. So consider this is one of the node. So before we have co-location, there's a couplet with some of the prod service pod running here. So in order to enable co-location with YAN, there are several ways to make it. So the easiest way is we deploy the node manager and make it co-hosted on the node. So assuming that we have a YAN node manager deployed here, so as then we can schedule some batch up here. But then the prod service as a YAN job I will interfere with each other. Therefore, we introduce our Lumia agent which will run on every node to negotiate with the couplet as a YAN node manager. So you might be curious about what exactly the agent could do within the negotiation. So here's a brief overview of its responsibilities. So basically, it takes each in charge of six areas. Firstly, the agent will help to collect the metrics from different source, like the C-group, PROC-FS, risk control, and so on. With these metrics, you will analyze the real-time status of all PROD at batch services on that load. And also, it reports the reclaimable resource to make the scheduler be aware of any changes. So the definition of the reclaimable resource we will introduce later. Also, it utilizes tools like the C-group at risk control to do the isolations. And it detects the separation status of the batch services. And then you will trigger the eviction according to the strategies. And last, you will use the metrics to try to auto recover from some failures, if possible. OK, so after the brief introduction on the two parts of the colocation, so let's go into detail, see how the first part, the allocation, is achieved. So we will continue to use YAN as an example for the batch service. So imagine this is a Kubernetes node. So it has 48 CPU in total. So it makes it simple. We assume there is no reservation, so all of the CPU belongs to the allocatable CPU. So assuming some of the services coming in, the Kubernetes scheduler comes on the CPU request and then schedule those port to this node. So now we have a status of 45 CPU scheduled. So however, imagine now it's at night time, so the user is sleeping. The real CPU usage for the whole machine is super low. It's around 7.2 CPU at this moment. So to make it safety, during estimating the real usage, we smoothly add some buffer. So by some mathematics, we calculate that the estimation on the CPU usage is actually about 12.7. So which means now we have 48 minus 12.7 resulting 35.3 CPU available to oversell on this machine at the moment. So this means that we can reuse this amount of CPU to run some batch jobs. So let's add one more dimensions, because just now we are talking about one moment stuff. So here is a time series for this node. So you will see the black line at the top is a no allocatable CPU, which is 48 in constant. And then the origin line below is a PROC requested CPU. So assuming no allocation happens at this time, so it's also a constant line in orange color. The red line is the actual PROC CPU usage. So we'll say it's spiking and then at turbulent. The origin line is the blue line actually is the estimation value using some of the buffer. OK, so let's take the aerial. So in this graph, the total green aerial is the available reclaimed CPU over time. So after knowing how the reclaimedable resource is calculated, so let's say how the scheduling is impact for both PROC at batch service. So to make it compatible with the current Kubernetes, we reuse the mechanism of the extended resource. So the reclaimedable resource are introduced as new extended resource. So in this page, there are now batch CPU and batch memory accordingly. Let's say at one point, the CPU and memory of the PROC service start to burst. So immediately, the agent will capture the utilization change. It will calculate the new reclaimedable resource. And lastly, it will update to the node status. So as a result, you will see the batch CPU and batch memory is reduced to 16 at 64 gigabytes. So for the Kubernetes scheduler, the update on the node extended resource will soon capture by the informers. So you will influence any subsequent scheduling decisions for the PROC service. For Hadoop YAN co-hosted on the same node, it will slightly be more complicated, at least say how. So firstly, our agent will notify the co-hosted node manager using some customized API for the resource change. So in the example above, the agent will tell the node manager that the CPU is 6 to 16 CPUs. And then the node manager will report its status to the young resource manager. At the meantime, assuming there is a job started, the application master will make a request to the resource manager. So here we assume a request for eight CPUs. So the resource manager check if all of the nodes fulfill the requirements. So luckily, we can find the node A. Even though its resource is claimed as a reclaimable resource, resource manager will still respond to the application master to say the node A is available. So after receiving the response, the application master will be glad to say that a node is available. So you will tell the node manager that the node is wrong. You will tell the application master that there is a node available. And then you will run some batch jobs. So cool. So still here, we managed to allocate the batch service. So wait a minute. Just now what we are talking about is just allocation. So in previous slices, we already allocate in our prod service and batch service in the same node by super greedy. So what if the prod service starts bursting after the batch job has been scheduled and the amount of the reclaimable resource suddenly drops? So under normal scenarios, the Kubernetes scheduler won't even prod based on the real utilization. So this is really scary. To induce it, let's take a look on this scenario. So at the beginning, some prod services with 20 requested CPU total is scheduled to this node. As then, because the node has 48 CPU in total, so by greedy, we allocate 30 cores of the batch job since the real utilization for the prod service is just 12. All of a sudden, a campaign starts. So a lot of users start to use our app. And then the real CPU usage for the prod service jump to 38 cores. So now we are in big trouble. Because of seriously, the poor machine with only 48 cores, it cannot support, which is in total 68 of total CPU usage. So both of the services will be affected. And the user will start to notice that the short P app is not responsible anymore. So intuitively, we need some kind of solution to reduce the CPU usage of the batch job immediately. Once the prod service needs a CPU. So how do we achieve that? OK, I will leave this challenge of expanding this to my teammate Irving. OK, thanks, Hailein. So in this section, we'll look at how we can isolate noisy neighbors. Or in other words, we'll explore how we can minimize the effects caused by co-locating batch workloads on the same node as prod services to resolve the problem that Hailein has illustrated earlier. So before we proceed any further, we first need to have a system to categorize workloads based on the latency requirements. We found that the original Kubernetes port QoS classes were insufficient to represent these latency requirements. And so we extended them to introduce a few new more classes of workloads, or what we call workload QoS. In this table, we show several different workload QoS classes in order of priority. Basically, prod workloads will have the highest priority in the system, and they can unconditionally suppress all lower priority workload classes. So to reiterate, in order to address the problem that Hailein mentioned earlier, we make use of several Linux kernel features so that we can ensure that batch workloads can give up their CPU time to the higher priority prod services whenever they need it. So the Unimide agent component that runs on every node is responsible for allocating many different kinds of resources in real time to achieve this. These include C-group features such as CFS quota, CPU weight, memory limits, and so on, as well as other REST control features, including L3 cache and memory bandwidth. However, in the interest of time, we'll only be focusing on a few examples in today's sharing. In this example, we can see how the C-group hierarchy is typically set up. So on the left, we can have a few ports under the Q-Port C-group folder. From the previous example, let's say the real usage is only a measly 12 CPU cores. And after smoothing and adding a safety buffer, we can round this up to 16 cores. So this actually leaves us with a total of 32 reclaimable CPU cores that the batch jobs can use. And as Hailein has explained earlier, how the batch jobs can be allocated to the node, we now have a new C-group that we call batch just under the C-group root. And we can set the CFS quota, as well as the CAPU set to be 32 CPU cores as computed from the reclabeled CPUs earlier. So let's say there is now a sudden spike in CPU usage for prod services. So when the agent detects this, it can immediately update the reclabel resource to five CPU cores and subsequently update the batch C-group in real time to reduce the CFS quota in CPU set correspondingly so that prod services can handle the sudden increase in user requests. So by changing the batch C-group dynamically, we are actually able to achieve real time suppression of batch workloads in order to ensure that prod always has a higher priority than batch. So what happens if there is a high CPU usage on the node that persists over a long period of time? So this might actually cause the Hadoop jobs to not be able to complete if they were suppressed for a really long time. As such, our agent will also evict batch workloads should their CPU request exceed the amount of reclabeled resources after an extended period of time. So let's now do a deep dive into another case study involving C-group V2 right back. In our classes, we sometimes observe that some batch jobs may read and write files very quickly, causing huge IOPS and affecting latency-sensitive prod services that also depend on the disk. Therefore, we want to find a way that we can control IO limits for batch jobs only in order to prevent them from causing side effects onto prod services whenever they were co-located onto the same node. Conveniently, both C-group V1 and V2 provide interfaces for us to specify IO limits for a C-group. On C-group V2, we can specify the IO max value in order to limit both the IOPS and the bytes per second, or BPS, written to the disk. However, is it really that simple, though? As you might expect, the answer is no. So the main issue is that configuring IO max will throttle both direct and buffered IO in C-group V2. This means that this actually controls the rate of write back of dirty pages for the C-group. So if batch jobs continuously write files faster than the configured IO max, this actually can result in dirty pages piling up over time. And eventually, this will result in a huge memory pressure for the whole system. And as we all know, having not enough free memory is a really bad thing. And this can actually result in prod containers start stalling whenever memory is reclaimed from the prod containers instead. Therefore, naively setting IO max might actually result in an even worse situation than before. So what can we do about this? Oh, sorry. Let me show you first an example of how setting IO max can go wrong. On the top, we set IO max to 100 Mbps for batch C-group. And at the same time, we also have a prod C-group reading 1 KB. So when the batch C-group writes a lot of files, writes a lot of data to the disk, the prod C-group might stall for as long as 46 seconds just to write 1 KB. So this, you can see, actually, it's quite a rather serious thing. So we need another approach to solve our original problem. So rather than letting dirty pages generated by batch C-groups pile up forever, we need to make the batch C-groups sleep once they hit a maximum dirty limit. Since IO max limits how fast dirty pages are flushed, together, we can use the maximum dirty page size per C-group plus IO max together to resolve the original issue. So in order to achieve this, we are currently working with our in-house Linux kernel team at Shopee to implement the first part, to implement patches for dirty page size limits per C-group. And I'd like to pass on the time to Hylin, who will share more about how we managed to evaluate the safety and effectiveness of rolling out code location in our company. OK, so thank you, Arvind. So after implementing and rolling out our code location in our cluster, so even though we reclaimed a lot of resources to use for batch job, we didn't really know whether it is safe for the business. So as such, we want to measure some kind of service level objective, or we call SLOs, so that we can control the race after introducing code location. So the first step is to collect enough metrics. We square metrics from three main areas, so there are no metrics, workload metrics, and business metrics. These three metrics cover the machine status, container status, and also the user program status. With these metrics on hand, we can start to generate useful alerts when the SLOs are violated. What else can we do with the metrics that we collected? We introduce another way to fix problematic machines automatically. So within our agent, we introduce a negative feedback loop, which is so-called performance guards. So the performance guards controllers will read from the in-memory metrics stores. The controller will check if the SLO is violated, and then generate some thing we call so-called prescriptions, which can be applied to same group to do isolation on resources quickly, and without need any human interventions. So after the overall globalese has been recovered, the controller will also detect it and gradually relax the isolation over time. So besides of the self-healing, these metrics can be also introduced contribute to scheduling. So metrics collected from different clusters are merged together into a global Hive tables. We have some Spark jobs running periodically to calculate and analyze the metrics, as in right to an aggregated data store. So the analyze is mainly focused on calculating on the persona for all of the services, including both approach and batch services. This persona statistic will be read by some of the scheduler plugins inside the cluster and to generate more comprehensive affinities. So for example, we can avoid scheduled services, which will burst at the same time to the same machine, and also we can avoid IO bound services to be scheduled to the same machine. We can also influence the scheduling using other business and low-level metrics. Yeah, but because of the limit of time, we won't introduce today. So let's go to the result part. So it's the moment we have been waiting for. So let's see all of the hard work we're putting has been paid off. So let's welcome Irving. Thanks, Hailein. So let me go through some of the results that we have seen as a result of the hard work that we have put in over the past few months, actually over the past year. So after enabling co-location in our clusters, we got some really promising results. Firstly, we were able to reclaim large portions of unused resources that were sitting idle during low periods to provide for Hadoop Yarn jobs and other low-priority tasks, providing the much-needed resources to teams that actually needed them the most. These actually amounted to more than 70% as you can see represented in green, more than 70% of CPUs that were reclaimed and could potentially be reused for best jobs every day. Across the clusters where co-location is enabled, we also tracked the CPU utilization rate across the clusters as a means to measure how efficiently resources are allocated. So when comparing non-co-location clusters and co-location clusters, we managed to improve the peak CPU utilization by up to 4.3 times, as well as the average CPU utilization improving by up to 3.2 times. We also want to ensure that such optimizations did not cause any adverse side effects to the prod services when co-located onto the same node. We compared the latencies for some critical RPC services that were deployed on both co-location and non-co-location nodes, and we observed that the prod services co-located with bad jobs had a less than 5% impact in the P99 tail latencies. So this is actually one such example to demonstrate that our resource isolation is effective enough to prevent bad jobs from impacting critical prod services whenever they need their resource the most. And lastly, we also measured the overall job failure rate for Hadoop yarn jobs that ran on our co-located Kubernetes clusters, and we found that less than 1% of such jobs failed every day, which shows that our co-location system is indeed sufficiently stable, even though these jobs, sorry, could potentially be evicted or suppressed at any time. Now, how does all of this translate into cost savings? So of course, every company's compute costs and they'll be different depending on their cloud provider and many, many different other factors, but let me just make an attempt to make a back of the napkin calculation over here. So assuming we managed to reclaim eight cores per machine in a 5,000 node cluster, we can estimate how much this would have potentially cost whenever, if let's say we were using AWS EC2 spot instances instead. This translates to savings of up to 4.2 million USD per year for every eight CPU cores we claim per node. And this figure can actually go even higher depending on how underutilized your clusters actually are. As you saw earlier, we actually reclaim more than 70% of our CPU cores and also depends on like the total number of clusters you have as well as the original cost price of Hadoop machines. So before we end today's sharing, I'd like to quickly share some takeaways that we learned throughout our journey in designing and rolling out the co-location system in our company for the past year. Firstly, we knew we had to design a system that could scale to multiple clusters across tens of thousands of nodes. We chose to adopt a bottom-up design such that most components could act in a decentralized manner. For example, we implemented an in-memory Lister Watcher implementation that allows us to gain the benefits of using CRDs in our code without potentially overloading a centralized API server. So such a design actually allowed us to reach more than 50,000 write operations per second in a single cluster, which would not have been possible with the native API server of the box. Next, we also learned many ways to mitigate and control the risk of rolling out a system that is complex yet potentially risky to the business as the co-location system. So we knew that modifying low-level C group and kernel parameters were risky from the very beginning and that these risks can and will be vastly amplified when deploying to even just a really small change to thousands and thousands of machines. As such, we have refined and come up with a very comprehensive risk classification system to get coupled together with strict release policies that all developers in the team must adhere to. We also heavily invested in release automation for our project. We have up to four stages in the release process and the higher the risk of the rollout, the longer the entire process, which could take several weeks per rollout. These are all automated with custom controllers and command line tools that are used within our team to minimize human errors. And lastly, we also want to share some several takeaways in terms of observability. We learned to capture as many metrics as possible from low-level metrics like using EBPF traces to capture several kernel functions like direct reclaim latency, all the way to high-level metrics that the business actually cares about. We also found that a good way to measure the effectiveness of our project was to use something that we call the co-location cost. Basically, by comparing the metrics for the same service on both co-location and non-co-location machines, we can actually get a very accurate estimation of the impact that co-location can cause to this particular service. So this actually greatly helps us to evaluate risks and identify problems before causing widespread problems to the whole cluster and to the business. So with that, I'd like to end today's sharing and you can scan the QR code to leave feedback on today's session if you might have and we'll open the floor to any questions. Thank you. Hi. Did you ever test Spark on Kubernetes with your presentation? You showed us combining online workloads with Spark on Yarn. But did you consider running it with Spark on Kubernetes as a resource manager? So your question is, why don't we use Spark on Kubernetes to do co-location, right? If you thought of using Spark or Kubernetes because in your example, you showed us Spark run on Yarn. Oh, yeah, yeah. Actually, we choose Yarn is because in our company, the DI, the data infrastructure team, they choose to use Yarn. That's why we are not able to use Spark on Kubernetes. But actually, we indeed run some Spark inside the node manager. Not sure if this answers your questions. Actually, I can try to follow up on the question as well. So our system is actually able to support both Kubernetes and non-Cubernetes workloads natively. So in this example, we show that we don't actually need to modify our existing teams technologies. They currently only use Yarn. So needing them to migrate to using Spark on Kubernetes could involve a lot more cost and a lot more time invested into a project. So basically, we're able to support, if you saw, we actually have, oh, sorry. So on the slide, actually, you can see that we are able to support both Kubernetes ports using the extended resource, as well as non-Cubernetes workloads. But we need to make a slight modification in the node manager and the whole Yarn architecture, which is basically just to implement a customized API. So I'm not sure if this answers your question. Okay, it's okay. Thank you. Thank you. Are you supporting as a Presto or TrinoDB in that? And also, are you then using HDFS mounted on every node of the Kubernetes cluster? Okay, so your question is whether we mount HDFS on every node inside the Kubernetes cluster? So actually, the answer is no. So we use a remote HDFS to achieve that. As then, also, we are adapting some of the technical RSS for remote shuffle inside our cluster to reduce the IO cost, yeah. Then for the question about the Presto, actually, we are integrating with a Presto worker inside our cluster. So actually, because of the Presto, you have a nature of consuming large amount of memory and because of the latency, it cannot release of the, because it claims a memory using the anonymous, claims a memory as an anonymous memory. So it means it cannot, it's not in page cache, so it means that memory cannot release quickly. So we are using our scheduling mechanism to find a stable memory among the reclaimable resource for to run, to make sure we can co-locate the Presto worker, I mean with stability, yeah. Great talk, guys. I really like the metrics and SLAs that you presented about how you are measuring the efficiency of communities. So I wanted to ask if I want to learn more about how the metrics and SLAs are structured and how it's implemented, can you make any recommendation? How can we learn more about the metrics? Okay, so thanks for your question. So I think we have several kinds of metrics as shown earlier, let me try to, yeah. So actually we split our metrics into three different kinds. So firstly, we have low level metrics that are on every single machine. And we also have workload metrics that maybe it's like CPU utilization, actually cache utilization per C group or per process. As well as some high level business metrics, for example, like latency, several rates that the business actually cares about. So the point we're trying to make is that we actually collect a whole slew of metrics all the way from low level to high level in order to make a much more comprehensive and effective evaluation on how we should roll out as well as to do certain automated recovery use as like Harlin has mentioned earlier. Yeah, I'm not sure if this answers your question. Any more questions? Just to kind of clarify, is this all like internal tooling that you guys have developed? Like it's not out there for general usage. And also does it work with like any batch job that might be running? Yeah, so actually at the moment, this is all internal system within our company. But I think it also depends on quite a lot of the situation in the company. For example, if you currently use the yarn architecture with, and we also need to depend on certain like kernel features as mentioned that need to be developed within the company. So, but that said, actually a lot of the code and our agent is actually able to be open source but I'm not sure if that's in our roadmap for this year in any time soon. And to answer the second part of your question as to whether any batch workloads can be supported. Theoretically, yes, but depending on the nature of the batch workloads, for example, like Trino or Presto workloads, which actually reserve high amounts of M-Mapped anonymous memory, we actually need to make certain maybe more long-term predictions and to provide more stable memory, stable reclaim memory for these batch workloads. So it really depends on a case by case basis. I'm sorry, I already asked a question, but this question actually made me like expand and what I also wanted to ask you is the implementation and like how are the metrics implemented? Are you implementing it internally or are you using any publicly available tools like DataDog or other tools for measuring the data points? Oh, I see. Okay, so maybe for some of the low-level metrics, these are mostly collected from like kernel, basically Linux kernel interfaces, for example, if we're reading from the PROC FS. So those actually are collected by, it's actually the same way that things like NodeExporter actually collects. And for Seagrid metrics, these are also collected in the same way that C Advisor would do it. So for the business metrics, these are actually mostly collected by our in-house observability team. And I guess you can say like it's more similar to what DataDog offers, I guess. Yeah, so I guess that's how we actually split the different kinds of ways of collecting the metrics based on what kind of metrics they are. Thank you so much. Very insightful presentation. Thank you. If you could elaborate on, did you need to do any modifications to the source code of Kubernetes or Yarn? How does your agent interface with these two different schedulers? Okay, thanks for the question. That's actually a really insightful question. So as for modifying the Kubernetes source code, actually we minimize, and actually we don't need any modification to Kubernetes. So our agent actually, in a way, we replace our Cubelert's CPU manager and memory manager. We basically extend all the logic and add on our own logic on top of it. And we basically convert the Cubelert's CPU manager policy to none so that we can actually take over control of the CPU for all parts. And for modifications to Yarn, because as mentioned, for certain non-Kubernetes workloads, we need to adapt certain frameworks slightly. So in our case, we modify node manager just by adding a single API to, so that we can actually change the amount of available vCores CPUs for the node in real time. So that's actually just a very minimal change. Just to add on, so for our schedule system, we are using dedicated scheduler plug-in, which also didn't modify the scheduler code. So inside that scheduler plug-in, we mimic the scheduler cache to collect to our data store and then to cache all of the prediction data and then which can be used for all of the following allocations. Hey, does your agent adding additional latency to the Cubelert and to the node manager at all? I would say no, because actually we are basically acting, we are actually modifying the C group after the container is actually created and ready. So if the C group already exists on the machine, after that, then we would modify any necessary C group parameters. So actually it's not in the critical path of creating a port or anything like that. We actually modify it maybe like less than a second after it's created. So it's actually not in the way of that. For node manager, it's we also don't add any latency to that because we just do a best effort update of the amount of reclaimable resource to node manager whenever we can. If let's say the API fails for whatever reason, we actually can fall back onto real-time suppression like using CFS quota as mentioned, without causing any adverse effects to the prod services still. Thanks for the wonderful presentation. Quick question on this slide. The aggregation and feeding the data into the cube scheduler, are you thinking in the future that you will be doing in real-time? I think it won't be useful in real-time because the prediction actually is used as a long-term prediction. As in, actually we have two kinds of predictions. So actually the agent itself has a short-term prediction which we are introducing in our slides as a smoother for the real utilization. As in, it uploads the estimation to as a reclaimable resource. As in the predictor here, which is used for long-term prediction, let's use some strategies like the Fourier transform. So it will, so we want to find a stable, I mean stable memory or stable CPU for a few hours. So as such, we need a lot of data. And then because we have so many services inside the cluster, it's not possible for us to do it in a very quick way or say in a real-time way. Yeah, as in because in our design of the agent, the agent is designed to, I mean to react fastly to do suppression and adjusting resources, do allocations, isolations, I mean do isolations quickly. So the scheduler itself don't have to take care of those things. Okay, thank you. No more questions? Thank you.


As children, you've often talked about education. As children, are we in a wrong system of education that leads us to be much more involved with the mind and the intellectual pursuit and rather than looking for the larger experience that you're talking about? Largely, there's no education in the world. There are ancillary units supplying cogs to the main machine. Human beings are like fuel for a larger machine that we have created.  See, the immediate needs of a society has to be attended to. Sometimes we have to do it. But to what extent? The fundamental goal of education is to enhance human perception. That is the only goal. But today education means they made an accountant out of you. I'm sure you were a bad one. I was terrible. But I managed to convince people I was good. In my own self, I knew I was terrible. So what I'm saying is, we're trying to make something for something, something that we think is useful. In the larger scheme of things, as life, you becoming an accountant, you becoming an engineer, you becoming a doctor, whatever else, is not of great importance. Socially right now, maybe there's an immediate need for somebody to keep accounts. We need some accountants. We can teach somebody a bit of accounts. But we don't have to go on producing millions of accountants and lawyers and engineers. But today's society we built like this, it's a must, okay? But in the real sense of term, as a piece of life, what is relevant to this life? Education is nowhere near that. Education is telling you your life is not important. You have to serve this society. What is a society? It's just an assembly of people. But no, people are not important. Society is important. Where is the society? I don't see any society anywhere. I see this man and this man and this woman and this woman. I don't see any society. I see only individual human beings. But people see a society because they got a word in their mind. I don't have words in my mind. When I look at people, I just look at people. Can you make all of them one? Each one is a unique creature by himself. So in the very nature of life is such, every life, whether it's an earthworm or a grasshopper or this tree or this blade of grass, just anything. It's only longing is to find its full potential. And that's all it should be. That human being should find their full potential as a life. Today we want to fight a war. We will make everybody into a soldier. Tomorrow we have a lot of sick people. We'll make everybody into a doctor. Tomorrow accounts. Day after tomorrow everybody is quarreling so everybody is a lawyer. What I'm saying is we are trying to make life serve something else. No, the highest manifestation here is life and life itself. Everything else, all social structures are here to assist and serve that. Unfortunately today we are enslaving life to serve social structures. No, social structures must be made in such a way that they will serve life, an individual life, not social life. And education is a fundamental tool, but education is twisted in such a way. All education on the planet if you ask me is fascist. Because it's all about enslaving people to serve the needs of whatever a few people are thinking. They are not even thinking straight, they don't even know. They are also slaves of the same system.