Tissue engineering and regenerative medicine have made significant progress over the last two decades with advances in the development of 3D scaffolds and hydrogels for the reconstruction and regeneration of damaged tissues. These scaffolds and hydrogels can be composed of naturally occurring or synthetic polymers, bio-resorbable inorganic materials, and combinations thereof. Additionally, decelularized tissues and current treatment strategies are being explored for use in tissue engineering. Research into the 3D behavior and multicellular interactions of native tissues has led to the development of in vitro models for further use in clinical trials. Challenges and future prospects of research in this field are also discussed. This article was authored by Sandra Pina, Viviana P. Ribeiro, Catarina F. Marquez, and others.


What a liquid sidechain asks David, looks like it's a private blockchain, but claimed to be having some degree of decentralization. If it did work, what does it mean to Bitcoin community and other sidechain projects, either public or private? First of all, Liquid is a product name. It's a creation of a company called Blockstream. A company that has a number of projects in the Bitcoin space, and also hires and pays a number of core developers, who are working on the Bitcoin Core protocol and the Bitcoin Core client. Blockstream announced this a long time ago, probably two or three years ago. They have hired some of the developers who came up with the initial idea of a sidechain. The sidechain is a separate blockchain that has a two-way peg, whereby you can move money from Bitcoin to this other blockchain, use it in this other blockchain, and then move it back. This uses a trustless two-way peg. In the case of Liquid, the purpose of Liquid, which is the first sidechain implementation by Blockstream, and has been in beta or trial stage for more than a year now, is intended to be like a back-end exchange-to-exchange pipeline for Bitcoin transfers, allowing Bitcoin to move very fluidly between exchanges without having on-chain transactions. The idea being that you have exchanges around the world, and they have to withdraw on deposit Bitcoin, to addresses that belong to other exchanges, and every time they do a withdrawal or deposit, they're not only using up space on the Bitcoin blockchain, but they also have to pay fees in order to do that. Liquid is intended to be essentially a payment channel between exchanges that allows them to move money between themselves directly. Kind of like a Swift network for Bitcoin, one of the things about Liquid is that it uses a federated signing model. It's not a mind-chain, instead it's a signed chain, so it uses what some people call a proof-of-authority, or a federated signing model, where there are some privileged nodes within the system that vet and manage the consensus rules on that sidechain, and they secure that blockchain by signing blocks and taking turns signing blocks. There is a degree of decentralization compared to a centralized provider like Swift, but there isn't the degree of decentralization you have in a completely decentralized and open blockchain that has a proof-of-work algorithm behind it, for example. On the liquid sidechain, what you're actually moving around is Liquid BTC, which is one-to-one equivalence of Bitcoin. You lock up one Bitcoin on the Bitcoin blockchain, and then that becomes one LBTC, or liquid Bitcoin, that can move on the liquid sidechain, presumably at faster speeds with lower transaction costs and without burdening the Bitcoin blockchain. At any point in time, you can move one Bitcoin back to one Bitcoin at a one-to-one exchange rate. That's the basic idea. This is not for end-users, this is not for consumers. There have been some discussions about opening it up to others, but this is something for exchanges to manage very large flows of Bitcoin between themselves in a network that's off-chain. It's a commercial product, so it has fees. As far as I understand, the fees are for the providers of the security on this system. Jerry asks, does the Bitcoin liquidity, I think he means liquid, compete with Ripple, XRP, for use-case scenarios? Does the liquid sidechain, which is the sidechain created by Blockstream, compete with XRP for use-cases? I don't think so. Part of the reason for that is, from what I understand, XRP has a very different model in both its consensus layer, as well as in how the currency is used within the network than liquid. I think people who are interested in XRP are not interested in Bitcoin, even if it is implemented through a liquid sidechain. But there is a spectrum of applications here, whether it is going from payment channels in a lightning network, or just raw payment channels and atomic swaps, a sidechain-like liquidity, or XRP. There is some overlap in some of those applications. I don't think that overlap is enough to make these systems compete directly against each other. I think they differentiate enough that they don't directly compete. Do sidechains inherit any of the security properties of the main blockchain they are connected to, or are they completely independent? If not, how can we trust them? That's a great question, Frank. It depends on the implementation of the sidechain, but in many cases, a sidechain is simply another blockchain that has a way to transfer value from another blockchain. You have two blockchains that are operating in parallel, side by side. You can't really say one is a sidechain of the other, any more than the other is a sidechain of the first. They are both sidechains to each other, but they are both independent blockchains. You have mechanisms which involve cross-blockchain transfers, forms of atomic swaps, pegging, and things like that, that allow you to move value from one to the other. That doesn't mean they are not independent blockchains. Then the question is, what are the consensus rules by which the blockchain is secured? A sidechain is a blockchain, and it has some consensus rules. Maybe it runs with proof-of-work mining, maybe it runs with proof-of-stake, maybe it has a federated proof-of-authority signing mechanism. Whatever the case may be, it will have the security characteristics that these consensus mechanisms bring to it. If you took two proof-of-work blockchains with substantial hashing power behind them and made them sidechains of each other, they would have different characteristics than if you connect a proof-of-work blockchain like Bitcoin, with a lot of hashing power behind it, and a sidechain that has either a very small amount of mining or a different consensus mechanism. How can we trust them? You can trust them if you understand their security model, and you think that security model is strong enough. But they don't really inherit the security of the other chain. There are some exceptions to that, which have to do with merge mining, where both chains are mined together. But those exceptions don't change the rule. Why have sidechains being overshadowed by other crypto projects like Ethereum for contracts and polymath for security tokens? Part of the reason is that we haven't seen a successful and popular implementation of sidechains. Sidechains are relatively recent technology. Many of the underlying requirements, although sidechains were proposed four or five years ago, many of the underlying technology only recently entered into the various blockchains in order to make sidechains possible. I think probably only two have launched so far that I know of Rootstock and just this week, Liquid. But other than that, we haven't really seen broad use, and both of those are still in beta stage. That's why sidechains may not be as popular as completely separate blockchains that have been bootstrapped with their own consensus layer and operate with their own security. I hope that answers your question, Tamara.


Senator, you happen to be out of Moscow and back in London. You happen to be in London. Is it Reagan who happened to see the Queen after Mrs. Gorbachev? I'm very happy to see him.


All right welcome back it's still the breakfast and plus TV Africa you just watched the conversation we had on Good Friday it is a good day but some family members are not actually having a good Friday they're not only having a good time because it's been 80 years and their loved ones are still you know trapped in captivity you know we talked about it when we introduced the show that's Chibo cause over a hundred of them are still in captivity and the parents are saying that the government is not showing genuine interest concerning them it used to be like the major stories it used to be you know on board captions across you know newspapers and television radio stations it used to be a very topical issue it's in eight years right now and it's as though the conversation is actually being muted to some extent not much has been said about I understand that we've had more issues of insecurity and kidnapping in recent times we've had the dapchi kidnapping we've had just not too long ago two days ago so we had you know girls and been kidnapped in a college of health science in Zamfra state it goes on and on but we still have to remember that these girls are yet in captivity even after eight years sadly April the 14th 2014 276 school girls were abducted by Boko Haram from schools I mean the secondary school in Chibok town that's in the North East and then some of the girls if you remember managed to escape while others were released following campaign efforts and government negotiations government will obviously they don't negotiate but that's not the case now despite all efforts to free all the people's 109 just like you have mentioned all the girls through many captivity and Leisha Ibu is also still part of you know this persons at least 16 have been killed so far that's according to the report now since then at abduction thing is the fact that 80 is after nothing has been done we would say nothing because it feels like we have moved on we have forgotten about them we've also forgotten that these children 109 of them belong to families and you can only imagine the pain and the trauma and how it's so difficult for parents or their parents to be alive and the thing that these children are nowhere to be found now according to reports this has been made available between December 2020 and October as a 2021 1436 children school children and 17 teachers have been abducted from schools in Niger but I'm groups men so it feels like we haven't even learned anything that's another thing also we you have an experience and it's okay because you were not proactive but one would think that you would take proactive measures to ensure that there's no real currents but this constantly happened and so you see have more children that have been in captivity and this has also forced a lot of schools in the north about a thousand one thousand five hundred schools in the northern schools is then you have this cool some schools have been shut down truly and so education in that part of in this part of the country is also being threatened and it's sad for government I will constantly say this and that's the basic things these are basic things that you expect over anyone like I always make this reference to the fact that if you have a father I mean if you are a parent or you have a family the basic thing it would not be rocket science for the children to be asking for basic things oh then it shelter then a clothing then eat you know food and of course education would just come in now and government has a primary responsibility of protecting lives and properties and and government has failed and the one that's even more surprising is the fact that when this government came into power we're talking about the president Mohammed Buhari's administration when they came into power top on the list was the fact that they're boasted and they're proposed to Nigerians how that we're going to solve the issue of security and insorgency and looking at this what has been done I don't even think that the Nigerian government still remembers that you still have these kids still missing and what are we doing to ensure that this doesn't happen therefore despite several initiatives that's been put out it's so difficult it is so painful and really our hats with the parents and this children who have been away from their family for eight years you can only imagine the pain and the trauma that they would be subjected to some of them would have been you know gone through several experiences way trauma you can only tell eight years is a lot of time eight years is like another term for government it's actually to term so you do for it is really a so it's really a time but our government and our people have really this children yes we'll just let the conversation you know lie for some for just now and let's just keep on pushing and reminding government to do the needful and rescue these children safe in all of this a safe school initiative you know has actually come out but we will also hear reports of this free Chibo girls saying that they have been treated with contempt in school a whole lot happening you know at the parent actually demanding to Lisa but we'll take a break right now and we'll give you an update on the UEFA champions legal Nigerian legal just some sports generally in a moment to join us again


Hi, this is your host Plubhatiya. Welcome to the 2023 Prediction Series. We have with us today, once again, Robert Schull, co-founder and CEO of Reckon Rob. It's great to have you back on the show. It's a pleasure to be back. It's hard to believe it's been a year. Before I ask you to grab your crystal ball and share your predictions with our audience, just quickly remind us about Reckon. What do you folks do? Reckon is an infrastructure as code software platform. So we help companies build end-to-end automation and workflows for any type of infrastructure across every type of system that they have to integrate. So literally helping them build the platforms that run their business. Now it's time for you to pick your crystal ball. I don't know it's virtual bare metal or a software driven crystal ball. Pick your crystal ball and tell us what predictions you have for 2023. You know, no matter what type of crystal ball you have, it always comes back to people in process making the crystal ball work. It's your spiritual energy of it. And I think that that's actually a lot of what we're seeing for the industry in 2023. Our predictions are that platform teams and platform engineering, which is this coalescing around a lot of the challenges of cloud native, but also where DevOps has been heading in site reliability engineering, that we're really seeing enterprises sort of turn over this narrative of developer enablement and figure out that they have to do real work behind the scenes to make it happen. So we think platform engineering and the topics and discussions around platform engineering and the tooling that supports it like what Reckon does are really going to have a very good year in 2023. Aided by the idea that economic challenges mean that people need cost and governance and controls and better use of the people they have. So a lot of indicators driving towards platform engineering being a major topic in 2023. There's a couple of other things that we're tracking that I think are also interesting here. One of them is actually on the hardware and infrastructure side. So for people who care about that, there is a software interconnect layer that is getting a lot of attention where we've been taking systems, taking next generation PCI bus and actually connecting systems together in a really converged way. And while that's a bit of a stretch at the moment, some of the work that we're seeing in that technology space is really exciting. So really some fascinating things coming out of the hardware side around being able to have much more distributed interconnected systems and then also of course ARM processors are making a tremendous amount of progress from that perspective. And I think that those are things that while people who are dealing with a lot of cloud don't think about how much this can impact the industry, it really does change the industry. The other thing that we're tracking here is the big vendors keep grinding through what they're doing. We're watching the VMware Broadcom acquisition, which we don't expect to have a lot of drama for people, but we're also seeing the big cloud providers bumping into each other more and more. And I think what we're looking at seeing is more direct competition between the cloud providers and then strangely enough, that might also mean more collaboration between the cloud providers and more interconnect and more stories around those items. But for the most part, we're expecting steady sailing at the cloud level. Thanks for sharing those predictions. If I ask you what is going to be the focus for Racken in 2023? Yeah, our course is very simple. We are diving in on the platform engineering side. Racken has been building platform engineering tools for years now and it's wonderful to have the industry coming around and saying, hey, we really do need help interconnecting systems and building consistent infrastructure, doing dev enablement, dev process enablement that also respects the operators and the infrastructure teams. So from our perspective, this is really an awakening of the industry to the types of problems that we've been solving all along. And we're very excited about having those discussions, really helping people see how to connect all the dots as they build these platform engineering systems and really take on the challenge of creating collaboration because ultimately a lot of the platform engineering pieces are about collaboration and so if you and I have talked about collaboration quite a bit and infrastructure is code and how those things work. But finally getting real collaboration in the enterprise across these teams and helping them be more efficient and productive, that's going to be really exciting to be a part of for next year. Excellent, thank you. Talk a bit about what are the challenges that you see will be there in 2023 not only for Racken to solve but for industry in general. One of the things that we see with the assumption of additional economic headwinds so we definitely do see companies pulling back on hiring or changing tact, canceling projects, sort of focusing on delivering the work they've got. The challenge of having people with the right skills, enough time, doing some really hard projects and work is going to continue to be a challenge across the industry. So while we have a lot of people in the industry, some of the skills that we need to do the type of platform engineering, high-scale automation, distributed control work, those are still very hard skills to get and find and retain in the industry. And so we continue to see that as being one of the things that people really struggle with. Because even if you can start a project and get executive buy-in to do the type of work that helps consolidate this, you also have to retain the talent to make it happen. And in a shifting market, that can be very hard to do. Rob, once again, thank you so much for taking time out today and share these predictions with us. And as usual, I would look forward to talk to you again. And of course, I didn't say it in the beginning, but we will be holding a scorecard next year to see how many of these predictions turn out to be true. So no pressure, but see you soon again. Thank you. I will stand by every word of it. Thanks, Swap. I'm looking forward to reviewing it next year.


This video will talk about the multiplication property of equations. Basically, it says if we multiply the same thing on both sides, then we're going to stay equivalent. Or if we divide the same thing on both sides, we will stay equivalent. And then just remembering that dividing is like multiplying by the reciprocal of both sides. So if I want to get rid of, or if I want to divide by b on both sides, it's the same thing as multiplying by 1 over b. So let's look at these examples. And remembering that what you do to one side, you have to do to the other side to keep it balanced. So we have 10x equal to 20. Well, if I want x by itself, it's 10 times x, so I need to do the opposite or divide by 10. But if I divide one side by 10, I have to divide the other side by 10. So now I'm going to have my 10s cancel each other out. We have x and 20 divided by 10 is going to be 2. And we could check it. 2 times our 10 is equal to 20, and 10 times 2 is 20, which is 20. What if we have an example that looks like this one? x divided by 3 is equal to negative 8. Well, if we're dividing by 3, we always want to do the opposite operation to that x. So we want to multiply by 3. So we multiply both sides by 3, and we get the 3 becomes like 1, 3 over 3, and we just have x on this side. And negative 8 times 3 would be negative 24. And again, we can check it by saying negative 24 divided by 3 is supposed to be equal to negative 8. And negative 24 divided by 3 is negative 8. We know that it's true, and we found the right answer. And then we have a third example where it looks like a fraction is being multiplied by x. And we want to divide that fraction off. And to divide fractions, we just multiply by the reciprocal. So the reciprocal of 1 fourth would be 4 over 1. And we have to do that to both sides. It's really the same thing as multiplying by 4. So the 4's are a factor of 1, and the 1 over 1 is a factor of 1, so we just have x over here. And 6 times 4 is 24. Let's try. 1 fourth times 24 is supposed to be equal to 6. And 1 fourth times 24 would be 24 over 4. Equal 6 and 4 into 24 is 6. So we know we did it correctly. Now let's solve some equations that have multi-steps in them. Remember that when we do multi-steps, we're still trying to get to x by itself. So it's kind of like an onion. We want to peel the layers away. And the outer layer in this particular problem would be this minus 3. We're subtracting 3, so we have to do the opposite. So add 3 to both sides. And so we have x over 7 on the left-hand side. Equal to 1 plus 3, which is 4. And now we're dividing by 7, so the opposite to peel this layer away would be to multiply by 7 on both sides. So there's my factor of 1, and I just have x equal to 28. And if we checked it, we'd have 28 over 7 minus 3 is supposed to equal 1. 28 divided by 7 is 4 minus 3, and 4 minus 3 is 1. So 1 is equal to 1. There's our answer, x equal 28. Alright, peeling this onion. We have this plus 2, and we need to take it to the other side so that we can get to this x by itself. So we take this first layer off by subtracting 2. And when we do that, those cancel each other out because they're opposites. So I have negative 3x. We're subtracting 2 from both sides, but that would be the same thing as adding a negative 2. So we'd have negative 9. And now we need to divide by 3 to do the opposite operation. And x is going to be equal to negative 9 divided by negative 3, which is a positive 3. And if we checked it, negative 3 times 3 plus 2 is equal to negative 7. And negative 3 times 3 would be negative 9 plus 2. And negative 9 plus 2 is negative 7, and that is definitely equal to negative 7. Alright, so what if we have a really involved problem like this one? Where we start? Well, we start by combining like terms. We got to make it as simple as possible, and then we can see what we have to move. So we have a 7x and a 5x that we need to combine. So we really have 12x plus 4 on this side equal to 14x plus 3. Now remember, it doesn't matter if I move the constants first or if I move the variables first, but what I like to make my x positive. So if I'm going to do that, I have a 12x and a 14x. 14x is larger and it's positive on this side, so I want to subtract the 12x from both sides. I'm not dividing yet because I'm just moving it across the equal sign. The whole thing. I'm taking the x with it.  And now I need to get this 3 away from my x. Now that I only have one x term, I know I have to move this 3, so I'm going to subtract 3 from both sides. And that will give me 1 is equal to 2x, and it's 2 times x, so I'm going to divide both sides by 2, and that tells me that 1 half is equal to x. And I'll leave the checking up to you for that one. 4 times the quantity 2 plus 15 is equal to 20. You see parentheses. We need to clear them. We've got to simplify the sides, and that includes distributive. So we need to multiply the 4 times everything inside. So we have 4t plus 60 is equal to 20. And now I have a t term and a constant on this side and a constant on the other side, so I want to subtract my 60 to get to that t term all by itself, and subtract 60 on this side. And we're going to have 4t left over here, because these are opposites. And 20 minus 60 would be a negative 40, and then 4 times 2 means we have to divide by 4. So t on the left-hand side equal to negative 40 divided by 4 will be negative 10. And again, I'm going to leave the checking for you to do. All right. And then the final one here is going to be distributing a negative. So remember, when you distribute a negative, you always want to watch your signs. So a negative times a positive is going to be a negative 12p, and a negative 3 times a negative 1 will be plus 3 equal to negative 12. We've got a positive 3 on this side, so we want to subtract 3 from both sides to get to that p term by itself. So negative 12p on the left-hand side is equal to negative 15. And if we divide by negative 12 on both sides, then p is going to be equal to a positive, because it's a negative divided by a negative, 15 over 12, but we can reduce that. So it should be equal to, they're both divisible by 3, so it should be equal to 5 over 4.


Hey everyone, let's get started. Welcome to this session on bin packing pods and managed Kubernetes. Thank you for being here. I'm Vinay Suryadevara, and I'm a senior software engineer at ClickHouse. My co-presenter is Jenfei Hu. He also works as a senior software engineer at ClickHouse. So on the agenda today, we'll start with a short introduction to what is ClickHouse and ClickHouse Cloud. Then we'll talk about the overview of the problem that we are facing with our infrastructure and why we used bin packing to solve it. Then we'll get into the details of how infrastructure is set up, what were the various approaches we used to solve our node utilization problem. And then we'll also talk about the rollout we did across our fleet, what were the savings we achieved, and some of the learnings that we realized during this whole exercise. And then we'll end with Q&A at the end. So what is ClickHouse? ClickHouse is an OLAP database. It's used mainly for analytics use cases. It's used to generate aggregations and visualizations on your data. And it works best with mostly immutable data. It's been in development since 2009. And it was open sourced in 2016. And it's gained a lot of popularity since then. And it's one of the fastest growing GitHub communities. ClickHouse is a fully distributed database. So it supports replication, sharding, multi-master, and cross-region setup. So it is production ready. Last and the most important thing about ClickHouse is that it is extremely fast. Using various techniques such as column-oriented storage and state-of-the-art data compression, ClickHouse provides insights into the customer data in milliseconds, which makes it one of the fastest databases out there. So now that we have an idea about what is ClickHouse, what is ClickHouse Cloud? ClickHouse Cloud is the serverless offering of ClickHouse. It has various features such as idling and auto-scaling so that you can bring down your compute when there's no activity on the cluster. And similarly, auto-scale the cluster when there are workload spikes. We also have compute and storage separation so that you can scale compute independently of storage. This is how a ClickHouse Cloud instance looks under the hood. We use Kubernetes as a compute layer, and we use object storage for the data persistence. We currently still use PVCs for some metadata, but we are in the process of moving away from that. We're currently available on AWS and GCP, and pretty soon we'll be coming to Azure as well. And later in the year, we'd also have the bring your own cloud offering in which the customer data can stay in the customer VPC and customer cloud account, and they can use the ClickHouse Cloud control plane to deploy ClickHouse in their account so that large enterprises who do not want to part with the data can use this solution to deploy ClickHouse on their premises. So now that we have an idea about what is ClickHouse and what is ClickHouse Cloud, what is the problem that we were trying to solve? And before we get into the problem, the title of the talk is bin packing pods in managed Kubernetes. So what is bin packing? In general terms, bin packing is an optimization problem in which items of different sizes must be packed into a finite number of bins each with a given capacity in a way that minimizes the number of bins used. So if you look at this picture, on the left side, you see that we have an initial set of objects that individually have some weights, and on the right, we have bins with max capacity. And using bin packing, we can fit all of these elements in only three nodes instead of four saving us the cost of one of those bin. Specifically in Kubernetes, bin packing refers to the process of efficiently scheduling pods onto the nodes in the cluster in a way to maximize the resource utilization and minimize the number of nodes needed while also satisfying all of the constraints of the pods. This helps us save cluster resources such as CPU memory and can lead to a lot of cost savings and improved efficiency. So now that we have an idea of what has been packing, what is the problem that we were facing? So in our cluster, in our Kubernetes clusters, which is multi-tenant and present in multiple regions, we noticed that the node resource utilization, CPU and memory were non-optimal across the fleet, and this meant that we had higher infrastructure costs because we were running more nodes than was required. So we explored a few different solutions, and the one that we finally settled on was to update the Kubernetes scheduler scoring strategy to bin pack pods onto the fewer nodes. Then we had to roll out this solution to all of our Kubernetes clusters which are present in multiple regions, and also it's a multi-tenant feed so that we have multiple instances, click-house instances on the same node. So we had to find a way to reliably load out these changes so that existing customers do not face any disruptions. So we'll talk about all of these in detail in the coming sections. So before we take a look at the infrastructure setup for ClickHouse Cloud, some of the terminology that we'll be reusing across our slides. ClickHouse Keeper is our replacement to ZooKeeper. It is used as a coordination component for our server pods. ClickHouse Server, these are the compute pods for ingest and query. These are the pods which process the data and process them to the object storage. ClickHouse instance or cluster just means that this is a custom resource in Kubernetes that consists of Keeper and server stateful sets. Both ClickHouse Keeper and ClickHouse Server have similar needs with respect to bin packing. And so, for the purposes of this talk, we'll only focus on the server pods for simplicity, but all of the approaches that we used for server pods can also be transferred to the Keeper pods. When we refer to a node, we just refer to an EC2 Azure or a GCP VM. Node utilization, so when you say node utilization, it means it's the sum of resource request of all pods on the node divided by the allocatable resources on the node. The key thing to note is that we are not looking at the actual usage of the pod, but the resource request, which is the minimum resources required by the pod. And finally, CubeScheduler is the default pod scheduler for all the pods in the Kubernetes cluster. So, let's take a look at how we set up Kubernetes in ClickHouse Cloud. We use managed cloud Kubernetes in each cloud service provider, so EKS for AWS, GKE for GCP, and AKS for Azure. We run multi-tenant Kubernetes clusters, and what this means is that pods from two different ClickHouse instances can run on the same node. We also use namespaces for isolation so that pods from two different instances do not interfere with each other, and we use Kubernetes network policy via the CiliM-CNI solution to prevent cross-namespace network calls. So, continuing further, we use Cluster Autoscaler, which is a pretty well-known component for dynamic node provisioning, so for node scale up and scale down. Most of the ClickHouse server pods are in a similar shape in terms of CPU to memory ratio, and so we have a homogenous workload for the ClickHouse server node groups. We also make use of over-provisioning, which is a concept in Kubernetes where we reserve some extra capacity for workload spikes so that when these spikes come in, server pods can be scheduled on these extra nodes, and we do this by scheduling some low-priority preemptable pods on these nodes, which can be evicted if server pods need to be scheduled, and these evicted pods now will tell Cluster Autoscaler to provision more nodes for buffer. And lastly, we have a weekly release schedule for all of the pods in ClickHouse across our fleet. This is important because it will come up later, and we do periodic rescheduling and restarting of pods. So, we deliver all the ClickHouse image updates via this weekly release schedule. So, this is how a sample Kubernetes cluster would look like. We have nodes which are VMs in all three AZs because ClickHouse instance has pods in all three AZs for reliability. We also have a few system workloads that are deployed on these pods so that we can monitor the usage on these pods. We have a few ClickHouse server pods that are already running on the nodes, and we have an operator in the Kubernetes cluster that is responsible for managing the custom resources. So, when a new ClickHouse instance is created, it creates a custom resource which is managed by the operator and which, in turn, creates stateful sets, and the stateful sets create the pods that they manage. Similarly, when another custom resource, when a ClickHouse instance is created, the same process is repeated and the pods are created again. And we see that pods from two different instances are living on the same node. This is what we meant by multi-tenant. So, this is how the infrastructure would look like in a Kubernetes cluster in ClickHouse Cloud. So, now that we have an idea about how our infrastructure looks like, what is the problem that we were running into? So, when we did a regular audit of our fleet, we noticed that the node utilization in the fleet was only about 40% to 60%. And this was bad because it meant we were running nearly twice the number of nodes we needed to run the same workloads. So, a few factors in our cloud setup that contribute to utilization. We have a dynamic fleet, which means that since we have idling enabled, ClickHouse clusters which do not have any queries or inserts running on them for more than 30 minutes are now scaled down to zero pods. We also have autoscaling, that means pods might occasionally need to be restarted with a different size and different CPU and memory. And finally, in the regular service lifecycle and creation and termination, we have pods being provisioned and deep provisioned as well. And all this means that utilization is something that keeps changing over time and is quite dynamic within our Kubernetes clusters. So, the goal, which after identifying the problem, the main goal for us is to increase the node resource utilization across the fleet and via that we can realize some cost savings. So, we have a few approaches to improving the node utilization. But before we talk about the approaches, any solution that we actually adopt needs to satisfy a few requirements. The first and the most obvious one is that it needs to increase the node resource utilization across the fleet. The second and the most important one is that for existing customers, we want to minimize disruption to their workloads. Since we're a database, we might have long-running insert queries on pods that we do not want to interrupt. And we also do not want any degradation in the experience for customers when they're provisioning new instances. Since now we have fewer nodes in the cluster, we do not want the node provisioning time to be an additional burden for the customers. And the last requirement is that our solution needs to be multi-cloud friendly. Since we run in three major clouds, we want one solution that works for all three of them and not a bespoke solution for each cloud. So, those are the requirements that any solution that we adopt should satisfy. So, let's take a look at what are the different approaches we explored. The first one was overcomitting the resources on the node. So, in this case, we set the resource requests to be less than the limits of that node. And this means that the pods of the pod, this means that the pods will be QoS burstable, which means that their usage can vary from anywhere from the resource request to the resource limit. By decreasing the resource request, which is the minimum amount of resources required by the pods, we can schedule more pods on each node. The key assumption here is that not all the pods use these resources up to the limits at the same time, and hence, we schedule more pods on each node. So, looking at an example of how this would work, let's say we have a node with 64 GB total memory, and we have four pods with resource requests of 16 GB each and memory limits of 32 GB each. So, now, on this node, we can schedule four of these pods. And when they only use up to their resource requests, everything's good. But technically, if you look at the limits, we've committed 128 GB on this node, which only has 64 GB. And if all the pods start using up to their maximum limits at the same time, then that's going to cause issues. And for that very reason, we did not end up adopting this approach. And like I mentioned, if all the pods start using up to their maximum resources, it can either, in case of memory, lead to own kill, or it can lead to throttling of the CPU. And this is not good for us, because in Kikau's cloud, we have a QS guaranteed setup, because the customer sets the resource limits, and we guarantee that they get what they paid for. And so, if we change that to QS burstable, it will not be a good experience for the customer, and it can lead to the noisy neighbor situation. The second approach is to make sure that the user doesn't have to use up to their maximum limits. So Cluster Order Scaler has a setting where we can specify the utilization threshold. And if on a certain node, utilization falls below that threshold, then the pods on that node are evicted and that node is marked for scale down. So it doesn't mean that scale down happens immediately. It considers various things as pod disruption, budget, de-installations, et cetera. It's just marked as a candidate for scale down. And this is a very handy way to do this. But we didn't end up adopting this approach. And the reason is that this violates some of our requirements. Frequent evictions of pods on nodes is too disruptive for stateful workloads such as ClickHouse, which is a database. It also violates our requirement of not interrupting long-running queries on these pods. And we also do not have enough fine-grained control on evictions. So we looked at another component called de-scheduler, which is a Kubernetes 6 project. It's meant to be a counterpart to kubescheduler, where when kubescheduler makes a decision on where to schedule a pod, it looks at the state of the system at a certain point in time and it picks the best node for that pod. But the state of the system varies over time and after a certain point, the original scheduling decision might not be valid. But kubescheduler does not go back and re-schedule the pod. And this is where de-scheduler comes in. So de-scheduler can look at certain constraints that we give via plugins and it can evict pods on nodes which violate those constraints. One such constraint or plugin is the high node utilization plugin using which we can set a certain threshold in the node and if the usage threshold falls below that for that particular node, the pods there are evicted. So you can see that this is actually quite similar to the cluster auto-scaler and this solution also does not work for us for the same reasons where we have frequent evictions and we have interruption of long-running queries. So we noticed that we had the same problem with both approaches 2 and 3 where we are focusing on evicting the pods and scaling down the nodes but not focusing on where the evicted pods are being scheduled and if they just go on to another lesser utilized node, we are back at the same problem again. And they also have frequent evictions of pod which I already mentioned. So we decided to focus on optimizing packing during the pod scheduling phase instead of thinking about evictions. So we started looking into the Kubernetes scheduler a bit more deeply and we noticed that the way that Kubernetes scheduler does scheduling right now, it first identifies all the nodes that this pod can fit on and then it does the scoring phase for these nodes where it picks the best node based on the highest score and by default the least allocated scoring policy is used and that means that when there are two pods that there are two nodes that a pod can be scheduled on Kubernetes will pick the cube scheduler, will pick the node with the least allocation. So it does this so that it can evenly distribute the load across the cluster but the flip side of this is that this makes nodes scaled on unlikely because there is a certain amount of threshold and cluster autoscaler cannot reclaim the nodes and so we have an inefficient cluster with higher number of nodes than required. So in the same way in Kubernetes scheduler in the configuration where we set least allocated Kubernetes also offers an option for most allocated scheduling in which case in the same scenario that I just talked about if a pod needs to be scheduled onto two nodes if you use the most allocated scoring it's going to schedule the pod onto the node with the higher utilization and it actually turns out that the best solution for us was actually the simplest one and this looks like a very promising candidate. So this is just an example of the config file on how we would update this in the control Kubernetes control plane. Inside the plugin config section for the node resources fed plugin we update the scoring strategy to use most allocated scoring and we provide equal weights to CPU and memory but that can be changed based on your requirement. So let's take a look at how this would look in our cluster. Let's say we have three nodes with Clickhouse server pods on all three of them and if for any reason one of the pod is restarted either due to idling or idling or autoscaling. So now Clickhouse when it tries to schedule this pod again if it's using the most allocated scoring policy it will first find the nodes which can fit this pod which is node 3 and since node 1 has the higher utilization it will then schedule the pod onto node 1 and that means that node 3 can now be reclaimed by cluster order scaler and now our cluster is in a state where we only use two nodes instead of three and the nodes are packed better. So this looks like a promising candidate. Let's take a look if it satisfies all of our requirements. The first requirement was that we need to increase node utilization across the fleet. By virtue of packing higher number of pods onto lower number of nodes we get better utilization across the fleet. The second requirement was that this needs to be low impact to the customer. When we update the scheduler scoring policy this does not immediately trigger a restart of all the pods instead in the pod lifecycle whenever the next restart occurs naturally that's when the pod will now be scheduled onto a node which has more utilization. And the third requirement is that our solution needs to be multi-cloud friendly and since this is a Kubernetes native solution this applies equally to all clouds and so all of the requirements seems to be matching for this particular solution and so great we just need to update the Kubernetes control plane to set the scoring policy to most allocated and we should be good, right? Well turns out it's not so simple. In AWS and Azure we do not have access to the Kubernetes control plane to set this particular setting. GCP offers a version of this called optimized utilization that essentially does what we needed to but this only works in GCP and this is not portable to both AWS and Azure. So given this road block how do we set up the most allocated scheduler policy when we're running on managed Kubernetes? Turns out Kubernetes has a handy way of doing this. We can run a secondary scheduler that is natively supported by Kubernetes. It runs in parallel to the default Kube scheduler and we can annotate the pod to mention that it needs to be scheduled by the secondary scheduler. So what we can do is that click house server pods can be run by the secondary scheduler and the non click house pods can still be run by the default scheduler. We use the upstream Kube scheduler image. We use the upstream Kube scheduler image and we just tweak the most allocated scoring strategy. It's having some technical difficulties. I'm going to stand here. So we use the same Kube scheduler by just tweaking the scoring policy and we deploy the scheduler in the Kube system namespace so that we can prevent evictions and scale down. So this is how we set the scheduler up in the managed Kubernetes. So now that we have a solution that works for us and we also know how to set it up, I'll hand it over to Genfei to talk about how we did this roll out in a non disruptive manner and what were the learnings and cost savings we realized. Thank you. Thank you Vinay. Now we already know the answer for the problem just the question is how we can roll this out. So before we dive into the details let's take a look of how the roll out works. We create the deployment for the secondary scheduler next to the default ones. And just creating that scheduler basically a know-all but we need to update the clear house pod to use this most allocated scheduler. That would be just a one-time change and from that point on all the clear house pods we start to use this most allocated scheduler. And the pods will be restarted but we have pod disruption project configured. It's just a one-time migration. For our production safety we will start from some smaller cluster first to gain some experience and then we graduate roll out to our entire production. And finally after that is done we do the evaluation and measure how much savings we achieved. Alright so for the scheduler set up we create the deployment of these three pods and the purpose for three pods is for the high availability. The pod itself is basically the same Docker image as the upstream. We just need to update the scheduling policy. We also enable the leader election. The idea is that when the primary pod crashes the second one can notice that and then take over the scheduling decisions. We actually did some reliability test. Basically by constantly schedule a lot of pods and in the middle of that we kill the primary scheduler and then see how the secondary ones scheduler pod can take over. We find that it can just work nicely so good to go. We also need to do some measurements. There are two types of measurements we need to do. The first one is the ad hoc analysis. There's a tool open source one called EKS node viewer and that is dedicated for the EKS resource analysis. This is the screenshot you can see from their web page. They show the CPU and memory utilization, number of the pods running on the nodes, or so even the cost for different machine type based on the AWS pricing data. That comes very handy for us when you are doing some ad hoc analysis during the road. The other type of the measurements is for after you finish the road to the final evaluation we need something more sophisticated than EKS node viewer. For this one we use our internal data warehouse. And the interesting fact that it also runs on the clear house cloud because we love our product and this data warehouse instance we show the cost over time so we can do more comparison before and after the road. And the data warehouse we import the AWS cost and usage data periodically. We also use SuperSet, an open source software as a UI layer on top of this data warehouse for some analytics. All right. So having all these measurements and preparation down now it's time to road. As I mentioned we start from the smaller cluster first. Actually after road is out to a smaller cluster the initial savings were not that significant. But we are not too worried because for a cluster to run you need to have some flat cost anyway to run your system workloads and you also need to have your machine type. Those are just there you cannot avoid. But the important part is that we gain the confidence how to operate this in our production settings so that we can proceed. So this is a EKS node viewer diagram for a larger cluster before the road. The important thing I want to highlight is that first the average utilization for the node is around 50% which is not very high. Also the utilization per node is ranging from 30 to 60%. This is something we want to improve. This is the diagram after the road. As you can see the utilization for the node average has improved from 50 to 70%. During the road VNA just shows a clear hotspot being stopped and then rescheduled again to a more packed nodes and by doing so for a while for the entire cluster some nodes have been saved and cluster autoscaler cleans them up so that that is where the 10 to 15% cost savings we achieved with this road. All right, so having done all those things for our production clusters now time to look at the cost savings. In this diagram for our data warehouse software the X axis is the time and the Y axis is the money that we pay for the easy to compute machines. You can see on the right-hand side there is a very obvious deep that actually comes from our road which is quite promising. So with all this you may have one question left. Well, this is very good but how does the utilization change over time? We understand that right after the road when you pack the cluster the utilization is very high but what would happen then things change over time. You may remember we have service being idle or stopped or terminated those process can create this schedule some paths from those and create some holes in our VMs, something like this which drops our utilization. Luckily we have these regular upgrades for our clear house software which essentially try to update the clear house server paths with the newer Docker image versions and that is down in a controlled manner and gradually change one path at a time with disruption budget but overall the important thing is this basically move the path to the most allocated ones and achieve the same packing effect we just saw. So this brings the utilization back to a higher point. We do this every once in a while for our upgrades. So now I want to share a few findings and issues we have. I will go through them one by one. So first system workloads we find that sometimes an event in node utilization even if it's very low the cluster auto scaler still doesn't reclaim and scale down that node. Why did that happen? After some investigation we found that nodes actually run some of our system workloads such as Argo CD and AWS CSI controller all those things. And these workloads they have use of ephemeral volumes like a local host path or kind of storage on the node and because of uses of that cluster auto scaler will not be conservative trying to not evict that part and so which prevents the nodes being reclaimed and saving the cost. The solution is very simple because we know the nature of the process. So we just add a save to evict to the system workloads so the cluster auto scaler can then reclaim those nodes. On scheduler paths this is very interesting actually at some point we notice that certain paths are just stuck at the pending state and the cluster auto scaler and most allocated scheduler we deployed have different opinions on these pending paths. Basically our most allocated scheduler think oh there's no space for me to schedule this greenhouse path it will be pending however cluster auto scaler also think I don't have to scale up because you can just evict that over provisional path with lower priority what Vinay just mentioned for this for provisional and we want to figure out why this happened. So we actually searched and found these tags from Kubernetes document the summary is that when paths are created the scheduler will form an queue and trying to schedule the pending paths we are also trying to find some victim paths with lower priority to evict in order to accommodate this path this does ring a bell to us because you mentioned the scheduler and remember our clean house uses the most allocated scheduler our over provisional however is still live as the default settings use the default scheduler this create basically that two paths use different scheduler and we think that may create kind of independent workloads and unable to be evicted for the over provisional so the solution is very simple then we just make the over provisional also use the most allocated scheduler right after that we see the clean house part being scheduled all right co-start time by co-start time I mean clean house part being request from wake up from the idling or getting started in the first time the time to take the part being ready and ready to serve the customer request in theory as you can see now the cluster is more packed so then a new positive coming is more likely to trigger a new EC2 machine being provisioned and so co-start time can increase because of the EC2 machine load machine pre-setup time and we measure that and confirm that it did not happen and we think that is because our over provisional buffer gives some buffer space for eviction and allows the possibly in schedule result provision in no time a new node the likelihood is reduced finally as mentioned GCPs they have a native option called the optimized serialization of the customer at the cluster level you just enable this which achieve the same effect however for Asia is similar to the EKS you cannot change the control play for this configuration have to do this yourself deploy a secondary most allocated scheduler all right time for the summary so we are in a multi-tendent Kubernetes hosting environment doing a source product for our customers and our goal is to use the resource utilization for the Kubernetes clusters to save some cost however while doing so we would like to minimize the disruption to our customers we evaluated a few approach and we end up selecting Kubernetes scheduler with most adequate scoring strategy figured out a few issues during the route and that save us 15 to 20% cost which is great some take away message if you want to do something similar we think these are helpful that you can consider the Kubernetes scheduler scoring strategy to see if there is something that you can use the second one is to use the QS granted so that if you want to avoid the noise neighbor situation and the third one is take advantage of your software upgrade process so that you can do the scheduling as an opportunity to improve your utilization the last one is the over-provision that gives you some buffer time when you have a more packed cluster that's pretty much about it and we also have a blog to details about how this is done in our technical blog come and check it out if you are interested our colleague Manish he will talk about how to use Kubernetes seed for sets and how to do the scaling challenges here just in this room what a coincidence tomorrow afternoon and last but not least we are also hiring if you are interested in building a cloud offering for the database and open source technology come let us know we are interested thank you thank you I have a question about the practical way to do it you are just using the scheduler name spec inside do you have anything else? you just specify the policy scheduler name in the policy spec that should match whatever you specify for your second scheduler that should be enough so you are putting that on your click house database pod and not on the system pod and the rest of the pod sorry can you say that again you discussed the fact that you wanted to keep the default scheduler for the other pod I guess you are talking about why is it my question is more why is it interesting to keep the default scheduler for the rest of the pod the question is why do we even need to have the default scheduler the answer is that you need to explicitly opt in for our most allocated scheduler if you are deploying some third-party deployments for example Grafana or those things they their default values still need to work and I guess yeah I guess it's just what works and their footprint is not that huge anyway so we are just keeping it safe and yeah that's it yeah we haven't felt the need to use the scheduler for the other type of workloads that we use in the main cost savings that we discussed and the highest number of pods usage is for the click house server I would like to ask if this also works in a multi-cluster environment does it make a difference or really not multi-community clusters yes so we haven't explored that exact scenario yeah our click house workloads currently is not deployed in multi-cluster fashion so every scheduling decision is down within a cluster scope so that's not a challenge for us yeah I think it would depend on how scheduling works in this scenario thank you very much I wanted to clarify about the secondary scheduler was it a controller or did you just redeploy the cube scheduler yeah we just redeployed the cube scheduler like you mentioned it's a deployment in which like three pods run with the same cube scheduler image and a custom scoring policy you can take this mic I wanted to ask you said you wait are your workloads click house workloads are rescheduled naturally and do you plan or know how you could enforce this automatically so you don't have to wait until new releases are done or something do you have another controller for that or so when you update a pod spec all the pods that are using that spec like when you update the scheduler to use the secondary one it will trigger a restart automatically for all of them in our case we do that in a more controlled manner because we have the pod disruption budgets we don't want to restart all of them but if you don't care about that you can just do it by updating the pod spec okay but that sounds like you would do it manually and it's not automatically done somehow so we are so first of all our release process is done automatically it's managed by the second question and second thing is that it's not just release automation for example auto scaling or node drain or whatever reasons that customer change or configuration that requires restarts that we are all restart this pod and pack it to another more packed nodes which take advantage of this effect yeah but we automated it like after the pod spec was done in our code base we used the release process to automatically update it thank you


OK, now my name is Kimo Wünig, my name is Wulge Macht. We are both working for the mobile devices team at Susie Linux and Nurembed and today we want to present our power management infrastructure we are using which has been developed over the last few years. After 8pm was completed and 8pm I was introduced we got a lot of systems which is not working out of the box using the Nurembed power management piece and we are required to build an architecture to get as most of the systems running out of the box with the power management. I think we are giving a short overview about what we are going to talk about. With the cut on the left hand side I think you can read it anyway. First of all I will give a short overview about Linux power management in general which will be moving the kernel and the user space. And then I will talk about the PowerSafe architectures picture and what applications we use to make use of the PowerSafe or PowerSafe management subsystems. And this will be followed by a short introduction to the new thing of runtime device power management. It will be introducing what you are currently able to do with putting devices into some other sleep state at runtime and thus you are able to set power as well. Afterwards we will have a live demonstration and will demonstrate how much power is actually consumed and will make use of the power management features. At the end of the demonstration we hopefully will have shown that power management is very efficient when we use it. And the last thing we are going to present is a short what's next section where we are talking about various things which are coming up in regards to power management of the Linux. Now Linux power management is part of the kernel for sure. It provides all the subsystems and interfaces we are using with our project architecture to make use of it. One of the most important things is the CQ frequency scaling that you are able to switch the CQ frequency while at runtime. And nowadays if you have got a dual core machine for example you are also able to switch cores on and off while running the system without turning it off. The next subsystem will be ACPI. It provides information about every styles of components of the system as well as it gives interfaces about triggering and some actions to the subsystem implemented by the machine. And we have another subsystem called device power management because it's like in the subsets we are able to trigger the power state of the specific device that uses the system as it does. This layer is encompassed by a second layer. We are having the power set team at the top and Paul as well which are directly talking to the core interfaces and they are on the other end talking to the client. We are having it right there. Let's see how they are communicating with each other. We have two main IPC inter-process communications. The first thing would be system cores between the power state team and Paul. They are using both system cores to access the core interfaces like CPU, FAC, ACPI and the device power management. On the other hand we have the clients who want to communicate with the team and this is basically all the client dealers nowadays. We have the clients communicating with both with the power set team as well as Paul and we have additionally power set to speaking to Paul. Here we are moving some functionality from our power set team which has implemented stuff like battery calculations to Paul to have power range. Now that's it for the graph overview. We now have some information about the power set architecture. Okay, at first we have the power set team. It's a root team caring about things that need root privileges like caring about multiple batteries and performing corresponding actions. For example, if the battery runs low and suspending the system and switching CPU to NC and of course doing runtime device power management but this is currently still under development. We both support an ADM and ACPI in the same manner. A second big and important part of the team is its scripts. Why do we use scripts? For example, for unlawed modules or restarting services and why did we put this functionality into scripts? They are easy to customize and you don't even need a compiler. You can simply give a user a new part of a script or you can say you can change line 20 and so it's very easy to provide new functionality. You also handle hotkeys in those scripts and that makes it possible that users can define own actions on window-specific key presses and starting an application or something like this. On top of the power set team we have a set of libraries. At first the general power set library summarizing battery information calculating CPU load or reading out throttling information then we have two further small libraries providing convenience functions and for communication with the whole demon or for communication over the deepest demon. These libraries are used by our clients to present the power set features on the desktop. For example, they interact with the demon, triggering suspense states, setting CPU quality, setting different power-saves themes. The other main part of the clients are its own implementations like how to suspending the system after a specific time the client has to care about whether the user is idle on the desktop and then the client triggers the spent over the power-saves libraries to the demon. The client also implements other features like locking the screen when the lid is closed or showing messages from the demon to the user. We have a key power-save for the KDE desktop, a WM power-save for Windows Maker or GDM power-save and we also have a command line application that makes it possible to fully control the demon. That can be helpful on servers where you do not even have a desktop. Now I want to give a rough overview of how our scheme architecture works. I picked out our main two schemes that are in our default configuration, namely scheme power-save with running embedded battery or scheme performance when the AC adapters black in. I also put some example features around those schemes and now because you can configure the demon as you like you can put those features into the schemes put them inside like you want so you can compose your own set of schemes. It's also possible to create completely new schemes it's really easy and this can be very interesting for the user and he can create his own custom schemes that shifts his needs. You might expect that the positive demon of this architecture is only a solution for Susie but that's definitely not true. We already got a lot of feedback from other distributions and here you see the distributions and we know that the positive demon works and the architecture works and there are also packages available from those distributions already. Okay, let's hand over to Demo. Next thing we are going to talk about is runtime and last power management. As I look at what we are currently having it's like pushing the global system state to suspend to disk or suspend RAM or standby something and we are also able to switch to CPU frequency it's also quite easy to adjust on most systems we are also able to adjust the brightness of the display but what will we actually get for the case of switching the global system state because we are having time as we are able to restore a system as we lessen it when we suspend the system but when you look at CPU frequency and LCD brightness we are only saving power on the expense of something else you are losing performance when switching to a low frequency and you are losing the readability of the screen when you are switching down the brightness So next thing which is coming is runtime and device power management here you are able to switch off devices which you are currently not using and thus you are not having any loss when you are doing so imagine you are on the train or somewhere else where you have no AC adapter next to you so if you are not using the USB or sound subsystem at all then you are able to switch off those devices and even if they are currently not used we will see in our craft next on the live demo they are consuming power because they are powered up and with runtime device power management you are actually able to suspend the whole device and then you are saving power in the details about it the ACPI specification gives us four sleep states for the device power management D0 to D3 while D0 is the fully powered state the normal regular state which is mandatory and the D3 state is the off state where the device context is lost and you need to re-initialize everything after you power on just saying now the two states are not mandatory as the others they are optional and they are mostly not implemented on the rest of them so it's actually your next phase with the echoing of the numbers and species of SEPAR that's wrong, D0 to D3 don't represent these states they used to represent PCI states but now they represent off states but they are now using the D state on the PCI and then D0 to D2 to CIS and then you are kind of using an old kind of 10.0 the next thing is the exam pop switch to the power state if you are correct I think so on the recent course you only got a 0, 1 and 2 multiple 3 state and the two would correspond to D3 yeah, exactly so on an old system you would use the second line first of all you have to identify the device you want to power down and afterwards you are using an echo to the SOSFS path to execute the switch off the power device which is the power state of the device on recent course it would be echo, dash and 2 to get to the D3 state which is mentioned above as always we don't want users to go to a terminal to hack these states to activate the power management the run time device power management and thus we have enhanced our power set architecture to do this in the power safety mode and it's already implemented and this is going to be configurable by using a client of the power safety mode it's also important to make that this interface is likely to change, there's currently development going on then so because don't be sure that this will be the way you power down the device in the future so what we have actually done to test run time device power management to test the consumption we have a test set up you can, you have one target system that gets messaged you can see this in front from your point of view in the right the laptop will remove the battery of the laptop to get real accurate values to not measure the charging of the battery we have an ampere meter connected to this laptop to record and measure these values and we have another system running that runs a software recording the values and displaying the values in this case it's QGDMN and now I put out an example of a module of a device the IPW2100 module you see a graph and how our test results look like at this point the device is fully powered oh I have to mention that in the y-axis you see the ampere the volts are constant we measure it with 20 volts the az adapter provides 20 volts and so we are measuring the ampere on the x-axis is the time in seconds so in the beginning we are about 0.820 ampere at this point we power down the device you can see a tension peak the line increases that's the price we have to pay for powering down the device further on the line drops and you can see that a lot of power can be saved by only suspending one device and the IPW2100 module is a very good example of how much power is consumed without the device even used and these are about an example only the line drops to about 0.77 about 0.1 ampere finally the device is powered on again and it returns to an initial state about and the peak in the end is just the test ended at this point so now we have a live demonstration using the power measurement functionality any questions so far? we are having a set up it was not obvious from your presentation so far, I've probably missed a little bit of picking but anyway, how you detect the idle device what is the way? it's probably being done by the user on interaction so you are saying ok I don't want to use sound at the moment but it's going to be a far end of the implementation to see whether the device is being used and if not, it should be powered on so it's kind of an obsession level in a way that at the moment you have to say ok, switch down the USB subsystem for example and then version wise start going central ok and now you are fixing that that you are going to switch off the device switch off the device we are sending it to the USB 3 depending on the driver or the hardware it's going to be off or not ok, first of all we are running at full speed we have a system running with 1.6 TRs we will see it when the system is up what you cannot see is that the brightness of the display is at the maximum so we are currently using the most power as possible and we also have deactivated the device runtime with device power so we should have the machine as it would be powered on the desktop when we are running on AC for example we are using K-PowerSafe to show the interaction between the client and the device and in this information level we are running the information of the system I'm not sure whether you can read this stuff except that we are running with the performance scheme meaning the high-secure frequency which in contrast to the high-secure frequency it's at the maximum it's at 1.6 TRs we have our battery set we have brightness level being at 100 and one hand power management is being switched off and currently our system if you can start the machine we are missing y-axis on the left-hand side you will have to look at the values and you will soon see some graph hopefully it starts now if you look at the graph that's the value being consumed that's the 1.05 do you mind taking it on the board? right now the first thing what we are going to do is we are switching down the CPU frequency down to 600 MHz and we will see how much power this will save there you can see we are running at 600 MHz power consumption of the whole machine you can have a look at the setup afterwards it's quite easy and now we are seeing that we are dropping down to 1A which is already a gain of 0.4A simply by switching down the CPU frequency about 1.1 ok now the next thing you are not able to see because of the display thing I will lower the brightness of the display you won't see on the screen but trust me the screen of Wi-Fi gets darker and darker and when you are looking at the graph it's falling down, it's dropping and even though I've only run to 50% of the brightness you will see that you are almost gaining as much as with the switch of the CPU frequency which might not be obvious but you are thinking about what your gain is and we are switching the brightness further down to 0% it's still a little bit under it goes down a little further it dropped to about 0.8 and as a last step of those three steps we are performing we will switch on the run time device power management and in the moment we have a switch up for the test setup which will deactivate the USB sub-system will deactivate the sound sub-system as well as wireless card and the line adaptor as well and this will be created at the moment by switching the profile at once just integrated it for the test another thing we have added peak and it's falling a little a little bit more another peak for another sub-system that is power down too scared is about to drop but now it's dropping that's what we have expected we are setting another seal about 1.5 we have better results depending on the machine the device drivers are smart enough to detect that the device is induced but already power down the device is on a less consuming state so we have been testing several machines and it's very dependent on hardware whether they are gaining anything or much so that's how we do it 7 5 and G is in the end so we have saved about 6.5 and here 0.6 0.6 almost 50% ok what we are actually I'm using at the beginning with full throttle we had about as I already mentioned in 20V are consumed we have P equals U multiplied with I and so we have in the beginning we have 20V multiplied with 1.4 and here it makes 28V so in the end we had 20V multiplied with 1.0 0.75 and here which could make 15V what's at the end and you can see we nearly saved the half of the actual power consumption here in the beginning which I think is really great what you can achieve with power down devices and all those power machine functionality the thing why we choose to have a setup like this one is that the APACS system already provides information on how much power is consumed or the battery information is consumed but this is not accurate enough so we were forced to have something like this for getting real results for several machines we were investigating and it was a live demonstration so we have a short introduction of two new things coming up in power management first thing would be what part? first part is user space software I just want to give a very rough overview about what it's all about I do not want to go into detail it's a very new technology it's already in UAMM kernels and hopefully will go to mainline soon until now software suspend was done completely inside the kernel the writing of the memory image and so on and now with the user space software suspend you only have the kernel only provides the kernel interface have snapshot currently and the kernel provides several control controls for this kernel interface like natural freeze to freeze processors or snapshot available swap to query the kernel about the available swap space and there are several others and so now the complexity of writing the image and so on is shifted to user space currently there are two main tools one called suspend for suspending the system writing the image and one resume for resuming the image resuming the system so it's an original state this has some advantages this makes things like compression or encryption or boot splashing possible while suspending the system to disk that's just a rough overview of what's new and we already did some testing and it seems to work and we hopefully can use it in the future I think next thing we will be investigating is a fetch called Gintx at the moment we have a static value for hearts for the timetix issued to the CPU exactly and with Gintx this is going to be a variable which is dynamically chosen and this allows the CPU to enter the open saving mode C3 defined by the ACPI specification more often and thus we are very empowered we have a colleague who was testing the Gintx patch on several machines and the gains are quite cool but the problems like losing interrupts remain and they definitely have to go away for integrating Gintx into a regular working environment that will be possible I think that's all first of all thanks for coming although we had some problems thanks for staying one thing we are just using the whole call interface and we are happy to have call hackers like Harman Latsekow I was working on the ACPI sub system to work I was doing a lot of work on the new user space software we had really nothing to do with the kernel of the implementation we are only finding the bugs so I think I should also mention that there is an open source wiki page about the power safe project not only the power safe demon but also about all related tasks you just have to search for power safe and open source wiki and you will find it there is also a link to a power management report where we test it and present the values from the different machines we tested so you might have a look you like thanks you can tell us which brand of laptops to say more about usually the best system we had was the other system which was pretty old I guess it wasn't a new system but we were able to gain over 60% in the system especially runtime device power management was very helpful in the system because idle devices were consuming a lot of power without being used so this was also the last part when you are starting to set up special devices it's supposed that the device has to support that feature with hardware and driver dependent I guess the driver had to implement and care about saving its data and so on we also had to place an immediate laptop and there we had nearly no results of power saving with the runtime device power management because the device is hardware responsive so and everything is very experimental we also encountered a lot of crashes when using the runtime device power management because the device wasn't up again and it was being used excess from user space do you maintain a database of which devices are able to save power I mean if you have to go for an item and then you have to choose and you have to do several everything else to choose I think power management is quite a good person to choose one or another one first time that the chips are quite intelligent if they are not being used they power down a lot without anything being done from the current I think that the IPW is a good example for consumers a lot of power is needed the IPW one and a half year ago it worked perfectly and then was two weeks or a month where the current oops is why I'm saying that's because the driver is able to do that or it's because the hardware I'm not sure but I think the driver the implementation was broken at the moment but I'm really not sure about that but do you know how many other neatness devices will allow you to detect link while in power saving mode so then you can automatically power them up or down depending on whether a cable is plugged in we don't know currently I've not seen a a module or a device where you can detect the link while the device is powered down but then maybe it's done let's say I want to buy a laptop I'm going to a store with a live CD is there any way I can test that power to save features of that laptop because I don't have a multimeter present actually you are able to have a look at the power consumption using ACPI so this might be a hint if it is good or not but don't take it for sure it would make sense to have some database like when we are doing our tests and I think especially when we are talking about chips if an ethanol chip is adding a lot of power in this system it will be the same for another laptop brand same chip so this might be a hint if you have some database with the information thanks for coming