Sorry for that. So how to do building an open stack using CloudKit is a little bit of a new key. So first of all, I think it's a good place to start doing a review of the definition of building, a telcos definition of a building system. So as you can see, we have three world-defined processes here that take place in a building system. We have the method in process that is the process to get all the information from your cloud, information that you want to build. And as a result of that collection, you will get a collection of samples that you will store to be used later in the way you want. Then we have the rating process that is the process to analyze that collection of samples in base of business rule that can be set, for example, for the marketing department of your company. And in base of that analysis, you will create the build line items that you will store to be used later also. And finally, we have the building process that is the process to assemble that build line items into a per customer build or several per customer's build to start the building collection. So today, we will go through all these processes, and we will connect these processes with the corresponding open stack service. For example, I will start talking about methoding with Silometer and Yuki. Then, Stephen will talk about rating with Locity. And finally, I will show you a brief demo about how you can do bidding reports and how to show graphics in base of that metrics with an open stack dashboard. So as you already know, Silometer is the service to do methoding in open stack. It was launched in 2012 to provide the architecture to collect all the metrics from your cloud, from different services across your cloud, then store that metrics, and then expose that metrics via an HTTP API to be consumed. But after the kilo version, Silometers, the telemetry introduced several changes into this project, mainly to resolve several kind of performance issues that were reported. So they introduced Yuki as a completely different project to store all the metrics collected from the cloud in a new format. They also add a new dispatcher, a Yuki dispatcher that is used by the Silometer collector now to collect the metrics and store the metrics into the new service into Yuki. They introduced also another completely different service that is called AODH, that is for manage all the alarms and the notifications by the result of consuming the metrics, analyze the metrics from Yuki in the cloud, and finally, Silometer, that in this new architecture, just manage the events that comes from our cloud. So host the architecture of this new telemetry system after kilo. As you can see, you have all the Silometer-Asians structure that keeps the same structure that in the older versions. You have the compute, the Asian computes that get information from your instances through the hypervisor. Then the Asian notifications for the events from your APIs, Asian central that get information about the hardware using SNMP and EMI protocols, and all the Asians that collect that information from your cloud send that information through the OpenStack notification bus. Then you have the Silometer collector that collect that information from the OpenStack notification bus, and now split that information in two. First, store the events in their own database and then push the metrics using the Yuki dispatcher to the new Yuki service. Then Yuki process and store that metrics in their own back end. And then another service that is AODH, consume the metrics from new Yuki now, analyze that metrics and manage the creation of alarms and notifications, and store that data in their own database. So in this architecture is more modular. Each one of the modules has their own database, has their own storage back end, and also has a specific role in this architecture and store a specific kind of data. So what's new Yuki and why is so important in this new architecture? Yuki is a multi-tenant time-series metric and resource database for OpenStack. The main change thing is here. It's that Yuki now stores the metric in a new format, in a time-series format, and in an aggregated manner. So this enables Yuki to store big quantities of data, being easily scalable, and also without losing the performance. And that's the most important part, without losing the performance. So to do this, Yuki used two different back ends for storing the data. First of all, Yuki has an indexer to index the resources to then match the metrics and use my SQL back end to do this. And then have another driver that is the storage driver that had the role to push to store the final metrics into usually a distributed storage. So for this purpose, Yuki used a storage like Swift or Set that are distributed storage that are easily scalable and can store a big amount of big quantities of data. Also, Yuki has, as the other OpenStack services, their own API. It's a stateless process. So if you have more metrics coming on, you can store more processes. And also the asynchronous processing, that it's also stateless. So if you want, you can add more process to process more data in parallel. So how is the new architecture with Yuki and Cylometer? As I have said, you have the Cylometer relations in every part of your cloud. This is the architecture that we build in new value. So the Asians are getting information from computers, from the APIs, from the hardware, and pushing that information into the notification bus, OpenStack notification bus, with the Ocelol Diabetes. For that purpose, we use RabbitMQ cluster. Then we have a pool of Cylometer collectors. Sorry. There are a pool of process that get and consume that meshesys from RabbitMQ in parallel, and then have the new Nokia Dispatcher configure to push the metrics into the Yuki API. And as the Yuki API, we have a pool of Yuki API processes under a load balancer that consume that metrics in parallel, then index the resources into a SQL vacuum. In this case, we are using MariaDB, Galera Active Active cluster, and finally push the raw metrics into our self-distributed cluster. So as the first time that the Yuki API pushed the metric into the self-cluster, the format of that data is a raw format, we have to aggregate that data regarding some rules that we created in our Yuki API. So for that reason, we have a pool of Yuki metric processes that get the data, the raw data, from the self-cluster, aggregate that data regarding our policies, and then push the data again. And that pool of Yuki processes are coordinated by our latest cluster. So how we did this? The integration with Cylometer and Yuki is quite simple. You have to adjust a new dispatcher, that is, the Yuki dispatcher, into the default section of your Yuki configuration file. Then you have to add a new section that is the dispatcher Yuki. You have to add some flags that are optional. But the really important flag here is the URL. You have to put there the URL of your Yuki API. In this case, we are using a load balancer. So you will put the API of the load balancer. And then we have the also message in which we put all the addresses of our ribbed cluster, and the Kiston out-talking, and the service credential, like in other open stack services that are obviously connected with Kiston. Now, self-induration. As I said, we are using Ceph as our preferred storage. We have been testing all the storage backends that Yuki provides. Ceph, it's the one that gave us the better results. And we have been working with the telemetry team to improve the performance with this type of backend. So how to integrate it with Yuki? It's quite simple also. You have to add in the source section the Ceph driver. Then you have to add all the information about the pool. You have to create a new pool to use in Yuki. So it's yuki underscore three. It's the name of the pool in Ceph. You have to put the name of the pool, the username, and all the Ceph key ring and config file. All of them are credentials that you have to use when you connect something with Ceph. It's like when you use Cinder and Ceph with multiple backends, the same configuration. And if you will be using multiple metric these to process information in parallel, you have to add the coordination URL. That is based on radius. OK. So once the metrics, we have to create different policies to aggregate the data, as I say in the previous slide. So first of all, you have to create that policies in Yuki before start the metric collection and the metric pushing. So we have here the archive policy. You can define several archive policies. The archive policies are composed by different parameters. In this case, we have the time span that is the period of time that you will store, that you will store your metric. And then you have the granularity that is the interval between the samples that you will store in that metric. So it's like the resolution of the metric. So in the first example, you have a time sum of seven days. So you will keep that metric for seven days. And you will be pushing a sample every 20 minutes. So you will get 504 points. And then you have to add also the aggregation method, like summarize, average, minimum, maximum, et cetera. Those are the aggregation methods that you will use when you use the metric. Then you have to create an archive policy rule, because you want to match every metric and put every metric with an archive policy. So you have to create the archive policy rule that works with a metric pattern, that it's a regular expression that will match the name of the metric, and will put the metric into an archive policy. Then you have to create the resources. The resources are the resources that you get from your cloud, for example, instances, volumes, hypervisor, images. You can add metadata to that resources. In case of an instance, for example, you can add the project DD, the project name, user ID, et cetera, the metadata that you want. And then you have the metrics that Synometer provides for each one of these resources. In case of an instance, you have the CPU delta, CPU utilization, instant sub time, and a lot of more metrics. So once you configure your Synometer relation to push the metrics into Niochi, Niochi will match that metrics in your resources. And then the archive policy rule will match the name of the metrics and will aggregate that data in base of the archive policy that you have in set. And as a result of this matching, you will get the measures, that are the samples that you want, timestamp, granularity, and values. And that's the measure that you will be able to see when you do a query. So some improvements in the latest versions. So the addition of the new resource IPH to create resource, so before this feature, Niochi creates automatically all the resources in a fixed manner, so you couldn't change that. Right now, you can create your own resources. And the second point, you can create the metadata for that resources. You can set metadata for that resources. And you can use that metadata to do a group patience. And that's the first point. So you can do native group buy. For example, if you will get the instance sub times, you had to ask instance sub times to Niochi. Niochi will give you all the instance sub times from all your instances in the cloud. But you want to, that result will be aggregated, for example, by tenant, by project ID, then by flavor ID, and then by instance ID. So you can do that kind of queries. And then you can filter also by metadata. You can just get the metadata for a specific tenant or for a specific flavor. And the last thing is several performance and stability improvements. We have been working really hard with the telemetry team, providing feedback, and trying to solve some performance issues that we had at the beginning. And we get a really good performance, for example, using Ceph. So this is an example. SirRoudox is a different library that the telemetry team created to use Ceph with Niochi to get a faster access from the metric DE and from the Niochi API to our Ceph cluster. So now, Stephen will talk about CloudKitty. And then I will show you an email. Thanks, Max. OK, so you've got your data stored. That's good, but you want to do billing, so you need to apply some calculation on your data. So that's where CloudKitty comes. The goal of CloudKitty is to connect to all your components in OpenStack that are gathering metrics. So it's decomposed in a few parts. That's the red parts on the schema. So basically, first step is to collect the data. We have drivers in CloudKitty to collect data from the Celerometer API or Niochi API. So you can have both activated at the same time or only use one. Every red part is a driver, a SteveDog driver. So if you want to create your own driver because you have some data in a database, for example, some custom customer data, you can collect the data too and push it in the rating pipeline to do specific calculation based on business data, for example. When you've got your data, you need to apply calculation on it. So it's the rating part, the part in the middle. As I said before, simple plugins, you can have as many plugins as you want. And the calculation is done on every plugin. So we have a few plugins in CloudKitty, some plugins to create your rating rules directly in Python. So you can create Python script that will work on the data and do calculation. Or we have a hash map, as we call it, rating module, which is a way to model your rating rules with some objects. So you don't have to create some specific equation to do your calculation. You only create objects and map based on the different object and do your calculation this way. Every configuration of CloudKitty is stored in the database. You have your basic configuration, which is in configuration file, like Kiston of tokens and all the Kiston integration and the basic integration of every collectors because you need to connect to Kiston. But when you're set, you can use the same configuration file for all your processes. And then you configure CloudKitty directly from the API. The goal is to be able to scale up and scale down easily and configure it directly from the API so you can give the right to some people to modify the configuration without the need to access to the servers. We have some integration in Horizon, so you can directly modify the configuration and the rating rules in Horizon. And it's better for security because you're not allowing some salespeople, for example, or marketing people when they want to create rating rules to access the servers and modify configuration files that might look a little bit shady to them. And when you're done, we store the data. So we have a native storage backend that was the first backend that we created. So we store it in SQL Alchemy. It's a SQL Alchemy backend, so basically it will be Postgres or MySQL. And we can store the data this way. All you can use, since the Mitaka release, a new key backend. So basically, you get a reference to your new key information, and you only store the rate. So the final, final calculation. This way, you benefits for all the performances from new key, and you don't duplicate the data. Again, you can create your own storage driver if you want to. And there is a common API on top of storage. So if you're using new key or our native storage, there is only one common API, and you don't need to care about what's behind. About the state of the Mitaka release, so it's our first release with OpenStack. And what we work with in Mitaka is the integration with new key. So we created the first iteration of the new key driver. Basically, we can collect data from new key. So all the data that is in new key, we can collect it in Cloud Kitty, so you can do calculation on it. And we support a hybrid storage in new key. So we don't store all the data in new key because some features were not there at the time of the Mitaka release, for example, the dynamic resources. So if you collect data outside of new key, you need to create resources. And we weren't able to do it at the time. That's why we didn't integrate it fully with new key. Basically, you keep a reference to the new key metrics and resources, and you only store the final calculation. So it's only aligned with the final calculation, and all the metadata and metrics are still in new key. So you can still leverage the scaling capabilities of new key. And we improved the scalability of Cloud Kitty. We added a DLM, which is basically a lock that you can distribute in a cluster. So you can have as many nodes as you want, processing nodes as you want, and shatter your calculation across all the cluster. And Q&A, we added a lot of tests on the storage part to be sure that we don't have any consistency. So about our work with NubeLU, we first met in Tokyo. They were really excited about the Cloud Kitty project because they needed to do billing, and it was a good way to integrate with an open stack and open source component and to make it go in a good way. So basically they told us they need new key integration. We knew that we needed it in some time to integrate with new key, but as it was not requested by people, it was back in the roadmap. And they really told us that they wanted new key integration, they were able to test the code and run it in their servers on real workloads so we can have huge feedbacks. So we put new key on the number one spot in the roadmap, and it was really, really good to have them as a user because they gave us review on code. They tested it on their open stack clusters with real data so we can get a real sense of will it be scaling? Is it working with real data and not test data? You get some data that you might not have thought when you write a test and all this stuff. And they committed patches to what we did. And last slide's about Cloud Kitty before the demo. What we will do in Newton. So the first step in new key was a good step. We improved scalability issues and we don't relate on that much code from Cloud Kitty part. So all the storage will be done in new key and all the collection will be done in new key. So we only do rating and we don't need to focus on storage for example, which is a good thing because people can focus on new key and improve at the same time storage in Cloud Kitty. So the goal is to improve the hybrid storage of new key, create a new storage that will directly store data back in new key. So you'll have your raw data and the result of your calculation. The goal is to be able to use the same API. So you can use new key API and graph. For example, if you want to do graphs, you can query only one API and you graph off your raw data and your calculated data. And you don't need to know two APIs which is new key and Cloud Kitty. We have new collector models coming up. It's part of better user experiences. In some ways, it's pretty hard to know what the data is looking like. For example, if you want to apply rating on an instance, you need to know what the fields and the metadata field will be. At the moment, you need to have some insight on new key and know how the data is modeled in new key. With the new collector model, we will self-document on the API what the object are looking like. So you can have an API call or in our Iceland directly consume API and show to your user, okay, this is an instance object. This is all the field. For example, you can apply rating on the flavor and then you can directly show them what the field looks like. And as I said before, improved storage model. So we will mostly use new key to have better performances. Perfect. That's time for the demo. So, okay, so now we're going to see how to use all of these APIs in the real life. And this is based on our dashboard. It's a new value dashboard. It has several changes regarding user experience and usability from then. But we also added some features like the possibility to do reportings in different ways to explore that reportings, to create charts that are like a graphic way to show the users the metrics and the measures and also the dashboards that we will see in a couple of minutes. So suppose that you have, this horizon is connected to our cloud. We have all the salometer agents pushing data into our new key system. And we will be calling to that APIs to the Cloud Kitty and to the new API. So first of all, we will create new rules because we have the metrics in our new key system. So we have samples, but we need to create the line items with currency values. So I will disable the rating model first because we don't want to create inconsistencies. Then we will create a new rule set that is a set of rules, of course. We will choose these device write bytes. We will choose the unit in which the rule will be base and then an aggregation function. In this case, summarize. Perfect. So once it's created, we will manage that rule set and we will start into create rules. So we will create a new rule. It will be a flat rule and we will put here $10. So this rule is quite simple. We will charge $10 per gigabyte written in the disk of an instance, per hour of course. So you can create more complex rules in a rule set, like the threshold rules that are rules that, for example, I can say I will charge $10 per gigabyte, but if the user writes more than one petabyte, I will charge $10 extra per gigabyte. Or maybe if you can create rules in base of metadata, that are metadata rules, that for example, added to this base rule, if the user will use another flavor, for example, a platinum flavor, we will charge $10 extra. Or if they use a gold flavor, we will discount $5, for example. So once we have all our rules created or our set of rules set, we will enable again the rating module and we will go to the report section. Showback, the report screen. And now we will create a report. That's what we want to do finally. Okay, so now we have the measures, the samples, and we have the line items. And we want to show that the line items in a report. So we will put the name, we will put if the report will be able for common users or for administrators, a little description. And here we have all the available metrics in our NEOGI system, in our CloudKiddy system. So if we want, we can show the costs, or we can show just the measure. So you can choose the granularity from your storage policy. You can choose between several granularities and the timestamp. In this case, one day of granularity and seven days of timestamp. Then we can put a description of the metric because instance, it's the instance sub time but nobody knows that. So we want to show that it's the instance sub time of all the instance from my cloud. We will just show the cost, not the measure. And we will select the group by, the metric groupings and the order. So this is the group by feature that I told you. We will get our report divided by project ID, in this case, then flavor ID and then instance ID. And we will have the partial cost of every one of these levels. Then if you want, you can, for example, filter by a specific project because you want to create a report from a specific project or from a specific flavor. So this is creating, I have created another one to show you, perfect. So this is how the report looks like. So you have the period that it's the granularity, that from your report, we will choose here just one granularity that is one month, 30 days. And then you have a navigation bar in which you can move from different periods. So we are seeing the month of March, you can move to the previous months, to February and so. Then we have the total cost of the month, in this case, $75,000. And then we have the partial cost of the projects, for example, in this first screen. And you can notice that we have two metrics here, the CPU, Delta and the instant sub time. And we are summarizing these two metrics. And we have the partial cost of each one of the projects. For example, the admin project, migration test, that is another project, another project that you have in your cloud. If you, for example, drill down in one of these projects, you will be the partial cost of every one of your flavors in your cloud, for example, the large flavor, medium flavor, et cetera. And if you go deeper, you will have the partial costs of every one of the instances in your cloud. So if you go to the previous month, February, you can see the same information. The total cost, $38,000, then the partial cost of every project in the last column.  So what you can do with these reports. So far you can export the reports in two different formats. One is PDF, and the other is the CSV. So for example, you can export the report and open the report with another application. It can be Excel or can be your own billing application. So you can import that report in your billing application and you can, for example, generate electronic bills or paper bills as you want. Perfect. Another way to use the metrics, it's with graphics. So we have created a new feature to create different kind of graphics in base of metrics. Those are for us, the charts. So we will create a new chart definition with the name of the chart, with a single description. And the chart is created in base of just one metric. It's not like the report. In the report you can choose several metrics. So far in the charts you can select just one metric. And then you have to select the same parameters. The first, sorry, the chart type, if you will use a pie or a column or wherever you want. Then the granularity and the time span of the chart. Then you will choose if you will show the measure or show the cost, you can choose between each one. And the agroupation, like the report, in this case we were grouped by project ID. And then we save this. So once you have all your charts generated with different graphics, we will create dashboards. The dashboards are the way to show, to use that charts. So we will create a new dashboard. We will select if we'll be able to see by a common user or by an administrator user. Then you have to add the charts that you have created. For example, in this case we will select a lot of charts. CPU Delta, CPU time cost, network, average, last month cloud, boom, done. So you have all your graphics there in your dashboard. This is a template, so you can reorder your graphics. Maybe if you have some more important graphic, you can put that graphic at the beginning. Then scale up or scale down your graphics. Those are like JavaScript based graphics so you can interact with the graphics, filtering by, in this case, it's a CPU that you can filter an entire project. Then you can add again of the project. You can interact with your graphics. Then we will put a name and then a description to our dashboard and we will save this dashboard. So these are really good way to show other people, to show other departments how they are spending their money in their cloud. Or maybe to show to a director what's happening in the cloud, how it's divided the cost of regarding the projects or the flavors or the instances. So then they can take better decisions. And it's a really quickly way to see what's happening. You can go over the different parts of the graphic and you will get the partial cost of every, for example, in that case, were flavors. And another functionality is that you can detect if you are having some issue. For example, in this case, this is a bandwidth usage. So if some instances, it's having peaks in the use of the bandwidth, you can just select a period of time, zoom in and see what's happening with that particular instance. So you have, as I say, different kind of graphics and all of them are interactive. And another thing that we can do with this is select this dashboard as our home dashboards. So for the admins and for the common users. So for example, you can select the dashboard as the full home of a new user. And when a user, a new user logging into our OpenStack dashboard, they can select the building home and they have our Picasso with all the graphics and all the charts, the selected charts that we choose to them to see. So that's all. Thank you to everyone to be here and we will respond to some questions. Yes, here we have our contacts, sorry. Just. I had a question on the graphics. Can those be embedded in other dashboarding technologies? For instance, is there a URL where you can call, get that graphic embedded somewhere else? Yes, of course. You can do it. Cool. Yes. We are using an engine, Java engine. It's really common. So you can use it in the way you want. Okay. We are just calling the, using the RESTful API from CloudKiddy and from... You can get a JSON document back and... We get the metrics and just show the metrics. Thank you. You're welcome. So for your backend SAP cluster, how hard does this metrics gathering hit that? Do you recommend a dedicated cluster or is this going to interfere with other uses of that SAP cluster? Well, that's depend of your architecture. I mean, the problem is not the size because the metrics are quite, also compressed, compressive. Yes, it's really indeed compressed. Yeah. So you don't have a lot of size but the performance is different. So the problem is the IO in your SAP cluster. So as you will not need a lot of space, we prefer to use, for example, a little cluster of SSDs or something like that with high performance and low space. Yeah, because basically in your key is slicing your data in small files and compressing them. So it uses your indexer to find where is your data and get it back and show it to you. So you need to access multiple files if you want to have a huge time span. You need to access multiple files so it can be IO intensive. So the key is the IO, the throughput, not the size in the cluster. So we have like 200 instances and all our self-cluster is about one gigabyte of metrics. Someone else? I have a question about the rate calculation. So is there any modeling? Actually you are doing like per server, per power consumption or like a space. So merge it together and say, okay, that's a recommended rate or you already know. No, you have all the possibilities. You can create your own combinations. We are working today in a capacity planning feature and a comparing feature to make comparisons between different metrics and to do some statistic operations, to have a capacity planning feature. But you can do whatever you want. Another question is what's the scale, actual limit right now? How many instance or user or host that you support right now? Well, I don't know the final size that New York will support. We have a cluster of 10 physical nodes with 20, 25, or 13 instances each one. And all the metrics that Ceylonter provides that has been pushed into our new ecosystem. But it's not a big cloud, we know. It's like a medium or small cloud. But we expect to test this in a bigger cloud in the next months. Do you have any plans to support collection analysis of say, public cloud providers like AWS is your Google or not? Perfect, so Nuke's support, Nuke it's a 10 series metric database. So you can push the metrics that you want. You can create the resources and push the metric that you want. Right now we are working in some agents. One for being aware and another one for public clouds to get measures from those clouds and push it into our dashboard. So in the next months we are working to create this kind of agents and try this new, it's like a new business because you will be getting information from other clouds and pushing into our dashboard and showing and do billing with that information. It's an idea for the next months. I don't know if there are official agents to... Now you can do it multiple ways. You can even create a driver for Cloud Kitty so you can pull data from a public cloud.  Or you can create your polar for new key. So you fetch data, you store them in new key and we process new key data like we process every data in Cloud Kitty. It might be better to do with the new key polar because you're directly storing data and you can run Cloud Kitty when you need and when you want and you don't have to ensure consistency of your data in the input. You know that it's in new key and as soon as it's in new key stored then it will be always there. What kind of tools you guys use for performance, measuring performance, and what was your ideal architecture for measuring performance? Tools for... What was like, like you use a rally or anything like that to measure performance? How did you measure performance of your new key? Oh, okay. The performance in new key? Yeah. Okay. Well, you have different ways to do that. Basically, you have three parts in which the performance is important. The first one is in the storage backend, in SEF. So in SEF, SWIF, or the backend that you want. So there you have different tools like FIO or Rados benchmark or something like that. Then you have the part in the gravity MQ cluster. We know that gravity MQ could be a problem. So you can... You can monitoring your backlog, your message backlog. So you can see there if you have a message that is still are not processed and you maybe have to add more metric these or tune your Ravite MQ cluster. And the another part, it's all the APIs from then or the YPI spool and the metric D processors. So you can, in this case, add more processes to support more incoming metrics in your APIs. And in the case of metric D processors, you can add more processors and you have to use Redis to coordinate that metric D processors to process more amount of data. Because you have a backlog in Ravite MQ and you have a backlog also in SEF. The metric is pushed into SEF in a row format. So you will be a list of metrics in SEF. And that metrics, that row metrics should be processed by the metric D processors. So if the backlog is too big, it's not so good. So you have to add more metric Ds to process data in parallel more and more quickly. Yes. I have a question. The metric list retrieved from Celerometer as time goes on or in a very short time, it will be a long list, right? Included deleted resources with metrics and this list will be displayed in a drop-down list. Does this question, this buzzer is due? I didn't catch that. No. Do you mean when we process Celerometer data, when we collect Celerometer data to process them in Cloud Kitty? The metric list is displayed in the drop-down list. Actually, it's metric list from Noki because they only support Noki. Oh, Noki. Cloud Kitty, at the moment, you don't get directly the metric list. You know what you're supposed to collect. You can activate collection of some specific metrics in the configuration, but you don't have a dynamic list of metrics you can collect from Celerometer. We might plan on adding that in the future, but we are mostly focusing on Noki. We're doing Noki and Celerometer in parallel, but we want to have 100% support of Noki, which means collection and storage and full storage in Noki, and then we will go back to Celerometer because most people, when they have a big cloud, they want to transition from Celerometer to Noki storage because it will scale better and they have better performances and when you want to do building or graphs, you want some backend that will respond really fast to be able to graph it. Your user don't want to wait 20 minutes to get a graph, for example. So most people will directly use Noki API. That's why we want to have a full support for Noki in Newton, and then go back to Celerometer and try to improve the support. Is that answering your question? Okay, so we are running out of time. So thank you to everyone and if you have any other question, we will be right there to answer that. No, not this time.


Okay, I guess it's one pass now, it's about time to get. I'm still in a tutoring, I'm going to talk today about SystemDs, the CoroS. The CoroS is some relatively newly coined terminology, it was actually coined in an LWN article about SystemD and we like to term very much so we adopted it and this talk should highlight a little bit what we actually understand as the CoroS, why we think SystemD is the CoroS or should be relevant for the CoroS and all the things around it. Yeah, I mentioned that yesterday already I like my talks interactive so if you have any kind of questions please interrupt me, I much rather prefer if we turn this talk into the direction you guys want rather than just what I like to talk about. So if you have questions just show up and answer them right away. Okay, so as mentioned the CoroS terminology for LWN, it's a bit of a vision of what SystemD should be now, when we started with SystemD we of course started with, we wanted it to be an inner system. An inner system is responsible for bringing up the system, that's why it's called inner system. But as we progressed with this vision we figured out that maybe it's not sufficient to just focus on the inner system alone and SystemD turned into something of basic building, a set of basic building blocks to build an OS from and for that we now have this nice term CoroS. So what the reason we shifted the focus a little bit from just being an inner system to being a CoroS. The reason is basically when we looked at solving the inner situation we figured out that well just being PID1 and bringing up the system is not enough because actually the inner system is not the only thing that is crucial for bringing up the system. Usually in traditional Linux in Unix there's a huge number of scripts which initialize certain parts of the system as well which are not part of the inner system traditionally but which are shipped along with it and are very much distribution specific. These can be various things for example like starting UDEF and doing device discovery and loading kernel modules. It could be something like registering new binary format that could be emptying slash temp, it could be all kinds of little things that each by its own are very simple and minimal but you all have to have them to have a proper operating system to be built from. Distributions had different scripts for that but they all very much did the same things like cleaned up slash temp and all the things that I just mentioned a couple of other things. Did it all with different code? Is it all different quality? So we looked at that and figured out okay we have cleaned up the inner system but it really would be interesting to clean up this basic set of inner scripts as well and replace complex shell scripts by a relatively simple and minimal AC program. So yeah we then started to move a couple of things into this to be proper meaning that system returned into just being one process into set of processes a set of very minimal processes that are responsible for very specific parts of system initialization and yeah bit by bit this vision extend a little bit like we move more and more stuff into it for example nowadays we have a replacement for console kit which manages sessions. The reason we move that in is very similar because we figured out okay service management is all about resource control and by about setting properties for services like process properties and the C group properties and all these kind of things and we figured out well services are not the only processes running on a system. You have all kinds of processes running for example also user processes I mean if you log into your GNOME session and that's not a system service that's user code basically and it is very interesting to set all the properties that you can set for service managers for the services you can also set for the user services you want that because I mean one interesting use case for example for C groups is that if you have Apache running SQL and Apache has 1000 CGI scripts and my SQL only has 20 worker processes you still want my SQL and Apache get roughly the same amount of CPU so for that you can use C groups but the same problem also applies to sessions if one user logs in and creates 1000 processes and another user logs in and only gets three they still should get in total each of them is the same amount of CPU and that's where C group comes in as well so you figure out okay so a lot of things that apply to the service problem also apply to the session problem and so we figured out okay maybe if we have that code anyway we should extend that and also include something consequence like the manager session tracks one user log in applies properties to them wraps them all nicely in C groups and these kind of things so yeah basically the vision grew a little bit by bit we added stuff here and there everywhere so yeah to summarize this vision of what the system in that case should be the core I should be is a minimal set of components to build an OS from of course by with the time the minimality increased bit by bit I guess that is I mean most computing programs end up growing growing growing that way so by minimal we actually don't really mean the absolute minimum you need to to to build an operating system from like if you want to do that then there are other solutions like busybox kind of things we don't want to have like go into that term but what we kind of try to focus on is providing everything that 90% of the use cases require and 90% of the use cases require also covering all the important bases like desktop stuff server stuff and other stuff as well the core OS having this on system D does not mean that system D should be one monolithic thing that covers everything but it means simply that we have a set of services that are tightly coupled that integrate nicely that behave similar that share code that reduce duplication but that are still individual processes so if you install system D these days it will cover a lot of ground but it's about I don't know 50 or so individual processes at boot we started at least I don't know 10 or 15 of those so yeah it's about it's not about making things monolithic not about pulling things all in the same process it's about simply streamlining the code a bit unifying things unifying things between distributions and unifying code paths where we all have the same stuff the core OS should be universal it should cover all the architectures and it should call all the users as mentioned servers containers and desktop all this kind of stuff we want to cover with system D we kind of want to focus on the same bandwidth as Linux itself focuses on and the same way as Linux works on mobile servers and embedded on desktop we want system D to cover the same ground as mentioned not everything like if you want to go really embedded then go for dvbox but everything that is like the more common stuff universal does not mean that we want to cover other kernels or anything like that we strictly want to focus on Linux with system D as the core OS we want to manage all the high level OS objects traditionally on Linux kernels and certainly concept to kernel new where basically files and directories and these kind of things and processes and a couple of other stuff like sockets we want to add a couple of or we added a couple of high level things that are now managed by system D of course services services we actually define them in terms of C groups mostly so that nowadays you have the identity of a service actually exposed as kernel objects by means of C groups and we did the same for user sessions for containers and apps where everything basically now exists in all layers as a stack on the kernel level of C groups and in user space by a user space maintained object so yeah that's our vision a little bit like basically finishing the high level object view on the system where basically postings left off and didn't go any further and it's basically just realizing the fact that I mean traditionally on Linux services we're just a single process each but nowadays that's not really how systems work because they haven't been in a long time because Apache is hardly one process it's a set of processes, one process, a couple of worker processes a couple of CDIs kind of things so yeah this vision of the core OS we try to basically complete this thing and have the same definitions that we have on all layers of our stacks of these high level objects system D as the core OS one of the key aspects we try to focus on is introspectability introspectability means whatever state the system is in currently you want to allow the administrator, the user, the developer be able to look into it more specifically than many cases most of the core components we have you can connect to primarily via debuts or other like individual components have different uses there but we basically give the administrator and developers complete access to the channel for system D mostly readable access that sometimes we can control it transparency means also that you can look into all the details specifically this can mean for example filing system D because we figured out a lot of things that new system booted up is very useful information and we generate this nowadays with system D build in so that when you boot up it will just measure everything it can tell you how much time the bias took, how much time the boot loader took, how much time the kernel took to initialize how much time the initial round was cooked to initialize and how much actual final user space it took so yeah we want to focus on making everything introspective so you always can figure out what's going on, how long did it take and what was involved in which order it was executed as mentioned a couple of times actually system D as a core OS is supposed to be something you build real world systems from so it's not the absolute minimum that boots it's more than that this of course is like if we decide if something belongs in the core OS and belongs to system D we of course always have this question like yeah where do we put the limit that's going to be more about that later on but yeah we knowingly put the limit not at the maximum absolute minimum which some people probably would do then we want to make sure that whatever you put into system D should be zero maintenance by that we mean that whatever happens in whatever condition you run this stuff it should not suck up more specifically like let me give you an example what we mean by that if we look at things like SysLock work now and SysLock traditionally if lock files were collected and went to disk rotation would come by time after a while rotate the files and free a little bit of disk space and delete those files basically and then rotation is finished and then after now it would come back into the same thing now this kind of design like from the first look at it kind of looks safe because after some time the disk always gets cleaned up and if the disk runs full then you just have to wait an hour and everything is done but if you actually look closely at it it's fundamentally flawed in a way because if somebody manages to generate a lot of lock messages in a very short time period he kind of can DOS a system and until the next time the lock rotation actually takes place the system will be basically clock nothing can write to disk anymore because the disk is full so in the system we context we added a little bit of logging functionality into it we really wanted to avoid these kind of vulnerability and what we did basically is that in the general context in the system we log in frame on contact everything we lock to disk we first check if we can actually do that if the disk is full and stuff like that and hence do the rotation in line synchronously to how we write stuff so yeah and then this thing is this kind of goes like this is kind of thinking of designing system that whatever happens we stay running and we don't knowingly enter a condition where it doesn't work anymore goes through the entire design I mean this ultimately is a lot of other things for example we put a strong focus on allowing the administrator and developers configure their services in a way that they automatically restarted if something failed this is something actually that was solely missing in system 5 in it because in system 5 in it when you start a service that basically all that happens and nobody watches if the service ever dies nobody is there nothing is there that could restart the service when the service dies and make sure it goes up again which is actually huge there's a huge hole in the entire model of Linux traditionally because I mean how can you do like a reliable service if something fucks up and you cannot automatically recover from that but this also means a couple of other things like in system 5 for example we have socket activation which allows a lot of interesting things because we can recover from damage dying without actually losing the connectivity because the socket part of the communication part of the damage always stays around and is maintained by the inner system and if the actual processing part dies then of course the specific transaction it's currently working on will fail and will be terminated but the actual connectivity is now a lot so yeah the entire process of design that we had in that was always to keep in mind whatever we do we want to have it in a way that it is reliable that it gives ISOS automatically reliable by the way so we want to have it in the design or at least give the administrator and developer the means to make it reliable by doing automatically starting this one the minimize test matrix is an absolutely key thing for us minimize test matrix by that we basically mean we want to remove a lot of variables from the base system the minimize test matrix basically means like if you look at the distribution like Debian where they have 10 implementations if you want to test everything then the test matrix becomes huge because you have to test every combination against every combination and the more combinations you have the larger the test matrix goes and effectively this means is like the more variables you have in the system the more unlikely is that you will actually find the bugs in every possible combination so we want to minimize the test matrix with stuff that we do in the system so basically we want to remove variables that make no sense anymore this has various effects like for example traditionally on Linux this is really simple program called Getty. Getty is responsible for bringing prompt on a terminal screen so that you can type in your username password and it will lock you it's an absolute trivial thing it will just really just ask for your username password then check TAM like the security the authentication library and if it's okay it will spawn a shell for you there's nothing like there's a few programs in our state that are trivial and more trivial than that now as it turned out over the history of Linux we got like seven implementations of that the core distributions ended up shipping at least three of them A Getty, Min Getty and something called Getty all these programs were completely trivial and could all do the same more or less functionality we looked at that and figured out this is bullshit all distributions pick a different one by default and all distributions ship them all we shouldn't do that and then there was all this irony that somebody created this Min Getty thing which was the minimal Getty which had a smaller binary size that actually took more memory than the other Getty that everybody else used anyway so we said we're not interested in testing all these combinations we don't want this kind of unnecessary variables in there and we just said yeah, it's just a myth we will default to only one of them to A Getty from your Pula Linux which is the best-maintained and one of the most featureful and that's how we ship our stuff and that's how everything will work by default but we will still make it possible that if people want to do something else they can plug it in but they have to do manual configuration for that yeah, this minimized test matrix goes through our entire design as well we try to look at the different components of our stack and really ask ourselves if we want the variability in there if we want to have this ability for people to replace a component unfairly in some cases it totally makes sense for example if people want to run Apache or Nginx or whatever kind of a piece of it it's totally fine I should always support it because like in the Getty case I don't think there's any point in making that overly configurable and suggesting this the effect of that is basically that if you run system D system because all the components we suggest you default configuration meaning you have to use that but yeah this default configuration because probably most people use that as the best test across the board and if it's best test then it's much likely to be actually robot another thing you wanted to really make sure about system D is to and then as used in the core OS is design the whole stuff in a way that it is useful for read only system images which is interesting for a lot of different use cases it's interesting for embedded case because yeah if you have the entire operating system in a way that it's entirely read only then you're less likely to run problems if you suddenly power off the machine but this also is very interesting for containers and things like that because if you are in a container containerized environment like cloud and stuff you usually have 5 front containers and they have very similar boot images if they don't have exactly the same images and if you can make them read only then you can be sure that you can basically run all the container from the same image and if one container changes the image that's just not possible because it's read only. So with system D we really tried hard to separate the read only parts from the operating system from the actual state parts of the operating system so that these kind of use cases work. Ultimately our goal with that is even it's probably not realistic to ever achieve that for normal systems but it's very realistic to achieve that for embedded container setup is that we actually want to be able to ship an empty slash EDC so that the definition of slash EDC is not anymore to ship defaults and these kind of things I mean it could still do that but it wouldn't need that so all the components in system D that we wrote basically they have default built in and if the EDC is empty it will still boot up with all these defaults and will just work and will not require any configuration but if you drop something in a slash EDC that is basically when the administrator changed the defaults and overrode what the default were. So the basic ideas it's a little bit inspired from what Android does in this era like if you have an Android phone they basically have three parts of the flash, three petitions one is for the system image, one is for actually user data like music files and photos and things like that and one is configuration and with this model of the allowing empty slash EDC and allowing with only slash user we kind of come close to that with model and as mentioned this is super interesting for more embedded kind of things and super interesting container kind of thing but probably not necessarily over all the back of really the depth of it. Another thing that we want to go for is state lessness. It took us a while to realize how awesome state lessness is. State lessness basically means that if you boot up a system it doesn't require to save any state to work. Let me explain this a little bit like I don't know most people probably use windows before if you have a windows operating system image and you boot it up on one machine it will write a ton lot of registry entries because it will just say okay I found this hardware let's say I'm crossing the disk about that and if you use the same OS image and boot it up on another system with slightly different hardware then suddenly there will be a ton lot of more registry things and everything just goes really bad. With everything we want to design for have designs so far with system D as the core OS we really made sure that the entire thing is state less meaning I can have the same image and I can boot it up in a container and I can boot it up on this bare metal machine on server and I can boot up on this bare metal desktop or something like that and it will work and it will not save anything to disk so that it's completely state less the same image can be used in any possible way and we actually really like this being able to boot the same image on bare metal on virtualizing solution on the container. It's actually really really interesting because with the system D image nowadays you can basically start your server with bare metal and then move it into container and at least all the system D components will automatically adapt to that and do the right thing. So yeah, statelessness we really funded this. Then here's another thing we tried to do with the core OS model we tried to basically copy a little bit how the BSD model works and the BSDs are, I mean I assume most of you know that is Unix implementation, the classic one, it predates Linux a bit and it's kind of lost traction, it's not popular at the next stage but their model of development is different from how Linux does that. In Linux like the kernel is developed by one group and the Lipsy is developed by another group and Utilinux by somebody else and system you by somebody else blah blah blah blah so you usually have, you pick a lot of different components from a lot of different people and then you build a system from that and you take different versions and you mix and match and you basically have this bazaar component that you put together in the way that you want it. The BSD model is different, the BSD model, everything is basically developed in a single repository more or less. So the Lipsy is updated at the same time with the kernel and the user space components are all in the same tree. We kind of wanted to not go all the way to the BSD model but at least move things a little bit closer. So basically effectively this means that we moved a lot of the basic building blocks of an operating system into the system tree, like Utilinux kind of thing for example and develop them in line so that the integration can be synchronized and we don't have to maintain all the different compatibility for all the different versions all the time. This helps us reduce the test metrics. This helps us to move a little bit faster because instead of updating 20 packages or 100 packages you can just update one. This does not mean that you can build the entire operating system just with one package but it will use the set of packages you need to build operating. So I mean traditionally if you want to build a Linux operating system you kind of need 60 packages or so. You need Lipsy, you need the kernel, you need Utilinux, you need agetty you get and these kind of things with by adopting this little bit going model that goes a little bit into the BSD direction in the system tree we hopefully kind of reduce that to just require kind of a set. The other thing is, yeah we try to innovate a little bit within the Unix design. There's actually, I mean there are a lot of people who are saying Unix, oh my god Unix is awesome and SystemD is totally not Unix and so SystemD sucks on this kind of thing. Actually we as the System Guys do believe that Unix is great inspiration. It's created something durable and we should always drive inspiration from that and we actually believe that in the system we often do that. More specifically this basically means yeah we try to, I mean in Insisting there are a lot of components that are kind of replacing specific components that people traditionally understood as the Unix. For example we have like the journal which kind of replaces this lock but not really replaces it. But this thing is and we believe that Unix is mostly about the concept that they used. More specifically in Unix everything was a file. In SystemD we said okay if everything is a file then we can go one step further and actually say that services are a file too. So in the SystemD world due to using C groups to group processes and C groups being exposed in the file system, we can actually say okay yeah we went one step further to be more Unix-ish. Because yeah in the file system you suddenly see old services and not just processes, that's the problem. But that's just one example that many other examples like for example in Unix traditionally you have this TTY logic. TTY is teletype, it's basically on traditional Unix system you have basically one central, you could have one central server and a couple of terminals connected to it and each terminal was basically just a keyboard plus a screen. And yeah this is how a machine worked in the 70s, 80s and 90s but not anymore really these days. But the concept is actually kind of cool because you have just a single machine that you have to administrate and it's cheaper and it's more energy efficient that you have to administrate but you have a couple of users on the same thing. With system we said okay it's a cool thing but we probably need to bring the concept to what is today required from a workstation because just the keyboard on the screen is not anymore what people expect from a workstation. Nowadays they expect the sound work but there's a mouse that is graphical that there might be in the webcam that might be something where you can plug in the USB key, your storage device will actually show up on your screen and not on your neighbor's screen. So yeah we looked at that and thought okay we can make that happen again, we can bring this bit of Unix history back, updated to how systems work these days and we did that by multi-seat support. Multi-seat support basically just means that due to creating these concepts that I mentioned earlier of sessions and seats in users, these high level operating system we just said okay yeah then let's right away from the beginning introduce multi-seat support. So basically it means that system is actually capable of tracking which device belongs to which seat and if you log in on a system system it will always keep in mind okay you log in on that seat and hence you have access to this set of hardware and if you log out and somebody else can log in then hardware is assigned to that user depending on which seat you log in. So yeah that's another thing where we kind of looked at back at what Unix actually was about and said okay that's cool let's bring that back, it kind of lost history but it was a good idea. So yeah and a couple of other things where you basically just said okay this is totally Unix. Again, I'm pretty sure a lot of people totally disagree without saying system is not Unix at all. Another thing that we try to follow with the system as the core OS is that we really should take what Linux offers and provide the user with it and empower the user with it. That basically means in Linux since time began basically people cared a lot about portability so they wanted not to limit themselves to one specific operating system or Linux but they were kind of focused on APIs such as POSIX and developing the APIs against POSIX and little else so that it would run on the BSDs and Solaris and all these kind of things at the same time. This is an interesting concept and it works quite well the higher up you go in the stack but we're systemally being like yeah basically the building block for the West which is way further down. We saw that it's probably not the best concept to follow. Since the Linux kernel actually offers so much more beyond POSIX that it's really useful to have. For example I mean starts with C groups like resource management. Running services these days is mostly like one of the primary facets of service management is the resource management. Regardless if you have an AMD device or a huge server, if you have an AMD device you probably want to run like 10 services on them and you have very little resources and you want to really distribute the resources you have in the way you want across several servers you have and hence resource management really matters. If you have a huge server you have a lot of resources but you probably run even more services on them because you have a lot of customers who sell a bunch of machines too for example. And if you have 500 machines there you probably want to use your machine to a very high degree because you made investment in the machine and buying it and hence the more customers you can sell it to at the same time the more money you'll make. So whatever side you look at resource management is absolutely key for running a modern computer. So resource management in classic post-exit is really minimal. They have resource limits they call their prep process but it makes them kind of useless because not even a web server like on a web server that doesn't even make sense anymore because a web server usually has a lot of workers right with CDR scripts and things like that and setting a prep process limit hence has very little effect because people can just park off and they get a new set of limits. So yeah we decided we looked at the Linux kernel and found so many awesome things in that that make real life like administrator lives and so much more useful that we say we will not limit our self to purpose. We will actually expose what the kernel has to offer awesome things awesome technologies and that covers everything like resource management as I just said a lot of security things that the kernel exposed hardware management all these kind of things. We really want to give the user the full power of the system and expose it as if it was normal and not add this artificial limitation and other things that we try to do with systemally as we quarrel as if we run into problem we always want to fix that problem instead of working around it. That basically meant that if something we need in the kernel then we will fix it in the kernel rather than work around. This is traditionally like probably done differently when people build in a system. I mean if you look into the initial scripts of various distributions you find tons and tons of sleep loops where they wait for something to happen and they sleep for one second and check if it happened now and then if it didn't they sleep again and all these kind of things that just grows hacks and we basically when we sat down and started with this and we said we'll never do that. We will really try hard to always fix the problem where it is and fix it properly instead of adding a work around it. Another design goal with systemally as we quarrel as is that we never want to be excused. That means while we will replace a lot of functionality that has been traditionally implemented into various unique services we never want to force people to actually use that. People can continue to use what they want. For example, more specifically this means you can run services based on time. You can say run this service every five minutes or so, or run this service on November 5th, 2013 at 6 a.m. in the morning. This of course has functionality that had been existing before in CRON and people are CRON and CRON is often because it's really simple to use. You just write one line and you can just find that. So when we added that systemally it's really useful to have that in system D because system D is really good at starting services. You can specify all the resource limits, all these other properties that are available for you. It can restart things, it can watch stuff, it can collect logs of the service which is much better than what CRON can do. But we acknowledge that people love CRON and people are across any kind of product from the page. So yeah, we never are excused. We don't have people to run whatever they want even if we provide something similar or equally powerful or somewhat related in the basic operating system. So this is like everything that I was talking about, just this fluff, divine thing that we kept in mind while turning system D from this inner system into this CoreOS kind of thing. The next part of the slide really focuses on the specific parts where we kind of took one of the existing components and replaced in system D itself. I hopefully can explain why we did this. So the obvious case of course system D replaces system 5 in it. Most distribution had something called Start Stop Demand which they, because actually spawning services from childs could suck, they wrote a little bit of a glue thing between called Stop Stop Demand which makes it starting to look a little bit nicer. It's all gone. In its scripts as mentioned, all the distributions had a different implementation of that. It's just a set of basic scripts that everybody used to set up to those modules, blah, blah, blah, blah. And we replaced that. These are kind of the obvious things because yeah, it's what system we started out. We kind of replaced INED. INED is this thing. Many modern machines don't even have that anymore, but around in Unix for longer you do know the thing. It's basically it allows you to run services only when somebody actually connects to them. Meaning, I don't know if you have a, if you run Git for example the Git Demand, there's no need to run it all the time. It's efficient if you actually run the Git Demand. Somebody connects to it and wants to check out your sources or something like that. So traditionally people used INED for that. It would just listen on a port or on a list of ports and IP addresses and then as soon as somebody connected would actually start to serve the actual Git Demand. This existed in Unix since the Saturdays basically and it was used as a way to kind of overcommit the functionality of the machine. So you have a very, very low resource machine and you can run a ton of services on them because the services are only started when they're needed. And systemally we have something called socket activation which makes use of this but it's a lot more powerful than the classic INED. But it's also compatible to INED. So whatever, if you have an INED service like Git Demand or whatever you have, it will actually work with this system as well. In a system you can do a lot of other things like you can have support for more particles than just internet sockets that are Unix sort of things. And you can say ton loads of parameters for the sockets like TOS fields and these kind of things. It is a lot more useful. Then we kind of reply to Cron on App. Cron being, I just talked about Cron before and App doing something very similar. Actually just two weeks ago something I actually implemented the calendar time support for for time archivation. So yeah if you have an app added system and don't actually need the functionality that actually real Cron provides to you nor the interface with Cron then this should absolutely be sufficient. Like if you want to start job space and time system you will do it out of the box. Another thing that we reply is Readahat. Readahat is something that's not so interesting on service but it is kind of relevant on desktops and an embedded hub or some embedded setup. It's basically something that optimizes food speeds. It basically, it starts during food and watches what part of the heart is actually accessed during food and then in the next food it uses that data collected and will load all this data into memory very quickly at food, like very early and then if the the food process actually needs that data already in memory. So it's a, it gives you like, depending on what kind of part we have it gives you like 20% of the food up. And we said, we looked at that and they were like probably like five different implementations of it and they all sucked in different ways and then really hard in different ways. Like for example some of them used the audit framework which is something for Securities to watch the files and system accesses which kind of broke audit if you enable this functionality. It's crazy. And other things used, yeah. It was all awful and we looked at that and said, okay, Readahat is actually something useful to have because even on modern hardware if you have SSDs and things like that it actually speeds up your food and there's a sufficient large number of people who actually will use this like amateur people and investor people. So we said, okay, we're moving implementation of that into system B. It's nicely integrated. It started at the right time in food very, very early on and collected data and will just work. Of course, this functionality is completely optional because on the server you don't need that. You can just compile time without like you remove it at compile time or remove it at runtime. All right, mention this. We replace content like session tracking, user-set tracking, MC tracking. This is all done by some component called log-indy and system B. It's also an optional component because if you have an embedded device where no user ever can log into because I don't know how would you log into your refrigerator, you can remove that part. But we said it's common enough which would be in system B so the people who need it can just use that and it's nicely integrated. We'll work and interface with all the rest of it. Then earlier this year we introduced something called the journal which can replace Syslog in a way. Talking about the journal probably could be another talk, or hours that we talk about that. So I'm not going to cover too much about that. But yeah, basically we looked at Syslog, figured out there are huge problems with the current Syslog because I mean it basically was designed in 1983 if something hadn't changed at all in its functionality. It doesn't do anything what people probably want these days. So we came up with a journal which can take the same input as Syslog including it's compatible to a certain degree with Syslog. But it provides you a lot of different functionality, much more powerful. So we added that to the journal because we inherently believe that collecting logs of services is absolutely essential about service management. And system D primarily is a lot of management. More specifically this means if we watch services dying and can we start them then this is only half of the equation because if you restart a service you need to give the administrator a way to figure out why it died so that he can do something about it for the next time. And this why question you can only answer with the logs. So we came to this conclusion that there's no way around it. Collecting logs and handling logs and indexing logs is absolutely essential for service management. It's just one asset of it. And keeping this entirely separate doesn't scale because I mean classics are not indexed and it's not trustable. So that's why we moved that into the player. Of course moving this and mentioned is never exclusive so people can continue to run the block. Here's a simple one we added watchdog support.  It's an embedded device. It's a small device. Even my laptop, even the snapbook that I'm using there has a hardware watchbox device these days. It's a very simple device that if the software doesn't ping it every, like it's configurable at the time but let's say every minute, then the hardware will be automatically reset and rebooted. So the idea is that if for some reason software hangs, the machine gets automatically rebooted and becomes usable again. The user space part or like the software part of watchdog support is absolutely trivial. All you need to do is wake up every now and then and send one or the other to the device. And we figured out, okay watchdog support in a way is absolutely essential to many, many use cases like GAMP service and embedded and all that kind of things. Not on the desktop. So we said, okay, executing one IOC is probably something we should be moving into system. Especially since we recursively actually need to support watchdog support services as well. Meaning that if for example a patchy stops responding, we probably should restart a patchy as well. Not just when a patchy is dying. And if you do that for the individual services so that system supervisor services does watchdog for them, then you probably need something that also watchdog system be and we can use the hardware for that. So yeah, we said, okay, we can just move this watchdog support into system. And it's trivial. It's like five lines of code. And yeah, we'll cover a lot of use cases. It's much nicer. Yeah, ACPID. ACPID is a program that whose only purpose is that when you press the power button on your desktop or on your server the machine shuts down. And if you, yeah, and if you press sleep button that's the only purpose of ACPID. And that's an absolute trivial thing to do. Absolutely trivial. If the key press comes in you shut down the machine. There's nothing about it. Traditionally though, people always have to install ACPID. Like a demo for this absolute trivial thing. Another thing is the inner system already controls like if somebody press control it reboots on its own. And that's already handled by the inner system anyway. So the question is, why do we actually need an individual demo to react to power presses if we already have one that's way more complicated for reboots inside PID 1? So when we said, okay, this is ridiculous. Everybody needs the handling of the power key. So we moved that one line down. So my time is over basically. And we moved up to a couple of other things like power management stuff. This is now the system D. Parts of auto fast. You let this move into system D handling of crypto devices and a couple of things and we innovated a couple of things and yeah, we have a couple of other things on the line. But this is basically the last thing I want to talk about. Where to draw the line? If people look at this then of course we moved a lot of things into system D and the question is, well, that's a lot of things in there. Isn't system D bloated? Because I mean previously we had so many different components that did different things and now we moved that all thing into one and then this one thing grew large and so it's bloated.  Because it doesn't actually move everything into one single process but it just changes the distribution model how we do these things. Like instead of having 20 or so or 100 or so different packages it just moves a lot of them into one package and it moves a lot of duplication of code. Like if you look at, for example, a cron and look at INED and look at classic system 5-init and start-stop them and you realize that most of the stuff they do is actually just boring processes and all of them have different functionality like some of them can set alignment variables before starting something and some of them can do some other properties when starting them. But yeah with system we said okay system you should be really good at setting the environment up for services to run and we want to regardless how it's triggered and provide the same powerful functionality regarding resource controls, security and all these kind of things. So we said yeah let's unify all this, let's just teach system D to have a couple of different triggers without a service and remove all the duplicate code that spawns services with various amount of configuration.  It allows you to build your system for much fewer packages and yeah. So I don't know, of course different people have different ideas to draw the line. We still think that we haven't reached the line yet where we should stop moving things in the system D. But as we started out with just saying system should be an in-system and nothing else and now it's changed a little bit positions saying it's a CoroS I guess it's a little bit of a dynamic thing. But yeah anyway it's an interesting question to figure out where does to draw the line. But anyway this is all I have for now. Nobody asked the questions so far. Please do that now. Do we have a mic? How does it work? The first question I want to ask is about the adoption. Who is using this? Which distro is using this? I used this on Arch Linux for quite some time. I used system D on Arch Linux for quite some time. Which OS is using it now? Which OS will move on to this? So what is the future? So basically all distributions have it. All the big commercial distributions use it by default like meaning Fedora that uses it RAL7 will use it, Magai use it, Madriva uses it, Frugalway uses this Arch Linux by default. Debian again to have it in the distributions but don't use it by default. I mean they're kind of very conservative distributions and they're very slowly especially Debian that. So it would be surprising they already switched. Basically the big one outlier is Ubuntu. They're stuck to Upstart which is their own thing. And they maintain that and they don't have that and I do not expect that to change hands. And then there's of course all the weirdo distributions like Slackware and I'm never going to switch. But anyway basically all the big commercial distributions did and all the community distributions at least have it. You started developing this replacement for init, right? Then you made choices between things like in-getty, a-getty or between signs. If you like any one of those you adopted them or you just replaced like the thing which you did with ACPID. So this is applicable to the desktop environment you're going to choose one of them or you're going to do your own. I mean to clarify that we didn't even make the choice regarding ACPID you still can run ACPID and turn off this power button handling in system D. So we will not make things impossible. This is what I meant with this exclusive exclusivity thing that even though we provide this functionality in system D, we will always allow you to continue to run what you previously did. We will just have this nice suggestion by the way you can just use the built-in stuff that's much easier and it will just work out of the box. But you can still run any kind of, you can run min-getty if you wanted. We will not stop you from that. I mean there's even documentation on the internet somewhere where people figure it out how it works. And for the desktops of course it doesn't have no implication at all like D or GNOME or something like that. It's completely irrelevant. This is all low-level stuff. We do not make decisions for you. We basically just say, we make suggestions to you. We say if you use that one we have tested very well and we support it very well and if you have questions about it we can answer them right away. But so basically we don't remove the variables entirely. We just say, yeah this is the option we think is the best one. But we don't want to force anybody to do that. I listened to your talk and I couldn't help but wonder, most of this philosophies you talked about. Some of them are slightly conflicting. I was wondering about the future of the project. I mean this project needs really good BDFL or somewhere to measure when to draw the line or when not to draw the line. Can you discuss the future of this project? The question is always where to draw the line and then the approach to that is basically to decide on the individual case. I don't know what the future will bring and there are certain things that we believe is absolutely essential to the basic building blocks of the operating system where we say, okay obviously this belongs to the system. There's some case where it's obviously not, for example being UI or anything like that. It's definitely not on the card. And there's a lot of things that are in the middle and we will decide on the individual case. To keep this in mind insistently, even though it gained a lot of modules almost all of those modules are actually optional. There are very few that actually are required. The only thing that's required is basically PID1.  That's basically it. But all the other components like the replacements for the unit scripts most of them you can totally disable and that support you can just do that at configure time. So even though we have a lot of things in this build tree and you build if you do not specify anything with all of it it doesn't mean that you have to use that. We just say well this is really nice integrating our stuff and shares a lot of code so it moves a lot of complexity and a lot of duplication. But again if it's totally up to you to use that and in embedded cases for example I expect people to turn off a lot of it. So whatever we add it's very likely that things are going to stay optional. We will have to figure that out in individual cases but the thing is like the systems change a little bit how they used to be and how they are nowadays but basically everything that probably the vast majority of people will need to build an operating system might like and it is lower level, low level and that probably belongs to somebody. But again, individual cases. Yeah, hi. So looking at your vision of I'm here. I was looking at your vision of integrating some of the utilities that run in a distribution and my question is how do you see this effects say updates and other things that from the point of distribution. So today for example if Cron has a bug and getting out a new version of Cron takes is quite fast. But when you're moving into a monolithic app which does a lot of stuff. It's not monolithic. Okay, I mean even if it's modular so for example if you integrated Cron functionality into system D and a bug comes up in Cron does it require a new version of system D to be released for the maintainers of a distribution to pick up? I mean if there's a bug to fix then there's a bug to fix and it really depends on distribution how they package system D even. Like for example if they are concerned by that they could split up system D in a lot of individual packages and then they could individually update them. But honestly I mean looking at the bug history so far I mean we don't have that many bugs and currently at least in Fedora it's completely sufficient to do one update every two months or so at least the distribution is even more selling than that. So it really depends so far there's not been a problem at all with that and the bugs that were there usually were not the super critical ones. So I have a fatty positive view on all of this. So but yeah I mean if the thing is like system D like the system D RPM is not huge. It's a couple of megabytes but it doesn't really matter if you update all of system D. But does it cover all the services that you mentioned right now? I haven't actually looked at system D in Fedora yet. Does it handle Cron? I mean many other things that you're mentioning can be pulled into system D once that happens in the future. Then normally when there is a package for example we have seen it in KD something a small component of it has a bug and there's a lot of resistance to come out with a new bug fix for KD because it changes the bug fix as the versions have to be kept compatible and the modules are out there and there's a lot of resistance for coming up with a new version when a small component has a bug fix. I mean I'm not entirely sure I understand the question but you can update system D and it's not that big like if you have a bug then update system D is totally fine. But the thing is like I think actually that updating things like this is much easier because you get the test matrix down. You just apply the patch to system D build and you don't have to test this package against all the other packages. I mean most of the distributions allow you to mix and match not only different implementations of things but also different versions of things so you can run basically system 5 in it against any kind of Lipsy and these kind of things. But basically I don't think that's much of an issue because the package is not that large and it's completely up to the distribution how the package is so exactly nothing changes from the previous thing. None of that makes sense. I mean right now in Fedora system D is split up into four packages or something like that but there's probably plans to add a fifth or something like that and I think Debian splits it up and more but it's really up to the distributions how much they do. But I mean the other thing is like one should not assume that Cron has actually a lot of code. I mean just think about what Cron does if it actually executes something based on time that is not really rocket science. So they actually like when I added calendar support to system D calendar time support like calendar based events this was like I don't know 200 lines of code for parsing calendar specifications and that's kinda it. That is nothing. That is really nothing. So yeah there's a surface where where bugs can be gets smaller system D because you have so much less duplication because we now use just one code password and then have a little bit that's just time handling instead of having the code has for responding things in Cron and ATIN and so on and so on. Hope this can help. Yeah so Linux is traditionally been a loosely coupled subsystems which we alluded to as you know the Bazaar model. And you said that you know there are advantages and I see some of them in moving more of them in a single atomic unit but what are the disadvantages you see with that because especially when it fits in the distribution model or you know about the lifetime of those packages which are probably also being worked on in parallel by different groups. People see different like of course one of the disadvantages of moving everything together is compatibility because it does work differently than Cron did before. We try to deal with these issues by saying well if you need to compatibility with Cron then install Cron that's what I meant with this non-exclusivity that people can just continue to use what they want. But basically yeah it is about compatibility with the problem and then doing things new of course always comes with a learning curve. People know how things used to be and now they are slightly different. That is a problem absolutely. I don't think it is a big problem because we really try how to make system D as easy to use as possible. But yeah I don't know. There are other drawbacks of course. I think the net result though is that integrating these things has drastic advantages because it simplifies the code unifies the code and reduces the test matrix radically. How do you what is the biggest advantage you are saying to move to system D One day I moved to system D. I don't have Cron Syslog works before I mean Syslog is older than me. It works well Cron works well. So you are basically a big move right? To enable like Syslog it was just one line of configuration somewhere. To enable journal CTS it was like one huge command like system CTL or journal or something. How do you plan to move all the people to system D because it's anyway a very new thing. Almost all the commands change, configuration change and what is the biggest advantage you are saying to use this and throw away everything you know. I don't know. The thing is like I don't really try to move people to system D necessarily. Whenever we add something we say this is the offer you have but continue to use Cron if you want to. To convince people to actually use this stuff is basically just by providing very good documentation and providing a lot of functionality that didn't exist before. For example in the Cron case we will probably publicize a little bit more that we can replace Cron these days by pointing people to new features that we can do in system D that Cron can't do. And there's a lot of things to that. For example we want to add to Cron that you can specify things that you start even if the machine is off and then we use these wake up timers where the hardware actually boots up and then can have timers that can unsuspend and then start the machine if you want. And then for the cloud people we want to add some things that implicitly to every time event you can add a configurable jitter so that if you have a cloud like a cluster of machines it all start at the same time. Not all the Cron jobs are run at the very same time and clogs the network at the same time so that you avoid the DOS thing a little bit. So if we add this jitter which is relatively easy to add then this suddenly becomes interesting to cloud people. And there's a lot of things like that where for example we can integrate anacron and these kind of things and already actually the system D Cron replacement language is a lot more powerful than Cron itself is in very specific cases for example in system D you can express I want to start something on the first, second or third day of the month which is a Tuesday or something like that which you cannot actually express in Cron. And in system D you can express I want to start something on that very specific data and actually specify the year for it and you can't do that in Cron. So all these things are mostly smaller things but I think we can make this interesting for people by just saying yeah if you use that system D build and stuff you get something that is drastically more powerful than Cron used to be. And then there's lots of other things to that like system is actually more power efficient because the way traditionally yeah I think I should totally but yeah the thing how Cron traditionally used to work is it woke up every now and then figured out if there was anything to run, executed if there was if it didn't it would go to sleep so it would do this definitely every minute. Now in system D we never do that we actually wake up when we actually have to wake up with this program with timers now waking up every minute by itself is not expensive because like you wouldn't ever notice that the power graph but it suddenly starts to matter if you multiply your machines if you have containerized machines if you run 500 machines in one and every single container wakes up every minute or so then that basically means you wake up more or more than one second of the machine all the time and then the container set up and if you have this in the cluster the same thing happens. So basically yeah we are more power efficient we have a lot of things so we try to make this more accessible to people by saying we have more features and yeah it's a matter of telling people but I'm way over time I think and so thank you very much and I have a good lunch and yeah I hope this was interesting because a little bit of an idea what we trying to do with system D and what our vision behind it is just a small announcement we all haven't heard of NH7 already one of the best indie music festivals in India the festival is 15-16 in Bangalore and they are kind enough to give us 10% discount so please head to the the fostered install which is the right of the entrance and we are also giving our delegate kits at the swag area please don't put