I am extremely honored and humbled for the sixth year in a row, the 42nd President of the United States is with us. As our global chair of the regional networks of the patient safety movement, we are fortunate to have President Clinton continue this journey with us to eliminate preventable patient deaths in our hospitals. Some of you know that our commitment-based approach to patient safety was a concept we shamelessly stole from the Clinton Global Initiative. In fact, we announced the patient safety movement's goal of zero by 2020 at the Clinton Global Initiative several years ago as our commitment to CGI, amongst other commitments I had made over time to CGI. Today, this commitment-based approach and your collective work has saved between 81,533 lives and 212,579 lives a year. Yet, patient safety movement would not have been possible without President Clinton's early support and continuous encouragement and direction. With his unwavering commitment to eliminating preventable deaths in our hospitals, we have been able to convince some of the best minds and hearts and we have grown to a movement across 44 countries and even spurring more movements around the world all for patient safety. But President Clinton's compassion for helping people reaches far beyond his involvement with our movement. I've had the great honor and privilege to see firsthand the work President Clinton has done in the United States and around the world to decrease childhood obesity in the U.S. by getting soda and juice companies to reduce their portions and to cut the cost of HIV medication by 10 folds so that millions of people could have access to HIV medication instead of a few hundred thousand. These are just two examples of the great work he has done and is still doing for humankind. President Clinton has unique abilities rooted in his genuine concern for others and his enormous intelligence and an energy to take good ideas and make them reality by getting people like us involved and inspiring them to act locally and globally. Maybe most importantly, President Clinton has been a great teacher. He has taught us that we can do anything and solve any problem if we work in collaborative and productive ways together. He led the U.S. as the 42nd president and now as a global citizen with his belief that the power of our example is greater than the example of our power. Ladies and gentlemen, the 42nd president of the United States of America. Thank you very much, Joe. Thank you, ladies and gentlemen. First, congratulations on having your first international meeting. I thank Secretary of State for Health and Social Care, Jeremy Hunt, for inviting us here. Thank you to Liam Donaldson, the WHO Patient Safety Envoy, Permanent Secretary to DFID, Matthew Ryecroft. Thank you and your Chief Medical Officer in the U.K., Dame Sally Davies, who also is a new member of the Board of the Clinton Health Access Initiative. So I thank all of you. This has been a remarkable process since Joe made this commitment in 2012 that CGI's annual meeting. We didn't know then that by now there would be about 4,600 hospitals in the network that 83 technology companies would commit to share data, to develop algorithms, and predict dangerous trends that already some 80,000 lives were saved last year in the United States alone. And I just want to say this. I want to thank Joe for trying to be honest in counting the number of lives that have been saved in preventable deaths in the United States and in the networks around the world. I love these meetings, and the only thing that I have to say is that even though I don't get to sit in on the meetings, I try to read the proceeds of what happens because I rarely have a chance to go to a meeting where I have almost nothing to contribute on the merits, and I just learn. I'm sorry I didn't hear Dr. Tedros yesterday, but I did support him to become head of the WHO because when he was health minister of Ethiopia, he faced an interesting challenge. At the time, Ethiopia only had a little more than 80 million people, but 58% of them lived in 60,000 villages of 1,000 or less. And if we could transport this meeting, or if we could have then, to Addis, and we all got food poisoning on the way, we would have been fine. But in the other villages, there were only 700 health clinics for 58,000 villages. So we were asked to partner with him, and we put together more than 2,000 more. His goal was to make sure every Ethiopian would be within at least a half a day's walk of a clinic. And I hope they've continued to make progress on that since we did our part. When Joe said he wanted zero preventable deaths in hospitals, in all the networks, he said, you can't hope for zero. You have to plan for zero. So he's planning. One of the things that it's impossible to know is what the positive impact and new challenges will be with the rather rapid introduction of robotics into hospital care. I dodged robots delivering medicine in the Benioff Children's Hospital in Northern California about a year and a half ago. And what will happen with nanotechnology and what will happen with artificial intelligence? But I know one thing, the decisions that real people make will still make a difference. You mentioned that we've worked hard on the opioid epidemic in the United States through our Health Matters Initiative. And not very long ago, I had the chance to meet with Dr. Feinberg, who runs the Geisinger Health Network, concentrated in Eastern Pennsylvania. They came to the attention of many of us years ago when they had 700 doctors in the network and they made a commitment to anyone who joined that if for any reason, they had to return to the hospital where they were under the care of the Geisinger Network within three months of being released, that all the costs would be borne by Geisinger. And they were the first network of which I'm aware, at least, where all the doctors were given a very thick binder of best practices that was updated every single month. And their position was, this is first the science and then an art. Do the science first. We're going to do what's best known. And as a result, they were able to drastically reduce the error rate, drastically reduce the re-envision rate, had they not done so since they promised to eat all the costs and that it could never be reflected in higher rates or higher co-pays, they would have gone broke. They have now done something that is very important in our work, which I think will contribute to these numbers, which is to turn over the actual prescribing of painkillers to their network of trained pharmacists. So the member doctors actually described the nature and extent of the pain to be addressed. But in almost every case, an opioid is not the first prescription risk. So their view is, OK, we have a universe of people who are caught up in the opioid addiction. That opioids have become a gateway to heroin because heroin harvested in Sierra Madre in Mexico by preteens is now cheaper on the street than a lot of opioids as we get tighter on the distribution of them. But the first and best thing we could do is to stop new addictions as we figure out how to treat those who are addicted. I say this to make a larger point. In spite of this global effort toward disintegration and people trying to hunker down in their groups for economic, cultural, social, and maybe psychological reasons, the best way to solve problems is with diverse partnerships of co-operations toward a common goal. Preventable patient deaths may be caused by many things. I noticed in one of your panels that you said that hand sanitation was a problem that we thought we solved but didn't. But there are many things that can happen. I also wonder whether there are problems we assume were preventable that weren't. A friend of mine died in a hospital in Los Angeles not very long ago, a few years ago, because he had a bacterial infection he contracted in France and he got the finest care available in the Western part of the United States. And when he died, they still didn't know what his infection was. I'm not sure it's fair to say that was a preventable death. I had a cousin, as I have said before here, who was falsely diagnosed with having the flu. And she showed up Friday night in a hospital and she was dead Monday morning because she in fact had severe sepsis. I'm not sure that was preventable by the time she got to the hospital. But the point is, we now have the analytic tools and assistance in place to make judgments like this and to do better. So I think it's important that you be re-energized here. The fact that the politics and much of the economics of the world may be drifting away from the model embodied in the patient safety movement should steal your determination to stay with what you're doing, not weaken it. We all know that cooperation works better than conflict. As I've said several times at this meeting, one of the most influential books on me in the last decade is the first of the trilogy of E.O. Wilson's latest books on human existence, its meaning, its function, and what works, The Social Conquest of Earth, in which he says that ants, termites, bees, and people are the most successful species ever to live because they're the greatest cooperators. That has to be recently amended because spiders notoriously have not been cooperators. They've been lone actors. There are more than 2,400 species of spiders. For reasons no one yet knows, over the last two decades, two dozen of them have started to co-offer. Their webs are twice as strong. They get better nutrition, and they're living longer. This is not rocket science. People have great advantages and disadvantages rooted in the fact that they have consciousness and a conscience. Our consciousness can sometimes make us think we're smarter than we are, and our conscience can sometimes play games on us and convince us that we're righteous when in fact we're flawed. But we have the capacity to go beyond unlike ants, termites, and bees, our own kind, to deal with whole new challenges. And that's essentially what you're doing. You are proving that over the long run, the 21st century, if it is to be the most prosperous and peaceful time in human history, will have to be the century in which diverse groups reach across lines that superficially divide them to meet common challenges and seek shared opportunities. And therefore, you're not only saving the lives that are before you, you may be saving the democratic enterprise, and you may be saving a belief in the possibility of progress. And I think it is really, really important. I think about all this now in different ways. And in terms of the meaning of life and the meaning of a single life in the context of a much broader view than when I was only wondering about whether I could win the next election. I mean, think of it. When this month, last year, in this very month, a year ago, I became the oldest person in my family, man or woman, in three generations.  And I'm not very old, I don't think. But when I lived to be 70 years, six months, and one day, I had outlived both sets of grandparents and my parents. My maternal great-grandparents and my paternal great-grandparents and my maternal great-grandparents lived well into their 70s, but they lived out into the country in an old-fashioned way. They ate sparingly, went to bed with the son, got up with the son, had a life with slightly less stress than that I've endured. And maybe I'll beat my great-granddad, but I kind of doubt it. If I do, I'll be grateful. The point is that I see this in a different way. Every day is a gift to me. And I also have lived long enough now to know that in almost every endeavor, there are no permanent victories or permanent defeats. All there is is the permanent gift of choice and life to try. This January, Hillary and I had what was her first vacation really since the 2016 election because she wound up working on her book through our previously scheduled vacation. I took her to Hawaii. And we went to the Big Island. And one day, we went up 13,000 feet to look into the Keck telescope, the largest telescope on planet Earth to see beyond our own solar system, beyond our own galaxy. And then they took us down 3,500 feet to the best place to see the night sky with an naked eye. And I could barely pick out the constellation Orion because unlike every place else I've ever viewed it, those stars were not alone, but bathed in thousands upon thousands of others. I had never seen so many stars in the night sky. And I asked the people there who ran the operation. I said, what do all of you think about the prospects of life, sense life on other planets? And he said, well, there's a great debate here. We're really divided. I said, what are you divided? What's the range? He said a probability of 95% to a probability of 100%. He said, in our own solar system, we've already identified 20 planets that seem far enough away from their suns and dense enough that they might be able to support life. Beyond here and the billions and billions and billions of stars left in the universe, it's inconceivable. It might seem small. To me, it makes it more significant. When I was president near the end of my service, we celebrated the 30th anniversary of the first walk on the moon. And the surviving astronauts came to see me. And the NASA guys, our space agency, they kind of liked me because I put a lot of money in there. And I'm a believer in all that.  So we posed with the rock. And I said, how old is it? They said it's been carbon-dated at 3.6 billion years old. In other words, a few hundred million years before, as far as we can tell, the first single-cell organisms formed out of the premortal slime on planet Earth. I said, can I borrow that? And they said, you can't have it. I said, I don't want it. I want to borrow it. And I said, it'll be safe. It'll be here in the White House. So I put it on the table. When you see on television that the president of the United States is entertaining a world leader in the Oval Office, you know, there's two chairs, and there's a little table in front of them. And there's couches on each side of the table. I put that moon rock there. And then whenever people came into the Oval Office, I got out of the Republicans and the Democrats come in and they talk, and they started to fight, and they'd raise their voices. I said, wait a minute, guys. See that rock there? It is 3.6 billion years old. We are just passing through here. We have a nanosecond in the scope of human existence to do something good or bad. So let's just take a deep breath, settle down, and figure out what we're going to do. It always worked. By being humble, we became bigger. By feeling smaller, we were able to think larger. What's that got to do with what you're doing? I think two things about all these lives that are being saved. First of all, especially if this movement can spread, as Tedros said, into the developing world, God only knows what these people whose lives you save may accomplish in their lives. But secondly, from the perspective I now occupy, since we're in London, I can say that every passing year convinces me that the great English poet Wordsworth was right when he said, the last best hope of a good person's life are the little unremembered acts of kindness and love, that the things that we can all share are the things that matter most. And all of our university degrees and all of our advanced learning and all of our skills and all of the whatever else we do, gain more meaning when we give the simple gift of one more day to other people. And I think about it every day because one of the gifts of a long life is that more and more people you care about aren't around anymore. I wear these shoes because they were given to me by a man who died a few years ago. I tried to help through the last three years of his illness but 50 years ago when we were young, we lived in Oxford and he took care of me. So every time I put these shoes on I think about what it was like to be young in England or in the words of Willie Morris, oh to be in April now that England's here. I'm saying this because it's really important that you guys not get discouraged. And it's really important that you not turn these numbers into something unhuman. You should keep score because underneath the score there is something, the dimension of which you cannot imagine. If you believe there's an after life you are bringing this sacred into this life. If you don't and this is all we got it all lasts as long as it possibly can. And while it's lasting it ought to be as good as it possibly can be. So what I wanted to do is to say I think the regionalization of this movement is important but you should take in the developing world. One of the things that I learned when I started our work with AIDS is that we just fell into this is that because so many people in Africa were dying of HTV and malaria we overlooked the fact that in poor countries cancer rates are more or less consistent with what they are in other countries. So when the Rwandans asked us to come in and help them rebuild their health system after the genocide and we put a hospital in every region of the country, friend of mine, Jeff Gordon, a famous race driver gave us the money to establish a cancer center in the last of the hospitals we built in Butaro near the Ugandan border. And then it was hooked into the international network of children's hospitals so that they could actually access all the information they needed to maximize the treatment impact on young children with cancer in Central Africa. We are just at the beginning of this but we have to face the fact that there are all kinds of problems we're trying to fix people which wind up causing them to die sooner in hospitals in the United States and in other countries. And I was, and that you just when you think you've got something stopped you realize you don't. That's what I love the title in your program about the hand sanitation problem we thought we saw. I remember when C-sections used to be performed because we thought it was safer for the mother and sometimes it still is. But I found it fascinating you were dealing with that. I know that tubal feeding has saved many lives but I now know how many it can cost because of what you're doing. This is all real stuff. Don't let unreal headlines blind you to real trend lines. Whenever I speak to young people in the United States I said, you know, the headlines may be lousy but the trend lines on balance are good. If you make any kind of decision in your personal life or as a citizen based only on headlines you have a better than a 50% chance of making a mistake. You had to first ask that the headlines reflect the trend lines. If they don't then you have to calibrate the decision based on the headlines which may be today and the trend lines which are tomorrow. Because all of us have an obligation to make the trend lines better. That's really what Martin Luther King meant when he said the arc of the moral universe is long but it bends toward justice. And then everybody forgets the second part we have to do the bending. And so I wanna leave you with that. What you were doing is about the meaning of life. If all you did was wanna process people to the health system you wouldn't pay so much attention to this. The revenues can be generated about the same way. Now in countries with really humane comprehensive health systems it matters a lot like in the UK and we know in the US it probably cost us $30 billion a year to have all these people die when they shouldn't and that's all really important. But that's not why you should do it. You should do it so one more person can find out whether there's life in the universe. So that when it comes down to it our people will be conditioned to embracing the commonality of life and enjoying the differences in the context of that. So that when people show up here in a spaceship we don't kill each other. You should do it because there's a child somewhere who's claimed to grow up, love nature, fall in love, have children is no less than your own. You should do it because you made a choice all of you one way or the other to save and enhance lives or you wouldn't be here. And when you're in that space you have to accept the fact that you won't always succeed, that you may make a mistake, that circumstances may intervene beyond your own control but every single day you can get better at what you're doing. And the thing that I loved about the commitment Joe County made six years ago is that he was basically drawing a conclusion from the data he had and the experience he had with advances in his own field, non-invasive medical technologies for vital measurements. And he concluded that this was a problem we could whip, that it would be cheaper than brain surgery per person. Our complicated cancer surgery are putting people back together after a horrible car accident are dealing with the consequences of explosive devices on battlefields, that if we did it right we could probably save more lives for less money and less time than most of the things that we're organized to do. And that a big challenge in every human life is making sure you're organized to deal with areas of greatest opportunity as well as the areas of greatest problem. So that's how I think about this. We are basically reordering this incredibly complicated ecosystem to seize the simplest and most profound of all opportunities and to fulfill our obligation to give other people the chance to do what at least people of my age who have more yesterdays than tomorrow have already done. It is a noble good thing and there's plenty more still to do. Thank you.


That's the exercise for the end of the talk. So today I'm going to talk about the Flight Recorder for C and C++ programs. The idea is to have instrumentation in your programs that is always there and therefore is there when you need it. So what is a Flight Recorder? Those of you who had a program where they crashed, please raise your hand. Those of you who knew why it was crashing. Okay, so now you have to think, what was I doing right before that? Why is it crashing now?  I love this quote. Programming today is a race between software engineers striving to build bigger and better idiot-proof programs. And the universe trying to produce bigger and better idiots. So far, the universe is winning. So when you have a crash like this, you know, you wish you could reproduce, but of course when it's really tricky, usually you can't. And these kind of bugs are called hyzen bugs, because those are the bugs that only happen when you don't look at them. So the other thing is that sometimes in order to reproduce the problem, you need someone to actually do the work for you. And so you need some kind of guy, you know, they're usually called users or something like that. And you need them to do something for you. And, you know, this picture shows you how programmers tend to see their users. Now, another really nasty effect is when the problem seems to go away by itself. You know, it's magic. Just don't look at it. It doesn't happen anymore. Close the bug. So if your bugzilla was not touched in two years, you can close it safely. It's a rule. You know, it's basically, it's okay. So this piles up in a, you know, we have this unresolved mysteries part 177 that we need to care. So the problem there is that you're making someone happy. Do you want to make your customer unhappy? No, of course not. So you need a way to address that. So what does a flat recorder bring to the table? The idea is to collect information before the problem happens. So you recall everything while the program is running. And you print it only when you need it. A good example is you get a signal, you know, your program crashes, has a signal 11 or something, and then you print at that time. But you print the stuff that happened before the crash. So that means you record continuously. In order to record continuously, you need to have instrumentation that is always there.  It's not something that you toggle in or you, not something you if deaf in. It's always there. And something that is always there has a magic property. It's there when you need it. Now, once you have this instrumentation in place, you can do more, and that's the nice trick about it. For example, you can use tracing using the same instrumentation. So you can basically activate specific traces and say, I want to see that happen. Instead of having all the printers of your program come at once, you can get only those that you're interested in. And you can do continuous measurements and graphing. So I'm going to demo that later. The end point there is that, or the bottom line, is that you have one instrumentation to put in place and multiple users. Okay, so what is this thing? How does it work? Basically, it's simple circular buffers, and you record data in these circular buffers continuously. The new thing, for me at least, is that it has multiple buffers that can record at different speeds. A new thing and looking at Karen compared to what she knows from a previous life. So the benefit of doing this that way is that you can preserve old and important records. For instance, you can, at the beginning of your program, record printflex statements for all the input arguments of your program, like common line parameters, options that you get from environment variables, all that kind of stuff. And if you don't have a write it, it stays there. If you don't have a crash, you get that at the beginning of the dump. At the same time, the most recent events, for instance, you have an interrupt every microsecond. You can still get all the data about the last 128 interrupts at the same time as the stuff that happened at the beginning of the program. Because they are in separate buffers, so they don't overwrite each other. Okay, how do you use it? It's fairly easy to install. You just clone as a sub-module. At some point, I'm going to make a package like Promise, that myself, for the best of the past year. But for now, you just have to clone a sub-module and add two files to your source files. Recorder.c is the Recorder API itself, and Recorder.c is a lockless ring buffer implementation. And it's fairly easy to use. It works mostly like printf, except you use Recorder instead of printf. Most of you might be able to manage the difference. And you have to put the name of the Recorder, your stuff things in as the first argument. So, standard printer format. No backslashing at the end, just because I'm lazy, and I always forget them, so you don't need to put them. So that makes it fairly easy to pull existing codes. For instance, this is the Spice code, the remote viewer for virtual machines. And in that Spice code, there was a Spice info macro. The only thing I needed to do was to replace whatever lock function was there, like printf or something like that, with a record, and I connect to this mechanism automatically. So that makes it fairly easy to use. Okay, there is one caveat, though, is that the Recorder dump happens late. And I don't want to pay the price of formatting, so what happens is that the data is recorded, but not formatted at the time you record it. In other words, when you do a record like this, you end up with an entry, some structure, that has a timestamp, a source file information, an order, which is the order in which the things happen, so that you can sync events that came from multiple CPUs, the format string, and then up to four arguments per entry. So you can see, for instance, there that the %s string will be recorded as a pointer, as in C, and so this is not safe to print by the time you dump. So only the string is stored, the string is not copied, only the pointer is stored. So since things happen at dump time, the formatting happens at dump time, this means that in that case something bad happens, the dump function will only print the pointer, so the value of str will be some pointer, because it's not safe at this point to reference the pointer. Now, in some cases you know that something is a constant string, like in this case, for instance, I know that it's a built-in constant, so it's safe to print in that case, and so there is this special format of the %s, you add the plus option, and that tells the recorder you can print that. Keylet number two is that the recorder, as it is written today, has macros for up to 12 arguments. There are four elements for record entries, and you can have up to three entries. It's not a limitation, a hard limitation, it's just that basically the macros to do that look like this. The macros are somewhat complicated to write. You're welcome to submit a pass to go up to 16 if you need it. Just be ready to write this kind of crap. So how do you define or declare a recorder? The idea is that since this is instrumentation that is always in place, it better be self-documented so that folks who use the program have a way to know what each recorder represents. So you have this recorder macro that defines a recorder. It gives it a name and a size. The name is in its own namespace, so you don't risk clashing with existing variable names. You can keep them short. And the size is the number of entries you want to keep. It better be a power of two for performance. Not a big deal, but it's better that way. And the last argument here is a description that shows if you pass the help option to the recorder saying, please show me what recorders you have in this program. So that's how it shows on the outputs. The recorder declare macro is used to declare a recorder in a header, so much like a function. You have the prototype so that you can use it. And you have the actual implementation somewhere in a 1c file. So use recorder declare in header files. Use recorder or recorder define in the implementation files. Okay, so how do you dump the recorder? You use the recorder function and what it does is basically sort all the entries and give them in the order in which they were recorded. Again, this is thread safe, so you can have multiple CPUs banging at this thing at the same time. There is a variant that lets you select which recorder you want to dump, so you may have a condition where this is a specific kind of error and you know that when this happens, what I'm interested in is this or that data and then you can basically select it. You have this recorder dump and you pass away all the things you are interested in seeing. You have callbacks to format the output of the recorder dump any way you want. Typically you would want to send that, for instance, to a standard logger, to standard output, whatever. You decide whether you want that to go to a file or anything else. And there are callbacks to format to record entries because I prefer to have date and time, date and file name, et cetera, and maybe you prefer to have it the other way, your call. There is a function that is really helpful which is recorded dump on common signals and what this does is basically add a recorder dump for each of the signals of the standard crashing signals, like 811 and so on. So the arguments are bitmask of signal to enable and disable and if you put 0, 0, it's basically a default set. Okay, so how does tracing work? You have this environment variable called RecolorTraces which is read automatically if you call RecolorDump on common signals. And so you execute your program saying RecolorTraces equals the name of the recorder, regex of the name of the recorder you want to see and you can separate multiple ones with column or space. And as I said, you can use regex. Now RecolorTraces is read automatically but some applications may have specific requirements for how they dump their debug outputs. So for instance, Spice has their own way of doing debugging and so in that case I added some special code that reads spiced traces and in that case formats things, changes the formatting output so that it looks good for the Spice team. Now the executed record statements for all the active recorders, so the other recorders that you marked as I want to trace this will print right away. In that case, of course, you get the strings correctly in all cases. And you can programmatically configure which ones are active using RecolorTraceSet which typically you would do, for instance, to enable all the things that end in warning, error, etc. And then you can have naming conventions in your recorders to make sure that anything that would be something warning would show up right away. Now another interesting thing when you debug programs that are running more or less in real time is that sometimes you want to tweak what happens in them. And so the Recolor also has the possibility to define what I call tweaks which are basically variables that you can access from the outside so that you can change the program behavior. The demo is probably going to explain much better than just explaining it like this. But here's an example of code. So we have this code that depends on some sort of maximum queue length and I want to emit a message when we reach a given queue length. I want this queue length to be configurable and I define a tweak or trace and I say, okay, when I hit more than 300, for instance, then show me more details. So you use the macro RecolorTrace given a name or RecolorTweak given a name. It's really value-assisted to the same thing and it lets you show information. On the common line you would use the name equals value giving a numerical value to set the value of the tweaks.  If you set it to 0, you can disable a trace. So for instance, you have a warning that is really annoying you. You can say this warning equals 0 and you don't see it anymore. And so I said tweak and trace are equivalent because basically since I needed this flag to decide whether I was dumping or I was tracing or not, I decided I might as well give it a full word and let you use it for whatever reason you want. So that's why tweak and trace are aliases. It's really the same thing. Okay, so I said we can do some graphing in real time. So basically the idea is that the Recolor entries, if they are stable, if your prints are organized in some sort of column, so the data in the first column is representing the same thing, then you can use that, export it as it goes and you export that in shell memory and you can have another program that picks it up and draws it. And so now you can see what happens in your program in real time. So we'll see that in a moment. So let's say for instance that I have a size of stats recorder somewhere. Then I can have different record statements like initial size is this, the length is that, second size is this, the length is that. What matters here is that both here have a person F as the first argument and a person U as the second argument. So they have the same type. That's the first thing that is important. And the second thing is that they have to represent the same data otherwise your graph is going to jump all over the place. Makes sense so far? So in order to be able to graph something like this, you need to export it first. So it's much like tracing except it's tracing in order to see it in real time graphs. And in that case you give them names. So you give the name of the recolor equals and the name of the columns. So that way you can have, you can give arbitrary names to a given session to see what happens inside your code. Okay, so time for a demo that will clarify a lot of this. So I have this little test application and in it it has a recolor code speed test that is defined. It has two tweaks called sleep time and sleep time delta. And it has this thread here that loops forever. And for each speed test it records, as you can see, four values, the thread ID, the number of recordings it made in this thread, a module of 500 of the above. And the time it took between two record statements. So let me start this speed test. So let me start with, for instance, eight threads and four 10 seconds. And so basically the records are happening but you see nothing outside. Okay, 10 seconds was a bit long. You can see that each of the records takes 43 nanoseconds on this system. And when it's done, it's selectively dumping, for instance, some stuff that happened at the beginning. So if you look at the record of dump here, you can see that you have stuff that happened very early on. This is a record order. So this happened at the very beginning. And then we have information that happened in main. So this happened pre-main. This was basically constructors. Stuff that happens in main there. So you see that it launched eight recorders. And then you have an example where you don't see the string showing up. And then you can see that here the order jumped from 33 to 1027. So you know basically that there's a lot of data missing here, presumably stuff that was recorded, but then discarded because it's a circle of buffer that was not large enough. And you see that not much time happened. It's basically a few microseconds between the two. And then you move forward still a little bit. Now you move forward 10 seconds. And your order is something like 227 million. So basically in 10 seconds it's recorded 227 million records, most of which were discarded.  And you see that, for instance, thread four was recording. It was its recording number that much. Modular is 64. It took zero microsecond compared to the previous one, et cetera, et cetera. So this is the stuff that happened. Okay. So now let me do the same thing. But now I'm going to do it with recorded traces equals speed test. So what I'm doing now is I'm activating the tracing for the speed test entry, which is the main one. And now as it goes, okay. Now you see one of the main benefits of this approach. Now I'm printing as I go. And you can see that my prints are after 75. So I went from zero microsecond to 75. When you just print stuff, it's costly. It's expensive. So by not doing it ahead of time and basically setting time. Now for debugging, it might be okay. But for performance, you don't want to do that. You don't want to log in a file. You don't want to do that stuff. It's expensive. Okay. So next thing is now I'm going to do something where the speed test you see has four columns. So now I'm going to say that the first is the TID. The second is the recorder ID. I don't care about it. Then there is a module and then there is a delay.  And now again, nothing shows up. Execution again happens at full speed. But now let me just put 1000 seconds so that I have time to do the next thing. Now I can start a recorder scope here. And I'm going to say that I want to watch the delay. Okay. So this is the actual delay that is happening. So I said it was working full speed. And you can see it's not completely full speed because there are some outliers. And that's basically when the program is posted. What you see there is the effect of scheduling. Most of the time it's close to zero and sometimes it's much higher. So I'm going to actually force it to delay. And I'm going now to use the dash C option of the recorder scope to set the tweak that I defined here, the sleep time. And I'm going to set that tweak to 15 milliseconds, for instance. And in that magic now my... So what's happening is now I'm executing this code. Record the tweak. See that there is a sleep time and does another sleep of 15 milliseconds every time. Okay. So that's a bit slow. Let me adjust that. Oh, yeah. I thought that was too slow. That's because I used 150,000. So that's 15 milliseconds. And now you can see the actual scheduling time. So you're observing your scheduler at work. So you were asking for a delay of 15 milliseconds. And what you actually see is between 15 and 18 something. So now I'm going to introduce you to the next talk about the build system. So I'm going basically to do a parallel build at the same time. We're going to see in real time the effect on the scheduling in my system. And you see that you have a few more outliers now. And we're putting some pressure on the system. So it's slightly less stable. But it's pretty good, right? So modern schedulers know how to do that stuff. And now when it's starting to build all the stuff in parallel, which is coming soon, then you're going to see more pressure. And from now you see the pressure, you see the ticks. But sometimes your problem is delayed. OK. So as you can guess, I really love implementing stuff that has absolutely no purpose in life, right? So let me do, now I use dash s. So dash c is for command. I can send basically arbitrary commands to my application so that if I send help, for instance, it's going to dump the help for the recorder. That's basically using sending command over shell memory. Now I use s, sleep time, between 10,000 and 100,000. So can someone guess what I'm doing here? What does s stand for? No, you won't guess. Slider. So now I have this nice slider. And I can adjust the behavior on my C program in real time. OK. Let me put that in front. Because now normally that's the time when you applaud or something like that. If you don't, I'm disappointed. OK. So it's not completely useless, actually. It has many, many purposes. So now this is macOS 10. This is not Linux. And the reason I'm doing the demo on macOS 10 is for this. Yeah, I know. Once again, I'm ahead of time. Speak too quickly, as usual. So macOS and all the BSDs have these nice features called Control-T. Who knows what Control-T does on BSD? It sends SIG info to the running program. So Control-T, demo effect. Oh, I'm in the scope. Sorry. Control-T, record the dump. It sends SIG info. So now I have what happened right now. So your program is mid-range. Control-T, boom. I really wish I had that in Linux. So I find this so useful. I decided to try to implement it in Linux, and it's really crazy to implement that stuff. I decided to give up for that. But it's really useful. So if someone wants to implement Control-T in a kernel, terminal handler, IOCTS and all that stuff, feel free. It's really useful. OK. Anything else you want me to try while I'm in demo mode? No question there? OK. Just one thing that we might see here. So let me get back there. Just to show you that this thing can be fast. Oops. OK. So this can be fast. And in that case, if I do my Control-T, what I see is that... Where is that? OK. There's a bug there. So you see that typically, now, it's slightly more expensive than before. It's about one or two microphones when it's actually exporting the data. It's slightly more expensive to export data to shell memory. But it's still pretty comfortable. OK. So I was talking performance all that time. So let me give you some actual numbers. So this is on a PC and a Xeon E5 at 360GHz. So it's pretty much a start-up PC these days. You see that record is basically in the same order as the best case Madoch or a very short mem copy, like a half a K mem copy. It turns out that record is actually spending half of its time in get time of day. Just getting the timestamp is taking half of the time of the stuff. So there's a record fast. If you have a group of records, record fast basically reuses the timestamp of the produced one. And it's twice as fast. So you're talking, you know, look at the grid here, right? The grid is now 59 milliseconds. So basically, you saw 43 on my system here. It's in the same order here. So again, comparison mem copy is in the same order. Magnitude Madoch is more expensive than that. The best case Madoch is if it picks up something from the freeze right away. It's in the same order. If it has to walk the list, make some allocations in the worst case, it's much, much worse. Madoch is a killer. Don't use Madoch. S and printf is like, you know, want to use that, right? It's so expensive. Even printing to dev.nul is expensive. Even if you don't keep the data, it's expensive. Now, if you print to file and try to keep it, God forbid, don't do that. On the Macbooks, a different operating system, MacQuestern. Recall is slightly slower. It's actually faster than that now. It's below 50. I did some optimizations since I got all the data. And you can see that MacQuest is much better than Linux at printing to dev.nul. So it's a very fancy optimization where I don't know why that it does. It waits your time optimizing for the case nobody cares about.  So even in that case, you see that your well-sub micro-sugong, well, in the range of micro-sugong kind of stuff, for a first generation Raspberry Pi, 700 megahertz. The CEO of NVIDIA has a nice way of talking about this. It's a cute machine. Okay, that's cute. 700 megahertz. Okay, and if you take latest generation Raspberry Pi 2, well, sorry, latest when I'm at home. That's the latest I have. So it's interesting because you can see that printing to a log file really depends on the speed of your file system. In that case, it's a slower SD card, so the printing is slower. But basically, it's in the same order. So the bottom line of this is you can use it. It's really inexpensive. Even in the tightest loop, it's hard to find a case where you would not want to have the instrumentation staying in place just because it would cost too much. And it doesn't take that much memory either. Each of the buffers, you know, if you take a reasonable size, you waste 4K of data space somewhere. So 4K these days is, you know, it's four time-extinct layer, nothing. Don't call it. So it's lock-free for scalability. That's the measurements with one thread, same machines as before. Now, if you have 32 threads, you can see that basically the time is... Oops, sorry. Up. Ah! All the row keys are in the same spot on this keyboard, so it's... So you can see that basically the time is almost the same, you know, for one, two, three, four, 32 threads. Now, this is actually lying. I'll explain shortly why it's lying. And you get more of the same numbers depending on the zones. That's for Raspberry Pi. And you can see that Pi 2 is much faster at talking. Much better for multi-CPU operations. Okay, so I said I was lying, and the reason I'm lying is because the cost per thread increases. This is completely memory bound. So I don't know if Russ is here. I don't know if you saw the talk from Russ this morning about NUMA performance. This is exactly the kind of stuff you don't want to do in NUMA, having multiple threads that bang on the same counter, the same circle of buffers, et cetera, because you basically effect data from one cache line to another all the time. And so what is really limiting there is the memory bandwidth. So what you're seeing there is how much time it takes to write my record entries, and that's basically it. That's all there is to it. With that, time for questions. Please, questions. I beg you. Super. I'm sure mine won't be a problem. All right, so I'll ask the irritating question. What's the license used? I think I changed it to GPLv3 recently because I was using GPLv2 and someone complained. And frankly, I could not care less. So you asked me, I'm the only copyright owner. You want something that is under the barfooted license? I'll do it for you. Okay, I think this is very cool. And I've seen something, similar kind of application libraries that are helpful for programming that are under more permissive licenses. And not something that's... Yeah, that's right, or MIT or... Yeah, or even more permissive. Give me the most permissive you have. Okay, because I think that would, I think within places that would be very cool. So what I'll do is I'll give the, find me the two most permissive licenses and I'll apply one separately on each file in there. Any other question? And by the way, the next step for this is basically packaging. First, I've never really packaged on Fedora yet, so I'll use that to learn. The next thing is, that means putting it in a shared lib. I'm still struggling a little with the initialization with C++. The problem being that in order for the recaller trace set stuff to work initially, you want that to happen at constructor time. And so that means basically that any recall statements you might have in constructors might not trace properly. They would be present later, but they would not trace properly. So I'm still trying to figure out a way to solve that problem. That's the only issue I know of at the moment. Who wants to use it? Please raise your hands. Yeah, got it. Okay, thanks a lot. And...


This week's IonMPI, brought to you by DigiKey and Adafrit is from? Krusei, I hope I'm saying that correctly. I watched a video that of course I've forgotten already, but Krusei makes mechatronic and electronic control stuff and you know it's interesting we haven't yet put PLC on IonMPI because usually they're like massive and they're expensive and they're complicated and I'm like what would I use this for but I saw these adorable you know slim style PLCs and I was like this is so cute and I'm a sucker for anything cute and electronic so I picked one up so these are like the tiniest they're basically you know the size of a like a fuse holder basically it's a tiny little logic controller with four inputs and four outputs and you can run off of 24 volts or 120 or you know AC voltage or 240 AC voltage and inside is logic that you can program and what's interesting about PLCs is you know when you first yeah such as you come from microcontrollers and you look at a PLC you're kind of like well why don't I just use you know a microcontroller for this what's what's the big deal if it's just taking two inputs and ending them and then counting them why not just use it you know why am I buying a thing for a hundred bucks why am I not using something that's you know a $5 microcontroller board and this is the reason so all of these certifications that there's so many of them that they don't even fit on the screen legibly this has them it's UL certified it's you know IP rated it's tested it's assured it's reliable electronics and for automation you really need something that is reliable and will run continuously forever without you know a little bit of dust getting it damaged or you know mis-programmed bootloader you know brown out you know erasing all the flash so these are you know meant for automation but I think they could also be used you know at this size and cost for you know other robotics and you know home automation and robotic automation automation type projects so one thing that is neat about this one is it's Bluetooth programmable which is it is interesting I remember seeing PLC's that were you are programmable but this one actually uses a Bluetooth dongle now it's a I looked up the VID it's actually a blue giga which means that it's the Bluetooth dongle yes it's Bluetooth but it's like Bluetooth like you are basically like you connect to it over USB but it presents itself as a com port and that's actually really good because native USB is like a real pain to program on every platform and by using a basically you are to Bluetooth it it'll work under emulation it'll work like under Windows 7 it'll work under Windows XP it'll work under Windows 10 11 whatever you don't worry about you know weird sign drivers or native Bluetooth implementations you're literally just sending you are data through the program they give you so I think that was actually pretty wise I've noticed you know a lot of companies that have Bluetooth programmability they just go with these blue giga dongles because they are very simple so when you program it first off you know I tried this myself I downloaded their programming software for the PLC plugged in the dongle and it just worked it detected the com port just fine it detected the device just fine and then when you connect to it you get this kind of like again I'm never really used these so it's kind of interesting because it's like it's like flow programming but like everything happens at once so it's kind of like a cross between left FPGA CPLDs and you know block-based programming so at the top you can see it says like logic and you know you know math and whatever at the top you can select different tools so like here I selected the logic so you can see there's like and gates and nor gates and inverters and whatever and then I think I can click on control and you can see there's like counters and analog inputs and filters and and you know some of these are very simple they're like basic logic inverters and some of them are a lot more complicated like you can have timers and you know PID loops and incrementers and dividers and and you know you can have fairly good complex user interfaces and again one of the nice things about PLCs is that everything kind of runs at the same time so you don't have to worry about logic flow control as much because it's all kind of like instantaneous it's not complicated programming situations you know there's like no Wi-Fi interface here JSON parsing but it's really good for like take some inputs process them and then give you outputs without having to worry about you know cross-contamination of logic streams okay so after you you know when design your program you can save it you can compile it and then you upload it over Bluetooth and yeah it worked it works just fine it's kind of cool so if you you know you can program it and debug it all over BLE whenever you you know want to upload new code and you can also password protect it so people can't download it or reprogram it another thing that was interesting is because this PLC does not have a LCD or TFT display you know be wondering okay what if I want to get debug output or I want to like you know if there's a count output and I want to see the count or I want to do another analysis well all again over Bluetooth what's really nice you can use an app to do the debugging interface debugging and user interface when needed right because a lot of PLC's they you're not constantly pressing the buttons and reading the output it's only for like when you're doing the analysis and debugging and so you can basically download the app for iOS or Android install it on your computer and then you can see like this simulated LCD output and you have button inputs as well so you can do diagnostic checking and download data logs and stuff like that so I think what I thought was really neat about this is like I've never really done anything with PLC's but I was actually able to get going and get it working quite quickly and it was kind of shocking to me how easy it was so I think that even though this is a very cute adorable little PLC it could be also really handy for anyone who's doing either advanced stuff and just doesn't need a lot of inputs and outputs only for inputs for outputs but if you're also just like hey I want to learn about PLC's and build something at home just practice with a PLC to get that kind of training which again in you know when I went to college there was no course for it so if I want to learn this stuff I would DIY it pick up the BLD dongle and the PLC and you can just play around at home just like automating stuff in your room right to play around with the wheel a inputs and outputs and long inputs in PWM and just see what it's like and then you can see the the debug output diagnostic output on your phone available on digikey and it's in stock which is the thing that's great digikey.com slash short slash ZT78W55V and you can also search for it by typing in 96689 sorry 88983301ND and you can also just type in the words Cusay CB8R which is the name of this maybe I'll show it really fast on the overhaul. Show it really fast and we have a 2 minute and 45 second video that you wanted me to play. Yeah yeah yeah so just show it just to show how this sizing. So this is it and these are the the four inputs this is power you power it from 24 volts but yeah you can see it's like you know a lot of people are like oh I need I need it to be UL certified and I need it to be you know certified in every country. It's made in France it's got the BLE Mac on the outside and then here's the relay outputs 6 amps a piece it looks like 6 or 8 amps at 25 C and then there's LEDs for power and Bluetooth and it's just like so cute you can panel mount it with these two bolts or you can din rail mount it so you can see the din rail slots here so a very elegant and beautiful little controller. All right we're gonna play the video and then we'll wrap up by MPI and we're gonna get a new product. Hi I'm Robert and welcome to the Cusay Tech Lab today I have the opportunity to introduce you to one of the most exciting and revolutionary products in the small-scale automation market it's Cusay's Millennium Slim. So simply put a logic controller is the brain of your application it takes inputs which are wired here and within the logic controller they're going to process instructions and send those instructions in the terms of a signal to an output. Cusay specializes in making logic controllers that are easy to use easy to program and are favored by both experts and beginners and Millennium Slim represents the latest in Cusay's innovative line of Millennium controllers. Millennium Slim is the only internationally recognized industrial grade logic controller in such a narrow package size in fact at 17.5 millimeters narrow it's the same as a timer relay in its profile and it literally can fit in your pocket. This means you can put logic control in almost any application anywhere and it's easy to mount too. You have two options there's a din rail mounting feature you can see those features here or you can see screw holes here and that allows you to panel mount it in your application. Millennium Slim is in a compact profile but it packs a lot of functionality it's got four convertible inputs two of them are high-speed digital which is ideal for encoders or tachometers and two are PWM or pulse width modulation. On the output side the four outputs can either be relay or solid state solid state gives you the option of having PWM which is ideal when you've got an application that's frequently turning on and off. It has Bluetooth wireless communication already integrated so one Millennium Slim can easily communicate with another up to 20 meters away. This gives you the opportunity to control a number of different applications that interact with each other. The product is designed for use with Krusey's virtual display app. You see it demonstrated here on this smartphone Krusey virtual display is a way to view or modify the program either on a smartphone or on your laptop or PC. You can also download the data log on your smartphone using Krusey's virtual display.


Hello everyone. This is a talk about XCPNG. So this is more the story behind the project and not technical talk. But anyway, the title is Building an Open Source and Turnkey Virtualization Platform. So first, I'm Oliver. I'm the CEO and founder of VATES company doing Open Source since 2012. I'm a former CIS admin doing a lot of Linux, BSD, and Zen stuff. Also, I'm the founder of Zen Orchestra and XCPNG project. You can find us on Twitter and here is the website of XCPNG project. So first, to be sure we are talking about the right thing, the title was about a Turnkey and Open Source Virtualization Platform. So what does it mean? So it means that a virtualization platform is not just the hypervisor. It's the hypervisor but also the network stack, the storage stack, and an API. So in the case of XCPNG, it's basically Zen as the main engine and open V-switch for the networks, MAPI which is the storage stack inside XCPNG, and finally ZAPI which is the API for it. On top of that, you can put Zen Orchestra which is basically a web solution to manage the platform. When I said Turnkey, Turnkey means that you just have to download the ISO and install it on your hardware. So it means you don't have to make some specific configuration to have your Zen running. You just download, put on the stick, put on the hardware, install it and that just works. Plus, you got everything that you need through the API. So it could be using a client like the CLI or a web UI. Anyway, every command is going through the API. So it means, for example, if you want to create a network, you don't have to do that manually. It's just H to the box and create a network using OVS. No need to that, you just tell the API, I want a network for my VM. Finally, when I said Open Source, well, we know what Open Source means, but I want to just tell it's not about only the license. It's GPL V2, that's cool, but it's also something that's having a really, really nice community and nice. I mean, by that's a lot of people coming from everywhere and working on the projects, not just giving feedback, but also improving the documentation, etc. And finally, it's also about the governance of the projects. When I'm in Open Source, I mean also that all repo, all the doc and the developments, it's completely publicly accessible. So that's, for me, really important. It's not just a matter of the license. So this is just a view of a dashboard. So if you can represent what does it mean? So this is a web UI, Xenakis round-top of XCPNG, so classical stuff like Overt or Proxmox, whatever. So a bit of history for people here who doesn't know as any detail. So it's not new. It was in fact the first open-source hypervisor released back in 2003. Then Citrix acquired the company doing Xen, called XenSource, and Citrix decided to launch something called XenServer, which is basically a virtualization platform. It was closed source at the time. And then a bit later, Citrix decided to focus more on desktop virtualization. So it means the product that are using XenServer on top, but more for people accessing to their desktop, Windows desktop from a remote location. And it means that Citrix is more focusing on that and not more on server virtualization. One year later, the first release of XCP was done, and basically it was the open-source version of XenServer. And XCP means XenCloud Project. So you start to know about XCP. So you can imagine why it's XCP-NG because it's XCP new generation. A few years later, XenServer was open-sourced. So it means the end of the XCP project because it was the open-source thing, but now everything is open-source. No problem. And this is when we started to work on Xen Orchestra, just to give you some timing. In 2017, Citrix decided to remove some really cool feature inside XenServer. So they decided to remove, for example, live storage migration and that kind of stuff. And all big features, not in the hypervisor itself because Xen is fully open-source, but in the platform won't be open-source in the end. So we decided just a few times later to start the XCP project, XCP-NG project, to remove all those restrictions and to do a lot more than just a fork. So you can ask the question, why we are doing that? Because it sounds a bit 2008, to fork a virtualization platform. Everyone is talking about the cloud and the containers, et cetera, et cetera. But on the field, if you are not Netflix or a real large company like that, everyone is still using virtualization. And people want to have something that works, that's ideally open-source and that can scale. So, well, our reasons to do that, we are multiple. First, Xen Server was a great tool, really a great tool because it was a real turnkey platform. But now we know the future is a bit close source. Sadly, there wasn't any developer community outside Citrix walls because it's more a commercial project for Citrix. Also, Citrix had other priorities than working with the server virtualization world. So it makes sense for them to have other priorities. Not a problem, but the problem with that is there is no convergence between people who want to make server virtualization and people who want to make desktop virtualization. Also, not all components were open to contribution. When you release, it's GPL compliant. You release a table, but you don't have the master branch. So it's really hard to contribute from some project inside it. So we saw a great opportunity to open everything and to allow people to work on that and to improve the project. Also, for us as a company working on Xanakistra, it's a good opportunity to grow the Xan ecosystem by not just having one platform, but another one which is more open and also be able to answer people's needs not just on the management side, but also on the layer, which is on the bottom of Xanakistra. Because now we can have a common objective with both software to go into the right direction to improve, for example, some performances that are really important for us and also fix all bugs, et cetera, et cetera. But doing a fork of a platform is like forking a distro. So it's not really easy. Just to show you, this is basically Xan server or XCPNG platform in that square. So it means you got a lot of components. So we are not forking every component. It wouldn't have any sense to fork Xan or Linux because they are fully open source. But the goal is to be able to take the stuff, to understand how it works and to build that in a more open fashion in short, okay? But to do that, you really need to understand which component is talking to other components, et cetera. So the interactions between all those components, this is one of the challenge. So to recap, the three parts, the three main challenges that we have to overcome. The first one, and maybe the most important one, at least at first, is to get a good community walking with us. So it means not to be only a small company doing some stuff in our corner, but to be able to gather people around a project and to have people starting to do contribution and that kind of stuff. So we weren't the only company doing that. This is one of the challenge. The other one is, as you may see, in the technical challenge. Technical challenge is we need to be able to fork that and to release something that works, okay? So it's just not getting the whole stuff. We need to understand how it works and we need to also be able to demonstrate how it works and to improve it, to fix bugs and also for people using Zen Server, maybe to be able to migrate to our solution. And finally is the adoption for Zen newcomers. For example, people who never had any experience with Zen, so it should be easy for them to use XCPNG. And the last challenge is to have a really open project, open AmyMathet to get an open governance, for example, to be inside a foundation or also the last part is to continue to work for a long period of time. So it means we need to have on the corporate parts a support business model. So we can continue to pay a developer's team to work and to improve the product. So did it work? I have some answers. The first answer is, well, we did a Kickstarter last year and the first objective was about only 6,000 euros. So it was a low bar, but we estimated that we could take too many risks with that. So we will try to see if there is some interest in the community about doing that. And for this first goal, the objective was to release a kind of prototype. And we had other goals, but without thinking that we will reach all of them. And finally, in less than 24 hours, so it's February to March, okay? So it's one month. And in less than 24 hours, we got the first goal and then continue. And we've seen that a lot of people were interested in fact to have a really turnkey on an open source characterization platform. And in the end, even companies became sponsors by sending us money, not with the Kickstarter platform, but by sending checks or wire transfers. So in the end, it was more than this amount. And then it started and it started really fast because we did the first fork in April, basically exactly the same thing than Zen Server, but without the feature restriction. But then for the next releases, we've been able to add new features. For example, we have been able in October to also follow the pace of Citrix for the Zen Server and be able to sign our RPMs to add software-read support during installation, to add new features. So it's really a fast pace cycle. So today, regarding the community and the project challenge that we had in the previous slides, we've got a lot of people who builds a really, really vibrant community of people using it. A lot of active users on the forum, so active, I mean, by that people with more than one post. Got a lot of posts, a lot of threads every day. Got a lot of unique downloads. So it's really a good sign that people are trying the product, are talking about the product. Plus, it's still really easy to upgrade from a working Zen Server. It means it won't erase your VMs. It will just, you know, like an upgrade, in fact. So we'll reboot and it will be on XEPNG and it works. We are now a member of the Zen Security pre-disclosure list.  They have, in average, 15 days to fix some problems. So we are inside that. So that's really helpful to be able to fix problems really fast before it's public. We are working to be inside a foundation. So it's not official. So I can tell a lot more about that. But we are working on that to really have an open governance and not, you know, be the only company behind that. We want to gather other people around it. And also, as I said, no license or future restrictions. So about the technical side. So as I said, we did a lot of stuff in one year. So we already, for example, fixed bugs in the upstream. In that stream I made, for example, in Zen server, we discovered some bugs and we fixed them or we helped Citrix developers to find the root cause of some problems. So it was really, really cool. We got a testing repo. That's really nice because it means for people who want to try some maybe cutting edge features or, you know, like support of storage that weren't existing before without taking risks to put that into production so you can let the community to work with that. So that's really, really helpful. We got cloud stack compatibility and finally we embed the Z standard compression inside XCPNG basically. It's an algorithm, compression algorithm made by Facebook which allows us to compress faster and better than the existing algorithm in Zen server which is using JZip. So we've got a comparison of backup time in gray, light gray without compression. Then in dark you got a JZip compression and in red with Z standard compression. So we did a lot of performance improvement for making a backup of VM. For example, when you export the VM, it's compressed with Zen standard. It's really, really faster, up to six times faster than JZip. So it means in less than one year we were able to add really cool features in the short amount of time. So what's next? So this year we want to have more firepower and to do that we are growing the team working on that. So we target to have up to six people working on it. We are working with some research labs and universities. So we found that a reasonable amount of papers were done in France about Zen. So that's really interesting to work with a local university knowing Zen itself. So it can lead to interesting stuff. And we target a team about 20 people for the end of the year if you count all the researcher and the engineers and the contractor. Again, it's a joint effort. Regarding the research and development, we had a lot of topics to cover because as I said, there is a lot of room for improvements and that's really great. You can see that we want to improve on any side of a virtualization platform. So we are targeting improvement on the storage side, on the network side and compute side and to do that with some universities. So you have publication, that's really great because you have also benchmarks, you have the data so you can see the improvement that's have been done into the research. So it's very, very, very a lot of different stuff but to give you a quick example of one field that we are working this year is to improve the NVME drive, obtain drive. If you don't know those drive, those are 3D X points memory from Intel which is really, really, really fast. A lot of high ups. But the problem is the legacy storage design for a virtualization platform were made when HDD were there. So high latency and not really a lot of high ups. But then if you decide to create a specific layer which is thinner than the original one you can maybe reach almost the bare metal high ups of the hardware. So you need to do a lot of improvement to go there but that's doable. We got an architecture and we know the general direction when we need to go but now this year we need to go to the next phase which is a proof of concept and some benchmarks. And then if it goes well then it will be implemented and again pushed upstream because the goal is to push everything upstream. Also because as I said it's a distro in the end like a DBN or whatever. We need to improve the thing. So in XCPNG like in Zen server the DOM zero the privileged domain is based on CentOS. So we need to have a platform like Koji to automate the RPM build process. Koji is used by Fedora. We need to also make some automatic QA and CI. So if we add for example a patch of something that's ideally a platform should be able to tell everything's great, it's built, it works automatically. So this will help the project to have more contribution because every time someone adds something automatically it will be tests. Also the great thing about it it's that you can have nightly builds for people who want to test really, really cutting edge stuff. There's value stuff so to improve the mirror management when someone will become, want to become a mirror for download XCPNG. There's a lot of project are doing. I think we will use a project done by used by the people behind the VLC projects. Okay, so we can use really interesting open source project to help us about that. So also other features but I won't enter too deep inside of the details. So and to recap also regarding the Xanox side it's a management platform but more than that you can do backup resource delegation. So it's not cloud stack or open stack. Okay, but still it helps to manage your infrastructure up to let's say 2,000 or 3,000 VMs and then if you have a large infrastructure maybe probably cloud stack or open stack will be better. But anyway we want to integrate more features between the two software because this allows us to create really interesting connection and to also innovate on both interaction between the two software. So we are working with both Xanox team but also the XEPNG team working together on the ideas to push both projects into the right direction. So how to help? Well we got a really nice community so if you want to just try it, download it and then if you have any problems to talk with people there. Also we are hiring so if you are interesting to work on system development like QMU or Zen or that kind of stuff even the kernel or if you are using or developing in OCaml because the API is in OCaml we're interesting so just come to see me after the talk. As a company there is a lot of interesting programs if you want to make research and developments at the European level. Also just in the end using it will be helpful because more people using it, more hardware will be tests. So it can work from your I would say medium desktop machine up to your server inside the data center but it was built in mind to run into let's say servers into a data center. So as you've seen there's a lot of stuff coming and we did a lot with a small team and having a larger team will be really interesting and we hope to build a lot of new features and exciting features for this year. So stay tuned on the xp.ng.org website to get all the new stuff. And that's it, if you have a question go ahead. Yeah. Now the cooperation. For now it's, okay, what's the difference now with Zen server projects? Right now we tend to limit the fork because we are pushing upstream everything that we are adding. So for example for ZSTD stuff we added that into the upstream because most of the tools inside Zen server are open source. So we can push for the API, ZAPI on the GitHub, et cetera. So we try to maintain the smallest fork as possible but drive innovation and maybe to use our innovation faster than the Citrix cycle which is different than ours because we don't have the same priorities in the end. Yeah, merging features faster and yeah, to be more flexible than Zen server really cycle. Yeah, yeah there. So what do you mean is, does it work with CloudStack or sorry, I didn't hear the. It doesn't work. OpenStack, yeah. We don't have OpenStack support right now because you know OpenStack can work with Zen server on the bottom, CloudStack too. People like CloudStack already help to make it compatible with it but not yet at OpenStack. I don't know the current status at OpenStack right now but maybe you shouldn't ping some developers. It's the same API so it's not really complicated to adapt to it, it's just changing the name, a string basically to have the support of it because it's exactly the same API. Other questions? Yeah. Okay, so what the main driver for the XP NG project. Right now it's Vates, the company behind it but for example on the Windows stuff like the Windows client and the PV drivers it was someone from the community. So the goal to be inside a foundation is to be able not to be the only company driving the project. We have a lot of people working on it. We want to have a lot of people into it and we have professional support on it but it doesn't mean it will be the only company to provide improvements inside it and the goal is to have more companies to work with us. For example we have some people who would like to have for example the VX line supports because they are using a big data center and they are ready to work with us to improve that. So it's not just something by white company and we don't want to be the only company behind that in the end. Yes? Okay, so the question is the ability to use only for smaller use cases for one server because for example Proxmox got a web UI bundled directly with the installation. So you just for one machine it makes sense to have everything bundled. It's not the case for XEPNG but yeah you can deploy Xanarchistrate relatively easily. If you just want a basic UI we got a virtual appliance that's just really to download and to use if you want to install it from the source. Yes, you need a VM to make it work but bundling a web UI directly inside it could be problematic for something that's made to be work in clusters in pools. So we are thinking about that but there is no trivial solution to deliver something like that right now. Yes? Makes, sorry, make what? With a web UI, okay. So the question is maybe to create a flavor of XEPNG with a web UI bundle inside it. This is some stuff that we have discussed about already so we are thinking about this. And by the way we got stickers if you want at the end of the talk. Just thinking about it right now. Okay, that's it. Thanks a lot.