delivering value in blockchain networks today and let me grab, give me one second. Okay, and Jim, you're live streaming as well. Okay, great, thanks. So Alex is going to talk about distributed ledger and how it can create business value, which is really what we're trying to get to. It's not about the technology, but the value we can deliver. And he's going to talk about strategies for comprehensive digital transformation. And certainly DLT is one of those important building blocks in creating ecosystems and so on and delivering value. So with that said, I will say that Alex's background is deep in this space. He's been working on digital transformation for a long time and has spent a lot of the time in the government sector. And all his background includes an MBA from MIT and he currently works with publicists sapient on a variety of different projects. And so he brings a lot to the table and looking forward to the presentation. Before we get started, I do want to talk about, let's see if I can get to the break. First of all, we're hosted by the Linux Foundation. And Linux Foundation has an antitrust policy that's available on the website at linuxfoundation.org. And the antitrust policy basically says that Linux Foundation meetings involve participation by industry competitors. And it's the intention of the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. So please do not participate in any activities that are prohibited under applicable U.S. state, federal or foreign antitrust and competition laws. Secondly, Hyperledger project does have a code of conduct that we all abide by. And the code of conduct is available on the Hyperledger wiki. And that hyperledger code conduct says, we are committed to creating a safe and welcoming community for all. We treat each other with respect, professionalism, fairness and sensitivity for many differences and strengths at all times. So again, that's that policy is available on the code of conduct. And with that, I want to turn this over to Alex and let him get started. Thank you, Jim. And good morning, everyone. It's always a great thing to start with good conversation Friday morning. With the pandemic, Friday is the same like Mondays and Tuesdays, a little bit distorted, but at least give me a sense of these in a good Friday, we have time to kind of think more broader about what's going on. So I will let me also share my screen.  But being interesting tool is not enough, right? So basically, being interesting tool is interesting for somebody to read a nice article. As a matter of fact, there's a lot to be said about blockchain technology and a lot interesting things on interoperability and interesting layers you can put. And I myself worked on implementing hyperledger pretty much for all the blockchain solutions I worked on. So there's a lot to be said about that. But today I want to really talk about something that I've seen being very problematic in our marketplace and also in public sector in particular, but also I'm bringing both experiences from public sector as well as commercial sector sort of all as a practitioner and as a pioneer of this. So probably I was one of the very first pioneers to launch blockchain within the government and work with agencies on that. So I'm going to show you some of my perspectives on what we're back and today. And so hopefully it's going to give us some more some better insights. So I am I am in sort of all of this tied to strategy and consulting. I'm a big technologist too. But you know, we always start with what's what's really out there. And what's really out there is a digital transformation. DLT is one of those tools. And at heart, my company is basically doing digital transformation. That's kind of what we have focused, not installing blockchains or installing AI or so on whatever really making companies to be what we're called digitally native. So how do you sort of be that digitally native? Just a quick story, I think probably most of you have heard of publicist sapien. So we are essentially, you know, the strong the leader in digital transformation space, as well as in all the user experience space and technology space. So that kind of gives an interesting edge or interesting way of how we go about a lot of work we do. And we do work a lot across the globe, both across the many industries, including the public sector and that by the government. So we take the stuff we learn from public sector, bring the commercial and analyze versus. So and you know, probably you who read the forest or wave or garden or boards, you'd see that we on those sort of leading corners when it comes to digital business transformation, particularly some of the accelerators. There's a really interesting topic, like as some of the digital transformation you know, expert who is in this arena has been doing this a lot and even back from my MIT days. You know, it's always good to see technology, but recently what I found is a blockchain paradox.  So I talked to a lot of folks, I talked to everybody from students to CIOs of major corporations to govern CIOs. And typically, and as some of my presentation, I asked people, what is your knowledge of blockchain? And this is the answer I got from students. Okay, so and that's okay.  But then I go to CIOs good to all others, the answers are not much different. So it's interesting. So we've been talking about blockchain for a while, people have don't understand it. So, so why is that?  And you know, if you think about what, what essentially Vitalik has a long time about blockchain, right? It's not really, as you said that at that time, it was really not putting the taxi driver out of the job to use the Uber analogy, but really it's putting Uber out of job, right? Because Uber provides that in a way that's trust and that safety and that sort of all the verification so that you as a customer of Uber don't have to worry about it as a driver where all the blockchain could certainly do that. And it is in technology with enormous, enormous potential. So I was just wondering why this paradox is coming from. And it's not just students, it's CIOs, it's government employees, it's commercial, it's everywhere, right? Obviously, I don't count unicorns as a blockchain. I mean, but we don't all live up from unicorns. Like we live from, from what's really going in a mainstream marketplace. And I was just scratching my head. And this, and this, and I always said that's a paradox of transformation really comes a lot with the issue of understanding innovation, understanding the exponential power. Okay, we, we as humans, even mathematicians, whatever you want to call it, we just not good with exponential. And, and it is, it is, it is, it is very problematic.  Or even if things pick up, we just say, ah, that's not that still, that dominant. But I just want to leave, start when talking to the digital, I want to start with this effect, the exponential effect. And a very simple exercise, right? So, you know, you look at the, if you look at the, you know, stadium and American football, they look at the soccer stadium is much more closer to my heart. But I look at what the, you know, the stadium and if you were to put in the middle of this stadium, if you to put the, what I would call the exponential tap, basically the tap, they will just drop of water drops one by one, but in the exponential fashion, right? So meaning like in, you know, if in the first second, you got one, right? Then you get two, then you got four, then you got eight drops and on and on, right? So you get the logic, right? So if you put exponential tap, not regular water tap, exponential tap, that drops, you know, a little drop by little drop, but exponentially, right? So just a question to this audience, what do you think, how long will it take to fill at the top of this stadium? Any volunteers who like to tap the top of the stadium? It's okay. You don't have to say. I'll throw one out. 45 minutes. About 45 minutes? Okay. Any other volunteers? All right. You don't have to, I mean, it's also legitimate that lots of folks say this will take a year. I mean, if you think about it, it's just a drop by drop. I mean, exponential is fine, but it's drop by drop. But if you think about it, take a look and do the math through this. You know, it's a total area. This is high. That's the stadium capacity. This is how it drops like going on. But as you love the math, you can do that yourself. And as, as James has said, it's actually 40, it was very close, it was 49th minute. So maybe that's not surprising to some, actually surprising to a lot of folks, but to some, maybe not surprising. You expect the 49th minute will hit there, right? But what is more surprising is where is the water 10 minutes before this? So what, where is the water in 39th minute or 40th minute? Any volunteers? Where do you think the water is? At 39 minutes? Roughly. I'll say 10% full.  Is it like, you know, you know, 100 inches? I'll say 40 feet. 40 feet. That's fair. Any other takes? Well, be interesting.  I would just like, you know, eyeball it's something I knew like Jim. But to surprise, it's only mere six inches. So folks, we're having a six inches deep water in the stadium and we're in 39th minute. And then 49th minute, it's going to be overflowing the stadium. So really what's, what's going on here, right?  As a matter of fact, it's a six inches. So you sit in there and you say, that's not that thing's going to happen. Come on. This is nothing. It's just been dripping forever. Sure. It's little inconvenient, six inch water, big deal, right? I got my boots, right? I've seen this before, right? So basically what's happening is this whole notion of linear versus exponential. So over time, exponential looks like linear. It's actually roughly the same, but it's not because at some point there's disruption happens. And when disruption happens, it really turns into what we call the real exponential, right? And if you think about it, that's what's happening today with technology. And people just don't see it. They cannot comprehend it, right? So like to many times people say, blockchain was such a good technology. Why we didn't have 20 years ago. What was so bad about it? We had it. What could we did? There were, there were, there were nice pieces of work around blockchain that long time ago. But yes, we didn't have technology. I mean, who could have distributed databases wet on one megabyte hard drive or what? 64 kilobyte machines wouldn't even have internet, right? So, so when you take into that enormous exponential growth, think about the cloud computing, high power computing network, you name it anything, the chips, the screens, anything, right? So it is adding on top of each other, right? So it's even surprising as professionals. I know I sat down some months a year or two ago, sit down to do this great page for a quick question. And we're going to do it in AWS. We're going to use great, great stuff from Python libraries. And it took about two, three weeks to do it. But it's 95% precise. It was, it was phenomenal. So, so cool about it. Then, yeah, just a week after, you know, here's, here's the whole service available in AWS. You just click the button that's there, right? And it's, it's actually better than not than 95%. And I thought three weeks was pretty fast. But this is a few clicks now. In AWS. So it's an example. It's, it's, it's going enormously and it's affecting each other by having all these little things available to each other, one by an accelerators. We're at this point today where AI is not going to require a deep experts. It's just going to be people clicking at things and getting to getting it done, right? So I said this many times, we're going to, when they have an RPA robot, that's going to be putting AI out there, right? So, and then like, but that's all great and fine. But why these things are not working, right? And you soon realize it's not the technology. It's not this, it's not, it's not the speed. It's none of these things. It is really, it is, how do you use this? Like, how do you use blockchain? How, where do you start with, right? We're not installing, right? So that's really the, the heart of it. One of the, I've seen is that when it comes to digital transformation, blockchain is a tool to remove business friction, right? And it's well known business friction. That business friction is obviously comes from business process. And I can tell you that is a, that is a great user blockchain, wherever you can smell that, that's where it is. And obviously that smells like lots of like production of costs. That smells like, like making things faster, right? And sure enough, if you do some of the, some of the data set from 67 by 67 live enterprise blockchain networks, not talking here about millions of other experiments, enterprise level blockchains, but 67 of them, many of them at a high for ledger, if not all, is the 72% of cost reduction. So again, removing that business friction, new efficiencies. So it is a little bit of incremental revenue generation. That's all fair and good. But the first and foremost is about that friction. So I'm kind of looking, looking at that friction. So basically, what basically how that sort of works for removal of that friction is really, is really because blockchain enables this network value chain, right? So it's not a linear chain anymore. If you think about it, you know, Tom has to do something with Steven and Steven has to do with Mary and Mary has to do with whoever and so really great and well, well, it's not like that anymore, right? So we have now actors, we have bots that can be on their chain and operate and close transaction, we have many, many, many people. So you have smart self self executing contracts, whatnot. So it is a different dynamic dynamic that just destroys the business process as is. So the one of the things to think about blockchain, if you implement a blockchain to do the same thing, it's not going to work. You have to think how to change business process. It's just like Uber app, right? Do you, do you know that taxis have actually very decent apps? Super good apps. If you ever use red top caps, super nice app. It just doesn't work the same way as Uber is not efficient. I can never rely on it. I mean, and Uber app works not because it is a, you know, special to them, but it is, it is, it is because it's against the solving of that business process problem and the way how these things are scheduled and done, right? It's not, it's not the same. That means being a digital at heart. So when you look at the implement blockchain, look at more digital at heart, right? Don't look at just a little bandaid here and there. And here's, here's the things we've seen from public sector and elsewhere in commercial. When we do it, yes, adoption and interest is growing. I wholeheartedly agree. And I can tell you a lot of people want to take a look at a lot of government agencies, a lot of folks, a lot of CIOs, right? In many conversations I'm hearing, oh, just the database, let's, you know, and I'll just install it and it'll be okay. It'll replace even our Oracle databases. And so that's just kind of the wrong way to look at it. And it's not technology solution, honestly. It is a really modernization framework. And I'll share with you my idea as how it all works in the ecosystem. And it is by really leveraging other technologies and other ideas to lean on top of each other to get the effect you, you look in. But if you do not start from the very first is from the user and from the business value, that's going to get a final lot. You're going to find yourself in a lot of troubles. And that's hence the roadblocks we've seen.  Where's business case? I'm calling from Actioc for the public sector. They actually have designed a very good book. Casey probably is familiar with it. You can go and download, which is signed points to the business case. It's super interesting. World Bank user, just many others have used it. I use it myself. But it is a very good starting point. I have seen that in particularly government sector, that starts to be slippery. And there's no, there's no regard for users here. Many times I've been in a long-term conversation, she just talks about backend systems. I don't know contrary, what the users went to see. And so users have limited buy-in. And typically, users get tortured with the blockchain. I don't think users care about blockchain. Users care about their business problem. There's a lot of lack of agility that I've seen in product engineering. People do blockchain project. That such thing doesn't exist. The only thing that exists is your product that you're building. What are you doing? You're building a new contracting system with blockchain? Great. That's your product. Make it very agile. You don't need to do million tons of analysis. On the contrary, analysis is pretty straightforward if you thought on what it is. And that's to get to the really focused on a business and not constant firefighting. So it's a really interesting thing we've seen around there. So out of this, three big things. How to put the blockchain in the center of transfer major, that I've seen working across the industry, including public sector, is really make it a part of a foundational area to enable transformation. Not the only tool. So if your project is called blockchain or DLT, your project is doomed to fail. I don't know any project that's just named like that. It has to solve some certain problem. What I noticed working really well with the blockchain and we're doing lots of accelerators of different kind, is really start that out of heart. If you really start that out there with the user experience, what is that? What are you building? What people will be perceiving, right? And then you start to layer, okay, what's kind of data is going to be organized like that? You get that blockchain as your distributed ledger. You have it there. Even today, we're seeing some increased speeds where we can use blockchain and applications. We couldn't think of it yesterday like digital marketing. When transactions move a million times a day, I use a lot of blockchain in a federal contract and obviously you don't have a million transactions per second. It's kind of slow, quote to quote. So you have a lot of slack. But even in a fast-moving applications, it can work. So you have that as a ledger and then layering on top of its smart contracts and AI. Because you build the blockchain, you're going to get better captured data. When you get better captured data, you can build all those services on top of it to really understand and get inside of that data, right? So thinking about that, that's going to get there. Just putting a layer of blockchain, so what? It's not exciting. I haven't put it in Excel. But the way you go about it doing it is important. You cannot today, you cannot go on this until you microservice the environment, until you do testing automation, until you do DevOps. Until you can turn a dime to deploy something, see, touch, move on. That's what will bring the IT modernization. Without that engineering maturity, there is no blockchain implementation. It doesn't work in the Garage 3 guys doing it, never done QA on it, don't know what automation is. Oh, and we'll deploy it once in three months. Well, it doesn't work. Your idea is great. But you need that engineering muscle to get to that. That's not easy. I've seen many large companies struggling with it. It's easier said than done. Oh, just, you know, daily deploy. Well, easier said than done. Well, daily deploy the blockchain solution and make sure that database is an integrity and all of that. There's a lot of things around that. But if you think about that as a sort of ecosystem, then it's going to start to make a lot more sense to all of you. And as I said, without this, with a really strong user experience, it's just no technology success. It's just not going to get there. It is a typically blockchain now visible for people. Yeah, I have to yet to show me any user to understand it, and you have to yet to show me anybody really who cares about it on a front end, meaning those people who are really going to be doing the apps.  I don't think so. So from that perspective, this really started with the human centered design and discovering the problem back in track, failing the use case and going through it is something that he's going to get, get somewhere. And some government agencies are better than that. Some are not. So it is something that we as a practitioner need to bring down down there. And, and with that, you know, blockchain only works if people adopt it. You just build it and hope people going to use it, even if it's the greatest ROI could take away and I'm going to work. What basically is what's really working here is that, you know, that's this, this has to deliver value for users and user, right? Yes, also business value for corporation or government agents, but also today and user. So it's a combination of it. So, so efficiencies, they are in obsolete data, but there's a lot of other ones that are there. And, and that, that needs really to be shown and demonstrated. Those people need to be brought to the process to try it out to view it. It cannot be a corporate installation of this. Probably not going to work. And, and thinking about this, right, that's like this little simple framework. If you look at the ROI and user pain point, right? So that's simple framework that you can do how to kind of mix that because sometimes ROI does not align with user pain points. So I mean, if it's a line, right, and it's a star up there in top corner, it's great. That's awesome. But good luck with that. That doesn't allow all the time, right? Also, the sort of the weird idea that's an X corner, which I know user pain points and ROI, that typically people go fast through it. The much interesting is that box over there, where you have a lower user pain point, but enormous ROI. So think about it. So, so the government agency could have had enormous cost savings, but really it's not solving user problems. Yeah, it's a great idea. It's super good. Management is buying, but you go to users, they're like, sure, it's not making my life any easier. It actually doesn't make any sense. Yeah, helps the organization, but not me. That requires serious change management and requires actually looking at what really lever there is for the user and how do you do it? And you got to do that constantly. Right? So it's not a noxious. I think it's necessity of how you think about these digital transformations, particularly moment blockchain, which is sort of still today mysterious invisible tool. I know we all go deep into this. We've seen these things. I talked to a number of CIA was in a large enterprises, even high tech enterprises, and I didn't tell you it's not day to day to them. They are still very careful about it. So the more we can focus on things like that, I'm successful. We'll be rolling out the technology. So I don't want to sound like I'm putting the party down. So being is a big blockchain pioneer in public sector and being also a, what I will call it, really excited with digital transformation. I want to leave the impression that I see the doom not at all. I just see these issues that I didn't bear to my heart and I've seen them going on around there. But there's a lot of sign of hope. And I find it in data. You see a lot of data. So when I ran with the number of adults, so as a simple, simple analysis of what a blockchain solution, I got to offer them how they should be implemented, right? I think it's really good among a lot of these folks, particularly public sector agencies, they're looking there to start small looking to start a group of concept. They're looking to do that. There's no big banks. And my hire immediately goes, you know, oh, that's kind of cool, right? It's a responsible way to roll out, a responsible way to make this happen, a responsible way to grow. So that's a big sign of hope. Also, when you ask what's most critical in adoption to make this work, everybody get is a human centered design. You cannot start with that. So do not start with technologists in the room thinking about how nodes work in for our ability and, you know, and putting that a fascinating people with zeros and ones stored in a chain. That's not the way to go about it. It's really, it's really, it's really about that human center. And people kind of really get that. And it's, and it is, and it is ultimately what's, what's critical for adoption. So I'll stop here, but I want to leave you with this, with this really thought to that, you know, as you can tell, I like this sort of over, over analogies. But if you think about back in 2000, we were talking to our kids and talking to everybody, don't talk to strangers on the internet, right? Now we're getting a stranger's car, right? 2016, 10 years afterwards. That's about what I could, what I recall. We kind of went through it and we're advising everybody to use Uber, you get in your car and you go, right? We're kind of in a blockchain in between 2006 and 2016, right? We're still telling sort of the corporations here to lend their people, well, don't talk, don't talk blockchain around, you know, don't hang around and there's what's going to exactly happen. We can, apparently we can add things, but I promise you, think about the exponential curve innovation that's happening. I think before you know it, it is going to hit as, you know, like it's back to 2016 for Uber, it became the mainstream and it wasn't. So the same thing is going to happen in a blockchain. The same thing that happened with the cloud, the same thing happened with other things. We've seen a lot of that going on. And I think that is a, which is actually very exciting. So for us as a practitioners, particularly those who love technology, dear to it, I'm going to put it myself, it is really kind of putting that, you know, in a right perspective and understanding how we can work with organizations to make that happen. So I'll stop here, transfer the floor, we're back to you, Jim, for any additional questions or any additional kind of thoughts here. But, but this is a, this is, this is something that I personally as, as we say, people have seen across many, many blockchains of the projects, but they're commercial and public sector. It is the, it is what I would call the main obstacle in making this work. Yeah. So it's an interesting ending slide. Thank you so much for the session. It's interesting to end on the Uber slide that you have, just because sometimes there is controversy around Uber and exactly as you said, dangerous getting in their car. But that said, I think you covered almost the entire planet that's possible in terms of the concepts on value. I want to open it up to the floor and just say, if you have questions or ideas, you know, please either raise your hand or unmute yourself and just go ahead and ask Alex what your thought is. Of course, I always have a million questions, but anybody else want to ask a question? I'll throw one thing out there, Alex, your example on the stadium, the exponential one. So my background actually is distributed data before blockchain. So I was engineering that all over the place. So distributed data real time was not new, right? Blockchain had nothing to do with that. What's interesting to your example, and I use this and I can see, sounds idiotic, if you want to predict the future, look to the past, because I'll argue many times we have what I call solution design patterns in the past that do work. They're just not applied on new technology or new context. And so the most common thing I think is stealing those and reapplying them to the current context. And the best example is your stadium example about exponential growth. So when I listen and I look at Hyperledger, I just got off a project where Hyperledger fabric, we were at the point where we're doing 200 transactions a second, which is pretty good in fabric. It's not meeting the exact benchmarks IBM publishes with their different output cluster sizes and so on at all. But that's a reasonably good benchmark. And if you compare 200 TPS of real transactions to what you can do in Ethernet or Bitcoin, it's millions of miles ahead.  And it never will be. It's just it is a different world. But I'm optimistic about the future. And the only reason I say that is going back five years ago, I led a project for Apple on creating a data warehouse with Data Lake, data services and all that. And in there, we were just using MySQL, right? Just a regular database. But tuning matters. Not even more the capacity. And so at the end of it, I wound up being able to do 3.3 billion transactions, writes, not reads writes in MySQL in 19 hours. And it's like I compare that to what we just did in fabric. And that's almost 200 times faster, not quite than what we are able to achieve in fabric today. And I look at it and say, Oh, what's the difference? The difference is we have yet to apply those engineering patterns on the blockchain. So when somebody says to me, it looks like, Oh, it's kind of slow. Well, yeah, because we haven't done much on the engineering end today, honestly. And so in that sense, I'll say the landscape is wide open to look at other applications that other people would run away from. Because there's not too many places where I can say a single server did 19 3.3 billion transactions in 19 hours. And if you cluster that up, you know, by 10 times, there's not a lot you couldn't handle. And so I'll say again, it's the engineering design stuff. There's and the biggest thing to your point on constraints is what people and I'll say specifically engineers bring to the table, which is assumptions about how it has to work, how it has to be done, and what can't be done.  All of a sudden, all those constraints. Oh, yeah, there's another way to do that that we could we don't have to apply today's technology or today's solution pattern. So it opens up the door endlessly to things that people assume, just based on current results were sort of locked into to some degree. So that's a big thing. One thing I'll ask you along that line on the engineering thing I laugh, the concept of, as you know, many blockchains use what I call a Merkel tree, right, as a way to organize data on a blockchain. And so one of the simple things I always laugh about is like, okay, that simple slogan that says you can't see the forest through the trees. And what's missing is the Merkel forest. And somehow nobody came up with the idea that Merkel forest isn't there today. And yet it is in all these other domains. And so I just laugh and say, well, somebody's actually going to say, Oh, yeah, let's write the Merkel forest so we can see all the trees efficiently. But that hasn't been done yet. So there's a lot of room in this technology space to revolutionize it performance wise by a factor of 100 or 200 and make it a lot more flexible to the applications that you would consider important. Absolutely, Jim, I think you hit this and I cannot tell you how many questions I get on a daily basis about the speed. And I'm telling you with the noble starts with the hyper lecture program was the 1.0. Now, now we have so many, so many more things with all of that. What I've seen is really 20 transactions. Second, and, you know, we're kind of was interesting like, well, that's much better look at the Bitcoin, you know, people going to Bitcoin conference waiting for hours to get transactions to get a ticket in. Right. And kind of when you start with when you start with that, like I tell you today, the seven thousand experiment team with the hyper ledger and others other quarter, we're getting in thousands and thousands of times, we're getting to, you know, credit card speed, we're getting to this, we're already there, if not, so it's not just mainstream yet. So that you were right, Jim, that exponential curve is there and it's going to hit us. We all sit in there and thinking, oh, it's just 10 transactional second. And listen, I'm first one to say you have 200 transactions a second. You and I did a lot of blockchain implementation and contracting. And you're like, do you have 200 contracts process in one second? Really? You do? Wow. But for you, that human problem, right? But for the most part, that doesn't matter when we start mattering is some of those digital marketing decisions. If you could do digital marketing on blockchain and remove fraud, you'll be sending tons of money to advertisers and everybody else, right? But that requires enormous speed. So even with those 200, Jim, our service feed is pretty cool. We're in thousands and even more than that. So we we can definitely get there and get better and better and better. People still think we're on six inches. Nothing's happening. We're going to move in nowhere. It's just not true. So one day just going to go over just like cloud day. I remember talking to my dear colleagues at the DHS when they're saying we're never going to use cloud ever. Ask them today. It's like almost all the department went to cloud. So it's the beauty of recognizing whether we're only on those 6.2 inches deep of water. And I think that we are. One other thing, Alex, I have a question from the live stream that had asked whether or not we can apply Hyperledger and your concept to DeFi projects on finance? Yes. Totally. I would say that some of the finance project that we worked on, some of the banks, we definitely apply that. So I think it's kind of all, it's all pretty universal concept. I think most people get it. It's intuitive. But just sometimes when you get into the delivery, you just limit stuff into something very narrow and you don't think about all the call of this work together. Okay. Let me stop and pause for a second. Just ask the audience. Do we have anybody else that has a question as well? Hey, Jim. It's Jim St. Clair. Yeah, I had my hand up for just a second. And I just wanted to comment because I thought your comments were excellent as were as Alex's. And it's good to see you again, Alex. Speaking on behalf of the Hyperledger community as a whole, Hyperledger Foundation, I think we still have tremendous challenges with education because we walk into these blockchain conversations and I'll just use LinkedIn as an example with so many conversations. And someone will be presenting a blockchain discussion point that I happen to know they're referring to a Hyperledger implementation, whatever flavor Hyperledger. And then somebody pipes up and says something about, yeah, but what about the gas fees or something? Just the misunderstanding of the differences between Bitcoin, Ethereum, and Hyperledger and fundamentally what Hyperledger as an enterprise application is capable of doing and no criticism intended. But even just within Hyperledger Foundation itself, as you and I have talked about, you show up in a fabric session and say, well, they're doing this over in Ursa or they're doing this over in Indy and people's minds are blown. And so to this last question, I was pleased to be part of the committee for the Hyperledger Global Forum that's coming up to help select what's being presented. And there is some tremendously exciting presentations lined up, I think, in Hyperledger Global Forum that to our conversation here show where some of these different permutations and new models for Hyperledger are going around decentralized finance, identity, unification between chain transactions in Ethereum to Hyperledger Bezu, other applications of NFTs. So I would encourage everyone to sign up for the Global Forum coming up in June because there's some cool stuff and also give you a good education, I think, on the breadth and depth to where Hyperledger is going. Yeah, excellent comments, Jim. For sure on where the Global Forum said it as well. And Jim, since you actually unmuted yourself, I'm going to pick on you for another thought and see what Alex thinks about it. But the concept, I'll call it, I'll say the traditional model for customer experience now that's so exciting is we always say human centered everything. And I say, ah, that's the old school. That's not the new school. So what I mean is in the old days, we had everything that was, in a sense, a lot of technicians are always be tech centered. And you're right, that's the wrong place to start is don't look at the technology, start with a problem and what you're trying to solve. And in the, and I'll say the old school thinking is, oh, already start with the user, you know, the customer, in a sense. And I think in many cases, you hit it, Alex, especially with AI and automation, you know, leave blockchain alone for a second, you add AI and automation in there. And the question is, who is the customer, right? Who is the user? And the answer is, I don't think the user is human anymore. So don't tell me it's a human centered thing. I'm working on a smart energy app. And in the energy app, we did something because people own it to Jim's point, actually brilliant point, Jim, about what people understand. People only understand what they already know. So when we want to communicate this energy peer to peer trading app, we said, okay, so the person that paid us to do it says, I see a trading platform. So we created a traditional trading platform that users would use. And so I actually talked to a bunch of real engineers in utility grids and so on.  There's no value to that activity at all. It's something we could 100% automate, right? So you would literally in the future vision using AI is say, no, let's do the trading directly. Let's create the marketplace, but let's eliminate the people we don't need. You could have a dashboard, but we don't need people to be looking at a dashboard and actually taking actions. All you have to do is say, boom, would you like to have an option to optimize your monthly utility bill? Click here. You know, there's no charge for it. Just click on your profile for your account. And if you elect that, we would do automated trading underneath for you and show you a potential, or at first potential gains, but actual real gains that are varying all over the place, but based on the variable demand for energy. So we can automate that trading for you without using people in the process. And I'll pick on you, Jim, because I think you're in the healthcare space. And the way healthcare is moving, it's like it's on me to figure out, do I have a problem? It's on me to reach out to a health care service provider. It's always on me to be smart enough, educated enough and everything else. And the place you live, it doesn't look like the future to me. It looks very different, I think, in your space. Yeah, the first thing I'd say is I know Alex and I, gosh, going back almost three years ago, talked about these cases in the public sector. And speaking as a former management consultant, my attitude was always, and I think this supports what you're saying, Jim, define the business case first. There are so many cases where people were trying to, and I have this conversation in healthcare all the time and people say, well, what's the use of blockchain and healthcare? And I'm like, well, rewind for a second. We're not just trying to throw blockchain and healthcare. What we're trying to do is improve a process where decentralization of the activities leads to either a unique capability or an improvement of things. And I think from a healthcare perspective, and I don't want to necessarily call out Adrian, but he is an expert in this as well, the concepts of privacy and user control of information relative to AI are very challenging things. And we have strong cultural opposition to the idea that I should decide first and foremost how my data is being used and how I can send to its use, especially in healthcare. And there are trade-offs there that haven't been decided where, do I really, should I really be in a position to keep my doctor from knowing something? Maybe yes, maybe no, but that trade-off should be something that's facilitated or discussed as part of the business case. And right now the default case is we just have all of your data and we decide what's going to be done with it. And even the changes under the 21st Country Cures Act basically says, oh, fine. So you want your information? There, use some APIs and you can get it all. And our corporate position on the issue is, well, no, you should be able to get access to your data because it's your right, but you should be able to control that in the context of how you manage your identity and the consent for that data at the same time. And there's lots of ways to do it, but I happen to be pretty religious from a former auditing and compliance standpoint that the hyperledger frameworks around Indian Aries provide the most transparent and enterprise deployable ways to do that. Can I do some of that using Microsoft Ion and Bitcoin? Sure, but there's public DIDs there that may create HIPAA challenges. Can I do it with Ethereum? Maybe, but quite frankly, if I walk into a major health center and start talking about using Ethereum, they're going to laugh me out the door. But there are fundamental principles of decentralization that support that business case for user-controlled identity that you look at first. That's my take on it. Yeah, and you're 100% right. And I think healthcare, there's many applications for blockchain technology and so on and AI, obviously, but because of the nature of the pandemic and everything else going on, I would say healthcare is in a lead role in driving a lot of that stuff for sure. All of the problems, the challenges, they all come up in your domain for sure. And what's interesting, I think your point is well taken about the Indie Aries approach to an open identity model based on self-sorting identity, the idea that I can control my identity, I decide who I'm going to share it with, and the beauty of having run a project for a state last year that went well using that, we did business licensing for a state and with individuals and so on. But the beauty was, the whole thing was concentrary, which is unbelievable. So we had this massive record of the fact that the state grants you a license and says, Jim, you're now an account. The thing is, you actually had to accept that license. The level of consent management and control was phenomenal. And then the history is there on the blockchain. It's all there. So the other issues that do come up to me in our domain and maybe in healthcare as well, most of those can be attacked with one of two things, either data masking or aggregation. So it's up to where the laws are going to shake out on this. But if you look at, I can control my own health data. I don't want to share my health data personally with anybody, right? But the point of it is, if you take my data and then just aggregate the data, the aggregated data probably has less resistance, I assume, to being shared because it's aggregated. It's not individual. And the only other thing that we did this in banking was we used to do data masking all the time. So here's Jim's real health profile. But now it turns into Donald Duck. So you have the actual individual records, but they're not, in a sense, something that can be tied back to me personally. I like those standards. I mean, I think, like open banking, open energy, all those standards, all those standards are there. And I like, and I like, this is this is a phenomenal idea. It's all logical. You should decide what to do with your data. Like, just because somebody's lazy and gets your data, and I'm like, I got it, I can do it. You just can't do many many of those things. And they're actually some pretty good. And I think the health research gem is going to be changed relatively soon. How do you run those experiments? How do you compute? I want to run experiments on my own. I mean, we're reaching this stage is why do you need a million researchers I can run to data should be democratized so that you can do that. You can check yourself. Obviously, the certain obviously, consent and all of that. And we will get to that how to get that and I agree with you more enterprise ready systems like hyper ledger of nicely positioned to do that makes perfect sense from many, many perspectives, but also your Mason, I think you're hitting on something very nice that typically tease people and say, Well, the bots are humans too, they have feelings too. So we should do the bot center design. So that teasing says, and that's another ways is a lot of time people forget there are a lot more users and factors that I interact in how humans interact with bots is also important. And that's what makes you X and CX way more complex than it is. Looks at the very first kind of classical edge, but I'm just in the room and you tell me what you want. I want to figure it out. It's way more complex. Absolutely. I'd give Adrian a chance to comment because I know he came off video and his expertise in this area. What we're doing in my data is a great, great example. Yeah, this is a very much a business oriented session. And I think given the experience in the government sector, the opportunity in healthcare is in the government sector, because they are the payers directly and indirectly, you know, they're paid for way more than half of the healthcare that's on. The healthcare in the US is wasting a trillion dollars a year. They are not interested in any of the business process improvements that you mentioned because transparency of the waste is the last thing that they want. So there's very little hope in the short run of selling any of the stuff you're talking about into the hospital provider sector or even to some extent into the private insurance sector. But on the other hand, so much of the business is controlled by government through Medicare, through Medicaid policies, if not directly, that they in effect can impose change and transformation because very few entities on the provider side can afford to ignore the Medicare rules. And they don't need a law to do that because they don't even need regulation. They just need policy. And I think in selling blockchain and digital transformation to the healthcare sector, that is the fastest way to go. But they do not go easily. Again, because of, you know, the way politics works and people leave their jobs in regulatory and regulation and go back to industry, but that's my opinion. Yeah, Adrian, that's a brilliant point about, I'll call it the health establishment, if you will, and not just the private side, but more the public side.  So I'm on Medicare, but if I die, it will have been Medicare that killed me, honestly, because of the administration of it is just it's the complexity of the current systems, the data inaccuracy, the data quality, all of those issues are phenomenal going through that system. And as a guy who went through that, the way it ties out today in healthcare, it actually starts with IRS, it then goes to Social Security, then it goes to Medicare, and then eventually from Medicare, it'll go out to the insurance providers to figure out what you're going to pay and how you get paid and how much you get charged for something. It's a long process. And having literally lived through what I call the horror show of data problems that aren't correctable on government systems in many cases. It took, in my case, 10 months and I don't know, 57 phone calls, I believe, and a tempted lawsuit just to get a $400 bill paid for something I was covered for, which is not unusual, I don't think. I don't think I'm the only person that goes through that, but it's a challenging system to your point. And the question I'll throw back is on the both you and Alex is, what is the strategy for change? If I look at something like healthcare providers, maybe there's more options, maybe I'm not locked into one, but with the government, there's only one to your point, only one system. And so back to your point, how do you drive In healthcare, it's easy. You just aim for transparency because you can't manage what you can't measure, just like we avoided measuring the impact of gun laws for decades. And now we find ourselves struggling to figure out what kind of policies might work, flag laws work, et cetera, et cetera. The healthcare will only be fixed through transparency. And if you can link your proposal to transparency and still sell it to whoever the customer is, you're golden. But that's the problem. And you're right. So I guess it comes down to, in our world, I'll call it publicity, probably, to publicize the problem. Much like people try to publicize the pandemic or climate change, it's going to come down to that. And I don't know how many people have direct experience with problems through a healthcare system. But I'll say you're right. We had in an identity 2020 act that never got implemented by the federal government. That got shot down. It was voted on, didn't get passed and never made it out of the house. But it was an important step forward in trying to get transparency. You're looking in the wrong place. You have to listen to what Alex was saying about digital transformation. You can't be looking at this from the identity. No, I wasn't. I was going to say, back to your point on transparency, and you're 100% right, is we don't have transparency. And I would agree with you that only 98% of probably all problems would be improved with transparency for sure. And so the question is, what do you see, as I call it a transparency act, if you will, that we could look at, define, and then actually, in a sense, vote on and implement, because that does make a lot of sense. It does. It does. I mean, I'm more, I mean, obviously, there's many aspects to it, but I mean, I was in disclosure, I mean, I work with HHS, and they were the ones that first implementers of the blockchain and government space, so much faster than many others. So, I mean, they started, they started that. And so I've seen government folks really thinking about it and public sector folks, and I would agree, probably, there's a lot more in our region and the government side, that you can kind of push that, have a ripple effect change through the public sector, through them, sorry, through the private sector, just the way, like you said, that I think it's a, I think it's a, it's where it comes down to, and what I've seen most of it, like, you know, coming back to CLAW, or anything like that is, when you speak with all those folks, I spoke with a lot of them, is that really comes down to making this a very straightforward argument of what is happening and not doing big banks. I think a lot of folks come and talk to, I don't know folks about doing big banks, oh, we're going to just go in there, we're changing everything over and over again. Also, it doesn't work that way. I mean, shock how much of that is happening, instead of really, really rolling up your sleeves and figuring out how do we get that transparency and how do you sort of hear it, start there, and, and doing it in a, in a, in a very direct, like today, even like, if you look at it, the Department of Homeland Security today, they're experimenting in making green cards on a blockchain. So it's moving on. It's moving on every, every front. That's the last, last place I would think they would, they would go with the blockchain, but it's not. So things are, things are moving, moving well across the board, which just have to kind of get more tighter with this, with this, with this, with this use case, and with the, with the transformation, how this works. I think we're going to get there. We're going to get there. I'm optimistic, despite, despite all of that. I mean, I, I'll never forget, as I said, I'll repeat again, my cloud conversation with very high level government people said, never going to happen here.  I'm never going to put my data somewhere in some, just who knows who controls it, right? Today is like, you have a good day in your government center. Now you're sure it's safe. So it's, I kind of work top of this. So, so you've got to be careful about that, really exponential. We had those six inches right now. So we're going to see a lot more change, a lot of going faster. So, you know, eat popcorn and watch it or be part of it. I think it's, it's, it's really what's up. Yeah. Yeah, I did want to say thanks to Adrian though, the point on transparency goes across the board for literally everything. And without it, it's very hard to drive change anywhere. So that's kind of like a core building block for trust is it begins with transparency, I guess. We have another comment. I want to point out from Jayakar in the chat, talking about a portable, unique digital medical record in hyperledger framework is a basic facility to control, I guess, COVID as accurate data is imperative for any new research and development, which is true. And that's a great idea. You know, the idea, it does work. I'll say even in the identity space, you know, we implemented what was available in wallet standards and so on. And yet those are moving verifiable credentials. W3C has one now. So it's great when we have in a sense of solution like this portable medical record and anything that becomes a standard is a huge win for us. One other thing I want to point out, Daniella has mentioned a couple of things. Number one, if you're looking at the global forum and I'll make sure I publish it on our website with a meeting reporting, there's a discount available, a discount code you can use to register for hyperledger global forum. And then also, there's a... And the ticket's only $50. And if someone's not, doesn't qualify as a non-profit or a government, there's some scholarships or just go ahead and reach out to myself, Jim. Yeah, can you cover the other point about the data? Sure. When we were just talking about data, there's a project going on right now at the World Economic Forum. It's called the Data for Common Purpose Initiative and hyperledger is participating in that work, primarily around the interoperability chapter. Maybe I didn't know if they had come over and kind of talked to you about that initiative. It might be something good to do a presentation on in this segment. Yeah, no, that's a great idea. And actually, you're right. I'm part of the blockchain forum over there and there's certain topics they cover, but that certainly isn't part of what we've covered over there. And then there also is the EC now has a new blockchain proposal out, which I think is also worthy of covering as well.  And then I'd say certainly your Data Common Purpose Initiative would be pretty cool to get a session here on that. That's excellent. Okay, I do need to run as well. Okay. All right. Well, I want to thank everybody today. I know we're a little bit over for the session, but I do have the session recording. I'm going to post that and Alex, if I can ask you if I can get a PDF of your slide presentation, I'll post that as well. And so it's been a great session.


Okay, let's start. So let me introduce my colleague Vladimir Kuklin. He is a principal deployment engineer and tech and fuel technical lead at Mirantis. And let me introduce Sergey Galvatyuk, our senior deployment engineer at Mirantis. And today I would like to tell you how we fought for open stack high availability. Apparently this battle is still going on, but we have some good results. So I would like to share the details how we achieved this while working on Mirantis open stack and fuel projects. Yes, all right guys. Some technical issues. Let me switch them. So it's not rotating. Let me once again. Okay guys. Sorry some technical issues. Oh, yeah, so. That was great. Okay, so before we start talking about open stack and its components individually let's define high availability and what kind of high availability this talk is related to. So high availability as you all you know is a characteristic of a system to attain high availability of service even if it is exposed to failures. So in this talk we are covering only a single failure of a component at a single point in time. We don't cover a false majority of events or uncommon physical destruction, not deterioration of hardware. Okay, here is a classical diagram of open stack. So what's the main problem of this? So the main problem that you should have high availability for all components. So in reality you should have at least two copies of each component. However, I would suggest to have at least three copies to avoid split-brain scenarios that may arise. So all components should have only healing mechanism or can be under arbitrary control such as pacemaker or zookeeper. So we are not covering no data center high availability in this talk. We're talking only about open stack. So what do we need to provide a chip for open stack? So first of all we need to provide high-available networking connectivity. In our project we have support for almost all types of bondings including Linux and open the switch. So for database we chose MySQL with Galera and we achieved really good results optimizing management of it by use of pacemaker. So MQP is also a crucial component of open stack architecture. So the death of one controller should not affect service availability at any time. So in order to achieve this we had to alter significantly deployment workflow of Rebit MQ along with open stack messaging code. And also we had some issues with memcached. It didn't work with no death of particular memcached instances, so we had to do some stuff about it. For storage availability one can choose, you know, proprietary solutions like EMC on ad-app or fault-tolerance software defined storage. We concentrated our efforts on sef and it's working pretty well. Also API services should be a lot balanced and they should redirect requests to alive nodes when something goes wrong and for Neutron you need to elaborate each solution that migrates all the entities from dead agents to alive agents and also for heat and cilometer you also need to ensure that agents can fail over safely from one controller to another controller. So Galera. Why Galera? Sometimes I hear from people that Galera is too complex and DRBD is an easy solution that is enough for my reliability. However, you cannot scale up really well with DRBD as you have only two instances. So what about master slave replication? Master slave replication is really good, well-known technology and there is no big difference between Galera and master slave replication in terms of open stack reliability. So here you could, so going back, so here you can see a classical diagram we have in fuel. So all services communicate with a database via HAProxy and HAProxy high availability is based on virtual IP which is moving from one to another HAProxy in case of problems. So however you can see that all communications go to one HAProxy. As you know, as was stated yesterday by Peter Boris from Percona and J-Pypes from Mirantis that open stack services have some problems as they use select for update statements and don't have a retry mechanism for failed transactions. So MySQL improvements, so we focused on MySQL and upgraded MySQL to 5.6.11. So that helped us to resolve many stability issues we have. Also we use the latest Galera module. So for Snapshot state transfer we moved from MySQL to we moved to extra backup and we got really good results as we we can do really large database synchronization during Snapshot state transfer or incremental transfer. For HAProxy we added simple checks via XNED to perform simple checks against database to determine where we have a database sync or donna or the sync state. So in case when we have donna or the sync state we HAProxy is not communicating to that server. So also we added MySQL Galera to base making control. However, our previous OCF script was not mature enough and had some technical issues and didn't reassemble cluster in many conditions. So before we wrote a completely new script that helped us to resolve many issues and doesn't require any human interactions to assemble Galera cluster. So how it works. So the general idea is to get the most recent data from all components and MySQL we get this data from from global transaction ID of each node and we save this data in base maker cluster information base. However, in case of problems we can get this data from grass state file and when we have all this data we can start primary component really easily with the most recent data. So however we also added OCF script as clone base so we don't need to create any primitives on new controllers. Next slide. It's Revit. Okay, so when we started working on Revit MQ cluster resilience we found that Revit MQ cluster behavior is not very obvious and clear and for instance when you restart Revit MQ do how to set and it just becomes very vulnerable to race conditions and there is not much things you can do about it. So what we did we wrote an OCF script that it is really complex and has really no complex logic and that utilizes base maker master slave resources and notification mechanism. So on this diagram you can see what it is doing. So when we do start of the servers we just start beam virtual machine. We do not do anything. Then when base maker relax master node for instance let it be node one we put this information to base maker database and start Revit application server. Then base maker sends notifications to all the other nodes and they join master node and also start the application. And our status function in this script is always checking beam Revit and whether all the nodes are clustered with the particular master. And if something goes wrong for example you cannot start application server for Revit MQ because of these race conditions and you know this application trying to connect nonexistent stuff we just reset it completely. Then speaking of Oslo messaging, initially we had HAProxy server requests for Revit MQ but it was not scalable solution. So we were very happy when Oslo messaging was merged and we could use several hosts in the Revit host parameter but the problem was that it was not working with connection failures. There was no support for hardware for MQP and connections were hanging for no, for a long time and nothing was happening. Consumers were not receiving messages, producers were not publishing messages. So we took first community heartbeat implementation and tried to use it. It was broken. We tried to use short kernel TCP keeper lives to kill connections in a minute after they became become idle. This also didn't help but the good news is that our guys finally elaborated the fix. It is already in RANDIS OpenStack and it is already on review on OpenStack.org. So if you think that you need Oslo messaging with heartbeat support please vote for it. So there is nothing interesting to say about high availability of OpenStack services. We use pretty standard way with active, active backend of HA proxy. So with active, active backend some of parameters are tuned to digest production load and we have pretty good results. However, we put some services under pacemaking control such as heat up, salometer and we moved HA proxy to separate name space using standard mechanism with VATH plus RP proxy to eliminate hand connections. So Keystone. We strongly believe that temporary data should be kept in key value storage. So and previously we switched Keystone to use memcached instead of MyScale and high availability of that solution was not really good enough. So and we decided to improve that solution. So we used standard Python memcached library which is broken as it which is broken and we tried to switch to Python LibemC library which is really good. However, it's written in C and it's not even let's say. So we decided to write a new driver and our developer, you wrote a new driver with pull of connection to memcached server with some logic to detect that servers. But however, we still have some problems with memcached as logic for choosing memcached server for sharding is still broken and we have fixed for it which is on review control. Okay. Speaking of Neutron. Well, there is nothing interesting about Neutron server which is under HAProxy. But the divorce in details when you try to manage Neutron agents safely. First of all, they leave a lot of artifacts like IP addresses, namespaces, running processes and so on. So now save scripts for Neutron agents. We needed to clean up all the stuff when we do start and stop actions. The second thing that you need to do is to reschedule entities like routers for three agents and Neutron networks for DHCP agents and well, eventually services for load balancers service which doesn't contain any opportunity to reschedule services right now. So our guys are working, so we were using API of Neutron because there are API methods to do this right now save script. But our guys are working on API handler and rescheduling mechanisms for Neutron. Currently there is some kind of automatic rescheduling code already but we tested it and actually it had problems with MQP failover. So in case of, it was not working in case of failover and this is really, no. It's a pity. So SAF, however in fuel we use pretty common way with SAF with monitor on controllers and we have OSD for each disk on SAF nodes. So however our developers worked hard to implement live migration and we added some code to open stack to have live migration. However we also added some code for object and image based storage sharing. So and right now SAF works pretty fine and all code is merged to open stack. So and now it comes to deployment. When you want to deploy all the aforementioned stuff, you know, in the cluster, it is not really easy to do. So we had a lot of things to write for our deployment workflow and these were, we need to, these were crossing place micro manifests originally created by puppet labs but we needed to fork it and add other stuff to there. So there is a chip proxy with configuration directory support to make it work in a granular manner along with manifest for puppet open stack modules and also custom OSD scripts for virtual IPs and the HA proxy to run them in a separate namespace to avoid problems that were mentioned earlier. So there are three slides about deployment. So what we needed to do, we wanted constrained support for puppet because we needed them. In order to run master slave and clone resources, we needed to write code to support them. And in order not to modify all the manifest, all the puppet manifest for open stack deployment, we also needed to provide service provider for pacemaker for puppet. So what it does right now, it parses local resource manager of pacemaker database, waits for status change, respects timeouts and monitor comments sent by pacemaker. And what is really good that in order to support it, you just need to change service provider and, you know, add some two or three stances to make vanilla puppet manifest work with open stack HA. And also in order to support really complicated OSD scripts like for rabbit MQN Galera, we needed to switch from default upstream shadow approach when you generate a copy of database of pacemaker configuration and then apply it again because it was overwriting attributes that are used by OSD scripts. So we moved to pacemaker XML patches and it is working pretty well. So as soon as we just got it deployable, we moved to stability. The problem was that our service provider was triggering actions globally.  So we switched to pacemaker asymmetric cluster where everything is stopped by default. And when we do start of services we enable them by putting zero location constraint. Then in order to make all these actions finally local-wise, we also modify status method of provider to make it dependent on resource type. For clones we check if it is started locally and for primitive resources we check if it is started somewhere in the cluster. Also we had to tune SQL driver timeouts and all the other stuff. And also we needed to introduce short kill nodes to keep alive because there are still, we need to kill hanging connections as fast as possible, not after two hours. So speaking of HA scalability, after we won this stability fight we moved to another one. We wanted to easily scale our cluster with controllers. And this is what we faced. Most of our customers do not have multicast configures in their data centers. I don't know why it is 2014 but it is fact. And we had to switch to unicast by default. And because of this we had to restart core sync each time. We want to add new controller. And in order to do this we altered core sync manifest to put pacemaker into maintenance mode because this is the only way to do this. In zero version of pacemaker plugin. And also we had to add, we had to limit parallel controllers deployment because not to exhaust Galera, Galera donor nodes. So testing. Testing doesn't come at easy cost. So we implemented a special library which is tiny piece of libvirt based orchestration. So and we perform our destructive tests on daily basis using that library.  And we start performing destructive tests. And it's integrated with our CI and we have results once we have any problems with HA. So we wrote special own framework which is fuel OSTF. However we also run tempest and rally. Okay so speaking of results what we currently supporting from HA's point of view. We handle single failure at a time. Either it is controller reset. Also if you reset all the controllers they reassemble automatically. It may be network partitioning, it may be failure of MQP, DB, database node or particular service. So all these test cases, all these use cases are covered. So our Sergey, this is your part. Okay so current plans and challenges. So we need to implement multiple rights for Galera. So as you know many open stack services still use select for update or doesn't have retry on failed transaction. So we are working with our developers to push this code to upstream to allow multiple rights to all Galera nodes instead of one. Okay and the same goes for European developing to test them. Because we are not really sure that they work completely fine right now. We are also going to introduce support for fencing for pacemaker using all the types of fencing devices. Also we wanted to implement centralized event driven fellow render vocations using common monitoring systems. Memcache still needs some polishing. We also want to concentrate on research of using 0MQ instead of MQP or maybe combination of MQP and 0MQ for messaging. And also as we have already working HA tests framework that can test your installation completely. We are going to, as soon as we, I think it will be really soon, as soon as we provide ability to deploy multi-node HA environments from particular OpenStack comments I think it will come in less than a month. We are going to introduce our CI gate into OpenStack Jenkins to see if any particular commit is breaking HA installation. So if you are interested you can always check out our project, our project's VQ page. Actual deployment is done by few library subprojects and we periodically write down all HA fixes into other part documents so you can check it out and find whether there is something useful for you. And you can always contact us at rcfieldfchannel at OpenStack mailing please. And finally I would like to thank all of the guys who are working hard to make this HA story work. This is Alexander Dedenko, Bakdan Dabrila, Roman Padalyak, Yurit Raday, Serge Millikan, Stan Lagun, Sergei Vasidenko, Matthew Mozesan, Dmitri Lin, Dmitri Baradayenko, Ryan Andrew Woodward and Cyr Lapova. I don't see here Tatiana Leantovich, Igor Katko, Artem Pachynko and Andriy Stazinski. And all of you for your attention. Thank you. Thank you. So if you have questions just ask with just... We have 14 minutes. Raise your hand. Could you please give the microphone? I think this session is recorded. Yes. Right. First of all, guys, Mirantis fuel is awesome. And good job, guys. Thank you. My question is around the roles that you provide, especially your controller nodes, all your API services, database and network, everything run on the controller nodes. Have you got any plans to split that controller role? The answer is yes. We are going to... It's our deployment architecture limitation right now, but I think in half a year we will be able to deploy any particular configuration and combination of roles you want on any node. Thank you. I've got another question. This is on your logging. All the logs go on to your fuel master. Have you got any plans to split that logging and send somewhere else? Actually, in FuelMaster we have option to send all logs to remote server. So we can use it even right now. But yes, we are going to redesign our log architecture and guys from FuelLibrary are working on that. Yes, sir. You mentioned about self-life migration. So how did the guy manage to do that? I think the best person who can explain that is Andrew Woodward. He actually worked on this part. I'll catch you later. You can tell it right now if you want. So you mentioned something about the pacemaker catalog and some way how you can track what services and in which order they start using the pacemaker. So you read information from the pacemaker database somehow to just track this. You mean about pacemaker puppet service provider? Yes. We are just parsing XML from pacemaker and parsing local resource manager part. And no. Eventually, well, it is completely based on the code that there is in P engines that shows you all the information about how in which state services are and CRM and PCS are showing this information. So it is rewritten in Ruby and it just does the same stuff. So you basically to start the services in the certain order you parse the pacemaker XML file. We parse it to derive its properties and status. Yeah. In order to start, we just put pacemaker cluster into asymmetric mode and then put zero location constraint. But if it is not started anywhere and then we unbend the node and then pacemaker starts the service and we wait for it to start or to stop whatever you want to do and see whether it is no. It is just like a regular service.  All right. Thanks. However, there are still some tricky logic as you cannot notify pacemaker just to load one service as it will reload in all nodes. So we just put constraints or just bend one node reload service and then we add it back to cluster. So just to eliminate such issues across all controllers. All right. Okay. Thanks. More questions? We still have like five minutes for questions. I think Fabio is surprised. Yeah. Actually, yes. We have plans and we are discussing with PuppetLab community to add to PuppetLab. Yes, we will be adding so but on you can actually use our OCAF script even right now. It's in our code. It's open source. Yes. The answer is yes. We are trying to be as open source as possible. Sergey, it is awesome. It just went slipping. Okay. Thank you guys.


Everybody forced here on release day with origins football here 2022 origins eight box half case number two. Give me one sec Just completing those let's read off our names and teams here Adam P Broncos Albert E Cowboys Austin M Seahawks Charles with the Dolphins Cody Ravens Kurt S Cardinals Darrell with the Colts Edwin S Giants Hamilton has the Bears James H Chargers Jeff and Texans Jerry B Washington Jim J Titans Jonathan C Steelers Carl T chiefs Kyle B Jets Matt M Lions Matt S Eagles Matthew J Raiders Maximilian Patriots Nicholas you Packers Well, good luck. Nick. That's one of the last teams taken there Paul G. Browns Philip and Niners Ruben Y With the Saints Ronald I falcons Scott T. Buccaneers Sean M with the Panthers also one of the last teams taken Thank you Sean. Good luck, buddy Shannon D. Bill Steven W. Rams Tony K Vikings Trey H Bengals and Vaughn with the Jags. Good luck everybody here. We go Mm-hmm, that's his team hits that I didn't know he was a that does that in the past That's kind of me too. That's like I'm a pass and then I say Jags in my second I'm cool. I'm cool with the Bucks and the Jags the Bucks. Yeah, Bucks are I wish Tom Brady didn't go there I just wish the Patriots were just paid I'll say that Good had another run in it, but I don't know. It's gonna have his career now All right, Buccaneers Chris Godwin first card number six seven ninety nine And Buccaneers Chris Godwin 150 to a two ninety nine Go Bucks Saints rookie Chris Olave You go Saints and blank slate case hit Mack Jones Patriots Love that Congrats Patriots That was a maximum million with the pads blank slate is a case hit heck yeah, one of the newer case hits this year, but they're from a Like the court Kings realm the Diamond King and all the all those kings breaks Nice Mack blank slate, baby. We'll get that in a mag Mac is back y'all Desmond Ritter rookie Jersey auto Falcons. There's a quarterback. That's the first quarterback Auto of the day. I think we've had Go Falcons ship worldwide Yeah, free shipping worldwide Patriots rookie on card auto Taekwondo Thornton great start for the Pats And great start for the Falcons here a rookie Jersey auto Drake London as well wow Ritter and London rookie Jersey autos in the same box as the blank slate. What the heck Man Falcons Ronald I congrats on that. I Said we were due for a quarterback. We got a Ritter. We also got the case hit hang man I just did the pickings picking stuff probably the hottest as a receiver right now All that stealer love It's good start to the break And we still have the young card too So we just ship skipped the relic went straight to auto relic on both of those Seahawks Tyler lock at 54 of 75 and a chief Travis Kelsey 22 of 199 Bears Justin Fields and rookie Calvin Austin the third Steelers He's got some playing time come when he gets healthy now that Claypool is gone Seahawks Jersey card hometown roots DK Metcalf 34 of 99 I would think I don't know who else would be Once he's healthy, I'd imagine he's gonna get the opportunity now Oxford, Mississippi there Jed sauce Gardner rookie on card auto On card sauce I would go nice with the cookbook I Don't we not think of that earlier all right eight bucks half case number two. I don't know Eight bucks half case number two Browns rookie patch auto David Bell 42 of 49 On the blue parallel, yeah the cookbook from case one needs to go with the sauce Get a ridiculous You know what what else would be in the cookbook you need it for cooking as the bowl bowl Did I love bowl bowl I call him Super Bowl Super Bowl, baby Got Falcons, Corderole Patterson, please come back soon Patterson for my fantasy team What on out of 299 he's good man. He's awesome. He's good Jags James Robinson 21 of 25 Giants Kateria's Tony and a rookie Krishna Watson Packers Saints Hatch card drew breeze hometown roots 28 of 49 from Austin, Texas Breeze easy breezy Titans Traylon Berks red rookie on card auto 74 of 99.  Let's get these let's get these filled up guys Number three Panthers Browns Broncos And again guys we do have these in stock for personals as well Personals will be after group breaks tonight. It's weird tonight. We haven't met 299 99 300 a box On average you're supposed to get two autos one of them is the on-card one of them is the jersey auto And in a memorabilia card two base parallels as well That is what you should get in every box It's a great way to break it down a product. Mm-hmm. I love I love the the configuration on this Got Colts Shaquille Leonard. Nice. He got Shaquille name on this one 130 of 299. There you go Colts and ETN Jag 78 of 199 Base cards bears feel we have a lot of fields base for some reason and Matt Ryan Colts Chiefs Clyde Edwards a lair patch card 49 of 49 the chiefs Falcons having an awesome break Desmond Ritter rookie on card auto Actually, that was a blue parallel Burks. I thought I see this because it has the red there. So I think that was a blue parallel Burks, right? Oh Burks You think that's you think that's red Okay, that is the right parallel because it has the blue. Okay. Yeah, I've seen what the team color was I forgot which one was which there you go Falcons again cardinals rookie jersey auto Trey McBride Dang, this is actually fine. Yeah, that's that's that's a nice one. I read her on card is nice And yeah, they're also guys we're looking for some crazy dual autos are pretty rare They're much rarer than a case hit like dual autos and the crazy books You can also hit some some pretty big bets But the bets are a lot rarer and this product you're looking for those on-card rookie autos But occasionally you get some pretty big bet hits to Chargers Keenan Allen 20 of 99 Niners Eli Mitchell 272 of 299 Got Broncos Devonte Williams and a rookie Greg Dulcich Broncos Dude Matt started him against me in fantasy and he had like 12 points.  It was like his flex Gold rookie patch Panthers Matt Corral last team mojo 3 out of 10 That's nice Corral a golden Corral if you will We are sue super super corny today you guys Golden Corral Three of ten Panthers on the patch card Titans Hassan Haskins rookie on card We're way too hungry to be breaking right now Once that once that Golden Corral check clears, I'll be pretty happy you guys once I get that big old payday from Golden Corral Cowboys Jalen Tolbert rookie Jersey Auto Food is pretty crazy today, but yeah Poinsford I'm sure for no dad jokes. Nope. We don't have that that doesn't exist Jalen Tolbert Cowboys Family-friendly channel. I mean dad jokes are kind of the thing, right? Oh, I was only saying it was a Golden Corral card. I'm not I'm not giving anything. That's not you know thrown to me. Okay guys I'm not catching any minutes out of the way. Yeah, I'm not like making a big stretch here Got browns to Sean Watson 43 of 199. We're just taking what's given to us. Honestly hackers Aaron Rodgers 114 of 299 Jared cooking right now. Yeah, they said let let forest cook so they said and I say I said LSE country. Let's ride Bears Dave Montgomery Eagles Carson strong rookie Got giants rookie patch Wanda Robinson 172 of 199 Lions Jameson Williams rookie on card auto. I'm excited to see him come back Now come come back come back to play football for the first time as an NFL player and bears again a Belish Jones, Jr. Rookie Jersey Auto First had Golden Crow in Florida is Golden Crow even good anymore. I haven't been never when I was a kid It was it was fine the best thing ever when I was a kid Yeah, like my grandma would take us there for like like when it was all the grandkids She would take us all there because I was like she was pretty affordable for you know a family dinner I remember like all the my parents like on someone was like nope keep the kids from the dessert table Chief Travis Kelsey 287 at 299 Ice cream It's terrible. Yeah, I figured most buffet places are pretty to keep it afford to keep it affordable now Like it's very tough to do Josh Jacobs Raiders 45 out of 199 Like buffets you you have to pay a good amount for nowadays from the week for them to be good to be fair Just not blaming anyone prices are just higher on everything Jonathan Taylor Colts and Matt Corral Panthers Rookie, that's the biggest variation just straight up big letters right there.  I got a I'm gonna get with will who does it for us and find a way Because the stream deck works like I can hit, you know like we Like I want I want to get the the voice mod back so I can do the auto tune again. I miss that I miss that Saints rookie auto on card Chris Olave. There we go Saints congrats on that one Origins rank for football products It really depends for an on-card rookie auto like no patch. It's it's pretty high up there Yeah, like I think of like just on card rookie auto like no frills You think of like contenders contenders optic and like origins are like the three that I that I think of personally and Skymore rookie Jersey auto chiefs. There you go chiefs another sky more Yeah, a lot of the players will have zero zero on here because they at the time that the product goes into production Especially for an on-card auto those are out a lot longer before so like you'll see Jersey numbers on some of the sticker Autos they had to make the card for him to sign They know a number so like Matt Jones didn't get a number until like preseason last year So he had double zero on a lot of cards. He was like number 50 and like all preseason He didn't get number 10 until like Until like preseason started. So yeah, you'll see a lot of double zeros because Again on card. They probably signed these at the rookie premiere. They probably signed these So early two months after the draft really realistically Got Marquis Brown Cardinals to 20 of 299 Rogers Packers 72 of 99 Colts Matt Ryan rookie Drake London variation. There you go Falcons Nice one nice one Got an Eric Dickerson rise to the Hall patch card Rams 24 25 Player one on the Dickerson Origins on card is awesome Chargers Isaiah Spiller rookie on card auto. I can make a food pun out of that.  Yeah, that one's too much. It doesn't it? Yeah, I'm not gonna do it. We're above that here in LSE. I don't go for the low-hanging fruit jokes Only high-quality humor here All right guys, we'll do a recap in a second The next half case number three you're looking at Panthers and Broncos Olavi has a course it. Yeah Yeah, luminance in year one to an extent The thing about Luminance is they're not in their rookie uniforms. They're in collegiate uniforms in rookie Unlike their what are they called the year ones and then the pen pals is technically their first on-card Auto in their uniform, but it's just like they're not like doing an action So this is like a step above those essentially these are sick though. So we got our relics here Rams Eric Dickerson, I'm just gonna call out team names and players. That's my guys numbers Steelers Najee Harris Giants Wanda Robinson Matt Corral Panthers gold to 10 Chiefs Clavards a lair Saints Drew Brees and Seahawks DK Mack have a lot more vets relics on this one Autos now autos in rookie Jersey autos we had charters Isaiah Spiller Patriots type on Thornton a Chiefs Skymore Saints Crystal Ave. I sit there Bears Vylus Jones Jr. Lions Jamison Williams some nice on cards here Jaylen Tolbert a Cowboys Titans a son Haskins Cardinals Trey McBride a Balkans Desmond Ritter on card Titans trailing Berkston 99 the cool color match on the red parallel there We had Vylus Jones Jr. Bears RPA to 75 with a sweet orange color match We've got David Bell Browns to 49 sauce Gardner Jets on card rookie Jersey auto Falcons Calvin Ridley Drake London for the Falcons Jersey auto. They did very well Patriots type on Thornton on card and Patriots blank slate case hit Mack Jones There were a few teams that had nice hits to me that was this is good stuff man It's good stuff half case number three coming up guys. Thank you