Hello, everybody. Welcome to Best Practices for GitHub's repository. It's a very popular question. Thanks. So my name is Costis. I'm working remotely from Greece for CodeFresh. I'm a developer advocate. I'm also an Argo maintainer, so I contribute to Argo documentation, mainly for Argo Loads and Argo CD. If you don't know anything about CodeFresh, it's the enterprise platform for Argo. We have a booth downstairs, so you can ask us more questions. And if you have ever taken the first GitHub certification, I was one of the co-authors, so if you have any feedback, you should tell me about it. If you haven't, go to learning.codefresh.io and start it. It's free at this point in time.  And one of the most common questions you have on day two is how you organize your GitHub repositories. I've been monitoring, you know, the Slack channels, the GitHub discussions, issues, and everybody's asking the same thing. How do I organize my GitHub repositories, or how do I make promotions? So if I have, excuse me, the three environments, QA, staging production, how I make promotions from one to the next. So this was a very popular question, and I was, you know, tired of understanding the same thing all the time. And I said, let's write some articles to solve this issue once and for all, and this presentation is a summary of those articles. It's only 10 minutes, so if you need more details, you need to read the articles. So when I talk with people, obviously they have many choices on how to do this, and it's not only, you know, the three popular ways like branches, folders, or repositories, but also the combinations. So a lot of companies, they say, oh, I'm going to use branches and repositories. So folders and repos. The combinations are endless, and people don't know what to select. So I'm going to give you, like, a starting point, and I'm going to start by what not to do. And then I will say what you should do. So don't use branches for GitHub's environments. And just to make things more clear, don't use long-running branches for GitHub's environments. It's okay if you have, you know, short branches, so maybe you want to have a feature. You create a short branch. You test your feature. You create a full request. And then you merge it. That's fine. I'm not against it. What I'm talking about is this scenario. You have long-running branches that always exist, and maybe they have the same names as the environments. So let's say staging, staging, QA, QA, and then production, production, or main. And the way that you do promotions is you say, okay, I have some changes in QA. I'm going to do a simple git merge and go to the next environment. Another git merge and go to the next environment. This is the pattern that I'm talking you should not adopt. Now, when I ask people why they start using this environment, like why they selected it, the answer I expect is we had an evaluation. We tried everything else, and this was the best choice. But almost always, the answer I get is, ah, we were using GitFlow. We knew about GitFlow, and we did the same thing. So this is wrong. If you go today to the page that explains GitFlow, there is a huge warning message from the creator of GitFlow that says you should not use this. So don't, you know, believe me. Believe the creator of GitFlow. Go and read it. The other problem is that people are getting confused and they say, okay, I'm using, let's say, GitFlow for my source code. I'm going to use the same thing for my manifest. This is not true. One of the first best practices for Argo is that you should split your repositories. So you have one repository for source code, one repository for manifest. They are completely independent. So even if your developers still want to use GitFlow for some reason, they can do whatever they want on their own repository. But on your repository, you should use trunk-based development and folders, as you will see. You should not confuse those two, let's say, practices. Each repository should do something different. But the most important point, if you ask me, like, why you should do this, is that in theory, yes, in simple cases, if you do a Gitmerge, you get all the changes, let's say, from QA to production. But there are corner cases, and depending on your company, maybe it's not a corner case. It's a usual case where you don't want to bring everything. So here I have an example where in staging there is a number of changes. And somebody says, usually, business, we want you to take this change and this change and that change, but not that one. So doing a Gitmerge will not work. So then you say, OK, what I'm doing, I need to start learning about cherry pick and start picking the commits I need and it gets complex quickly. So don't do this because it's getting super complex. So what do you do? Now I can say, do the same thing as Adobe. That's the correct answer or into it. Yes. Thank you, thank you. Use folders. Yeah, use folders. Do use folders. So what I'm saying is that each environment should be a folder in a single Git repository. There is only one branch. The example on the left is one of my simple examples that they have in the articles. The one on the right is the one from last year's presentation not going to come from into it. So this is how into it organizes their environments. And when I update my presentation, I will also include Adobe and say Adobe also does this. So that's the recommended way everything is a folder. And all the environments and folder, you only have one branch and then things become really simple. You can just copy files around. You can select everything that you, anything that you want. So let's say you're using Helm. A good practice would be to have multiple value files where let's say one value file is the container image. Another value file is settings that you want to promote. Another value file is settings that you don't want to promote. So during the promote, you can decide what you will take. The order of commits does not matter anymore. You are not only copying files around. And if you have five environments, you have one branch. If you have 20 environments, you have one branch. If you have 15 environments, you still have one branch. The complexity is always the same as far as branches as concerned. So if you have this corner case a lot where you need to know to promote stuff out of order, this is very easy to do when you use folders. So start using folders. Another important point is that if you look at the existing ecosystem of Kubernetes, these folders are very easy to support. So if you're using customize, customize says okay, I will give you overlays for different environments and they are folders. If you're using Helm, Helm says I'm going to give you values and you put values in folders and it works just fine. Argos CD has the capability to use folders so you can have a mono repo with multiple applications and it's very easy to say to Argos CD, this folder is this application, that folder is that environment without any issues. And application sets, if you're familiar with application sets, there is a whole generator, the git generator where you can have a folder pattern and say my applications are under this folder. So if you're using folders, the tools are helping you. You're not going against your tools. While if you're using branches, customize has no concept of you know that a branch is an environment. It says that an overlay is an environment. So these are the links to the two articles I talked about. So if you want to know more, you can read them. There is also a huge discussion in GitHub that one linked which contains my opinion and the opinions of other people. It's very long to read but if you know you want to do research, you should read it and let's say all the possible combinations. And if you want to play with this pattern, you don't need to set up anything. The certification that I talked about, there is a level two and one of the lessons is exactly that. How you promote with three different environments where you use folders. So you just launch it and in your browser you get an Argo CD instance, a Git repository with three folders and even a GitHub action pipeline instead of Argo workflows that copies file from one another. So if you want to see a quick demo of how this works, the easiest way is to go there. And the last one is the presentation from Intuit from last year that had the picture I talked about. I will also include the Adobe one in the next presentation. So that's it. Thank you very much. We are right on time so I don't think we have time for questions so find me afterwards and ask me anything or tell me that you are wrong and I will still use branches.


Welcome to my presentation on Dapper, which we used at iHomer for a product that we developed for customer of ours. Yeah, thank you for having me. Yeah, I'm Michiel. And again, I'm passionate about Microsoft since 2004. I'm 43 years old, working in .NET Core, .NET framework before that since 2004. Worked as a developer architect for multiple companies, larger companies within Netherlands. And I was mainly responsible for a couple of big companies that for their public facing websites, or some insurance companies and an energy company. I love tech. I not only programming, also other stuff like taking things apart, working on servers, working on cars, whatever. I'm very busy with my family life. I have two beautiful daughters, Kate and Olivia.  And I'm passionate about helping students with getting into an academy program that we have at iHomer. And before iHomer, iHomer was, I was a teacher at a bachelor's university, or an higher school in the Netherlands, in Breida. And since two years, I'm working for iHomer as a consultant. So who is iHomer? Yeah, we are with 50 plus consultants within our company in the Netherlands. Mainly only software engineers and UX designers. And what we are, we are all participants in our company. So we don't have any directors or managers or each department or whatever. We're just driving this company with 50 people. And as a basic, that's what iHomer stands for. We had a home shoring approach. And that means that we only work from home, or at least most of the time we work from home. And that's before the corona crisis. So we had a standard like 15 years ago, that we have an opinion that we can always work from a place that is nearest to us. And that's how we try to work in short teams with short lines. And we try to prioritize our humans as first. And also very close contact with our customers. So it's not about money and driving profit. It's about getting the correct project done for our customers with our customers. Today, I'm going to talk to you about a customer case that we built for a customer of ours. Unfortunately, I cannot tell you about the name of the company, but I can tell you what the solution is that we have built. It's because of NDA reasons that we cannot mention the customer. It's a customer which operates in a small to medium business market. And when another company needs money or investment for their business, this company will help you to realize your financing. And after when the financing is realized through a crowdfunding kind of way, the funding is realized. Then investors who have a sort of stake or share or certificate or whatever you call it in a company, they can trade with those stakes and those shares. And that's special about this company because it's not standard that you can trade afterwards after you did a crowdfunding kind of thing. So we built a trading platform for them. And what's important for them is that everything is in the right order and it's all correct and consistent. So you can realize that if you do a trading, like when I'm trading for a certain share or another person is also trading for the same share, it's important that the timing of these shares is correct and that there's no gap between having money or don't have money or having shares or shares are gone or whatever. It needs to be correct and consistent. And it needs to be traceable and auditable. One of the requirements is that it should be in Azure and it should be in .NET or at least not fully in .NET, but it should be in a Microsoft technology stack because the company is working with Microsoft systems and they want to keep the stack as close to their other systems as possible. So we chose for .NET Core because it's the latest .NET stack, of course, and we chose for Azure Container Apps. When we start a new project for our customers, we always try to define a sort of new mission statement, ambition, what this product does and what this product is meant for. And together with our UX designers, we try to come up with a plan which is iterative with designing a product and then developing it so that we get the best result for our customers. Because it's a trading system, we figured out that we can use domain-driven design, so DDD. And we use the processes called event storming to determine which actors are in the system, which domains are there, and eventually which microservices exist within the solution. So based on that, we can define actors, aggregates, commands like creating an account or deposit cache or having an instrument. And from there, we can create views or projections that the user or the actor can see and can interact with. So why Dapper? Yeah, that's interesting. We started with a team of about five people, five developers, and I started with two colleagues of mine who are very majestic Java developers or Node.NET developers. And in Spin Zero, we defined, okay, we need to build an event-driven system also in the design phase with the event storming. We figured out it need to be event-driven. And those two Java developers had experience with Axon. And Axon is an event-driven system. So we were on the hunt in .NET to have an event-driven system or framework which we can use for building the solution. And at first glance, we couldn't find something that was similar to Axon, or at least we couldn't find something that is very similar and behaves like Axon. So the two Java developers were like, okay, we can create ourselves and we create a .NET framework and we try to build one like an open source framework or something and build something new. On the other hand, me is like, okay, I have 20 years of development experience in .NET. I heard about Dapper and I said, okay, I will set up the infrastructure and we'll use Dapper for the infrastructure because I thought it was cool and I wanted to work with this framework because of the microservices and the service costs, etc. But I wasn't that experienced with Dapper at all. And after a couple of sprints of them developing a framework, and me developing the infrastructure, we figured out that the active framework within Dapper might be a perfect fit for our event-driven event-based system. So after a couple of sprints, we figured out, okay, this finally is the best way forward. So we stopped developing a framework and we created everything on the active framework of Dapper. Next, I want to talk to you about the architecture that we built and designed during building. And we use IcePanel for a sort of overview of that architecture. And next slide, I will show you a video of our architecture. So what you hear here is the first level of architecture, and you see the two actors, investor and employee, and some external systems like Dynamics 365 and our step frame, step system, which complies of a lot of microservices which you see in the background. And when you zoom in on a microservice, you see the dependencies that they have on the other components that we have. So in key vaults, service bus, events, et cetera, and some external events that they need to question. We also have a blueprint, which is a sort of what can happen if all is implemented, kind of standard thingy. And what you hear here is our CQRS mutation stuff running through the system. So from a mutation, we call the position servers and mutation calls an aggregate. And it's routed to the event bus and to an event dispatcher. And this dispatcher can send to multiple subscribers like event handler or a Saga manager and can send those events to the database. And that database saves the projection and that can be queried via the query of the Kafka well into our back office. And of course, every event that is created is stored in an event database. So that's just a flat table of events. Next up, I want to, I recorded a demo for you guys to see how the system, some functionality is implemented and see what our system does at least. We have two actors in here. That is an investor, which is Bob. And we have Alice, which is an employee of the company and has the back office. So I will start now the demo. Now you see all the services that have health checks on them and see if they're healthy. And the interesting, the last one is the gateway. And the gateway is a stitch up of all our microservices into one GraphQL service at the front. Next slide, I will talk to you about how stitching works, etc. When I get my password right, we log in as Alice as a back office employee. And in this back office, you can manage all of our issuers, so companies who have shares. Next I will log in as a portal, sign up Alice and log in with Bob. Alice and Bob, the traditional users within the... Okay, so first we go into the back office. What you will see is all the different screens that we have. And every screen is just an overview of items that you can manage. So financial instruments are like shares. Offerings are like new openings to issue a new share. Trading is of course something that you can do afterwards. You have an offering created. As shares can be sold, etc. On working days, so which days are closed. And this is also important trading hours, because you can always trade between 10 o'clock and 4 o'clock in the evening. Account information. These are actually our position accounts. And this is the ZJCN that's the company itself. It's not actually that name, but we renamed it of course. And this has a cash position or an instrument position. So how much cash is in your account and how much... It's like a bank account, but then in a soft virtual way. MCCon contains issues and cash. Okay, I'm going to create an offering now. And we'll show you that there's no position yet available. Oh, okay. I forgot about this. I wanted to show you what the initial state is between the services and how they communicate. Just with SIPKIN, this is a standard thing that we implemented in our solution just to see the overview of all of the services that are interconnected with each other. So you see a lot of lines. It's mainly health checks because there's no actual events happening yet. And we'll just now add an instrument to this conference company. And we'll add 100 items. Okay, after refresh, you don't see anything, but after refresh you see the position, that's the 100. And you see one mutation that something is added to this position account. Position account of conference. So our conference is creating a new offering and it has no subscriptions yet. So what we do now is Bob will create a subscription on this. We'll see in his position account that he has 100,000 euros, but no instruments yet. So he now will subscribe on this offering and he will get 30 items, 30 shares. Okay, he subscribed. And what you see now is that there's only one subscription now, which is just created, have a state of created. And we'll now just confirm the subscription. So Bob actually really wants this and he confirms that he has 30 subscriptions. But there's no reservation yet. There's only subscription. There's nothing closed yet. So we'll move on now in the back office again. And we go to the offering and we'll close the offering now because it's now open. So everything is done and all of the shares are subscribed and we can now close the offer. And what you now see is that there is a reservation of 30. And we confirm it. There are a couple of steps in between the process of confirming and having the subscription. It's not fully confirmed yet of closed process yet. So it's only reserved now that the money is reserved and there's no instruments yet traded or changed accounts. It's only 3000 euros is reserved. Now we process it and this means that the actual money is transferred from the one account to the other account and that the instrument is also being transferred. So this is now done, completed. And what you see now at Bob's side, I think, oh maybe I will show the transaction. So you see the different position count from and to. And what you see here is that there is now nothing reserved anymore, but there is 3000 euros deducted from my position, but also 30 euros. 3000 euros goes to the conference company and 30 euros goes to the company itself, which I cannot say the name of. In this case it's ZGACM. So we get 30 euros. And the conference company has 3000 euros and 30 stocks less because it had a hundred as a base and 30 was deducted from it. But that was our demo. I'll show you the Zipkin again because, at 11 o'clock of course, very strange to see my demo again. And you see now that there are a lot of actors created within the process of everything. So every time a position account is starched or an offering is starched, then there is an actor created to manage the state of that thing. So a position account is an aggregate in our system, position account of pop in this case, but that keeps alive until whenever in time.  So that's the interesting thing about this actor framework. Okay, I'll go to the next slide for you. So which type of components did we use? So a lot of things that we did is actors. So all of our aggregates and we also have SAGAs and SAGAM managers are all actors in our system. They have state. So they use a state store. Locally we use Redis and in Azure also Redis. But for instance, Popsup is locally on RevitonQ and in Azure Container Apps it's Azure Service Bus. We heavily rely on the, of course, on the Popsup and configuration. So all the configuration settings are stored in AppConfig within Azure and within Redis on the local site. And we also use reminders to, for instance, to, we call it, to, to get a sort of, for instance, that offering has a, as an expiration of 30 days. So we need to set a reminder that after 30 days the state is changed from open to closed because the reminders pushes it to close. Distributed log is also one thing that we not implemented yet, but we need to implement it because of the, the locking mechanisms that we need for our solution. So, my head is again in the way. CQRS and GraphQL are a perfect fit for us. So we use GraphQL for our commands and queries. And that's a direct match to mutations and projections in our CQRS framework. And we use Hot Chocolate as a GraphQL solution in our system. And now I'll just start the demo, or at least a video about our solution and how, how our solution is built up. So you see API gateways, which is, contains really an envoy and a stitching API of GraphQL. We have building blocks, services and a front end, which is contained in the web project. Yeah. We will now zoom into the, to the services and we go to the position accounts. And I will show you the CQRS steps that, that we follow when a mutation is created. We go to the mutation and we add a cache deposit. And what you see is that there, there is a, the first we get the current aggregate. So Bob's aggregate is getting from the state management and, and read into the memory of this service. And we handle the cache command. And then cache command is handled within the aggregate. And in this case, we get a command inside of our methods. And we check if the, the state is, if the, the aggregate is still created or open. And we send an event to our event service, to our event source aggregate. And this event source aggregate is a, is a building block within our, so every, every service uses this. And this will assist our event into the database of all the events, just the flat table of events, which is stored in, in JSON, et cetera. And we publish the event to our event bus. And the topic name is actually the, the name of the aggregate that the event is published to. Yeah. And the, the Pub sub name, that's the, the name within Dapper that knows which event bus we use. Then we have a, on the controller side, we have an event dispatcher, which gets all of the events that are coming from the events Pub sub. So we subscribe to this Pub sub and we have an event listener. And in this case, we only have one event listener, you can have multiple. And we handle this event to just write to the database into the position account table. And I think we also write our limitation. What's important to, to realize is that on the publish event dispatcher, you can only subscribe once on a topic within Dapper. In this case, we only have one event handler listening to this, to this event from the, from the Pub sub mechanism. But in our secret system, we need sometimes multiple listeners to, to listen to certain events. So we created this component, this event dispatcher to dispatch events to different sources. Okay. In this case, I will show you one that has, that has multiple. So in this case, the confirm subscription. So we go to the handle subscription of the offering aggregate. And we check if it's still a valid subscription and if it's still created and not closed. And we create an event for that, just a subscription confirmed event. Oh, no, this is the standard created event stuff. So it's the center of the event service, et cetera. And then the event dispatcher for this case is, is called. So the subscription confirmed event dispatcher. And we have a offering subscription event handler, which is just written to the database. And the other one is the reserve cash position Saga manager, which handles our position accounts. So we go to the implementation and we see here that we are getting the event itself get it from the state because we need that to get some values like the position account ID and some other stuff that we need. And based on that position account, we get the position aggregate. So in this case, maybe pop with that ID. And we handle the reserve. And this reserve, this handling, this checks if there's sufficient cash cash in this account. And if it's not closed it again, and if it succeeded, then it's confirmed and it will stop with the method and will stop. If it's not, if it doesn't have enough cash, it will continue and send out a rejection. As talked about in our previous video that I showed you, we have a gateway API, which acts as a sort of proxy in between our front-end and our services, our microservices. That stitches up all of our microservices into one big API towards the front-end. And in this case, we can stitch up two services that have a sort of connection to each other, like an order has a, an order service has a financial instrument ID. And I want the name of that financial instrument also in that schema of that order service. And that's, that's doable with this, this stitching solution. And that helps us in preventing having multiple interservice calls between services and keep the services isolated from each other. What I want to tell you about is that we also added some things to the actors. We created a sort of generic class stop actor, which is based on the actor framework, but something extra. And in this way, we can create unit tests, which are very, very important for event-based system, because we need to be certain that all of our commands and events are handled well. And for this one, we needed to add some mocked state manager to, to mock the state manager before we do a test. And in that way, we can create small tests for all of our commands to be handled correctly. So that's an interesting thing that we can, that I can mention. Now we're reaching the end now. Yeah, this platform is now underway for about a year now. And in the first year, we did a lot of sort of proof of concepts. And the solution that you just saw was already like multiple proof of concepts together. So the offering, the trading, all of these, these mechanisms we needed to try out with our solution. And we're actually now building upon this solution, actually creating a new solution and moving all of those parts that we just created into our new solution. And we're finalizing a MVP for this year for this company. But we see at IOMA, we see other opportunities also for these kind of frameworks, because of course, DAPRO is not only .NET, but can also be used for other languages and other frameworks. So we need to see a lot of potential in that. And I'd like to thank you for your time and your attention for this presentation. So if any questions arrived and the future me will answer them. Thank you. Bye bye.


find the hydrogen bonds and the donor and accept the nomenclature, we can look a little bit at hydrogen bonds and real molecules. I already mentioned water, right? And in that movie that I showed you, I mentioned that water, a perfect ice crystal at zero Kelvin would have exactly two full hydrogen bonds per water. That is not as simple as you might think, because each hydrogen is participating in one hydrogen bond. But the oxygen also has two electron pairs. So each water molecule is participating in four hydrogen bonds. But it's donor for two of them, and it's acceptor for two of them. So it's kind of, it's participating forming four half hydrogen bonds. So the total number of hydrogen bonds is going to be two per water under the ideal scenario. Now, once we start cranking up the temperature and going to, say, room temperature, so when we have liquid water, what's going to happen is that the number of hydrogen bonds is going to go just slightly above two, but as long as we're ice, it's not going to increase a whole lot. And then when we move over to the liquid phase, there will be a jump because the atoms will now, the molecules will now start to diffuse and move relative to each other. But it turns out that almost all the hydrogen bonds are still intact. Liquid water has an average of 1.7 hydrogen bonds. So if you do H2O, aqueous phase is 1.7 H bonds. And if instead they do, I'll say X-ray for a crystal, that would be roughly two H bonds. Not quite. If I were to boil the water then, if I would go all the way to gas phase, I would have zero hydrogen bonds. That's not quite true, but there will be some of them formed transiently, but in principle in gas phase, things are so far away from each other that they do not interact. We already talked about DNA. Remember that I mentioned that inside the DNA, the recent DNA paired up the way they did was because we had this basis. Purins and purimidines, A, G, C and T. And these four specific hydrogen bond patterns from a purine to a purimidine and either two hydrogen bonds or three hydrogen bonds under the normal Watson-Crick based pairing. That's what gives DNA its specificity. If it was, if the hydrogen bond was very weak and easier to break, the DNA would not maintain its structural integrity. And there are a ton of hydrogen bonds in this DNA spiral. And that's of course why it's stable and it doesn't deteriorate, which we should be fairly happy about. Because if it deteriorated quickly, we would form tumors and have errors in our genetic code. But it's not just DNA. It's going to turn out that almost all the molecules we work with in this class have hydrogen bonds. We're going to talk about protein structure. On the very far there, you have a so-called alpha helix. And in the alpha helix, we have tons of hydrogen bonds formed along a staircase. It's going to involve those peptide bonds I showed you before. The other panel there shows so-called beta sheets. We'll introduce the next week. And the beta sheets also have a ton of hydrogen bonds. So there are two aspects that we can learn already from this lecture. Remember how I said how the torsions are important because the torsions determine the rotational degrees of freedom. That is, they determine how a molecule can move. On the other hand, the strongest interactions that actually form or break are the hydrogen bonds. Note that form or break is an important modifier there because if I literally tore bonds apart, those interactions would be much stronger. But those interactions are so stronger that they are brick walls. And I won't push my head through a brick wall. Sorry, not even for a lecture. But the hydrogen bonds are so strong that they're important to form interactions, but they are weak enough that you can actually break them under some conditions. For instance, if I'm changing temperature or conditions. And that's why this is going to be such a miracle that we're going to look at when do hydrogen bonds form, can that explain when things are stable or not. And the part that allows the molecule to reach those different states will be the torsional degrees of freedom. So again, you need to know what are the important degrees of freedom in most biomolecules and what are the most important interactions to stabilize degrees of freedoms. Hydrogen bonds


to do development on. If yours do, then you should come talk to me afterwards so that maybe we can figure out where you work, and no, I love Hewlett Packard, they're great. Wow, I get in trouble in new and interesting ways every single time I stand up in front of people, and I think I just found a new one. So that's great. So anyways, you can do all of those things, which is ultimately beneficial to your business, to your business goals, really. It's about making things easier. So that's the nice side of things. That's the pretty marketing picture of how this works. This is also not a picture drawn by Hewlett Packard, and it was also not Copyright 2013 Hewlett Packard Development Corporation. So again, sorry for my inability to use open office, or Libre office, whichever one it is that I'm using. I'm pretty sure it's Libre office. But this is sort of more what an open stack installation looks like. It doesn't really look like this. This is very nice and clean and pretty and understandable. This is actually an old and simple version of what a current open stack deployment would look like. The camel circuits, what? The family circus diagram, yes. There should totally be a little dude. There should totally be a little dude like running around this, that would be, yeah. I love that, that makes me really happy, yes. This is, I want to see that emanated version of that slide. That just makes me exceedingly happy. So in any case, this is what an open stack deployment looks like, and as you might be able to tell, this is a little bit complicated. There's a lot of pieces here with a lot of relationships to each other. So many so that we have many people whose sort of sole purpose in life is trying to make this easier and better. It's kind of indicative of, and this is sort of where we're going, it's kind of indicative potentially also though of applications that you might want to deploy in a cloud. Because with something that's this complicated, you might want to be able to iterate on that easily to develop it. In any case, with an application like this, you wind up with some typical issues that crop up in your, wow, I have, we're being invaded by something. That's kind of amazing. One of the most interesting sounds I've ever heard. So you've got some issues that will crop up in something as complex as that previous thing that can cause you some problems. Probably there's other ones, but a few of the biggie ones that we've been sort of trying to deal with are bugs in the software. Turns out when you've got that many components, some of them might not work properly. I know it might be far fetched to believe in open source software, but we do have bugs.  Also in the installation, Croft and Entropy can come up. You've got thousands of machines. Things happen. Admins log in and make changes somewhere and all of a sudden you've now got a divergent fleet of things and then you do a new deploy and then something doesn't work on one of the nodes and you don't really know why and then eventually you figure it out. It's because somebody installed some sort of lolcat software on the server. I don't even know what that mean, but something happens. The server drifts from the state that you thought it was in so that your application of a new state of the server changes. And then the thing that we all know and love is hardware failure. Things can die and that can also make your very complicated application unhappy. So in thinking about the complication of an OpenStack deployment and in thinking about these problems that tend to arise is where we came up with what we're calling triple O these days, which was this is what happens when you let me come up with a code name for something. It stands for OpenStack on OpenStack. It's sort of the word has stuck around and gotten a little bit more prevalent than one might have expected. And nobody seems to be able to spell it.  But what it is is looking at a number of things from the perspective of treating OpenStack like a cloud application. So that we can do things that we sort of know how to do with other cloud applications such as continuous integration and delivery. We can keep the install and maintenance costs of running one of these clouds down because a lot of this, in addition to being about velocity, it also certainly isn't not about margins. And allows us to use all of the fun tools that we've written for our cloud end users to encapsulate and describe the installation of the cloud itself. And then sort of finally the common API part. We've got this API that allows us to manipulate infrastructure resources kind of neat if the guys in the data centers running the cloud themselves got the benefit of being able to use that really nice API for manipulating and controlling infrastructure resources. So we're really trying to be friendly to our ops guys, even though it causes them to want to go a little crazy at first. So there's a few sort of specific, along in the process of this, to deal with the problems that I was talking about before, there's a couple of different ways that we're approaching trying to mitigate the issues of the complexity. From the bugs perspective, and this is sort of something open stack wide, the more we can do continuous integration and continuous deployment is the larger chance we've got of catching bugs before they get deployed. Right, so rather than sort of deploying things and then going, oh, that broke, we try and do rampant amounts of testing. This isn't specific to Triple O, but it's certainly a philosophical point that the Triple O team ardently supports as an approach, and is sort of part and parcel to how we're thinking about the deployments. The cruft and entropy side of things, we are attacking from a golden images perspective, and typically when we say that, there's the people that are like, oh gosh, that doesn't work, we tried that before. So I'll talk a little bit more about the specifics of why I think you're wrong, but not why you're necessarily wrong, but why things might have changed, let me put it that way. Okay, why you're wrong, just why you're wrong is the way I'm gonna think about that. And then this is no news to anybody, but one of the best ways to deal with hardware failure is to actually develop an HA deployment of your thing so that if the hardware fails you don't have a service go down, not rocket science there, but sort of underpinnings of how we're thinking about what happens. So this is a diagram that there's absolutely no chance that you can read the words in the boxes, or possibly if your eyesight is better than mine, I'm gonna say I can't read the words in the boxes. That might be because I'm getting old and need glasses and haven't bothered to go get them yet. This is sort of the workflow of managing and deploying an open stack based cloud in the triple O sort of mindset, which is that everything in a deployment starts with the change the developer makes in their local code repositories, right? Turns out you can't deploy new changes if somebody did not write them, unless you've got a magical code generating bot somewhere, but I guess even that would be writing the code. So somebody developer writes the change and uploads it into the CI system, whereas it's tested, and at this point we generate an image using some of the tools I'm about to talk about, generate an image and then spin that up in a VM and test the image itself, that the set of software that comprises the things touched by that change, test it and go, hey, does that work at all? And if it does, then we can take that image and combine it with the other images that comprise the service over here, and then we can actually deploy that onto a set of bare metal machines, because we have a bare metal cloud that I'm gonna talk about in a second. Deploy the set of those and test that, and all of this is completely, like a completely automated workflow at this point, other than the developer writing the change, we still haven't automated that. But deploys that onto the bare metal nodes and tests that. So now you've got a multi machine, multi image deployment being tested, and then if that works, then you have a general sense of ease and pleasure about the change that you just produced. Especially if down here you're actually properly deploying a sort of a multi node situation and testing multi version, I've got some of my nodes in the older version, I've upgraded some of them to the new version, et cetera, but those are all automatable. So you do that, and then at that point you can publish the images into your public server, which can then feed the next iteration of this, because probably you're doing more than one change a day. If you're not doing more than one change a day, I don't know, well you probably are. But you can publish the images so that both they can be consumed by your production infrastructure, or they can be consumed by the next iteration of the next changes that are gonna come after it for testing. So to do this, we've written a number of small tools. We're big fans of writing small special purpose tools that can interact with each other, but also that don't require each other, so that you can use some of the stack or not, and other parts of the stack or not. And just for comparison's sake, tried to sort of group these into sort of sets of sort of areas of concern, and sort of done some indication here of where some of the other tools in the space take care of one or more of these tasks as well. So like in the puppet world for instance, configuration, state management, and the software installation are all handled by Puppet, and then Razer from the PuppetLabs guys is the thing that they've written to handle bare metal provisioning. In our world, we've tried to split each one of those up into its own tool. So that if you like our tool for software installation, but you still want to use our disk image builder tool, but you still want to use Puppet or Chef on top of it, that should be completely fine. And if it's not fine, then we've done our job wrong in terms of separation of concern and making sure that everything is modular. So the tools and the things across the top are the sort of the main focal point tools. Nova Disk Image Builder, I actually need a new column for this, I just realized I've fixed the rest of the slides and not this one. We have a set of three tools actually, OS Apply Config, OS Refresh Config, and OS Collect Config that work together to do things in terms of configuration and state. And then Heat is the thing that sits in the position that knows how to do multi-node orchestration. Absolutely, you'll hear no words from me about anything that do not involve completely open source things, and actually all of these are constituent projects of open stack. So, and I'll dig into that in just a second here. So, sort of directly following up that, these are the sort of, this is sort of the list of the components themselves that are important. The list has grown a couple of times since the last time I've talked about this. Underlying all of this for this to work is what is currently, what's currently sort of production usable is the bare metal drive-off for Nova. So Nova Bare Metal, Nova's the open stack compute component. That is in process of being split out into a project called Ironic. Ironic is not ready for production yet and will not be ready for production in this next release. So Nova Bare Metal is still the thing that you would want to use in an actual deployment today. Ironic is taking what was, a lot of the things that were sort of hand hard-coded in Nova Bare Metal to make the sensible one and make it more modular so that we can interface with more vendors, ILO systems and, you know, because we were starting to get the point where it was a plug-in to Nova which itself needed plugins and that's when you start going, okay, maybe this wants to be its own thing. So that's underneath that. To drive the Nova operations itself, we use Heat which is the open stack orchestration engine and then those are both tools that sort of exist in open stack and have sort of separate life cycles. The Triple O team itself is responsible for the next collection of things. There's a tool called Disk Image Builder which amazingly enough builds disk images, very inventive with names. There's a tool called OS Apply Config which is a very, you could almost say that it's config management but it's not. It's really not trying to be a chef or a puppet replacement. It's trying to be the minimal required management of just config files and not other state. Partially so that we can drive installation and testing of these things in a pure open stack environment without having to just write specific shell scripts or punt out to Chef or Puppet. OS Refresh Config works in contact with it. It's actually the thing that orchestrates the running of a thing that is going to muck with your config files because there's things that you might need to do like, say, queues a service before applying the new config file and then start the service again after dealing with the config file. That level of in machine orchestration is handled by OS Refresh Config. There's also a lovely little tool which is actually just useful in and of itself without any context of this called OS Collect Config which knows how to get your config data from all of the various cloud metadata sources that might exist. So if you've had a machine that's been spun up by heat there will be metadata in the heat metadata service. There will also be metadata in the cloud in it whether it's the local network interface, the EC2 metadata server, or whether you've got a config drive or whatever like that. There's a bunch of different places that something might have passed some metadata to your server. And OS Collect Config will go to all of them and get them and present them to you as sort of one single pile of JSON which is not particularly all that complex but winds up being kind of handy. And you can just pop on to any, pretty much any server and just run OS Collect Config and it'll show you all of the cloud metadata that is available to you on that server. Which I find handy just because I can never remember how to curl the fancy internet address that's there to get that. So anyway, so that's that and it's used by the other two. And then we have, so disk image builder is comprised of series of elements. So if you're going to build a disk image of software you need to describe what software you want to go into that disk image. And the building blocks that we use, we call elements, they've been called at least five different things since the beginning of the process. And all of those terms were overloaded. So we think, we think, we've been on elements long enough that we think that we're okay with that as a long running term. And there's a repository of disk image elements that are the ones that we use to actually deploy an open stack. And that's in the triple O image elements repo. And all of these, these are all the, you can look at it URLs. These are not the clone from this, no, sorry, these are the, these are actually a, dang it, I did it backwards. Yeah, yeah well, these are the clone URLs. These are not the look at it URLs. I realized earlier today that we have a new Seagit server in the open stack infrastructure and it does take putting URLs here, interesting in terms of indicating which activity you're expecting to happen from those. In any case, so I've been babbling a lot about the components. The short version of the story for a triple O deployment is that you have a heat stack which defines your cluster. Heat then drives an over API to deliver images to the machine. So this is sort of a typical cloud activity. Hey, cloud, spin me up a one of these. So heat drives the orchestration of that. And it does these things in a sort of dev test environment on virtual machines, but because we've got this Nova bare metal thing, it does that on full bare metal for actual production deployments. And if you've got the bare metal, you could do that in your CI CD environment as well because it turns out it's clouds all the way down.  Nova has the idea of pluggable virtualization drivers. So you can use Nova to spin up KVM VMs, Zen server VMs. You can actually use it to spin up VMware VMs. If you have the ability to do these things, you could do Hyper-V. I don't know anything about doing that myself, but I hear that it is a thing that Nova can do. But in general, it's a pluggable backend that knows how to talk to different things to get VMs. So the Nova bare metal driver took the vert layer interface and put an implementation of it that speaks Pixie and IPMI. So rather than spinning up VMs on the machine on which the Nova compute instance is running, it talks over the network via IPMI and Pixie calls to other machines in your rack and manipulates them as if they were VMs. So we've gone from the place where we have a cloud that we did to give us a virtualized abstraction to get us things that behaved like real machines and now we're taking real machines and making them behave like the VMs that we then want to act like real machines. So, and the best part about this, which is really fun, is if you want to hack on Nova bare metal or Ironic, there is a virtual bare metal power driver.  And thus you can Pixie boot VMs on your laptop so that you can just do development of, anyway. Yeah, but the fun thing about that is actually, for what it's worth, there's virtual box and VMware desktop drivers and stuff like that in there as well, so if you've got various ways of, I don't know, if you're like a vagrant fan or something like that and like doing that, you could actually do that with Nova commands instead to drive the VMs on your local workstation. Might be a little bit overkill, but you know. Why? What's that, yeah. Just, yeah, the easy way is just install OpenStack on your laptop and that'll make it really easy to manage VMs on it. But you'll treat them like bare metal machines so that it's not, anyway. So that's Nova. So once we've got that, once we've got a Nova API that actually can control the hardware and you can think of this, right? Nova already knows how to sort of scale horizontally out by adding more Nova compute nodes. You can have a data center's worth of hardware that's just basically controlled by a Nova API calls and we're talking like a couple of machines of overhead in the data center to manage all of your bare metal like it's a cloud, which is sort of cool. But then you've gotta be able to do something with that. Yay, I can, oh, that's weird. So the next thing is when you've got a data center worth of gear, you might not wanna be thinking of it one machine at a time because you might quickly go mad. So Heat is the OpenStack orchestration tool and the thing that it's focusing on is the multi-machine orchestration level. Heat's job isn't to manage the MySQL inside of your machine. Heat's job is to say I need a machine that looks like a MySQL server that's been defined in some particular way and I'm gonna make one of you there. And so then the things that happen in the machines are basically a handoff from metadata that a combination of Heat spinning up the right base image from the cloud provider that you want plus Heat handing some metadata to that spun up image. So it delivers that metadata and also Heat accepts return values from the thing. So when Heat spins up one machine, it says, hey, here's what I know about you.  And also you might not have known this piece of information about me before this moment because it wasn't possible to know. So here, here's some more information about it which can then be parceled out to other people in the cluster. So that's sort of the space in which Heat sits and the position that it sits. You can use it with something like Chef or Puppet. Chef and Puppet are really good at managing the configuration of a machine. They're really, when you start using them for the multi-machine coordination of things, things get a little bit weird. It's honestly not what they were designed to do. You can do it, but it's not the underlying thing of that. So the idea here is to decouple that so that the thing that knows about the machine then just tells the machine, you go do whatever it is that you do on that machine and I'm fine with it.  Salt, yeah, I could, so I apologize to various people. I've kept most of the references in here to just be Puppet and Chef rather than Puppet Chef. Salt, you could, salt, same thing. Salt, because of how it was written, has a little bit more understanding of sequencing and relationship and orchestration between servers. But in this context, Heat can certainly drive salt states on a machine as well. So certainly absolutely possible. The thought here is basically that there's two concerns. One of them is what is the relationship between your machines and the other one is what's happening inside of the machine itself. There's a set of heat templates that describe an open stack installation which are at, you can clone them from get.openstack.org slash openstack slash triple o heat templates. You'll notice that we tend to name things after what they are. So that's a place you can go get those and see what those look like. So if Nova Bare Metal is a way to boot, to boot Bare Metal machines, boot images onto Bare Metal machines and heat is a way to pass metadata to those images that have been booted, you probably kind of need a way to actually make the images that you're going to boot onto the Bare Metal machines in the first place. Or the VMs because it turns out it's cloud software and so you can actually use the exact same image to do both of them. And so this is where we wrote the tool, Disc Image Builder. And the idea here is to encapsulate and describe the sort of set of software. These are intended to be static images essentially, not to contain lots of specific specified metadata. So if you've got a thousand machines in Iraq you probably aren't going to wind up with a thousand images. You probably wind up with an image for each type of thing and then configuration data from heat gets passed into the machine at boot time or at cluster stack reconfiguration time, which then can be specialized by your config management system. This is sort of another way to think about this is the equivalent of packages except at the machine level, which is sort of what the Juju guys like to say about Juju. But I like to steal as much wording from the Juju guys as I can because it's fun. But the idea is that you've got a description of a machine that's repeatable rather than just the description of one of the building blocks of a machine that's reusable. And especially if you're thinking about thousands and thousands of machines then thinking about the individual, the multiplication of the packages plus the machine. Anyway, it's intended to be building blocks at that level. One of the things that's nice about this is that because the images themselves are sort of self-contained and accept a sort of a known and reasonably small set of metadata inputs, you can test the images and then when you go to deploy you can actually deploy the actual image that you tested, not just the instructions for creating that image but actually the actual binary bits themselves, which is kind of fun. It also means that you're a little bit more, you have possibilities for efficiency because you're not running the build steps to create what should be pretty much identical a thousand times, a thousand times. You're running them once and then you're just splatting out the results of that. Which is again sort of why we talk about that being sort of like packages except at the larger level because that's one of the things that you get out of packages, you don't rebuild Emacs every time that you install it. You just, well except for the elist files which you have to buy to compile, but I'm digressing. I think you get the point there. So there's again small tool chains. We've got this image builder which I mentioned earlier and triple low image elements which is a collection of elements which describe the software. So we've got the image elements which describe the images that we want to deploy with heat and we've got the heat templates which describe the collection and relationship between those in a running cluster. And then I talked about these a little bit earlier. The OS, the OS splat config utilities, the three of them. I've sort of talked about what they do but basically the interaction between them is that you have OS collect config, gets the metadata from heat or from cloud in it. One of the reasons for that is that we would sort of like the first boot and the first boot configuration action and the subsequent configuration actions to sort of go through the same mechanism rather than to go through wildly different mechanisms. OS refresh config then knows how to quiesce any fragile services that you might need it to do. Upgrade software as needed from Glantz. Trigger OS apply config to actually manifest any of the metadata into config files that might need to happen if needed to reboot the machine. Although that's a, there's another set of information about that later. And then after that has been done, OS refresh config ensures that any of the required services that it might have mucked with are running again. And then after you've done that, you might need to do migrations. If you've applied an update, you might have to do a data migration in a database or something like that which needs to get applied. And then the fun part is that once it's done with that, it sends heat back a message saying, hey, I'm done, which means that you can actually describe things like rolling upgrades in the heat structure because you can say I need you to do, I've got five of these guys, I need you one at a time and need you to not start the second one until you're finished with the first one. So heat actually has the context to know that the operation it's attempting to do on the first one has finished before going on to the next one and doing that one. So sort of managed orchestrated rollouts of that sort of thing is possible. Although I should be clear that some of that is theory, the heat guys are currently still working on rolling out the orchestrated rollout thing, but the structure of how we're describing that is there. And then the theory of getting that to go, yes. So in our version of the world here, we're talking about using Puppet and Chef to, potentially if you're into using Puppet and Chef, using those to actually manage configuration and not using them to manage software installation. Because running app to get install new package version 10,000 times across your 10,000 servers is actually not the quickest thing in the world to do, whereas spotting out the results of that is actually pretty repeatable and you've got 10,000 new points of potential failure each time you run the code that does that. So we're trying to split that heat in, excuse me, Chef and Puppet are actually extremely good at describing configuration changes because they are config management systems. So it's a little bit, you can use those if you want to. In our particular case, the OS apply config thing we've got here is sort of a special purpose one for an open stack installation. I would imagine that if you're doing a non open stack installation, you would probably use Puppet or Chef rather than OS apply config to actually manage the config files on your server. But the main thing that in looking at this is what you can get to is having heat be the one that triggers the Puppet or Chef run, except it's triggering it on a per server basis in a phased manner. And so you're actually able at that point to do things like either use Chef solo or Puppet apply rather than needing a Chef server or a Puppet master because the input, the sort of the things that you would normally get outside of your recipes or modules in a Puppet or Chef world are gonna be the specific pieces of information, the metadata about the server that those recipes would need to know to be able to do their job. And heat is the one with the view of the sort of whole collection of software. So it's able to say, hey, Puppet run with this file of JSON, it's sort of like from the Chef world, I think that'd be like a data bag, right? They'd be like, here, run with this. Run with this piece of input data and apply these input parameters to your recipe or your module. And so then you also don't get into the scaling problems that you see with things like a Puppet master and Chef server at tens of thousands of nodes trying to manage them, although I'm sure that, I don't know, I was about to say bad things. But yeah, so that's the, it's not necessarily a let's kill Puppet and Chef as much as it's maybe let's rearrange things a little bit. And in fact, that sort of dovetails into a really interesting sort of point here is that building some images and then deploying them using sort of your cloud tools, it's kind of really fast. And speed is not everything. Correctness is actually way more important than speed. But when you're talking about deploying tons and tons of things, if it takes four hours for you to actually deploy an upgrade because of the time that it takes for Chef to calculate the graph and then for it to partial out all the things and then on each machine it's gonna take 20 minutes to run all of the upgrades or whatever that is, that stuff adds up and it adds up to making you a little bit more worried about rolling out one of the upgrades because of the impact that it might have on your system. We can go from a rack of bare metal machines that's completely powered off to a completely functional working rack of software and with the software installed in six minutes. Which again, you don't do this very often, right? That's to be really clear, the times in your life when you do that are kind of small, but that's sort of to illustrate the sort of efficiency of just splatting out the changes that you need rather than having all that stuff run every time and across your days, yes. There's a what? Yeah, so that's not, so there's another, sort of dive a little bit further into the, into sort of ironic and Nova bare metal and how those work. The question was, assuming there's a system to be able to basically take a node out of production and put it back into the pool of things. So in the way that the Nova bare metal system works, you have say your Nova compute controller that with the bare metal driver in it and then I don't know, say a thousand bare metal machines that have registered with that. Each of those machines is basically a resource that can either be spun up or spun down and is like a static pool of allocated VMs. So if you've got a compute node that's got a certain amount of RAM and disk then there's a certain amount of VMs that that compute node can spin up. And in this case, this compute node has a certain number of bare metal instances that it can respond, that it can, it can spin up. So it can delete those just as easily. It's kind of like what? Yeah, yeah, it's a little bit lower level in this. This sort of assumes that basically just sort of letting you know that we've got this thing which lets us use your bare metal like a cloud but then basically most of the cloud functionality we'd expect to happen is present. So actually really what it is is to just mark it off maybe power it off with the IPI things and the next time you wanna use it you're gonna be splatting a whole new image on top of it. There's an interesting thing that somebody should find me at a bar at if you're interested in talking about this or actually you should find, thank you. Matthew Garrett who's sitting over there and talked to him about it. There's some interesting ramifications of a multi-tenant bare metal public-facing thing in terms of what you can do to the BIOS on machines and things like that that need to be sorted out before people start doing actual public-facing bare metal NOVA which may or may not be solvable even but some of them are really interesting. Anyway so that's, it turns out I babble a bit too much. I'm running out of the time so that is I'm gonna sort of talk faster and I even added more slides for Jim this time and I might have to show him those at a bar. So the general structure of this is that we have a couple things. We have what we call the under cloud and we have the over cloud because once you start talking about using clouds to run clouds things get tricky really quickly. And so in this case we have an under cloud which opens up with bare metal and then we have one or more tenants in that cloud that are themselves KVM based clouds that are running on the bare metal machines that are part of the resource pool of the under cloud. Sort of have to say that four or five times to yourself and your brain before it starts to really make sense but we're trying to invent some terminology there so that we don't get quite as confused with talking about this cloud and that other cloud and this cloud that's inside of that cloud and whatnot. But so you can at this point since you've got a cloud under the underhand you can add additional tenants and have sort of additional clouds. So if you wanted a production cloud and a dev test cloud those could just both be tenants in the exact same pool of bare metal so that you can elastic your clouds as easily as you can elastic the resources inside of the cloud which is kind of a neat idea. The under cloud itself needs to be a fully highly available bare metal open stack cloud. It's self-hosted because it turns out this is a cloud that knows how to operate bare metal that's running on bare metal so is absolutely possible for it to pixie boot itself after you get past the first stage which is a little bit neared but if you've got two nodes that are in an HA failover scenario you tell one of them to take the other one out of commission and pixie install it with a new version of itself and then when that one comes up you orchestrate the other thing to happen on the other node. That one hurts people's heads a little bit but then you can basically have as few as two machines for the control plane of that and then the rest of them are just bare metal resources to be the source of your tenants for the other things. The over cloud then is again fully HA because a lot of these things work you don't really wanna run a cloud that's not HA because then you're just everything's gonna break. There's a fully highly available in our case KVM based open stack that's hosted as a tenant on the under cloud. So you've got a KVM cloud and it's running inside of the machines that are inside of the bare metal cloud. This is orchestrated by the heat instance that's running in the under cloud. You can also have a heat in the over cloud but there's a heat in the under cloud that's managing the bare metal resources on the under cloud that manages that. And one of the neat things about this is that the disk images that you build for your services you can use the exact same images on bare metal machines as you can in VMs because of the way that all of the Nova stuff works. So you can actually if you've got services that want to live both in the under and over cloud you can deploy them from the exact same software source should you choose to. The initial installation is a sort of a special case which I won't go into too much detail but the general version of it is install a cloud all in one on your laptop, plug the laptop into the network card of the rack, register one of the nodes in the rack with the cloud on your laptop and tell it to HA expand itself out to the second node that it now knows about and then take your laptop away. Now you've got a, you had what was an HA pair now it just got broken and it needs to heal itself so then it'll expand out to one more node and now you've got a two node HA cloud in your rack which you can then use to install the rest of your KVM clouds. That's the short version of that because I've run out of time. There's two versions of the upgrade story which again I'm gonna blow through in three seconds. The simple version is all of this is HA so for all of your scale out things you just shoot one of them in the head and reinstall it and reboot it and you do that in a rolling fashion. This is the sort of standard scale out how you deal with your Apache web farm. You just kill one of them and reinstall it and then you do that to the next one and if all your HA stuff works and you have the machinery to do that then score. Like it's the same machinery you use to deal with machine failures and it's not a problem. However, that doesn't work if you've got actual things that have precious data. I mean you could in theory do that but it starts to get really costly and even an HA failover situation the cost of dealing with that might be such that doing that say 12 times a day might be a little bit, you might basically be running in a degraded mode all the time and that wouldn't be exactly great. So if we talked earlier about having, I sort of blew through the slide because I talked about something else. The, so part of the idea here is if we think about these things as cloud apps the general theory is that your cloud compute nodes are kind of ephemeral and that you have volumes that are less ephemeral that you attach to those nodes which is where you would put precious data because in theory, so it doesn't always work out in practice, but in theory the volumes are the place where the cloud provider is telling you this data isn't gonna go away but these VMs might go away. And so in the sort of VM cloud world you'd have a sender volume and in the open stack world that has your data on it and you can then spin up a new node and spot them out. You can sort of do the same thing here if you think about partitioning your bare metal machine in such a way that you've got the root file system, the thing that is ephemeral as it were and then the place where your precious data goes. And so then when you have a new disk image that you want to upgrade to you can actually just rsync-delete the contents of the disk image over the root file system and either do a full reboot or a kexec hop of that if a kernel update was needed or otherwise you can just manage restarting services as needed, but you should know in that case that your root file system isn't something that you're changing outside of the context of the tooling and the data that is being changed by your application is going into the long-lived volume. There's some gotchas to that but I have babbled about other things for too long so I would have to defer to that. There's a couple things we need to do to make that better. One of them is local sender volumes which is the sender is the volume API service inside of OpenStack and we need a way to tell the system that when you redeploy a machine that has requested a particular sender volume that that actually needs to attach to the bare-metal machine that it was running on before because what it actually is is a partition on the machine and it's not actually something that's being connected over the network via iSCSI, waving hands but it's a thing that you need to be able to reattach to and also in some of these things we're actually net-booting kernels so to do kernel upgrades, you actually need to be able to tell Nova, hey, reboot this machine with a new kernel but please don't mess with the disk image that's there. I just want you to reboot it into a new kernel and so those are a couple of the to do those tasks and I only babbled over time for just a little bit. That's sort of what I've got for today. We're out of time for the questions, right? I have two minutes for questions. Woohoo! I just want to thank Monty. Absolutely, thank you. Yeah, I'm always, I'm always available for... A couple of questions. So, the, the Nova Group Metal stuff is in now and is functional as of Havana. The, the specific triple-O tools, OS Star config and disk image builder and stuff like that, they all, they all work right now. They are, they are a official program currently. I'm not sure that we've, we've defined how utilities produced by a program such as triple-O like OS can apply config might relate to release cycle. So, they're, consider them a little bit orthogonal from the, from the open sacrileases at least right now as we're figuring that part out. But those all work at the moment. Ironic probably won't land as, as usable until, until Ice House. But that's the, the goal is to get that landed for Ice House. And, and then yes, the other ones are more about, about trying to orchestrate this rather than being, most of those are utilities rather than, than things. The, the, the features we need from Neutron in order to do the, the DHCP and Pixie booting appropriately in the bare metal context has landed for, for Havana. So there's, there's no changes needed in the core open stack projects that I'm currently aware that we're blocked on in terms of being able to use this for, for real deployment as of the Havana releases. What's that? Heat, I don't, I believe there's one more upgrade thing that needs to happen to heat before you can do the non-destructive upgrade version of it. For destructive, blow one away and give me another one upgrade heat can, heat can already do that and, and knows how to do that. So it's, it's a little bit, there's works and then we'll get better is sort of the, you know, as those features land. Yes, these, these will be up on a slide share somewhere and then all the code is, you know, on the online. So, absolutely. Oh gosh, should really learn to stop saying things about my employer on these things. Great, thanks a lot.