Hi everyone. Welcome to the Flux maintainer track. Today we're going to talk about how you can use Flux to amplify your get-off setup with OCI cosine. I'm Sunskol. I'm a Flux and flagger maintainer. I work at Weaverx. And my name is Kengden. I'm also a Flux maintainer and I work at Weaverx. This is a maintainer track, so I'm going to assume most of you are familiar with Flux. For those of you who are not, let me do a quick intro. Flux is a CD tool which lets you do get-offs for your community clusters. What that means is you can deploy your applications in a community cluster in a get-offs fashion by storing them in a grid repository or a hem repository. It's a CNCF graduated project. We achieved graduation status last year. We have multiple integrations with Terraform and AWS cloud formation and there is a free and open source UI you can use as well to view your Flux resources. And we are being used by companies like AWS and GitLab for their own get-offs offering as well as companies like Orange and DT for their 5G deployments. So good news everyone, Flux finally has a GA release. Thank you. What that means is that the core get-offs APIs that are the Git repository API, the customization API and the receiver API are all considered V1. Which means that there are going to be no breaking changes so you can upgrade those APIs whenever you want at your wish. Flux, if you don't know, is built of multiple components and multiple APIs. We like to evolve and iterate on these APIs at our own fashion so that we can give the best experience to you users. What that means is certain APIs are still not considered GA, which means they might have breaking changes. For example, Helm and OCI. Helm is considered close to GA. We are working very hard on it. It's tricky to get it right because Helm is Helm. So it's in the pipeline. There's also OCI repository which is a relatively new API which lets you fetch sources from OCI registries and then there is the image automation APIs and then there's also the notification API which lets you do all sorts of alerting and updates to Slack and so forth. The basics of Helm with Flux. You have your Helm repository CRD, you have your three CRDs that each map to a particular artifact in the cluster. The Helm repository has an index.yml Helm chart is an instance of that chart. So if you know about how Helm repositories worked historically, that makes some sense. It'll make more sense as we go on. The Helm controller with the source controller together applies the Helm release in the Kubernetes API. It's using the Helm SDK under the hood. Everything is compatible with the Helm CLI if you're using Helm already. It's a seamless transition. Also, the notification controller is involved so that we can send alerts when things happen. On the right you see there are various places that charts can come from. Harbor or GitHub or that's the chart museum on the bottom there. These are the resources. The Helm repository resource on the left we said maps to that index.yml in the legacy version. This is our legacy Helm release and repository. On the right you see the source ref points at that Helm repository. We have selected a particular chart since legacy Helm repositories can store many charts and many versions. We select a version. We're using a December wild card so that we get the latest version within a range. There's other configuration that you can do, but this is the basics. All of this has been working fine, but we have continuously run into issues by all of you is saying that Helm acts funky or there are certain problems. Sometimes it's too slow. From what we have seen, majority of these issues can be tracked down to one root problem is the fact that index.yml just does not scale. The approach of having to store all your charts information in one file is just not a very scalable approach. There are several reasons for this. First of all, it's because it's a YAML file and YAML parsing is just slow. Matt Farina is a Helm maintainer who recently did a benchmark test comparing YAML parsing and JSON parsing and JSON parsing was significantly faster than YAML parsing. So you have to parse the entire index.yml file and then load into memory and then look up the exact chart entry that you care about. That's a lot of CPU and RAM that you are spending on that. Helm repositories can contain hundreds of different charts and those charts can have thousands of different versions. That means you can end up with a very, very big index.yml file and there is no way to filter out the charts or the versions that you don't care about. So you end up downloading a lot of stuff that you just don't need. Lastly, verification requires provenance files. Provenance files have been around for a long time but the problem with provenance files is that it's another file that you need to care about. It's another file that you need to manage and distribute. And these are not some theoretical predictions that we are making here. You can see these issues out in the wild. Bitnami has a very famous repository which has multiple Helm charts. Recently they had to purge a significant amount of their index.yml because CloudFront was not being able to serve it due to traffic limits. So what is the solution? Of course, it's like the scatter of the index.yml file. How do we do that? So how many of you are familiar with OCI? That's great. So for those of you who don't know, OCI is this open governance body which standardizes everything related to images, how they are built, how they are packaged, how they are run. And for most importantly for us, it's about how they are distributed. So there is a distribution spec which standardizes how container industries work and how they are supposed to distribute images. The best thing about the OCI distribution spec is that it's supposed to be agnostic of content types, which means that it's built in mind keeping images but you can store anything. You can store an mp3 file in there if you want to do. So if you can store an mp3 file, why can't you store a hemp shot? This is a meme that I stole from Dan Lorank on Twitter. It essentially boils down the OCI distribution spec into this one image where you give it some certain parameters and it gives your URL which points to a tarball and in our case this tarball points to a hemp shot. So basically what you do is you can list the tags. So OCI distribution API has a very good API for listing tags and writing and such stuff. So you can just list the tags you're worried about and you can just go and fetch the tarball that has your chart. So what does this look like in practice for users of Helm? People who package charts and publish them. So Helm package doesn't change at all. It generates a tarball and that has the chart YAML and the metadata in it. Helm push changes a little bit. Now we're pushing to an OCI URL that's a registry on the right hand side. There's no third step unless you're interested in provenance which is that third step there. So cosine sign, we'll see in more detail. The spec has some changes. So we've added type OCI and OCI prefix to the URL. We've added a provider here and that's for authentication. On the right hand side we've also added provider cosine which is for verification of the provenance. So what are some of these benefits that you get when you use Flux, Helm and OCI together? First thing right off the bat is the fact that you have all your apps, images and signatures in one place. That is your container registry. You don't need to have 10 different tabs for several different things. You get passwordless authentication and keyless integrity verification. We'll touch on that in the next few slides. And most importantly you can get rid of your index.yaml which means that you have to spend as much CPU and RAM as you were spending earlier which translates into cost efficiency. You don't need to spend that much on English traffic which again translates into cost efficiency and you don't run into issues like Bitnami did with network bottlenecks due to size. Okay, so let's talk about how passwordless authentication works in this scenario. So how it works is it uses something called workload identity. Workload identity is basically a cloud IAM role binding to a particular workload. What you do is you create a cloud IAM role which has access to your container registry. So let's say you're running on GKE and your container registry is on Google artifact registry. You can create a service account which has read access to that container registry and then you can bind that service account to the pod or the node that is running in your Kubernetes cluster. Now since you have bound this role to your pod or node, that pod or node also has access to the container registry and can pull those images from that registry. The benefit right of the bat here is the fact that there are no secrets. There are no static credentials so you don't need to manage all of that dangerous stuff. Flux's implementation integrates seamlessly with Azure, AWS and GCP. We're going to demo Azure today. And most importantly it's native to Kubernetes. So workload identity uses something called service accounts and OIDC projection which means that you're part of the Kubernetes world. You deal with Kubernetes APIs. Right now, Flux's implementation only works at a global controller level which means that if you wanted to use different workload identities in the same source controller instance, you would not be able to but work is being done right now to make that happen as well. So you can specify the service account name on the Hemchart object or the OCI repository object itself multiple workload identities working around on the same source controller. You also get keyless verification. You get keyless integrity verification which is great because you don't need to manage PGP private public key pairs which is again like another elimination of the need of secrets which is great because secrets are dangerous. How this works is that it's bound to your OIDC identity so you sign your chart with cosine. Cosine is great because it uses OIDC behind the scenes and all you have to do is sign it with Google or sign it with GitHub and the signature is bound to your OIDC identity. Flux also supports matching against OIDC identities like the exact identity. It's not out in a release yet but the PR has been merged and the next release will have this feature as well. So what this means is that you can be super strict about where your Hemcharts are coming from. So for example, if you wanted to make sure that it is indeed our GitHub workflow which generated this chart and pushed it off to your container industry, we can do that. So as you can see here there's an issuer field which says token.actions.githubusercontent.com which basically is saying that the issuer of this authorization token is the GitHub OIDC provider and the subject is the repository that has the workflow which generated the packet shot. Okay so this is the workflow that we're about to demonstrate and in the beginning a chart is pushed as it's released and I described earlier the package step, well this happens in CI Flux pulls it into the cluster automatically in the staging environment and it deploys it. And you can do destructive tests because this is a staging environment. So we have in this case we're going to run Helm tests only in the staging environment. You can have different configuration. Assuming everything goes okay the step triggers dispatch event that creates a pull request which user can review and then merge and at that point the upgrade proceeds in production in an attended fashion. So we're probably a bit too early for the demo but we have a few more slides after this. So we're going to do the demo first and then we're going to do a recap and reassess what is the state of Helm and Flux. So I'm going to first walk you through the file structure. Is this font size okay? Okay great. So I have two clusters staging and production so if you want to look at the file structure there is clusters and there's production and there is staging. Both are pretty similar. They have Flux installed and they have one app for info. So this is the Helm repository for the code info registry. So as you can see it points to a OCI registry which is hosted on ACR Azure and you can see provider is Azure here. This basically tells Flux that try to use the Azure workload entity to access the container registry so that the user doesn't need to provide any kind of secrets with static credentials. So this is pretty much the same for both staging and production. So let's take a look at how Helm release looks like for staging. So here as you can see it's pretty standard but if you notice here it says 7.0.x for version. This .x is basically a wild card which basically says deploy the latest patch release. So if the latest patch releases 7.0.2, deploy that. If it's 0.3, deploy that. It also has the verification stuff here which basically saying that make sure it's the GitHub OIDC provider which created the authorization token and it's my repository here which is the owner of this workflow. We have a production Helm release file here. The only difference between the production Helm release file and the staging Helm release file is the fact that the production is pinned to a particular version. There is no sembler wild card here. Which is great. Which is what we want. We don't want to deploy random stuff or like the latest release into production. We want to make sure that we know what we're deploying into production. That's why it's pinned to a particular version and it can't move without our explicit approval. Let's take a look at notifications. So this is a provider. Provider is basically a way that Flux can contact other external systems and let them know about Flux events and alerts are basically a way of creating a Flux event alert. So as you can see here whenever Helm upgrade succeeds for this particular Helm release it's going to create an alert and that alert is going to be sent to a GitHub dispatch provider. A GitHub dispatch provider lets us trigger workflows based on certain events. And this is a GitHub workflow that we have. So this GitHub workflow basically gets triggered by the GitHub dispatch provider. And it's pretty standard. We're not doing anything revolutionary here. The most important business logic here is the fact that we are getting the version from the alert. So whatever is the latest version that got deployed in staging. So let's say that's 7.0.3 that got deployed in staging. We take that version and we use YQ to do some YAML manipulation and we change the value of the version in the production Helm release file to that version. So we're basically changing the pinned version from 7.0.2 to 7.0.3 in the production Helm release file. And that's how we make sure that we never deploy the latest thing. We always make sure that whenever we do deploy a new thing that's without approval. And then we go ahead and create a pull request with that change. Okay, cool. So this was like the basic intro to the file structure. So I'm going to go ahead and sorry, I'm going to go ahead and create a new chart version and push that. So I have a workflow. I have a GitHub workflow which will whenever there's a new tag version, it will basically create a chart and will push it off to Azure Container Registry. So let me just tag this here and push it. Okay, cool. So let's see. Okay, cool. So there's a GitHub workflow which is running right now. Let's see what it's doing. Okay. It's packaging the chart. Right now it's logging into ACR. Cool. It signs the Helm chart. Okay, so let's take a look at Azure. As you can see there's a 7.0.4 tagged chart here and this is its signature here. Right, so now that there is a new chart in our container registry, what we expect from our clusters is that the new chart should be deployed in the staging cluster. So I'm going to go ahead, let me just make sure I'm staging. Let me change to staging. And I'm going to go ahead and reconcile the Helm chart in the staging cluster. Okay, cool. So as you can see, Flux saw that there's a new version and it deployed 7.0.4 in staging. And we can verify that by doing a Get Helm release as well. Right now it's pending. It's in progress. Let's do that again. Okay, cool. So now staging is running 7.0.4 of our application, which is great. So now what this did was it created an event which was sent to Flux's notification controller and Flux took that notification that it got, the alert that it got and triggered a workflow which created a PR and we're going to go and check out that PR. So let's reload this. Okay, cool. So now there's a PR in our repository which contains our Kubernetes manifests. And what does it say? Oh, okay. So it says to change the production Helm release file from 7.0.3 to 7.0.4 and which is essentially what we want. We want to make sure that all updates to our app versions happen with our explicit approval. So if I don't approve this PR and if I don't merge this PR, even though staging is running the latest thing, production will still be on the thing which we approved it for. So I mean right now you could push another chart version 7.0.5 and that would get deployed in staging as well. But production would still be on 7.0.3 because we haven't approved anything yet. So I'm going to go ahead and approve it and approve it as well on his phone. Yeah, here we go. Okay, cool. I've approved it. So let's go ahead and merge this PR. Okay, cool. So now that this PR has been merged, I'm going to go ahead and switch to production and I'm going to reconcile the customization. Okay. So now let's see where was the status of our Helm chart in production. Okay. So as you can see it has pulled 7.0.4 because we merged the PR. And what's interesting to note here is that it also says source verified, verified signature of version 7.0.4. And this is because we included that verification section in the Helm release. So source controller or rather flux make sure that we always verify the signature of the artifact that we're pulling. If this signature was invalid or if there was some kind of a problem this reconciliation would not succeed. So let's go ahead and see what is the status of our Helm release. And here you go. And now we have 7.0.4 in production. But the difference is that we explicitly approved this. It didn't happen on its own. Flux didn't pull 7.0.4 on production and deployed it. We explicitly approved this via the PR workflow. Yeah, that's it for the demo. Okay. So let's recap what we've just done. So this is a well-documented workflow. It's been around for a long time. The main things that we've added here today are the password lists and the keyless parts for OCI and Cosign. We have the advantage of Helm tests. It's a battle-hardened workflow and it's extensible. So you can use it on other platforms. I've heard of people using this with Azure DevOps. They use a generic webhook instead of GitHub dispatch. So this is great. But there is a negative drawback if you are not building Helm charts. It's a rather large investment to get started and there are some very good reasons not to use Helm in 2023. We don't need to go into them. So what can you do if you don't want to use Helm? What Flux would suggest is you could use OCI repository instead of Helm release. So we haven't documented this yet, but if you follow the workflow, we'll update the docs within a couple of days, hopefully, to describe this. It's very similar. It's like three lines of changes. It's not really a big deal. So you get all the same benefits of keyless and identity using OCI repository. So it's not well documented yet and there are a few other drawbacks. That's the main one is that you won't have Helm test. So if you were using Helm test or Helm for other reasons, the life cycle features, the rollbacks, you'll use Git for that instead. And of course we also saw the semver wildcard promotion method, which we used in staging. So this is portable across most of the source kinds in Flux. You can set spec.semver on a Helm chart or OCI repository or Git repository and it will work in all of those places. This is a great way to get releases out faster if you're trying to publish and iterate quickly without a lot of friction. You don't necessarily need that friction unless you want it. We know we want it in this case, so that's why we demonstrated it. But really to be clear, this is a workflow for staging environments. I wouldn't recommend you do this in production without a great deal of ceremony applied some other way. One possible way would be to use flag arcanearies, which we're not going to go into today. We don't really have time, but... Does anyone here use flag arcanearies? Okay, that's cool. Yeah, great. So that's a faster workflow. It has less friction, but it's not really for production because we want that friction for production. We want a manual approver. We want all of that. It's good. So as far as the take away for users, you should be using OCI for helm everywhere you can, especially if you're using flux. And that's really the one take away. But as a vendor, what do you do? Do you support both? Do you publish to your legacy and also publish OCI, or do you just go with one or the other? That's not really up to us. From a flux perspective, there's no need to support both. There's no valuable work done by index.yaml for flux. But there may be other reasons that are not relevant to flux in particular that you should support both. But from an end user perspective use OCI if you can, because you're avoiding all that needless waste, you pay for that waste, and even if you don't pay for that waste, it's environmentally friendly to avoid waste. So what if your chart vendor doesn't support OCI? Well, you can talk to them and ask if they will do it, but maybe don't worry about it, because it's really not a big deal. Legacy is probably not going away. For years we've heard that legacy helm repositories would go away. There really is no sign of that happening anytime soon. It really only affects you if it affects you. So the way to know that it affects you is to monitor your performance. And we have a great monitoring guy that's been updated. There's a link to it here. So go ahead and check that out and see if you have a performance impact. If you switch it will be quick and clear. So that is our talk and we can take questions at this point. This is actually the QR code we'll take you to and it'll be online in a short amount. But back to this slide here. Yeah, that's the link up at the top. That's the link you want. I'm not just really familiar with Helmfile. I'm not sure how that works. The question is, is there a locking mechanism? There's no lock file. Yeah, it's either automated or it isn't. If you want it to be pinned to a particular version then you just pin it in the file. So here we have a wildcard and if you want it to be pinned to a particular version for like an incident or something or just to implement this workflow as we've done in production it's pinned to a particular version. So there's no automation. It's not automatically promoting anything unless you do it. So Flux doesn't really work that way. Flux basically pulls the Helm chart and then it just deploys the Helm release and it just makes sure it continuously reconciles against what is there in the Helm chart and what is there in the Helm release. There is no concept of a lock file where you are tracking what has been deployed at what time. It always fetches from the source. So it always fetches the chart and then compares it with what's there in the Helm release and install release. That's all. So Flux is not committing back. The GitHub actions workflow is generating the commit. So I'm not sure it happens in CI. Flux doesn't. Yeah, but if you use image automation then you can probably mix and match. You can have an image policy which says that oh, you use 7.0.x and then the image automation controller commits back and it says 7.0.4 but then you need to use the image automation APIs. You can't just do it with the Helm control. So it's basically an OCI artifact. I think he means the OCI signature. Are you talking about this one? So that's just the digest of the artifact. So it's a .6 file. That's what cosine generates. That's the signature of your artifact. So you can take a look at the workflow. You mean with PGP? But we wanted to demo Keyless because Keyless is just easier. You don't need to handle PGP key pairs. It's always easier to do it without keys. Flux also does not implement the PGP verification. It's not implemented now. I think Keyless means not the Helm problems but the cosine private public key. You can do a cosine keyful verification. Yes, there are keyless and keyless workflows. You just store the... Yeah, you can do that. You just need to store the key in a secret. Oh, there's a mic there. I didn't know that. Hello? How would someone integrate Flagger to do the production release upgrades in this case? In this case, it's really two orthogonal things. You can use Flagger wherever. You can use it in a staging environment or you can use it in production environment. Flagger is not tight to get or Helm. It's completely independent of all of these things. It's just about what's happening in your cluster. As soon as the Helm release upgrade succeeds and the deployment... Let's say the app is a deployment and the deployment changed, like the image version changed or whatever, Flagger would kick into action and it would start the canary process. It does not really matter what your Flux setup looks like. Flagger can be deployed agnostic. You don't need Flux and you don't need Flagger to do Flux. They're completely independent but they work very great together. Just a question on the production deployment side. The deployment was approved in GitHub and then obviously the deployment happened. You didn't actually trigger anything that happened automatically. Flux system is pulling from Git. Is that right? To get the latest changes? We also merged the pull request. After that. I guess the production system still needs to have Git access. You've got some Git access secrets installed in an environment. You need Git access because all your manifest are stored in a Git repository. You need Git access. Do you have any other way of having an OCI backed access? You could do it in an OCI repository. Right now the default bootstrap process uses Git but you could definitely store all your manifest in an OCI repository instead of Git repository and then do it from there. We do want to take Flux into a direction where OCI repository and Git repositories are very equivalent to each other as in whatever supported by Git repositories should also be supported by OCI repository. If you did that, my question would be then how does Flux system know to detect that change? Is it constantly polling the source system like the OCI repository in that case? It would be polling at the interval or you can set up a receiver and the receiver would work for Git repository or OCI repository in a similar way. Is that like a web hook or something? Yeah. In terms of dealing with legacy Helm repositories, I just wanted to call out for anyone who didn't see it. There was a lightning talk on utility to proxy legacy Helm repositories through OCI so that might be a solution for that as well. Great. Awesome. Thank you. Hi. In terms of verifying the signatures on Helm charts using Cosine, you said it wouldn't reconcile if the signature verification failed. Is that always going to fail or is there a way to audit that instead? Could you come again? Sorry, I didn't catch your question. So when verifying OCI artifacts using Cosine, when the source controller pulls them down, you said that it wouldn't reconcile that artifact if the signature verification failed. Is there a way to just audit that instead and output in the law? I think there's an event that gets submitted so you could look out for that Kubernetes event. But the purpose of the signature verification is to be sure that it is from the correct source so I'm not sure that would be implemented as an optional... Like if the verification fails then the reconciliation will fail. There is no try to verify but it's fine if it doesn't verify. That's not an option.


As the present administration of Governor Samuel, or term of being a state, gets to an end, the Executive Council has given approvals for the implementation of policies brought to the Council by various commissioners, as it affects the ministries and prostitutes. The Council in Hasitin today approved an additional furnishing of Government Lodge opening of Roundtown Street in Makodi Town, purchase of pretend security-feated vehicles, and a donation of official costs to outgoing local government chairman and local legislative leaders among other issues of importance. Our correspondents in Makodi, Frances Tehr, who covered the briefing, are now reports as presented from our studios. The Commissioner for Information, who gave a breakdown of items that were given approval of at the meeting presided over by Governor Samuel at home, said that the furnishing of the Government Lodge would cost 43 million Naira, as it is expected to be of a murder standard. In Aligu also said that the construction of roads in the north bank part of Makodi, where accessible roads were needed, would go up a sum of 151.2 million Naira. Because of the inflationary trends, things changed. So, additional fund was approved by the Commissioner that was standing in House. So, he asked for the Council approval of additional 43 million, which was approved, to make sure that facility at the Government Lodge will be up to date. If you look at it recently, we are trying to make sure that a lot of openings in Makodi are made through police, with quality roads. If anybody can go there, you see that that road from Polyfamilica to Church means, yes, modernisation. So, the contract for it was approved today and it's going to be constructed at the cost of 151.2 million, as far as not surface dressing. The Commissioner further explained that with a monetisation policy, officials of the state who recently resigned were allowed to go with their official cars, which has made it necessary for the Government to purchase pretend security free-tiered ones, to facilitate mobility. Remember from Bureau of Local Government, for the waiver of the vehicles given to the local government chairman and leaders, the hill-offs. So, as they are trying to aspire today, they will be going with those vehicles, because they have served militarily and the way a man has served the government, they need to go with it so that they will do something with it. Either all organs of government must be up to date to make sure that whatever we do should be in line with our manifestos and we told the Benos State people that when we are leaving, we will leave a legacy. According to the Benos spokesperson, the Governor-return is ready to ensure that the people of the state benefit from the dividend of democracy before the present administration leaves office in 2023.


All right, let's take a look at this Yeah, I need and you also set me reference If you look at the beginning though looking at this There's something that feels a bit more connected with her looking at the staff There I don't know there's something it is very subjective. I mean, you know, it depends how Performancy you want to do this and how she wants to look at the one or maybe she doesn't have to it Doesn't want to it just seems like throughout the rest of the shock there's a lot of Connection that she really follows follows It's all very nice all very smooth It's all good stuff nice poses. It just feels here eyes closed and then she kind of just looks up and And then just kind of casually grabs this and I wonder If that would be something like in a reference where she would actually look down To see where she grabs, but maybe that's your whole point So I don't know I'm not quite sure about the history of this and I don't know. There's nothing in the email That's not kind of the only thing that kind of caught me Or that caught my eye. It's probably just off It's kind of so casual and weirdly it just felt like a move I'm animating arm up arm down and I just catch it While the rest is so focused where I have to look at it. I don't know. I don't know. I know I'm reacting to this Just that section so much It's also because of just from an animation point of view it goes back this way Let's turn this on here And then it comes back pretty much the same way and it's just one of those things where you don't really want to do You know, you don't want to go one way and then come back the same way If at least there's a bit of a different arc and like imagine this is your a and b and this is your c You don't want to go a to be back to a you want to have some sort of difference in contrast If it's too repetitive if it just makes it too cg Yeah, that's I mean And I'm also have to be careful covering Characterize especially when the eyes are open. It's okay. I guess when it's here was your eyes are closed But I think like that then you could technically just do When we grabs it like this and it's just a different type of grab I don't want to linger too much on that beginning, but that's just kind of something I do like how that staff moves though how the back goes up first giving this more weight. I think that's cool Same thing here with this arm going up. It kind of goes down the same way It just feels hitting one pose and then I go back to where I was Where's the ending here? Kind of ish here. See how it's pretty much the same thing So that'll be my biggest comment for the beginning Once she's here those moves are really nice I like how There's a change there for super polished stuff once the foot's down here You just have feels like just a foot rolled out in but even She moves it over here the foot. I mean with the whole leg and then as As she does the foot roll down you could still have a slight rotation in y With that heel going over here pivoting off the toes. That's for some super polished stuff I was a big fan of foot details. It's like here where it feels like the Foot roll just goes straight up And then the leg comes out But if you're already going to do an arc like this and moving out you can imagine that a As your foot goes up you do a foot roll plus a tilt over this way like your leg Muscles are already kind of firing this way and your roll is kind of going off this way to kind anticipate That rotation over if that makes sense cool There's so much detail. I'm gonna look at where I'm What I'm focusing on. I should technically focus on the root first Not on detail things You got good moves here I feel like through here Just looking at this I will probably reduce the amount of Weight here on that hip and go a bit lower So that when you lift this leg now all the weight is on this because it feels like through here There's still some weight on it. I mean you are leaning on this But you could still imagine there's a little bit weight on this leg And now that you're letting go and lifting this all the body weight is on this hip side here So you could have that side of the hip shoot up just a bit more and right now as that leg goes up I don't really see the difference in the hips. So I'm cool. So you got your Weight on there Now you take a step forward and now weight is on the leg same thing here. You can probably go a bit lower So that when you do this again, you can feel how that hip is taking everything weight wise There's something this is going to be super picky, but I just feel through Here and I know that's the legs swinging over, but there's something this is super subtle, but there's something about the root Just doing this section. It feels like it's accelerating like there's not that much movement here It's pretty still. I know it's I was pivoting off here of that leg but then it moves really really fast and then Stops a bit quickly right there And I know this comes down and stops that move and you do have your hips Straightening which is good because that's this this part of the foot, right? So this is stopping the root going forward. So this shoots up As a stopping point which then should shoot up the hip Which I feel like you have right here But you could push that a tiny bit. It's there There's just something when I watch this Right there just something it's super picky. This might just be like a frame or two spacing, but It's just something where everything kind of moves over too quickly It might even be more what's going on up here So it might be a slight reduction in spacing in the root But it could be because the roots And the head and everything else is moving over this way So quickly and stop so quickly where I would drag this part It feels like the leg is going to swing around and drag and pull everything with it, right? Look at that swing Right there That's the swing that's going to take the root with it and then the root stops During this section right there. I feel like too much of the upper part is moving Maybe that's what I'm reacting to too much of this is moving where this should be delayed and draggy When an overlap through there The other thing is and I'm looking at overall style where It's not super Cartooning or stylized. It's fairly nice and smooth fairly sticking to the reference Sort of timing, you know, I look at our movements. Even how quickly she throws this Up in the air It's all pretty good And the reason why I'm mentioning this although careful you have A lot of movement right through here Especially this That's a big big move.  You're slowing everything down a lot So again a super picky note. This feels too much I would reduce how much the head goes back and the arms because right now we're slowing down a lot right through there So watch this in real time Right there. It feels like a pop But again, the reason why I'm mentioning this why it's so smooth is because when you do this That move right through there that leg could use a frame or two more Going this way meaning it's just a bit too fast Right there it just a tad just a tad It just feels suddenly supernaturally fast And stylized borderline stylized. It's not I'm not saying it's you know, super cartoony. You just suddenly the movement becomes Exaggeratedly fast versus here. None of it is except the section where there's a pop Yeah, I think that's what I'm reacting to then This is weird too I'm looking at the root and back to the root She goes this way And then stops This is the tricky one the speed at which the root goes back And the reason why I'm reacting to this is because the only way she can go this way is by pushing with this leg Pushing down to propel her back if that makes sense in terms of purely body mechanics so In order to do that the only way she's doing it right now is through a foot roll And again spacing wise this this is a lot if you look at the lines here Right there That's suddenly a lot of movement and it's also only over this frame. So again, this feels like a pop And the rest is a bit too smooth but in order to make it work that She moves this far this way The couple things you can either suddenly go down with the root and then go over So that not that she has to fully bend this leg and it would take away from this nice line But if there's any way you can just subtly imply that the root she goes down a bit to push herself off Or if you want to keep this line, she does a little bit of a move this way to anticipate and then you And then go back So there are at least two ways you can do this where you can make the physics work a bit better I'm just not quite buying That quick move through a foot roll and then the other thing that's a bit tricky Again, this is all because of the timing you're setting at the beginning the realism or somewhat realism When she goes back here, this was too fast. So she goes this way that That's too fast because right now she is straight up here The only way she can move this way and fall forward is through just physics She's up here and The off balance of this leg the leg going this way would make her tip over this way But there's nothing where this leg is like this and she can again through a Roll and calves and leg muscles have her go forward. This is purely. I'm pushing over Hold and drop. But if you watch this timing Too fast you would have to go here and give this maybe three or four more frames Just so she hangs a bit more and you can buy it that This leg and this move Will bring her back because this just seems like I'm pushing myself up and I'm dropping But this speed feels like I'm really pushing myself forward, which doesn't quite work Right there that it feels like a magic which you can argue magic magic But in terms of body mechanics, it feels like she magically moves over there And that stop is abrupt too She goes over here Right there Uh, I hate to say this but look at her um What crack here right there? That's where it starts to lock So if you look at that Is even a bit of a pop if you look at the lines here how he goes over and then Snaps into place and then it just kind of rotates over and then visually you have that weird lock of this line This line. Hold on. Let me just bring this back line here You can see how it totally pivots off of that visual line there So it's a bit of a weird combination of you moving over and then pop into place Stop too fast like there should be a stop stop ease in and then come back It's kind of over here. I'm exaggerating how far she has to go over here Not this far, but you want to go over here Maybe till here and then come back Into this and I think that is also going to help you in terms of not pivoting off this super straight hip line here Just kind of the visual weirdness there And it's almost it almost was like you want to add half a second To the end we're getting To the end here. She goes over there I buy this there's a bent leg which means that she can push her leg This way and she's leaning a bit. So I think that way shift I buy but then You go over there Oh, it's the way you cut it. So you have this frame and I have Two more frames of nothing. So I would just cut that Because right now we're heading into energy freezes But even then it feels like you're doing And you almost want It depends what comes afterwards But it's almost like you want to hold this pose kind of ease into this with a little bit of overshoot and come back There's something where it's almost like introducing a new idea and then we're cutting out of it. We're not finishing it So I think that would be Generally for just way shifts and the roots and all that good stuff. I think overall From here on like your hands and your arms seem really nice. There's some good stuff there. Hold on. Let me just go Let me check one arm here. Look at this guy And by this guy I mean the arm This feels a bit weird spacing wise When it goes over here right through there as a nice arc And then right there It's a pop Oh, we're back to where I mentioned the pop in here. You can see it in the arm kind of pops over and then it's small and spacing again Then it has this weird lock of almost through here That wrist is locked in space versus going over this way this way this way and coming back Watch out for that It's nice. That's cool. Let's look at this one Again a little pop there, which we talked about It's nice. Oops Right there Again a little pop there There's something that's suddenly too fast. Look at the spacing and then here from here to here It's suddenly too fast which and again That's where that leg suddenly feels too fast and pops but hold on hold on What is this? Frame there's a frame 121 for me. Hold on. Hold on. Let me just quickly check something here This might be it your render. Sorry. I have allergies. I'm just for some reason this morning really Kicked into high gear Okay, hold on so you have One frame. All right, so we're here And then you have one frame here. Oh, look at that. It's your render Your render is missing frames Ah, see that we're missing this frame. Ha I see so That with that in mind, let me quit this And look at that Well, well, well, I thought well, let's look at the render. It's going to be all nice and I could judge the motion blur so performance wise Be the same at the very beginning but Yeah, yeah, yeah your your pops are gone Ah, okay. So all my pop comments, I would say This regard it's like This regard a lot of my first 18 minutes. Holy moly But interesting that's still good to know so that if you put your render on your demoreal, um red flag don't And put your playblast on there check your render All right. Well, let's continue then Looking at arms Yes much smoother So we were stuck with this arm here. I have I'm suspecting this is an ik arm Or it could just be a visual Thing goes right through there, but it feels like it was a bit too straight giving this a little bit of an ik arm feel So how the wrist is kind of changing but that I don't know that's kind up to you to check This seems pretty good even though through here That arm seems almost a bit separate compared to what the rest of the body's doing giving this potentially An ik thing but right through there. It just Too ik is for me even if it's fk.  We're we're in a sticky frame again where that wrist just kind of locks Then comes down a bit too straight. So if you're going this way, you want to maybe that kind of a move A bit more of an arc in there if you're coming out here, which is nice, but here you're just getting into Sticky frame right through there coming down a bit too straight. It feels a bit too ik-ish Hold on. Let me see No, you still have the sticky frames at the end. So I was still cut out sooner Looking at the head So same This the stuff that I talked about at the beginning still applies how this comes back the same way same way We're no eye contact, which could be subjective. That could just be me But definitely none of those The pops in that leg and this way Yeah, and even there was a pop at the beginning as you go up through here That is all good. The staff is good for sure There's some funky right at the end here. Let me do this. We're back to This is doc those It's a tricky thing every now and then you have those visual things where it's kind of like the head is stuck through here Where everything moves, but it's almost like there's a clamp on the head And then through here visually it feels like that wrist is stuck in space Where it moves and then for Two frames see that it's kind of pivoting off here And then suddenly it goes up again This feels ik where it goes straight up, but then the wrist is kind of locked I have a high suspicion that this is an ik arm But to go back here in order to avoid this you would have to kind of move over with that arm And you might have to help yourself with moving the chest over a little bit And this is a bit of a strange feel Hold on It's very very picky, but there's something about It's almost like this elbow is locked. I know it's not. I know I see it going over to the right But there's something about here's a dividing line And now this four arm rotates this way and then the upper arm rotates this way And it feels like there's that that line where we just kind of equally evenly Unfold this arm Where you might help yourself with raising that arm a bit through this move So that the um the upper arm is a bit in a different spot And giving this bit more of an onwards unfolding and then you come back down into what you have here Little picky picky visual things Excuse me stuffy nose, um, then we talked about What's it called foot rolls a couple of things in your feet here Same thing here. This just feels like you're just activating the foot roll in that one channel one axis That's it. Even though she's leaning this way towards us So you're gonna have to pivot off this side of this shoe and tilt over towards us So if there was a camera here, this would feel probably this is the shoe and then this is your your shin Versus shin like that and then the shoe is like this, you know, I mean you still have a slight curve But it would if that's your Toe and that's the backside of the of the boot and then here's your shin, you know, you want to pivot off this side so that you have I hope it makes sense here so that then the Toes of your boot are like this Then you have your the backside of your of your boot and then your shin like this So you're kind of leaning more of this side, right Not sure why I need to go back to red but let me just do it And then speaking of legs carefully you have a bit of an over extension of your ik Then pop Pop into this and you pop out You can see how far you're kind of crashing your Achilles heel here But you can see that big pop. There's a lot of movement while the foot doesn't really react to it Given this that ik feel again where everything's kind of cut off That feels a bit better Even through here as she goes forward Um, you could tilt one way or in y By rotation where you have the backside of the foot going either way What is she doing here? It feels like she can go almost come towards us for more profile But just be mindful that throughout all of your, um, animation that when you have foot rolls Do not just activate the foot roll until it just goes up in in one axis So straight up you have to look at well, what's the angle of my leg? Where's the body leaning towards or going away from or and so on and so on Then you want to kind of make sure that the the foot roll is following that angle if possible get a nice Nice thing pose. I really like this how you go from this to that Into a nice simplify triangle Then changing the hands again This might be a bit weird that's finger just being separate through there. I'm not super fan of that one I mean, you know, some of those things are just between you When you frame through it if I watch this I don't see that finger. So like some of my comments are for sure ridiculous, but Framing through this one finger feels weird but with motion blur Let me just ignore the pops, right and look at the render version I'm gonna bring this up here so we can see What that hand is doing blur wise You know what I mean? I mean come on If you have That much blur. Yeah, you know my finger comment is not gonna matter But why you know we're frame by framing things and It's a little just but I see some finger waggle there, which is nice There's some nice stuff there Like that you're changing into something different, but again has like a one specific finger out But again the blur you wouldn't see it and then just look at Yeah, just really just really bugs me where in terms of this is pose a pose b Back to a there's not even a bit of an overlap or drag in the head. I mean there's a little bit there, but Just watch out for that.  I thought I saw some other Head issues in terms of drag and overlap That's okay. She leads there with that tilt right there, which is really nice That's cool like through here if she pushes herself off You could have she feels a bit It's staying the same line. You can have a little bit of chin drop like she could drag that head just a bit more Through there, especially through this this still applies that weight shift that feels like she's being pulled That has nothing to do with the one frame pop. So that still applies and Let's pretend you want to keep this even a bigger reason to Bring that head up a bit for some drag That's all cool here. Then she leads with the head at the end Cool, I think I'm gonna leave it at that. I think that's about it. Whoop. I'm watching something with that hand On this side Right there. It's my dog winding out if you can hear that. He won't still go out I think what I'm reacting to in real time is how you're doing this and it's It feels like you have a sudden straight line again. This could be just your ik thing right through there What I'm reacting to is that you're having this Then for a short moment a straight line through there And then a drop And then through there It's kind of that reverse where it's almost locked here But there's a lot of movement in the elbow And then we go back up and then the wrist drags Meaning that right through there again, this feels a bit like an ik arm Right there. There's just something funky, but it's also mainly how you go here then you flat now and you go straight down like this It just has this kind of feel versus giving this Just smoothing out a bit through something, you know where your your um curves a bit softer This still applies this still feels a bit slow and straight even I mean you could Go over here hold so at this point that arm would be further out this way and then through this Speed up that part just a little bit It also feels a bit soft and even in timing Right through there and I'd go I know she's slow and elegant, but Or just a bit even and I think that even is in timing combined with that straight line of that I suspect ik arm Gives a bit of a weird feel All right, I'm gonna end it here. There's so many picky things you can find, you know, there's always Again, I know I would look at all those little picky notes Fix your render in terms of the blurs.  It feels like that it's a stump Where that feels fairly broken So you want to bring up that wrist a bit here. I will bring up that wrist a bit here So it's it feels a bit more like that, but even that feels almost broken Like you want to go up a bit higher But the reason why I'm saying this. Yeah, but but but it's just it's fast and it's going to be blurred So it's kind of to you. Do you want to go full hardcore and really fix those poses and even in the blur? It might feel nice Even though looking at this now, I have to say I would fix this it feels really broken But again this in front of the hat with the blur. I don't think you're gonna notice so You know take a lot of speaking notes with a grain of salt in terms of well, if I fix the pops though But I add the blur back is that really unnecessary? All right, leave it at that half an hour critique. Um It looks good though. I mean, I like it. It's cool shot and the render is nice with the glow big fan of that Um, but yeah, I'm gonna leave it at that. Thanks everybody for watching. This is a long one. Thank you All right, there's an email you can sign up you can start whenever you want you can submit whatever you want You get 16 submissions either way a like and subscribe would be awesome. All right. Thank you


card number 20 world anti-communist league check this out death squad convention Buonas eras welcomes WACL I don't know what that stands for who are these people South America everybody wants a piece of the pie who's that general squad convention let's take a look at this number 20 world anti-communist league founded in 1954 by a group of Southeast Asian drug lords the organization now known as the world anti-communist league WACL world anti communist league went international in 1966 thanks to the efforts of KMT leader Xi'an Kai-shek South Korean politician Park Jong-hee Japanese right-wing gangster Radici Sakawa and Yoshi Kodama on South Korean Messiah Reverend Sun Ming moon oh wow former CIA officers assert that the US government acting covertly through the CIA provided seed money for WACL via Ray Klein CIA station chief in Taiwan from 1958 to 1962 and later CIA deputy director of intelligence Klein continued to attend WACL that's the world anti-communist league conferences well into the 1980s and his WW2 OSS associate John Singh lab became head of the American chapter 1981 the 1980 WACL conference was held in Buonas Ares Argentina hosting the meeting was General Carlos Suarez Mason architect of Argentine's drug financed dirty war against his own people general Louis Garcia Mesa and Stefan del Chai leader of the leader of Bolivia's 1980 cocaine coup were on hand at the conference these men with met with paramilitary leaders from Central America to discuss using dirty war tactics in their nations Mario said Sandoval Alar Alar Khan Godfather of Guatemala's death squads and Roberto do Busan L of El Salvador who had trained in the WACL paramilitary or military Academy in Taiwan and ordered the death of Archbishop Romero also attended holy within months Argentine advisors took command of dirty war campaigns throughout Central America tens of thousands of civilians had died at the hands of WACL inspired and US funded paramilitary death squads and Contra Force forces since then we know to the connections well we learned about the Bolivian one and this one right and involved with assassination of Archbishop Romero and what was the other one oh yeah Sun Ming Moon right the cult in Korea holy


Okay. Thank you very much for coming. I know it's the end of the day, and it's great to have a full room. We'll try and get through this quickly so you can get the beer from outside and go to the parties and everything else. But hopefully this full attendance reflects the importance and interest in this topic, which we're definitely seeing out in the field. So just introduce myself, Neil Levine. I do product management at Red Hat for Steph. I have Sean Cohen who's on the product management side on the open stack side, and we have down there Gorka who might join us for a little brief demo halfway through who's one of the core developers on Cinder. So this talk is, we're just saying here, we did it, I think it was a last summit. We'll probably do it again at the next summit because it's definitely a work in progress. For those of you who went to the Cinder talk this morning, we'll have heard some commentary on the state of volume replication and volume backup and so on in Cinder. So we're going to cover some of the options that we think are out there at the moment. Some of them work pretty well, some of them not so perfect, and we're going to show you where we're going here. So first of all, we'll explain a little bit about just how open stack and Steph works from a multi-site perspective to start off with. Does assume you have some basic knowledge, but we'll go through some of the components. Then we're going to go through four use cases or topologies about how we think you can lay this out here, which have varying levels of complexity and trade-offs and elegance. I will sort of paint the picture of where we think this is going out certainly in the Liberty session towards the Nirvana of perfect multi-site and disaster recovery. So with that, I'll hand over to Sean to kick things off. All right. Thank you, Neil. So the first topic is of course looking at the open-stack disaster recovery. This is actually my third talk on the subject, and as Neil mentioned, this is an ongoing effort. We cannot finish it in one cycle. That's obviously, and it involves a lot of services. Yeah, it's blocks, storage, it's just a simple piece. When your site goes down in flames, you need to all the services being available on the other sites actually to restore, and we need, there's a building blocks to take place. We look at the different topologies we have for open-stack. So it starts pretty much what I call disaster recovery for the poor. So I have a cluster, let's stretch it between two sites and see what happened. Then we have one open-stack cluster, and of course, moving to two open-stack clusters. When we look at the R configurations, this is where RPO and RTO comes into play, and I have various choice all the way from having an active and cold standby to an active and hot standby, and of course, the holy grail is active-active, which typically costs more. It seems go if you look at Amazon, for example, or any other commercial offering for Cloud as you would like to get more RPO and RTO, you will have to pay more. It has to do with the investment in hardware, but it's not all. So if we look at RPO and RTO, the basic thing you could do is just have tape back up and ship it with your truck to another site. That's the baseline, and then we move on to if I need more, then I can do replication, which is going to be a synchronicity, synchronously, and then I move up and up, and of course, mirroring is like the top one. But even if I have mirroring, which is continuous mirroring from one site to another, if I have a failure at the application layer, it will be copied as well to the other site. So you might have another disaster in the other site as well. This is why you need to have snapshots to protect you in both sites. You can actually restore the last operation. So that's pretty much where we are in general, but what are actually involved when we talk about disaster recovery with OpenStack? So if you look at the ingredient, data is not enough. Let's say I have a method to replicate my data to the other side. Am I done? No, this is just a start. So I need to capture the metadata of the relevant service workload resources in order to be able to restore it in the other side. I need to ensure that the VM images are present in the target site. I'm not even opening it here, but we have different configuration in the two sites. Meaning if the VM configuration has different IP settings, I need actually different settings in the other side. This is where we need to leverage heat, for example, to deploy different configurations in the target site. So replication of the workload data using data replication as a start, we can have application level replication or backup restore, which is the initial fundamental step to disaster recovery. So let's start with the OpenStack components. And as you know, Cinder has its own API for doing backup. And what notable milestone in the Cinder backup progress is actually the ability to do a backup with the metadata state. So if I'm actually backing up my volume, I can actually back up the metadata as well. So if I import it even to another OpenStack site or even brand new install, I will be able to take this volumes and restore them. In terms of AJ, just like any other service in OpenStack, we have working in AJ pair, but within a single site. Today, Cinder doesn't have the notion of additional Cinder in the other end to talk to, right? This is my own domain, I'm just one Cinder. So we don't have inherent multi-site or disaster architecture built in to Cinder topology. When we look at the APIs, we have volume migration already built into the API. There's other work to be done there, but that's an important milestone. We have the backup API as I mentioned, and we have the volume replication API that was started to add in ISO. There's more work to be done as we're gonna move on. And when we look at the OpenStack glance, the image repository, right? I need to take the images and make sure I can actually input them on the other side. Similar to topology, no notion of AJ, we have a notion of AJ pair, but with no notion of a multi-site. And of course, we have the Gliance API that we can utilize. Moving to Nova. So here we need the metadata of the database and the volume, right? Because if I need to restore mobile in the other site, I need to have the metadata kept or backed up. And in terms of topology, same thing, right? AJ pair, single-site, no inherent multi-site. When we talk about cattle, right? If should I back up my ephemal volumes? We wouldn't recommend it. You can actually use Snapshot to protect the ephemal volumes for your boot volume, but that's not something I think we're in the scope. And of course, this is where Snapshots in Glance comes handy. With that, I'm gonna hand it off to Neil and I'm gonna cover the Ceph components. So Ceph is slightly different to OpenStack. It's got a very different architecture. I won't cover that. But the main way that Ceph is used within OpenStack and the reason it's become so popular is as a common storage backend, both Glance and Cinder and Nova. And this is because it does copy and write. It can do thin provisioning. So it's an incredibly versatile and sort of good way of doing storage for all of the different services. The main mechanism we have right now with RBD is what we call RBD exports. And it's a sort of a fairly low level tool where you can actually export an RBD volume as an image. It actually comes out as a file that gets put into, you know, standard out. You can pipe it into standard in somewhere else. So it's pretty low level, but it makes it versatile. It means you can encode it into scripts. You can use SSH pipes and so on to sort of move images around. And critically, when you do these RBD exports and imports, they're incremental by default. So the first time you run it, it'll take a full dump of the image. The second time, third time, it'll be incremental only. So you don't have to worry about, you know, am I, is it incremental or full? It'll just do it incremental for you by default. So it's a pretty versatile. What we're currently working on, or I say we did, the engineers are working on is what we're calling RBD mirroring, which is sort of the next stage of that, which is volume replication or to work with volume replication API in Cinder. So right now we're still putting in the low level sort of foundations and we're sort of targeting that work for the beginning of next year. But this will allow us to sort of do effectively live streaming of an image from site A to site B. So you don't have to say go and back it up. You just say once replicate it and that's ongoing for any image within a pool. And if you're interested in the architecture, please join the mailing list where you can sort of get involved with, you know, suggesting how you want to use this feature. But that's currently in progress. People also use the Rados gateway or RGW, which is the Swift and S3 interface to your CIF cluster. So this is typically used by tenant applications rather than the underlying infrastructure itself. If you're using RBD everywhere, you may use RGW for your backups as you'll see. But typically this is about your user's data. The multi-site capabilities there, we have what we call sort of the V1 implementation, which came out about a year and a half ago, which is basically it's an active passive pair, but explicitly between sites. So an eventually consistent layer that sits on top of your CIF cluster, which will mirror the data from site A to site B. I think it's fair to say it was kind of like, you know, we call it a V1 with all the implications there. It works, it's a little bit sort of brittle to set up. But once you go up and running, it does its job. But what we're working on now is sort of a V2 implementation of that to sort of get to a sort of a full mesh active cluster design for multi-sites. So slightly closer to kind of how Swift does it, but still with the strong consistency of CIF at the low level in each individual site. Okay, so armed with knowledge of OpenStack, which has, you know, databases filled with metadata and they're running in HA pairs and they have some API capabilities and sometimes they don't. And hopefully armed with a little bit of knowledge about CIF. We're gonna explore sort of some of the different use cases. And we've got four that we've sort of lined up in order of complexity and sort of features. So the first one of course is, you know, the most obvious thing, hey, I just wanna take my clusters and stretch them. This is not recommended. Both OpenStack and CIF were not really designed to run across WAN links with high latency, you know, across geographies. They can work in campus environments and we definitely have users doing that in campus environments. But you gotta pay a lot of attention to the latency of your links, the reliability of your links and so on. So it's kind of, you know, phone up for advice from consultants and everything else if you're thinking of doing that. But as a first order step, this is not really what you should be doing if you really are thinking about sort of proper DR between continents or between coasts of a country and so on. If you do it with CIF in particular, you have to pay a lot of attention to where you're putting your monitors. You know, there's a lot of settings that you have to sort of tune to make it aware that it's operating across high latency links. But not really recommended. So the first use case is really just using Cinder Backup, which, you know, we call this the Control-Z capability here, which is a very user-centric thing.  And you want to sort of give the user an option to sort of get back from that. And that's really what Cinder Backup is for. So here you're gonna take one OpenStack cluster, you're gonna keep it in site A, but you are gonna provision two separate CIF clusters in different physical locations, one which is the origin and one which is gonna be for the backup. So whilst this has kind of been designed for end users, we have some ways of allowing the admin to take advantage of it as well. So here's a kind of a very simple map of the topology. And yes, it's simple, which means it's simple to set up.  But, you know, not that fully featured in terms of sort of giving you a sleepless night if you're an admin here. So the issues you have with this, of course, is your Cinder service in site A has to be very, you know, that's the authoritative one. You got to make sure that the data is looked after, you're sort of keeping all the metadata safe and backed up and so on. You can pick different back ends, doesn't have to be SEF, doesn't even have to be RBD. You can use RGW or OpenStack Swift or any other objects, or even S3, you know, Shim to sort of to move your backups over. But as I said, if you do use RBD, it's incremental by default, which is a huge benefit if you're moving very large volumes across. The limitations when you're using Cinder backup are that, you know, the volume can't be in use. It has to be unmounted. It doesn't really handle backing up of snapshots. It's a fairly primitive API call, which really just works for sort of simple, unmounted Cinder volume. You want to send over to site B. So again, as a user, you can do that through the API or using Horizon. But the goal is to make admins take advantage of this with this topology to give them some sleepless, you know, get over the sleepless nights they may be having. And I will introduce Gorka to explain some of the work we've done here to improve this tool for admins. Yeah, I think everyone will hear me. Okay. Okay, in this next cycle, we are going to address some of the missing features in Cinder backup. For example, we are going to decouple the Cinder volume from Cinder backup service. So Cinder backup will be able to scale out. We'll also include the snapshot backups. And we're looking into scheduling the backups as well. In the meantime, how can you solve these missing features is using, for example, CronJob, Cinder, Client, Python API, and some scripting. This way, you can create automatic backups. For example, the administrator can backup all tenants' volumes and choose whether he wants to make these backups visible to the tenants or to keep just them hiding. You can also export all the backup metadata so it can be automatically imported later on. And for example, you can, as well, create a backup for induced volumes, which means that you have to create a temporary snapshot and a temporary volume to do the backups and then delete them. And you can also control how many backups you want to keep in the backend. So then, with a sliding window, and you delete the other ones. And now we are going to see a small demonstration on what this kind of script would look like. And something important to notice is that when we are going to run the list commands is three commands altogether, which will show the sender volumes available. It will show, as well, what the sender backup sees as the backups and what the script sees as the backups. And we will have, this is a very basic demo, so it's easier to follow because we will have only one demo tenant. No, did you start already? Yeah, no, no, nope, nope. Where's my control seat button? Yeah, exactly. Okay, the demo tenant will have only one volume, but it will be induced. It will have already one automatic backup done. And the admin will have one volume, as well, this one will be available. And it will also have one backup of that volume. So, here we run the list. We see that we have one volume induced. There was supposed to be some call outs showing where everything was, but okay. Now, this is the administrator view, you can see it there. And he also has one available volume, but he has two backups because he's seen the tenant's backup since he did it himself, even though it's the tenant. Okay. Now, we are going to fire the script to create backups for all the tenants. We are going to keep only one backup store in the back end, and we are exporting the metadata back to an FSR directly. And we can see how it is for the on-life induced volume. It creates a snapshot, then a volume, then does the actual backup, then deletes the temporary volume and the snapshot. And it has to remove the old backup because we only want to keep one alive. So, it will replace the old one. For its own volume, it only has to do the backup. Okay. Now, we will see the list of the volumes and the backups. And eventually, okay. There we go. So, what we can see here is that the volume IDs are, okay, no, no, no. This is going to be difficult to show. Okay, just a second. Okay. The volume ID is reported by Cinder, reports the temporary backup, the temporary volume ID, while the script is smart enough to know that that was created through a temporary volume. So, it reflects the original volume ID. Now, this is, what is, okay, sorry. With the callouts, it was a lot easier to follow. Okay. The admin list, there it goes. Okay. I don't know why it keeps. Okay. Then what, okay, I'll just hit and I'll start talking. It will eventually get up. Okay. What we will do next is detach the volume for the demo tenant, delete it. And as an administrator, also delete the volume. And delete all the metadata from the database. So, it simulates like we are in a new environment. So, when we do the restore, we also import all the metadata back into Cinder. Okay, there we deleted the, using my SQL. And here we do an import of the metadata as well as the volumes. And they, then they are back there, all the volumes from, visible from the demo tenant and the administrator. And now Neil will continue with the second use case. So, the goal here is that whilst the Cinder backup API, again it's very primitive, but you can sort of put a reasonable amount of scripting around it. So, as an admin, you can back everything up. You can make it available to your users if they didn't decide to back things up. And you can restore things if you want to in a reasonably clean state.   This is obviously only specific to Cinder. So, what we want to cover is, well, okay, well, if you are using RBD to back both Cinder and Glance, and you're using it for Nova as well. So, you're not just backing up Cinder volumes, you're backing up everything. How do you do that? So, this is what we call the admin warehouse. So, again, there's only one OpenStack cluster. You've got one active OpenStack site inside A. Again, we're going to run two self clusters here, one inside A and B. So, you know, less of the ploy still here. But this is not really driving it through Cinder backup. This is kind of taking things at a slightly lower level here. And really, this is kind of closest to the sort of tape backup. This is really, again, pretty primitive, but it's effective. It gets the stuff backed up. And perhaps adds more onto the restoration of the data. So, here we're going to, this is where you essentially, you're just taking a MySQL dump of your Cinder and Glance databases. And you're using RBD export to ship the data out. So, again, if you're using RBD export repeatedly, so it's taking the incremental backups, then you're going to get, you know, that shouldn't take too long, depending on your size of your cluster. And if it's absolutely huge, and this probably isn't ideal. But if you've got a reasonably manageable number of volumes, and you're just synchronizing the dumps, the MySQL dumps with the RBD exports, then you've got your data in site B. It's all there sitting there. Again, it's kind of sort of like a tape, in a sense, you know, it's all backed up, kind of be slow perhaps to restore it. This is not about a speedy recovery, but it's about having a fairly safe and easy one, which you're controlling, not using open stack services, but really the lower level components. So, MySQL dump, if you don't know how to use that, then I think there's other sessions you might want to attend. But there are scripts, there's one here, which go through an entire pool within Ceph. So, if you've got separate pools for glance, and for Cinder or your images and volumes, you can run the script, or just take a dump of all of those repeatedly, and push them out to a second site. So, it's pretty low level. It won't handle sort of cleaning up lots of snapshots, it won't do grandfathering of images and so on. But, to restore the data, you essentially import the MySQL, the .sql text, and you reverse the streams on the RB export script and just point it from what you've already got inside A, and just point it back to, sorry, what you've got inside B, now push back to site A. So, pretty primitive, but it gets the job done. And so again, if you're trying to avoid a sleepless night, this is the most sort of low level, non-open stack way of taking a lot of the backups. So, to go through the next two final future use cases, I don't know if you're sure. All right, so, thank you Neil. So, we saw that there's a lot of things that can actually be done already today, even scripted, to save you the hustle, but there's also more front door approach, and there's work to be done, and with that we're gonna actually move to the failover side. So, unlike the other two topologies, these topologies are actually based of two open stack clusters, and two self clusters, right? The backup, unlike in the previous one that can be done from the tenant user, is an admin, right, the operator responsibility. We're still talking in this use case, active passive as before, but we are using low level tools to allow us to handle the backups. So, instead of using just MySQL export and dumps, or RBD experts, we're actually using MSQL replication, and here we're using RBD experts as well, and this is how it looks. So, we have two senders, two open stack clusters, we have SQL replication for the sender, we have MySQL replication for Glance, and we use RBD experts between the sides. Now, so the replication is, but not including the AJ pairs, right? But we're doing between the two nodes, and unlike active active configuration, and I think Neil touched upon it a bit, it's the consistency between the data and the database is not guaranteed, because I'm the admin, I need to control when to take the export, right? I need to actually align the consistency windows, and again, you can automate some of it, as we saw earlier, you can schedule cron job to do the work, but it's still admin work to make it happen. So, that's actually already two open stacks, two set clusters, we're getting to a real failover phase, but there's more to it, and that's where we wanna go. So, our golden disaster recovery option will be actually an open stack live disaster recovery to a disaster recovery site, and how are we gonna do it? Through the front door, instead of using tools and scripts, we can actually use the front door open stack that has notion of the fact it's being disaster recovery to another site, and today we do it, the way to do it is actually using the front door APIs. So, Cinder, as you know, already has Cinder applications studying ISOs, we are working on improvements, as we speak to the Cinder application. As Neil mentioned, we are working also on the RBD mirroring that's gonna make usage of the volume API replication of Cinder, and at last, we can use Glance replication. The caveat in Glance, we need to use the same FSIDs on the backup clusters to avoid misconsistency, and if you look, this is just, I'm including a link, by the way, the slides will be available for download, I'm including the barcode at the end. So, if you don't know Glance Replicator, it's pretty much a cool tool, it allows you to do Glance push images, copies to the other side, and that can be leveraged. It's not built in as API, but as a separate tool, but it pretty much does the same job, it allows us to push the images to the other side, which is a better way than just doing the exports. And this is what's coming up in Liberty. So, I mentioned that volume replication has been here since Icehouse. However, we are trying to get more and more into volume replication, one of the things is actually aligning with consistency groups. So, the worst progress in the kilo release on consistency groups, but we still don't have the connection between volume replication and consistency groups, so we need to put both of them together, and the fact that we still have replication within the same sender implementation, it's not yet between sender. So, volume replication, Victor, there's actually design sessions this week on this topic, so if you're interested in it, please attend, and this will actually solve the problem of data replication between two single deployments, and with the CG replication, it will allow us actually to synchronize the different volume types as well, what needs to be captured within the replication, because as you know, we are serving application, the application workloads maybe spread on several volumes, including database, logs, et cetera, we need to capture them in consistency state. So, if you connected what I said earlier about the pain of the admin, just to do the consistency checkpoints, here it's pretty much will be built in into the API. And of course, we'll miss scheduling, et cetera, that will come next, but as you know, in OpenStack, we take around our approach, and I think that's the way we're going. And with that, I wanna hand it back to Neil to summarize. As you can see, multi-site backup is not a simple thing, and it's fair to say it wasn't designed into the core of OpenStack at the very beginning here. If you put all this to one side, there's a sort of philosophical thing here, which is if you wanna really do proper multi-site, really you take a cloud approach, which is you push this up to the application, and you say, look, it's up to the application to pick between multiple clouds, and ensure that the data is stored in multiple locations, and the application is handling the failover. This is kind of very much the public cloud, Amazon-style way of doing things here, but recognizing in the private cloud, you've got admins who have eyes on them, and they need to sort of ensure that it's backups. You have to do something right now, and this is kind of what these use cases are designed to do. So just to review here, you've really got a choice between making this a user-driven option, where you're using sender backup, which you can then optionally use scripts like the one at Gorka's developing here to sort of make it an admin-driven workflow if you don't trust your users to be doing backups. If you don't wanna just make this focus on sender, you want to expand this out to cover all of the storage that's perhaps backed by something, SepRBD or you're storing in general with your images and so on, you've got to choose. Do you really wanna go for just let me get it backed up and gonna take me a while to restore it, but at least I know it's there in some state. Or use case number three, which is I will run Active Passive, I'll accept some caveats, which is there might be some inconsistencies which I have to clear up manually, but at least I know I can fail over to a site B in some format. So you have to pretty pick between a medium and what we call the advanced use case right now. But as you can see the goal here, which we're working on both on the Sep and the OpenStack site is to really get to a true active failover site configuration and the work has all been done there, it's just gonna take a time for it to all mesh, but hopefully this has given you some ideas and in a few minutes we've got left, if other people are running DR in different ways, I'd be interested to hear your options and how you're implementing it, but otherwise thank you very much for your time. And there, thank you. Thank you, and with that we're gonna open it actually for a short Q and A. There's, as I mentioned, there's a bar cut to the slides with the relevant links in there. And if you have a question, please use the mic, yeah. In the case of the passive active, I mean active passive topology, the amount of nodes in both sites has to be the same or in the B site can be, I don't know, one single node or something like that?  Certainly on the left side, you can choose different policies around, you might want to have fewer replicas, so you have fewer nodes because you don't want to have the same level of data integrity. Certainly you might want to compress several of the services and co-locate them, recognizing that site B is not gonna have the same kind of latency and performance characteristics of site A. So now you can definitely save money by compressing things in site B. It depends what you're trying to offer your users. But as you're just copying the backup and the underlying topology that's supporting those backups can definitely be different. I have a question. How is RBD mirroring different than having a crush map that puts like two or three replicas to a different data center, to the backup side? Yeah, so if you're doing things at the RADOS level, RADOS is a strongly consistent storage system, right? So when you're doing a write, it needs to get a successful act back from each copy that's been written. So if you're doing that over sites with high latency, that can take a long time and applications are potentially gonna be very sensitive to having that high latency. So yes, I mean, if you do what we call split site RADOS, you can in theory do it, but it can definitely have an impact in your performance which is expecting and wants a very quick response back. This is slightly different with mirroring is where it's not at the RADOS level, it's at the RBD level. So you're worried about copying the entire image, not just the RADOS objects underneath it. So you'll have a fully formed copy of the entire image in both sites. But as kind of relates back to the earlier question, you may want to have them differently stored. You might have say a three X replica on site A, but you may only want a two X replica for the site B. So right now, doing that stretched RADOS S, you could do it in a campus environments, which will give you a similar kind of thing here. But really most people aren't doing backups in a campus. They're doing them geographically and at that wide geographies and at that point, you don't really want to be spreading your crush map that far. Okay, thank you. Next question for Sean. Hi, two parts. One is some of the basic considerations of Cinder replication, all these things. Some of them would be again applicable for Manila. And Manila is big in liberty. So are we thinking of some of the things at least on the replication backup shares for Manila now that we kind of have a control on Cinder? So, thank you for raising the question. I was almost at the top of the tongue today this morning when we had the Manila session to bring the disaster recovery, also in that scope. But yes, the overall, yes, it's in the scope and it's part of, as soon as you put your eggs in the open stack basket, you start to have those concerns. As I said earlier, I believe the work will be incremental because there are bigger, fresh defects and there is delta right now between Cinder and Manila. Manila still has a way to go in terms of implementing replication, which is not there yet, consistency groups, which is not there yet. So there is a parity yet to make it. The other smaller question is, now that we are talking WAN and other issues, are we thinking of compression-enabled extent awareness or over WAN, all those things which we had in traditional issues, compression-enabled application? I mean, it's more about on the storage side of things. I mean, certainly that's, it's not part of your initial design for RBD, unless Josh wants to tell me differently here.  But yeah, I mean, it's something we'll have to look at, but it's not part of the original scope. You know, that compression, it's about sort of getting as eventually consistent window down as close as possible. But I think that's an optimization once we've got like the basic architecture in place. And again, it depends on the back-end implementation, right? Some of the back-ends does come with the compression build. Would the demo, is that also available at the site? The demo you had, you know? It should be, yes. Yeah, it is. There's a link to the full demo with explanations, et cetera, that didn't come through. Yeah, because you do it offline. I want more questions. So with data backup, we are pretty much clear with volumes, with images. But in a situation when I need to restore or if I want to have a DR site, what is the approach for backing up and for backing up networking and launching virtual machines into the same networks, into the same floating IPs, into the same load balancer groups? It was outside of the scope of this talk and it's a very good question. So you are invited to my next talks on this topic. But again, as I said at the beginning, disaster recovery is a larger umbrella than just the block storage because there's networking, there's compute, there's other, there's as I mentioned, heat as part of the orchestration needs to be take place. So this is just, today we try to tackle just the storage side, but yes, as soon as you're already dealing with disaster recovery, that's pretty much the next door. And again, Cinder I think right now has the only a founder API to maintain the backup with the metadata, but there are other ways to do it as well in network and compute, et cetera. So it's not fully active, active as you know. But as I said, there's incremental stuff that can be done today, just like we showed that there's things you can do already today. Okay. When is the next session? He's had three sessions already today. He's kind of worn out. He's not thinking about tomorrow. Open the schedule, yeah. Yeah, yeah. So I think we have the road to enterprise storage session on Wednesday. Yeah. Forferty, thank you. Thank you. Thank you.


What is up a welcome back do you like to do it feel gonna make it so do we and we have a new video each week This week. We're moving out moving on up. Yeah moving somewhere else Yes, yes, so we shared with you a couple of weeks ago that we had rented a new space where we can move all of our production move our filming and The painting paint station Other ship in we have room for retail space up front, too. It's pretty empty right now We got one cool table in there right now all ready to go For the rest of its friends, but for this week this week was move week So we weren't able to fit in a project this week because we are too busy moving into the new space Warehouse shop workshop. You don't know what to call it I tried calling it the K and G labs, but Kim thinks it sounds too much like a meth lab Yeah, I don't I don't know what we're cooking up in here, but I'm not sure cooking up goodies. It's DIY I don't sure you're I'm not sure you cook up DIY. So so what do we call it? We that is something we keep trying we keep throwing out different things and our family and friends are like What are you doing? I was like how did that one sound? Warehouse, I was I was shop. What is it main office corporate? Yes No, no enterprises. Nope. I don't I don't know the hangout right now. It's just called the space Yeah, we're just calling it the new space eventually. It won't be new So it'll be just be the space the space now go to space That's not either we don't know what it is yet But what you guys can give us your comment below if you've got an idea. Yeah, what we can call it Do we call the new space? Well with all of that said we did not have time for a project So what we will do is show you exactly what we did do this week. Yeah, we'll take you along our little move journey Step one we're gonna gather all of our friends and family and trick them into helping us Step two and now we're gonna prep the space first We needed to know will everything fit and where is everything gonna go before we have our friends and family? Start lugging this stuff around. We want to be able to tell or show them where it's going I don't know if you guys know but Garrett was a systems engineer prior to our DIY Lifestyle and so he is very adamant about making sure all right. Let's mark off the space I'm pretty detail-oriented too. So it doesn't hurt my feelings. I like yeah, let's do it So we get some chalk and we're gonna make sure everything fits. We're gonna McDonald's this thing Yes So we went ahead and mapped everything out so we should have the steps drawn all out Everything looks like it's going to fit. So now we're gonna clean the carpets and paint some of these walls That's what we did on Sunday. My sister was like I'm here to help And so we were planning to paint one room the office where we will do well Garrett will do this editing and we're out Paperworking pay the bills and so We started there, but she was like I'm here. We've got people. Let's paint another room I was like, okay. Well, then our filming studio, which is where we are right now Needs a coat of paint. So the walls were a dark gray and I think that's too dark for filming We need lots of light in here when you film. So we painted the walls white You're very productive. I've cleaned all of the carpets We borrowed a friend's carpet cleaner and we cleaned all the carpets and they were a nasty They were there were a lot of snacks left on the carpet grease and grind surprise. We don't have for your friends in here Step three we're moving out. We're taking the show on the road Touring the country or just moving 30 miles down the street. Yeah, it's like the last one We took inventory of the two rooms that we have we're moving two main rooms for this first initial move We're not moving shipping yet because a laser hasn't moved yet So we needed to move our filming studio and our paint room two rooms should be pretty simple I had court look at a truck and she's like here I got you this 20 foot u-haul that should hold a small two bedroom house or a three bedroom apartment I was like, yeah, that is plenty We have a lot of kids so we thought moving would be easy like into the truck lickety split Yes, that was not the case two hours done. No problem. We had seven movers one truck packer And within 90 minutes the truck was full So we had to go to the overflow trailer and that filled up in like 10 20 minutes And we still didn't get it all We still have the closet in our filming room is packed full of all that leftover filming supplies And there's a bathroom up there too and the closet in that bathroom is packed full of supplies We still have one more to move Step four Time to move in moving in was a whole lot easier than moving out There's no stairs and two big bay doors We backed the truck right up to the door and wheeled everything out and started to put it where it goes Yeah, we were able to unload it in less than an hour that was that was the easy part of the day That was the easy part to put together all the tables all of the chairs We had to lay out all of the offices and everything where everything goes the paint room And those tables are so heavy. I don't feel like moving anything ever again Yeah, I regret the two rooms. We don't need to move our statement Well, then the last thing we did is set up the studio and here we are right now As of right now, right? All we have are lights and camera This table put together and that table put together, but our pegboard. I don't have my paints up yet We have more to do more to do but we've got to move right along because step five Get ready for the craft show in two days. We have a big craft show coming up we've never done it before it's new to us and it's a bigger one and We're not ready Yeah, so we have been prepping for this show because we know it's bigger. We know we need more inventory We have about I'll say prior to the move day. I had half of what we needed So once we got everything in here, the first thing we did was set up that paint room And I have started painting signs, but luckily Again, we have a lot of great friends and family and thank you to all of you guys For helping us out because they have come in shifts to help you paint signs We've had people come help us paint walls help with the carpets hang things fix lights Fix light fixtures fix outlets. I don't know. What have they done? They've swept vacuum Well, and then like I said just to help me paint signs for the farmers market. They've been yeah, they've been shifts One day my friend was here helping she brought her mom and her daughter And then the next day court was here helping me paint We have been painting fools for the last couple of days, but I think we'll be ready I'm excited to give you guys an update on our new farmers market next week step six To be continued We are out of time for this week, but we will keep you guys posted of what's coming in and what's going on with the shop We'll give you guys. I don't know I don't know how how often would you like an update? I'm thinking like Monthly well good things happen Well, we've got some cool things being planned that we haven't even started yet We're going to add a new doorway These back to offices where we are are separated from the rest of the office space up front So we're going to create a doorway there. We've never done that before never made a doorway before So I'm kind of excited to do that and then we'll put a barn door up never made a barn door before Yeah, I've always I've been wanting to do a barn door But I literally have no air in my house that I can put a barn door I have no big wall where I could slide that thing open So I'm really excited. I have two of them that I'm going to put in this space And we're going to build an indoor porch We're going to build a roof and a porch for that front retail space And I can't wait to show you those projects. So they are coming probably over the winter I know they're going to be they're going to be their own their own big effort Well, we are about out of time So if you're not going to join us for the patron after show We will see you next week where we'll do it build it and make it again And like we'll actually do it build it or make something this time. Yeah Oh, and don't forget. Let us know what we call this new space down below. K&G. What? K&G Barn Yeah, sure. He said barn ranch k&g farmhouse. It's not a farmhouse. Not a farmhouse. I don't know what it is What do we call it? We will see you next week I'm I'm all balanced out this week. Yeah, thank goodness