are we early? I think we're early. It's 330. It's 330. Okay, my watch is off. All right. Welcome. My name is Ed Balduff. This is John Griffith and we're here to talk about the status replication in Cinder. I'm gonna start for a bit and we'll turn it over to John and then I'll take over again at the end. So the first kind of bit is why are we building replication into Cinder? Basically, Cinder was a cloud native storage application and there's a lot of push to have a DR backup capabilities in there which are traditional enterprise capabilities. So if you're trying to put that app into cloud that doesn't understand how to be cloudy, it doesn't understand how to replicate at the application layer, this is a way to solve that. You'll notice in the slide here I have backup kind of because that's not really a good backup solution in replication, but it sort of kind of can be. So in looking at the problem, there's a number of use cases. There's a number of use cases, but we're going to talk about the limitations today to address a few limited cases. And then the challenge becomes different vendors, different terminologies, different ways of doing failover, failback with a replicated volume. Terminology, John will take you through some of that. And so the philosophy has been crawl, walk, run, but the first time we tried to walk, I think we fell down. And John, again, we'll take us through that a little bit on why we have version one, why we have version two, and maybe version three at some point in time here, I don't know if you call them a different version. And then there's some other hypothetical questions out there like, should it be aware of availability groups? Right now it's not. So it's availability group, availability groups are a Nova concept and not a Cinder concept today. But it's a concept out there in the cloud that you might want to say, I want to replicate this and I just want to replicate it to somewhere outside of the current availability group. But as a operator, you're going to have to manage that a little bit to start with until we've answered that question somehow. I guess I should have a question mark on that. So what use cases did we consider as this was being built? Never two different vendor back ends, right? So this is a vendor, there's been a couple of sessions about our vendors killing Cinder, or there was one yesterday, and this is one of those things where it's very, very vendor specific on how you do replication on the back end. The other thing that we will talk about a little bit is there is no reference implementation. So in everything else we do in Cinder, it needs to go into reference implementation first, the LVM reference implementation. The community has not figured out a good way in taking the time to get that in there yet to so we can test it. So there's some challenges around that. Till yesterday, as of yesterday, we have a plan, so. Okay, now we have a plan, but. No, it's not there yet, but we have a plan. It's not there, right, right? So that's good. I was not in that session. So the questions were kind of the use cases considered, two back ends in the same cloud. Does that make sense? Again, this comes into availability groups. Two back ends in potentially a different cloud, federated identity, how does that all work? Two back ends, one not in the cloud. That's another thing that we do account for that today. Number one and three we're accounting for. Kind of two is a different discussion. One to multiple back ends. I think the code will accept that. I think we don't know how much testing has been done on that. Well, John will take you through the status of that in a little bit here in a minute. Automated versus non-automated failover. How do we determine is it time to fail over, right? Is there really a failure or is there some kind of transient failure that's failing over? Because failing over is a pretty significant event. One of the things I don't have in here is consistency groups, right? If I fail over one volume out of a consistency group, now we can get in this weird state. So that's got to be accounted for. I don't know the status of that, but that was one of the things we were discussing. If you've got one node, one volume failed over out of a consistency group, what do you do? That's bad. So you've got to fail them all over. And then, you know, snapshot replication versus continuous replication. And how do you then promote the continuous replica to a consistent state? And there's a lot of things to consider in there. So we've looked at those. We probably missed a few. If you got some situation, let us know and we'll add it into the list to continue to consider. So with that, I'm going to turn over to John a little bit for kind of a history and where we're at today. So all right. So I kind of want to talk a little bit about, you know, what we've done in the past, how we've got here, what some of the different thoughts in the community are and some opinions. So I'm really good at sharing my opinions, whether you want to hear them or not. But so what happened a while back was some folks came up and had this, you know, one of the big features that storage vendors offer is replication. So obviously a number of vendors in the Cinder community wanted to expose replication as an API that's available in Cinder and that they could do. So a feature that they could expose. Now that all sounds well and good. The problem is, there's different views on what OpenStack should be and whether it should do that or not. Right. So you've probably heard the different arguments about cloud native versus traditional, et cetera, et cetera, should OpenStack try and replicate some of the thing, replicate, oops, should it try and duplicate and emulate some of the things that VMware does or not. Some people believe that it should. Some people believe that it shouldn't. Personally, I'm kind of more in the camp that I don't think it should. But the reality is the users are usually who makes a lot of these decisions and drives things in an open source community. Right. So there is definitely a strong voice out there asking for volume replication inside of Cinder. So we started moving forward with that and we kind of ran into some problems and it's kind of predictable that we would. Replication is an interesting thing. It's a very complex thing and it's something that every vendor that has it in their product does it differently. They have different semantics. They have different rules. They have different vocabulary, which makes it even worse. Right. So everybody does it differently. Everybody has different rules on how to do it. And now you have Cinder, which is a project with a choice of over 70, 75 different backend devices that you can plug in. How do you get all of them to agree on an API? Right. So the first proposal that came out was you don't. Right. So don't put this in the API. Hide this under something like volume types and make it something that nobody actually sees. Right. Leave it up to the admin and vendor unique things to do in the background and be done with it. So that wasn't very popular. So some folks went and they went ahead and they tried to implement a full blown solution. Did a great job. Wrote some really cool code, some really complex code. The problem was we kind of gotten a rush. So we went through a six month release cycle and we were getting close to the end of the cycle. The patch had been iterated over many, many times, reviewed many, many times. And the problem that we ran into was people wanted it. They wanted it merged and they wanted it merged now. So we made a big mistake and we rushed it through. Right. So the result was we went out the door with replication v1 and we only had one implementation in tree for one vendor. And it just so happened it was the vendor that wrote it. And it was a very complex piece of code. It was a very complex change. It had a number of API changes and things like that. Well, what happened shortly after that was a number of us you know came on board during the next cycle and said, okay, I'm going to write a version.  And what we found is the semantics that we've been reviewing and working with and everything else all this time, as we actually started to dig deeper and try and implement this on our own devices, it was impossible. Right. So it wouldn't work. We couldn't fit. We couldn't find a mechanism to make all of this fit and work for everybody. So we were kind of stuck and we didn't really know what to do. Went back and forth for a while. We made some different decisions, some different arguments. The thing that we found was we didn't have good documentation on any of this at all. The one person who actually really understood it in depth was no longer really available, which created a significant problem. And then back to the other thing is nobody was testing it. So that was a really big problem. The original one. So that's kind of an interesting thing. I don't know. So if you look at, for example, the kilo release of OpenStack, you can go through and I'm not going to call out the vendor or anything else. And look at the code, the replication code and try and trace through it. You'll kind of get an idea. It's hard to tell. I'm not exactly sure exactly how it's implemented, exactly how the configuration options work or anything else. Now, in fairness, if you were a customer of theirs, I will guarantee that they probably have really good documentation available for you. And they probably do some extensive testing and it probably works great. But I do not have any answers on it. I don't have any insight to those things. One of the things that about all of this, it's kind of interesting. One of the things that makes OpenStack pretty powerful and makes it really successful and work well is the fact that we are pretty strict and rigorous about our continuous integration. So every change that gets submitted into OpenStack and this Cinder or any other project, it goes through a pretty extensive set of automated tests. The problem is all of those tests are in a synthetic environment on a single node OpenStack cluster, right? It's not really a cluster, right? So you start doing things like replication and you have two problems. One is, as Ed mentioned, there's no replication in the reference implementation. So it doesn't get tested. The other problem, there are no multi-node OpenStack deployments in the gate. So there's no way to actually test it anyway, right? So there's zero testing from the OpenStack community perspective, which creates a significant problem. So we started at the beginning of the L release. At the last summit in Vancouver, I had a big rant and a little temper tantrum and everything else about what was going on and everybody said, well, yeah, I want to do something about it. So I did. So V2, what we did is we all got together and we said, all right, look, this doesn't work. What we have is it may be great for a small set of customers, but it's not good for everybody. So we need to work together and come up with a solution that will work for everybody and make sure we're all on board and we all agree and things are good. So we did a really good job at the start. So we spent quite a bit of time at the summit and quite a bit of time over IRC and things like that for the first few weeks after the summit, collecting ideas, talking about different things and everything else. And then it was time to actually write the spec. So we write the spec and everybody starts reviewing it. And the beautiful thing about that was it took three and a half, I think four months to get everybody to finally agree on the spec. So the specification to do this was iterated over more than 60 times before everybody could agree. So it was just crazy. It was ridiculous. And it went from no API calls and just volume types to a handful of API calls and volume types to no volume types and then back to a mixture and back and forth and back and forth and back and forth. We finally got to a point where we said, okay, here's the deal. At the mid-cycle meetup in Fort Collins last summer, we said, we either agree on this and merge it now or we kick it all out and we don't do this, period. Well, lo and behold, plus one, plus one, plus two, plus two, all of a sudden it merged. So we were good. So we went ahead and we wrote the code. We implemented it. We implemented all of the bits and pieces that were needed in the core sender code. And it was good. We were feeling pretty good. Everybody was pretty happy and we thought things were good. But the problem was, is we were in this mode again where we were now two weeks out, three weeks out from cutting the release candidate for Liberty. So while there were a number of folks that had been working in parallel and had some things ready in a driver that they wanted to release, we made the decision that, hey, you know what? Let's not make the same mistake we made again. So what you have right now is you have all this code in sender that talks about replication, but there's no implementation anywhere in Liberty to use it. Right? I think it's better that we wait and get that solidified and everything else rather than release something that's not ready yet. And as it turns out, now that the M release has opened up and people are submitting code against it and doing reviews and digging deeper, we're finding a number of things that we're changing in the API. Had we released that and done that, we wouldn't be allowed to do that because you can't make an API change like that. So. Cool. Okay, thanks, John. So I'm going to take you through a little bit of kind of how it's going to work. We have some prototype implementations. John's gotten ours to what I call prototype stage. So I then took it and tried to make it work. And so here's what happened, right? So first off, the driver has to do its part and it has to report that replication enabled is true. So that's a pretty easy change in the driver. You as a operator have to change the config file. So there's a fairly complicated line. It's a single line, but it winds up being multiple lines in the config variable called replication device. You can see here the proper semantics of how that works. I'll take you through an example here, which will be more descriptive. I'm just looking at my note here at the bottom in my detail above it. And I have a comma at the end of there, which I'm not supposed to have. So that does make it not work. So in the example, which is here, I left off the comma. So this is the example config file from the setup I did. And we have a little demo. I didn't, I'm not going to try to do it live because we all know how those work, but I do have a video of this at the end. So you set up the first solidifier here, solidifier one, that's the normal config stanza that you put in the config file. The second one is the device that's the source. So you could have these cross-replicate, but I set this up just as a one way. So solidifier two is going to replicate to solidifier one. You can see that the men's replication target equals true. That's the thing that enables it. And then replication device points to that solidifier one above there. In our case, the vendor specific ID here is the actual name of the cluster. So you'll see in the demo that the cluster was called Wolverine after a comic book character. And that's what we use in there. You'll also notice that the admin, the login and password have to be repeated down below in that second stanza. John's going to work on figuring how to fix that. But for now, that has to be there and how to log into that. Now, this is, it's interesting. So the man's replication target equals true. That tells us that the destination is actually managed by the cloud. So the cloud will see both the ID show up in both devices. If that's false, that actually then is the replicate to a device that's not in the cloud anywhere. So that's the second use case, right?  No, that's fine. One thing, just so you know, when you look at the code, you'll notice that's not there anymore. So one of the things that we did change and fix up is you don't have to do that anymore. Which is pretty good deal. So it's kind of redundant. As you can sort of tell, this is in flux, right?  I found some issues that have been fixed. So, and then you set up volume types. And there's a couple of keywords in here. Every vendor can have their own set of things in here. So the big one is replication enabled. This is just, you can set up a volume type, not give it a keyword of replication enabled. You could put all the other things in there that you needed and it would not go. But you need that and then you can set up replication type async, sync, number of replicas. There's a whole bunch of other ones of these that will be documented by each vendor. These are the three that so far are in the code that are standard. Kind of makes sense, but there's going to be, you're going to have to read the vendor documentation for this. So we'll probably have to make sure we go through and get the documentation good.  And I actually, I'm going to ask you a question. Can you comment on the status of the V1 and V2 code in the tree and how those two are going to interact and then we'll talk. So it's an interesting problem. The first proposal was, hey, let's take all this V1 stuff and just rip it out. Get rid of it, right? Because it's actually a little bit cumbersome and a little bit confusing. But the problem is there are probably a couple of people using it. As a result, there it lives and it doesn't go away. So what we have done is we have put markers around the V2 versus the V1 methods. In addition, we will be putting a deprecation marker on the V1 methods. So at least for another year, they'll be there, but they will report and say, hey, these are deprecated and they don't apply. Right? Or who knows, maybe somebody will decide to make these work for V2 at some point. But right now they don't. So this is actually not a complete list. There are at least two or three others that I can think of.  So like these two came up when I was looking through the code and I was like, hey, you know, John, make sure and he's like, no, those are V1 semantics. So I put them in the slide. There's a bunch more. But these, so if you see promote and reenable, I don't actually know what these do either. That's why I was asking John. And neither does the guy that wrote them. Neither does the guy that wrote it. Okay, so this is so if you see these things and you see this, this is a V1 semantic, it's not V2. So current terms enabled, disabled, these are a little more descriptive statuses, disabled and active, active, active, stopped. Have these changed at all since I wrote this slide? No, but I suspect they will. Okay.  So these are the, the bits that you're going to actually do as an admin, you're going to, you know, enable replication, disable, these are admin tasks. So if you're going to take the other, the destination array down for some reason, you can stop all replication to it as a sender command. You can reenable it. If you had a failover as an admin, you can force that failover. I think we're going to do some of these on per volume basis too. So this is somewhat to be worked out. And then list replication targets. So there's just, so there's some discussion on some of these as well. Things like replication failover and things like that and having it as a command. There's some of us that think that that's not necessarily something that we want to expose that replication should be, hey, you asked this volume to be replicated and it's replicated. If something happens and something goes wrong, that's a pretty serious thing. And you probably don't want magic to happen in the background, right? You're probably going to want your cloud administrator to actually figure out what's going on and fix things and put them back in a state that they should be in. If you have these things silently happening in the background, it could actually lead to a really serious problem in the long run. So there's some discussion on the value of these things. However, one valid point that's been brought up is how do I test that this is working? And the only way to test that this is working? Well, not the only way, but an easy way to test this is working is to have those commands and expose them to the admin. So you got a question? So it depends on the back end. For some back ends it is. For some back ends it's not, right? So in our case, yes, you can have multiple replication targets. For other back ends, you can't. The other problem, and one of the reasons why things like this are hard, there are some back end devices that don't really support replication at all. So we put forth a massive amount of effort and work for something that only works for five or six different products. Actually, that leads right to the next question. So here's the, as I understand it, the state of the drivers that are in work or are going to be in progress. The first four are up there in some state that you could get a hold of today. They're in review, they're in code, they're in tree somewhere. I talked to the, and Dell, I might have to move above or below the line here, but I talked to the EMC and the NetApp folks today and they're both planning to do this and it's going to be in Mitaka, but they have not submitted them for review yet. So is Sean in the audience? I don't think he is. I don't know where, Dell will have to see where we're at. They're in progress. I've been talking to them. Is it in review? It's not up in review yet, but they are working on it. So they're below the line, but soon to be above the line. Yep. So let's see if we can get this demo to work here. It's going to come up. All right. Cool. So again, let me just pause it for a second. It's of course not live. It's Memorex. Is it showing up there? Okay. So I'm going to go through here a little bit and I'm going to pause this as we go. You can see here I've just looking at the, what's it doing now? There we go. Now you've done it. Is that big enough? Can everybody see that? Okay. Big enough. Is it fun? Yeah. I get it a little bigger here, I think in a second, but that's the, there's the config file. It's basically the same config file you saw in this slide, just kind of catting it to look through or look around at it there. You'll notice in this, if you can see it, I did use the target name of Wolverine in there. That's just a name internally. That's again, it's a required field, but it's dependent on the vendor. So we use the cluster name. Just so you know what that's used for is that's so that your driver actually knows how to identify and contact the target back in that it needs, right? So for some vendors that may be a UUID, it may be an IP address, it may be something else. For us, we use a cluster name. So that's a unique thing for us. Oh, it's doing it again. Yeah, the demo gods don't. It's should have done it live. Yeah. Yeah. It's, it's one of those things where I keep, I'm, I'm clicking on the, let's just do it from here. Just don't pause and you'll be okay. All right. Hang on. I'm going to go, I'm going to go the other way and just pull up the video here. Hang on. Where is it? Here we go. Right here. My eyes are going. All right. There we go. Make it bigger. Don't raise any attention to them. This is all being recorded, you know. Okay. So as the video goes through, we'll highlight it. There's the source, the destination. There's the replica device highlighted. This is the default, which is, which is one, right? So we're just, we're going from solid fire one to solid fire two in this example here. So the, the way that works now, if you were, if you were to have multiple replication targets, what you would do is that replication device equals line, you would duplicate that line with another entry and you would just continue on for however many you want it. And then you can set in the volume type so it'll go to one. So in our case, for example, if you had four of them set, if you set in the volume type, you had one type that was replication count one, it would only pick one of those. If you said three, it would pick three for whatever you say, until the number of devices you have configured. Okay. So move along here. This guy, I'm kind of going to, so this is our API. We're going to look just, you know, point out a couple things here. You can see in the, the cluster replica targets here. And there's the settings. There's, there's the Wolverine. And that's how you can see that those two are paired up. The driver will actually automatically create the pairing if it's not there. But if it's there, it will use the pairing that exists. So now we'll, we'll kill out of that VI session. And then here we'll go through and we'll do a listing of the cinder types basically to see that there's these are non replica cinder types. Then we're going to go through here and create a couple of this one called solid fire REPL, SF REPL, and then set up the, the extra specs here. So cinder extra specs list to show that there's none there. And then create the keys. So the first thing, so you'll notice I have multi backend set up here. So I have an LVM and I have solid fire. I'm going to need to not only set up the replication, but I'm also going to have to push this to the volume backend. So that's the volume backend name that I created there. And then this is the set up the replica replication enabled flag here. And it's an interesting syntax there. So yes, it is quote and is true. I don't know who wrote that. So the reason it's that way is I was told that I could not merge the code unless I changed it. So it used to be just true. But there's a number of people that are diehards on how this should be. And the is true and is false for Boolean representation is now a requirement. So now you'll see that I've flipped over here to horizon. I try to kind of go back and forth. Horizon's a little bit better for demos. Easier to show. So you can see that the solid fire replication type comes in here. We're going to, so you could have set those up through horizon if you wanted to. I did it through the command line just to show that it's there. And then I'm going to go create the camera. I think I created through horizon here. So we'll create the volume from horizon. It's just a pull down now and you can make it a replication type. Obviously, if you're an operator, you might have a little more description in there and not just it's replicated, but it's, you know, this is between Virginia and Colorado or something like that. But you can see here, you pull that down. So one thing that's cool about this, the way this is implemented now too to keep in mind is the way it's written, it doesn't break or impact any of the existing volume type stuff that you use. So what that means is you may have a volume type with certain extra specs in it, whether that be QOS settings, thin provisioning, whatever it might be, that stuff all still works. So what you can do is now you can just supplement that with replication information and things still work the same way. The old code, that wasn't quite true. So that's kind of a big deal. And it's, it may not seem like a big deal, but it's, it's pretty cool. Nope, no, no, that's what, that was my point. Right. So if, if you wanted to actually set QOS settings on that, on that volume, as well as replicate it, absolutely. Yeah. I didn't happen to do that here.  In our case. So he knows, he's familiar with, with solid fire and QOS. So yeah. Right. And again, that's another vendor specific that, so you see, you're starting to see a lot of the vendor specific bits of this coming out. And that's the challenge, right? So, so you can see here, it creates the volume. That's available. And now I, I'm going to do a couple things down here. I'm just going to make that go away there. You're going to hide it under the player bar. Yeah. So I switched over, I did this under demo. So you can see it's there and then we'll do a show. And you can see the details here of, and then replication status comes up in the details of the volume. So it's there and it's enabled and it's going. The demo here is then going to walk us through looking at the, the solid fire GUI here to see the volume.  We, we have an account for each thing. So I can filter the volumes by account. And you can see it's red right there. That means it's, it's replicated. It's started the replication process, but it hasn't synced up yet. So it's going through. This is on the other cluster here. So this is on the, the destination. And you see it's red. And then by the powers of demo video gods, we can make this go faster. So you'll see it goes to this resuming connected. And then it goes to a different state here to give it a second and it'll, I sped this up so it'll get there. But this is all dependent. This is another thing that's dependent on the, the how the backend does that stuff and what states it goes through and, and things like that. So that doesn't come up into OpenStack at this point in time to see what the, those statuses are. Yeah. So, so we do actually, the good thing is, is we do do that now. Okay. So it took us a while to figure out or took me a while to figure out how I want to do it. But what we do now is we actually have a periodic task that the driver reports a status backup and, and tells it what that is. I don't think that's a good solution long term, but that's what we have for now. So have you implemented that in our driver yet? Yes. Okay. Yes. Yeah. So, so the, the driver, a driver that, or a backend driver that implements this. There's now a method that they're expected to implement that will actually report the status back up to the volume manager. And the volume manager can then update the database. Nope. Nope. It does not. Well. So, so this is where things get really interesting, right? This is where, where the fun begins. And this is one of the reasons why I, I'm actually not a fan of having this, this in the APIs at all. So, yeah, there's a bunch of different limitations depending on what your back end is, right? And it creates some really serious problems. One of the biggest problems in an OpenStack cloud, in my opinion, is, you know, our whole goal was, we want to abstract all that messiness on the back end away, right? And what we're doing now is we're starting to expose more and more of it, you know, which isn't what we were supposed to do. And so the question becomes, is that an admin thing that you want to show it to the admin? Or do you want to show it to the user? Because then the user is going to be calling into the cloud manager going, is this right? Is this normal? How long does this take? All those kinds of things. And so that's the next thing. So what we've done right now is we have limited all of that visibility and everything. It is admin only. So an end user that, that is using your cloud has no visibility into any of this whatsoever. So as far as they're concerned, you know, you could implement R-sync as your replication on the background as far as they know. We're not exposing any of that information to the end user. And hopefully, you know, we'll leave it that way, right? Because I don't think there's any reason why the end user should know. You can give them a volume type and call it replication and give it a description and everything else. That's fine. But I don't think there's any reason to expose the details of what's happening on the back end to them. Right. Sure. Yep. So we're going to go over. Like a typical workload would be, you know, for a database, for example, you put the database in hot backup mode, you flush to disk, you do a snapshot, then you replicate that. Is that accounted for in this? Or is that kind of, is that beyond crawl? So it's beyond crawl, for sure, right? I mean, that's probably, that's probably halfway through walk, in my opinion. So my goal here was in the Mataka release, the upcoming release, to have a stable foundation that people can actually start using and experimenting with and figure out what the next steps are. So all things in OpenStack, in my opinion, a great way to do it is you start really small and simple and then build on top of that. And where you're talking about, that is exactly the vision a lot of people have for this. How long it takes to get there, that's tough to say. So I think there was a question over on this side of the room somewhere. We're open for questions. So I got one in the back. I'll take the gentleman back there and I got one up here. I was just wondering, how did you get around the issue you mentioned with the CI server testing? Is this now included? Yes. So that's one of the challenges still to be worked out. I mean, we have a plan for a reference implementation, but we also now have to build CI systems where, and so this just doubles the number of CI systems we need to have, right? So there's a couple of things on that. In my specific case, I have an external test set that I'm going to attach on to the existing CI runs that I do and actually run this test on a periodic basis. I probably won't do it for every single commit, but I will do it periodic to start just because tying up all the resources is kind of a pain. Long term, we do have some things that I'm working with some folks from Linbit to do DRBD for the reference implementation. So that will give us the ability to do replication for LVM for the reference driver. And at the same time in parallel, there's a number of folks working really hard to get multi-node tests in the gate, in the OpenStack gate. So once all that stuff's in place, we'll be able to start turning all that stuff on and it'll just be part of the standard process. So we're not there yet, but we are actually getting there and making the right moves to actually do this correctly this time. Question up here with the microphone. Yeah, question on the, can you explain how the failback will work? Also, when you have multiple targets, how the failbacks will work? So this is the problem, right? And again, this is where everything gets so fun and interesting. There's a number of ways you can do this. If you want to implement automatic failback and you have a replication setup that is a managed cluster to a managed cluster, right? It's a little bit easier because failover in that case, what you can do is you can do things like say, hey, all right, Cinder, I just want you to change the database column for this volume and point it to the other backend, right? So you can do some magic and kind of make things work and that's okay. Now, when you start talking about balancing back and forth, it turns out, which I thought, oh yeah, well, why wouldn't you?  Yeah, guess what? Most backends can't do that. So most backends, you might be able to fail over once, but you cannot go back. So there's all kinds of interesting things that happen there. And then the multi-node, if I'm going N-way, how do I pick who I fail over to? Yeah, which one, right? So now you need ZooKeeper or something to figure out. Or that's an admin function where the admin goes, oh, this data center failed and that means our next thing was we fail over to that one. And if that one's having a problem, then we go to the third one or something like that. I think all of, and this is really why I strongly believe that all of the failover type functionality and stuff like that, I think that's something that in a situation like OpenStack where you have all these different choices and everything else, that probably needs to be human intervention. Either that or we need a mechanism that each vendor that implements this has their own individual plugin that they can submit and do what they want and automate it for their customers if they choose to. But from a community sender perspective, I think it's unrealistic to think that we're going to actually solve all of these problems for 80 different solutions. Yeah. Next question up here in the front. I don't use question. I do not remember seeing it, but is there any test failover API available? First question, second question. Is this synchronous replication? Is it like transparent failover? So two things about that. So the first question was about a test failover, and that's what that failover API command is there for. It's there for the admin to go ahead and force a failover to test it. That's what it's for. Not test test, but here I can force the failover. Make sure that the connections are there. And make sure it actually works. So you can actually test it out and see if it works. So it'll actually. It's going to move it. It'll actually move it. Yeah. It's going to move it. It'll actually simulate breaking the source cluster or the source back end. All right. So are there any additional steps need to be taken from NOVA perspective or from sender perspective after the failover is complete? Yeah, as a matter of fact. So the thing that you have to remember what's going to happen is you were, for example, if you have a NOVA instance running an attach to a volume, or if it was a boot from volume, you had an iSCSI connection to that device. And now, all of a sudden, that device isn't valid anymore. It's not there. So you're still going to lose that instance. It's still going to die.  And bad things are going to happen. So, right. And honestly, there are a number of people that want that. This is why this is an admin commander. This is a script of some kind at a higher level that then goes through and restarts the services. Okay, it had a failure. We moved over to the other side, start everything back up and it's a DR scenario, right? It's a disaster. You're running your disaster run, Brooke, and you're now pointing to the other side. So it's not just automatic and it's over. And you had a second question. So it's not transparent failover. It's not synchronous replication. Well, it is. So here's the thing. Depending on the back end, and I'm sorry, did you want to go ahead? So depending on the back end, it can be synchronous or asynchronous. So a number of us, and I think you guys as well, right, you can pick either one, right? The point is, from an orchestration perspective, when you're using the volume, just because you're replicating and you can fail, depending on how you define failover, just because you're replicating, there's no magic, right? So there's nothing that all of a sudden just makes the new volume or the target volume all of a sudden appear in the right place and you never hiccup or anything like that. So does that make sense? Okay, back to the question. This is very much a DR scenario. Back to the question about database, you know, obviously you might want to snapshot it and have that.  That's another six months worth of work. So we'll take the question in the back. I've got, I saw another hand over here, where was that other hand? So is it possible to implement volume replication using community mirror feature? Then it can let the back end which lacks of volume replication have the same function. Could do that? There's nothing in this that would preclude the driver doing a breaking the mirror and then just pointing to a different part of the array, right? I mean, it wasn't the original intention, but I don't think there's anything that would prevent that. So if I'm not sure I'm following, but I think what you need is one missing piece and that would be a multi-attach to have it attached to another component, right? So in the multi-attach hopefully we'll land on the Nova side sometime in the coming weeks. And I think when that's there, I think your scenario would actually work and it would probably work fairly well. I think I'm not positive. Yeah. So don't hold me to it. I thought I saw another hand over here and maybe a hand over there. Anybody one up here? I got let's we're going to go with her first. Sorry. Okay. I wanted to understand the scenario that if previously I have three replicas or three replications and one of the replicas now die. So it goes offline. So now you have two replications only, but you want to keep the replications still be three. So what's the procedure to recompose it? Yeah. So right now what what I do right now is I say you are now in replication state error, right? In the future, ideally what I would do is I would create another replication target automatically for you. The problem is you can go down this path for a really long time, right? There's definitely value in it and it would be a cool thing to do. The problem is is as I start writing this code and doing these things, I find out that, oh my gosh, I've got to do that for this corner case over here and this one over here and this one over here.  So at some point, I think we need to figure out and decide where do we draw the line, right? Where are we giving people a good, not a great, not a perfect, but we're giving them a good DR solution that's usable and protects them versus trying to be something that we're really not. And when I say we, I mean the open stack pieces. That's honestly, that's my opinion. But I am finding that that's not acceptable to some people, right? But yeah, you're right. We're gonna take one last question. I think we're out of time or probably over time, but go ahead. Just a quick one. I was gonna ask, do you know where the the SEF guys are in terms of this, if they're working on implementing this in their back end? That's actually a really interesting question. Well, yeah. Well, SEF has no, they have RBD marrying and they have some other things to do replication type things as well. But the question, yeah, exactly. So the question, well, I was thinking your question was, are they doing anything in the sender code base right now? That I have not seen or heard that they're doing anything. However, keep in mind, it is early in mataka. And there are quite a few people that work on RBD. So I would imagine that they will probably do something or maybe they're like me and they just don't want to, right? Exactly, right? Okay, with that, we'll wrap it up because I believe we're out of time. We'll be up here for a bit. Thank you, everyone. Thanks.