Hey folks, apologies. Somehow the last meeting didn't get stopped properly and so we got delayed on this one. We someone willing to share the meeting minutes from this meeting. So I had trouble joining the meeting that said, another meeting is in progress. Yeah, I had the same issue. And so I'm not quite sure what went on there. I hope the left to see what their advice is for how to avoid that in the future. Cool. Oh, thank you Ashley for adding the meeting stuff if folks to go ahead and please add themselves to the attending list. That's very helpful. Cool. Should we get started. All the we're going to be good to want to share the meeting notes, but in the meantime, let's go ahead and get started. So welcome to the next number of services mesh working group. If you haven't added yourself yet, please add yourself to the attendee list. We have this particular call which occurs every Tuesday at 8am. We also have a call every other week. At 3am Pacific time. We also have. We also participate in the CNC of telecom user group. Which occurs every Monday at 8am and every every first Monday at 8am. Every third Monday at 3am Pacific. We also participate in the CNC of Sydney network, which has been rebooted and. Is occurs every first and third Thursday of every month at 11am Pacific. This particular group is, is a little bit interesting now because they, they now spend time vetting things for the CNCF. So if you're looking to get more involved with the CNCF, that's a fantastic way to do so. We also, we also have a few events coming up. So on March 18th, there was a go San Francisco meetup, which I will be talking about cloud native zero trust. And March 30th through April 2nd is cube con and cloud native con Europe. At the RAI in Amsterdam. Schedules are up and SM con is going to be on March 30th. They collocated with QCon. The schedule is now up. So if you would like to see what we will be talking about at NSM fun. Please go take a look. We have open networking and Ed some in North America. We're waiting for the schedules to be announced. That'll be April 20 or 21st. We have cube con and cloud native con China. The CFPs have already closed and the schedule will be announced in May. We have ONES in Europe. The CFPs close on June 7th. And that'll be an Antwerp in September 29th. In November 17th through 20th is the cube con, cloud native con North America, Boston. The CFPs will open in April 22nd. So the day after ONES and they will close in June 12th. Announcements will be in September 14th. And I don't think there's other events that we know about. So we have, again, just a reminder, we have a project page. The link is listed on the, on the meeting notes. And are there any other announcements that we want to do before we move on? I'm assuming not. So in that scenario, social media community team. Ashley, you have the floor. Hi everybody. So last week as far as social media went was pretty exciting. We reached our next milestone of officially having 700 followers on Twitter. So that was exciting to see. We gained 17 followers in the last week. We followed 15 in additional counts and tweeted and retweeted 26 posts. So as I mentioned that surpassed the 700 followers. As far as other things that were tweeted about and SMCon related, we got some registration reminders out there. The schedule announcement went out last night. And there are already a few people retweeting their, their schedules, their sessions. I'm sorry. So that's exciting to see as well. Outside of that, some general core reminders, video recaps, some CNCF news as far as promoting some events coming up. OSS in China later in the year. Some CNCF and LF events just again, letting everyone know that childcare is provided at those events. So just trying to get some additional attendees there for some parents. And other events that were promoted this last week was the introduction to network service mesh. Meetup that happened in Austin this week. So that was promoted a few times, a tweet that went out thanking the attendees as well as some retweets from others that had got some news out there that had attended that event. Some other general retweets, telecom user group meeting notes, CNCF blog and some VMware open source tweets. And then as far as LinkedIn, a bit of a slower week. We only gained two additional followers, but at least we are still growing there. So hopefully this next week we'll continue to follow. And I guess at this point we'll mainly focus on pushing sponsorship for NSMCon as we are a little over a month out and also promote the registration mostly outside of that. The meetup that's coming up in San Francisco later on in March. And then other sessions related to NSMCon and KubeCon. Sounds excellent. So really quick note to the folks who are speakers at NSMCon, you will find that your talk is actually individually linkable from the events page. So if you want to go and promote your talk, you can literally have a link that takes people directly to your talk on the events page. So it's very good stuff. Thank you very much, Ashley. This is awesome. It's super good to see us go over 700 followers. Oh, one of the things I didn't want to actually raise. So there was a really good blog post, Frederick. Do you have the link to it that came out of China about network service mesh that we may want to promote? Oh, that's a great idea. Let me go grab it. Yeah, let me see if I can find the link myself. It's been translated to English, which is super helpful because I otherwise would not know how amazing it is. Let me go ahead to the agenda. Yeah, it's an simplified Chinese and English. So I've put the link into the chat if you want to grab it from there, but it's super well done. Apparently there's some very thoughtful folks looking at what we're doing in China and writing really good things about it. Great. Good to know. I'll get that out there then. Thank you. Cool. Awesome. I'll also remind folks that you should feel free to add yourself to the agenda at any point. I added a few things, but I know there's more going on than what I've added. Back to you, Frederick. To be lose Frederick. Yeah, sorry. I was on mute. Well, and so. So with that, we have the. We have the main agenda that's coming up. So. And you want to talk about what's going on this weekend. I can talk about some things that I've been sort of heads down. So I'm not following as much what's going generally. So if other folks have things that encourage them to add them to the list as well in the agenda. So the sort of things that sort of got my attention were. The, uh, the command forwarder VP agent repo has opened. And there's an initial code drop PR. I'm still trying to figure out why the CI is not running on that repo. So that, that hasn't been merged yet. Cause I don't want to merge code that hasn't gone through the CI, but the, the initial code drop is super simple. It's like a 50 line main file effectively. Not a Docker file. Um, everything else is being pulled through from the SDKs. Um, so that's landed. Um, and then there's some improvements to mechanism testing.   Right. You're going to have, you know, we've got like four or five different kinds of mechanisms. And then you're going to rewrite those mechanism chain elements for each forwarder that supports them. So they're a thing that people are going to write a lot. Um, and so I, I put together a generic test suite to check that network chain elements for, um, Test suite for mechanisms to the right thing. Cause their contract is fairly simple, but straightforward, which is they should set the mechanism preference. Um, before they call the next chain element. And then when the next chain element returns, they should do whatever they're supposed to do, um, to, for whatever they're doing for the mechanism, but they've got to wait for the return. Cause that's where they get the complete connection. That's on the client side, then on the server side, that's where they get the complete connection. So they, they just do whatever they're going to do. Um, and in doing that, um, ended up building up some very generic machinery. So for example, um, There's a whole set of network chain elements that do simple checking. Right. So there's one to check to see if your chain returns an error. Um, because as it turns out, um, Sometimes when you test a chain, it gets wrapped up in other pieces. Um, and which is the case for mechanisms. And so you want to make sure you crisply understand that you've returned an error. Um, you know, check to see if the client propagates to your PC options. One of the things that's different between clients and servers is the extra, uh, variadic, uh, GRPC options, um, thing. And because it's very attic, meaning that if it's not there, it's fine. Um, I literally forget every single time. And so this is a simple thing that lets you write tests that simply check for that. Um, Checking the contents of context, um, after and on return from a chain element is important because it actually matters. Um, Particularly in the case of mechanisms, when you have actually, um, set up whatever things are in the context. Um, and then just generic things to check other aspects of the system, just to make sure, for example, that. If you have a network service requests after the chain, it's important to make sure that you have a chain element. Um, the key here is that, um, We wind up. If you want to check to make sure that a request contains something, you've got to have a chain element to follow to catch that. Um, likewise, if you care about things being in the return connection at certain points in the chain, you're testing.  Um, And then, um, I'm going to talk about some basic stuff here. Um, that generally speaking, and I think you made some points about this, Andre and the previous call, you want to try and keep your testing as simple as possible. If you've just got a chain element that. Um, Does a thing and returns the thing. Um, then you probably want to be checking just that and not wrapping it up in additional complexity. But sometimes it matters. Um, When in the chain of things, this happens. So for example, in the case of the mechanism stuff. Yeah. Yeah. More interesting is if we have more complex chain elements. Uh, with integration to a VP agent or to some library for Kernel, We also need some integration tests for this kind of chain elements. No, totally. Um, You know, I that that's that's one of the nice things about the way that VP agent SDK is put together is the only one that actually touches VP agent is the commit. Um, everything else is just setting up the config for that. Um, And then one of the other things that's kind of interesting. That's coming down to the VP Asian 3.01 is, um, The GRPC guys recently realized that it's hard to test if you pass a struct into everything. So the GRPC client connection, which is a struct has been replaced in most places with the GRPC client connection interface. Um, Ingenerated code, which is an interface, which means you can mock it. Um, So we can actually mock that out with unit tests as well. Um, Once we've made that transition. So there's a lot of good stuff going there. Um, But the baseline point is let's keep the testing as simple as we can. And it happens that this chain element stuff makes it super easy to test a lot of these things. Um, Any questions, comments, et cetera on that. Yeah. As a person who's reviewing a lot of this stuff, Um, Definitely, uh, I think we, it would help if we can get some examples for people playing out. There could be examples in the code just to show. Show off some of the, uh, more interesting areas. Um, I know there's a, there's a lot on the agenda to do though. So I know that that may not happen immediately, but just something we can add later on. Well, there's been something bouncing around the back of my head as I've been doing this. So my head went, I felt like my knees actually poked me and said, maybe we should have a style, a style guide. And I think what he was meaning by that was less like indent four spaces, but more, if you're doing something sort of like this, then this is sort of the pattern that's been established and this is why the pattern is. Um, And that might be super interesting and helpful. So, um, See who added the scene of conformance with NSM. I can go if you want or can wait. Yeah, is there anything else you want to talk about on that ad or? No, I think that's about it. I was basically providing some pointers to folks. But I think Andre is underlying point of keep your tests as simple as you can possibly keep them is a good one. But I think we may also want to consider in the case of things that are going to be done a lot like mechanisms writing a few test suites here and there. I don't want to get too crazy with that. If you're just doing a one off, it's probably not worth writing a test suite. But if you're doing something that's going to be things like that are going to be done a lot of places, then it may be worth it. Cool. Okay, what's in the you have a fork. Okay, so there is a new initiative on the in the CNCF, CNF conformance initiative. Basically, it's kind of an outgrowth of what the test there was, but basically checking CNF cloud native network functions. So within this community, it's really network services. And we're needing to have some examples for the conformance suite that come out of the box. There are some trivial examples just that that we're using now. But one of the things that we need to develop is a good story of a journey from a maybe a DNF world or just the non conforming or non compliant CNF into one that's more compliant and pretty, pretty sure that NSM would be a good, a good fit for as many different pieces, but the declarative portion. So just from our experience from taking DNFs from lots of hard coded subnet IP addresses and all that stuff. Is there maybe we can grab some low hanging fruit from maybe some examples that NSM has, where someone might be hard coding some things and when the addition of NSM makes it easier to have things be in a more declarative fashion. So, I'm trying to put that out there. If you guys can point me to something or if you want to help get that into the test bed. I'm sorry, the performance would that be great. Yeah, I think the thing that comes out to me is in this I think comes from sort of standard 12 factor out of thinking, which is an application should learn things from its environment. And things like IP addresses are literally things that you learn from your environment and NSM. You know, so basically things should learn from their environments and the system should be dynamic because dynamic things are resilient static things tend to be really, really unresilient. And then the other one that comes to mind is a scoping argument, which is if I statically part of why static things are not resilient is if I statically configure a bunch of stuff in a bunch of places. The scope over which I have to get that right is the global scope. And the global scope is really fragile. It's taking your fucked. Whereas, if you can localize things to a much smaller scope, then the scope over which you have to get them right is very, very limited. And so you're much less much more likely to get it right. Does that make sense. So, yeah, if there's anything where where you guys have examples from from maybe some type of network service that people were working on before Network Service Mesh and it's open source, and then after Network Service Mesh that story and the code. You can use as a sample. If we could, if you can point me in that direction, and it would be, you know, another outlet for Network Service Mesh. It's part of sample code for the performance. Don't have to answer now that, you know, something to think about. Definitely, we can try finding something so that's not very many examples before, but we can see about trying to find something that's that meets every requirement. Well, I mean, even the, what is it the Sarah, you know, example, I don't know if we're still using that but the we make it to where it's easier to just declare what you want for the security context, whereas before people were trying to maybe car code ask for specific ports to be open and all that stuff. Yeah, because I, my, from my recollection events time period, there was a lot of of static like static configuration. A lot of a lot of work towards like trainers to to specify things directly. I recall doing demos of this type of stuff back around 2016 2017 ish. I don't know if I have any available anymore, but I'll take a look. We could use something similar like it was an evpn example that I used to that I used to to use. And it was a real pain to try to get this thing set up and working properly because of all of the manual things have to be configured in order to try to get the to Kubernetes cluster networks to to work with each other. And so I'll see if I can that's amazing. Yeah. Worst case scenario, I could try reproducing. I could try reproducing it but it's, I'll see what I can find. Okay, great. Thanks. Anything else we want to discuss on this particular on this particular topic or I mean that's that's all I had if you guys want to, I mean, the, the code base is out there as far as what the different tests that we want, and it's still very much open. So the declarative part is definitely, I mean, there's so many parts that and if I'm going to be part of, as far as best practices, I know this community has been like certain opinions on what would be best practice what is cloud native and the thing that has always jumped out to me is the declarative portion. I think that you should have a strong voice on on that type of thing. So, um, I mean the one thing I would actually suggest being very careful about with just just talking about declarative is you can declare all kinds of things that are stupid. I can, for example, construct a system where I have to declare all the fine minutiae of every little detail of the infrastructure that I'm in what I'm working in. And it's still declarative, but it's also not a terribly smart approach to the problem. So I think declarative is one piece of it. But I think also like, I, it has a sense of immutable infrastructure. I think it's either IPAM to be part of the infrastructure, right? You can learn about it. You can ask for things from it, but you don't get to dictate it from the outside. Does that make sense? Okay. Yeah, sure. So, go ahead. No, I mean, I don't agree. Okay, go ahead. So, one of the things that actually is being discussed within the CNF conformance was the fact that these small bits of functionality that we call CNFs are supposed to not store state, not have state at all. Meaning that things like, you know, IPAM can and I believe should be managed outside. I mean, from the example that I'm trying to construct with NSM and actually I'm working on something you know, which I hope I'll be able to announce in the weeks to follow. But from the example that I look at from the practical way of applying NSM, I see a lot of problems exactly regarding to IPAM. We have seen this also with Ericsson, with the bridge and then load balancer domain. I mean, I think that the way that we approach IPAM today is not the best suited to whatever we try to do. That's what I think. Oh, so in what sense? Because I think you and I would agree that basically something has to manage the IPAM. What I was getting at is an IPAM story that involves putting an environment variable into your pod, assigning it its IPAM in the world. And therefore that's statically defined, which means all your pod specs everywhere have to come into proper agreement. That was what I was railing against. Does that make sense? Yes, but to me, like no one in the world should be writing pods their own. I mean, there should be, you know, top level orchestration that actually distributes these IP addresses and figures out for you. Like having the inline IPAM, you know, all the DHCP and stuff. I mean, you don't get the top level picture. It's very hard to deduct it from the running infrastructure. That's my argument here. Okay.  If you want to manage in a different way, that's part of why those parameters are optional. Number one, because you may, you know, number one, because you may not actually want them, right? They're definitely network services for which you wouldn't want IPAM at all. But number two, you may get them from someplace else. Yeah, but even if you get them from somewhere else, maybe you want to, like, introduce them from the service. Otherwise, your clients need to be privileged. Otherwise, your clients need to be what? I'm sorry? Yeah, no, I mean, this is actually a good point. Somebody has to go set those, you know, and that somebody has to be privileged. Network Service Mesh handles that because the forwarder, which has to be privileged to insert things like kernel interfaces anyway, goes and does that. I mean, you're talking about user space CNS and I would expect most CNS to be user space in some fashion, and to be using some kind of a user space interface because kernel interfaces are very slow. But if you're living purely in user space, then you don't need to be privileged to do those things. Does that make sense? It does to me. I mean, just as much as we expect on the enterprise side, almost all enterprise applications are going to ask for a kernel interface, because they're written on currently stuff, but they're also not going to be super smart about their own IPAM. On the CNF side of the house, we would expect almost everything on CNF to be using something faster than a kernel interface to move packets around, whether that's SRIOV or MIF or V host or whatever. Does that match what you're expecting to see? You may be, it's me. I agree with you. Okay, good. I mean, just because I have a notion of the world doesn't mean my notion of the world is right, by the way, which is why I ask questions like, does that make sense? Also, I have a bit of like a problem with this, if I can discuss this now, because usually NSM mostly is virtual wires and only cares like point-to-point connections. So in a way, an IP address is completely useless. You could be on-link routing instead if you want to do that, if you see what I mean. Well, it's not so much the NSM itself is point-to-point connections, but we presume that there is someone on the other side who may be providing a network service that's point-to-multipoint, and that's the someone who knows what the proper IPAM is there. Does that make sense? Yeah, that makes sense. But what I mean with this is that in this protocol between the service and the client, sometimes I can see a need for more data than just the source and the desktop IP. I would like to put a big address as well, to have the form of a virtual IP. Like when we did this load balancer. Oh, okay. I'm missing the possibility of transferring more data over this protocol. And this data NSM doesn't really care about, but it's important to the client. So thank you for bringing this up. This actually brings up an important point that we should probably go just literally just go fix right now, which is like literally as soon as this call ends. So when we started out with, so what I think you're saying is that there are other things you might want to put in the connection context. Is that correct? Yeah. Okay. And when we started out with the connection context, it literally started out as just a map, a string string map.  And we did that we ripped out the string map, this, the map string map, the map string string. The truth of the matter, the truth of the matter is, you still need some kind of a generic context that gives you that map support, because even if you do eventually get to a place where you converged on all the things someone might conveniently want to pass there. Number one, you want people to be able to innovate without having to come and get something upstream. Right. So you, giving your example if you wanted to provide a bit, right. And I don't know the full depth of what you want to do but just on the face of it that sounds like an eminently reasonable thing to want. I don't know whether or when or if that would ever make its way up to being like a bit context that might not make any sense, or it might, I don't know. But what I do know is there ought to be in the connection context some sort of a map that you can use for that kind of thing. Does that make sense. Yeah, I added them up in my hackery but I never like I never. I just go push a patch up that adds that to the API, because it's something that I keep forgetting to go do, but it just comes up every few months and it should be done. So please feel free to go push that on. If you want but it's, I mean it's kind of, I just put the map into the struct of the context struct and then. Uh huh. And then you have to re-generate the DRPC but you know that. But I mean that that's a perfectly valid thing to want to go out to the API. Right so and you know if you go push that PR. I'd love to get that because you're absolutely right we're never going to be able to see all the things people are going to want here. Right. And all we can do is codify the things that we think are going to be super common. Like the IP context and the internet context and the DNS context. And then allow a safety valve so people can actually get their work done, which is what you've done in your hack and we'd love to see a PR for. So that and you know I yeah that that would be super super helpful. Thank you so much I do appreciate it and thank you for bringing it up. Right. If people don't bring these things up. Number one, even if they're things that like I said that's been in the back of my mind every, you know, for months now but it never quite got done. Or sometimes you bring up things that haven't occurred to people. And eventually I'm super curious to see what you're doing with that the parameter. Interesting as well. But I also put like a hack into the forwarding plane to be able to generate like extra interfaces with a bit. But that I don't want to push up because I don't think you want to have it. I certainly like to at least understand what you're doing there just had a curiosity. Maybe some kind of plug in concept to the data plane to be able to like catch stuff in the map. So that's certainly definitely an interesting discussion. You know that that's definitely interesting discussion and this gets to be. So I mean, and one of the things that we're getting more flexible with the forwarders is, you know, there's no reason you can't bring your own for that supports your own stuff. For example, I mean, generally speaking the one piece of advice I would offer you is that it logically speaking. Anything point to point is probably best thought of as a network service. Now implementation wise, where you put that network service from the purpose from the purposes of optimization. That may vary right, but the logical and the implementation are useful to separate because when you when you have clean cognitive breakdown of what you're doing and where then the where you put it to get implementation optimizations is unlikely to screw you up. But I'm sure you've seen this to get it again because we're both a networking and have been forever, where you muddy the conceptual end of it, and suddenly you've got like 15,000 features in one place where they're driving you knots. And when you can't pull them apart, right, so just just for the sake of argument, it sounds like you're using the script to do a point to multi point thing and you've hacked the point to multi point thing into your forwarder. Right. Yeah, what I did. What I did was that enabled this load balancing in VPP and it used the GRE tunnel, you know. Yeah, this is this is this is a fantastic thing right. And so, you know, my my my counsel to you would be as long as you consider continue to think of that as a load balancer network service which you happen to have optimized by sticking it into the forwarder so you didn't have to jump through a second VPP. I think you're going to find your life extremely pleasant. But this is also something that I thought a bit about most of the stuff we're doing in the service.  And to get performance, the application would need to do a poll mode load for something like this. I mean, application and action and the real service. But the local stuff we're doing are just infrastructural things. And it would be nice to have an API down to the data plane to actually delegate those kind of stuff like remapping or be long tag or something like this. That's a fantastic thing to discuss. I mean one of the things that you're probably seeing in some of the work with the chain elements is if you look at the chain elements, the difference between anything that you can break up into multiple steps, you can collapse into a single step. Yeah. And that's that's very much by design. Because there are there are lots of places where you care more about flexibility in the performance, a lot of them, but there's definitely places you care way more about performance than flexibility. It makes a lot of sense and also I like the abstraction with the virtual virus and services because it's a very strong abstraction and you can model almost anything with this. I get the problem when we implement this we would get like native EPPs running in each container. I would like to push the data I mean push the implementation down to the data plane. Yep. No, it sounds like you're doing a reasonable thing you're thinking about it in a reasonable way. You know, and you have some additional thoughts that I'd like to understand better about how to make this a bit more flexible. So, and so we can definitely carry on some of that conversation about how to make things more flexible because I like flexible. Yeah, that that's that the net net is a PR to add a map to the context to use for the sort of thing is probably an unalloyed good. So, yeah, I was more like with the new with the new approach to implementing the forwarders probably the best thing is to just, you know, create the forwarder that suits your needs. It's effectively just another endpoint. So, just, I guess that that would be the best the best approach I was to why we were talking I was trying to figure out if we can do some things like whatever the CNIs were doing with, you know, kind of chaining different forwarders but I guess that's just going to get too complicated at least at this stage at least until we don't move. Maybe at some point we can think about this but probably not today. Yeah, I mean, in some sense, like you still you can change things in some sense. Yes. So a whole of network service measures, any things that speak the network service machine for them to collaborate, sometimes across, you know, space and time and sometimes complete mutual ignorance, in order to do the thing that you want. But I'm particularly heartened under is that you sort of arrived at this solution on your own, because it is almost exactly what I would have suggested. Which means that there's some intellectual coherence flowing through the system where it leads to the right sort of thoughts, which is good. With the new forwarder architecture, is that done? No, it's getting there. I just pushed up the command forwarder VPPA and stuff so there's definitely a code you can go look at. I'm going to go take a look at where that's going. But it's, we just got into the border command bit, and there are some other pieces still coming together. But the idea of how the pieces fit together. And the point that you made about it would be nice if you could go and insert something with additional logic into it. So what you're going to find is basically everything is a chain of small chain elements that do a piece of work. So if you were to go take a look at the command forwarder you'd see all it's doing is calling something that returns a cross-connect network service. And if you chase back to the SDK VPPA agent, you would discover all that's doing is putting together a list of things.  That sounds good. Because then we can do it. Exactly. That's exactly the idea. And part of it is to enable experimentation because quick, frankly, there are going to be all kinds of bespoke things that people want to do. Because the world is, you know, the networking world is incredibly diverse. And we're going to try and hold the reference implementation to the stuff to sort of what you might call the sort of optimized for the average base. But we want to make it as easy as possible for people to solve the other kinds of problems they encounter in their life. And generally speaking, you want to minimize the amount of code that they have to fork. And if the only code you have to fork is the assembly of a chain, then that's going to make your life much nicer. So, cool. Great. Okay, I want to bring one more topic. It's not on the list. I probably should have put it, but it's just a quick, quick one. And I think Watson is still here. So he probably have heard this. I don't want to put names here, but there was this discussion in I believe in the telecom user group. And it looks like folks are trying to pin for the telecom usage, of course, try to pin the specific networking solution. And we have a good support within certain communities. But effectively, when NSM was brought on the table, the pushback response was it is not production ready yet. I'm just saying this, not like, you know, panic, panic, panic, we should release whatever, whatever. It's mostly like we as a community should be aware that people are expecting more of us. You know, it's easier to say the implement, but you know, No, I think that's an important point to get front and center. And I appreciate you bringing it up. I don't have anything to comment. Just like, yeah, let's move forward to the best. Yeah, the, the unfortunate part is a lot of people come flight, like they'll compare NSM to, to multis or Dan M or, or genie or so on. And the problem that that ends up happening when you make those comparisons is that people will just look at the, at the CNI multiplexer and say, okay, we'll just thing inserts an interface into into a pod. And they compare that to NSM because that's with the context that they're only looking at that and then they say, oh, and it's not ready. But what they don't see is that NSM is actually solving a different problem and that we've had that specific feature for well over a year in in production quality, I dare say. But the problem is that NSM itself has is solving a bigger problem of like, how do we solve the, the changing aspect, how do we get multi organizational inter domain working, how do we get, how do we get these various vendors, like SDNs to cooperate with each other, and do it in a secure way. And so, and that's before you even look at things like, okay, well how do we line in a PNF or VNF into the chain as well. And so the, the scope of what, what we're doing is, is very different from others but people have inflated our project with, with basically a CNI multiplexer in terms of in terms of scope. And that's part of why we're getting this, this actions. And they are right, we do need to get a production version out there. But it's, but there's there's a little bit something, there's a little bit more that's, it's a little bit bigger that we need to try to, that we need to try to tackle. That's, that's clear. I'm saying like, if you are unbiased and you come to the Kubernetes landscape, and try to find the networking solution that will treat your needs being catelco. You would find certain projects, whatever they are, are they CNI based or whatever. It doesn't, it doesn't matter. I mean, you're just looking at what's available.    I think that we're doing great on that side, but on the other side, people are expecting, okay, and the same is greatest idea. I like it. I think it's going to solve my problems. I want to use it today. And I don't know, maybe we just have to try to figure out what to when is today, you know, that's just, you know, Yeah, I agree with that. But, um, yeah, we should be, we should be relatively close, especially when you compare to where we were at like the last, the last few condoms and before so. But we definitely need to work out with that. What those steps are to get something to get something out and I would argue the most important things that we needed to finish adding in where things related towards the inner domain. And specifically around the security and, and permissions use case and we now have some stuff in the repo that still needs a little bit more work, but it's very, very close to being done. And then at the very minimum, we should be able to release a version that works with the kernel with kernel interfaces, and, and wire things together in that respect, and we can spend more time with subsequent releases trying to push out, like, how do you get the SROV work that the community isn't working on and how do you get the things related towards, towards additional monitoring or additional, or how do you get additional things with like, with, with other, like VFs or so on. So, so I think there's, there's some interesting stuff in that respect that that we can push towards but yeah, I think we should try to carve, we should probably try to carve out sooner than later what the parts that are stable and build a story around that as an initial release. But that's my opinion towards it. Okay, maybe I don't know, I mean, I don't have any particular plan or answer today is just like, I think this is a healthy discussion I think your distinction Frederick is actually a good one because essentially the, the, the CNI multiplexers saw such a narrow corner of the problem space that, you know, it's essentially saying we're going to settle on the problem, we're not even sure we can use to solve the broader problem but because there's a thing there. We're going to settle on it. Because it literally it's not clear that there is a generic broader solution for problem I mean you can always go get it bespoke preventer thing, but I think that's what they're trying to avoid. Exactly. And that's the, and that's part of the part of the challenges like how do we, and the people who are in the in the calls for the most part. I don't think that there's like any malicious intent or anything like that like they're they're looking at it from the context of, as best as as best as they can and the stories that they're hearing our NSM is not ready. We have multis and Dan and and so on and those are ready. And so, and the thing that we want from NSM is multiple interfaces so why would we want anything. Why would we want anything more. And then when they started visiting the harder problems, because they've already pre selected a solution they're not thinking of NSM as a, as a thing. Those harder problems really are where I think, ultimately, life gets interesting, because effectively at a very fundamental level, I don't think you can solve those harder problems with a solution where everything is set in stone at pod start up time. And, and so I think that fundamentally makes any solution that is constructed that way a dead end at the end of the day. I will agree with you, but the thing that I get all the time is show me those examples where there is actually are needed. Like, which is the example that actually can demonstrate how NSM utilizes this dynamism this flexibility that it brings every CNF ever. I'd like to demo those examples. Every CNF ever because here's the thing. If you, if you, here's the very fundamental thing. If I have a CNF, right. And there are other things that need to connect to that CNF. If I want to do any kind of chaining at all, then I have, I need to have some kind of a connection between CNF one and CNF two. Now, those two CNFs, one of them, no matter what you do one of them is already going to have completed a startup process when the other comes up. The CNF completed startup process will not be able to change to accommodate the addition of a new client in any reasonable sort of way. Number one, but number two. If you have CNF one and CNF two and they're supposed to be connected to each other and CNF two crashes dies or otherwise goes away CNF one is now marooned. Not only is it no longer receiving the service that it needs, but it literally cannot get the service that it needs from anyone else. That means everything about its networking was set in stone at the beginning of time. That's not how services working Kubernetes. Services are doing a very different thing. Yeah, this is our carrying a payload over a Kubernetes intra cluster network. There's this fundamental presumption that every pod can reach every other pod. You want to reach CNF to reach every other CNF. Wow, I mean, you still have policies and things. Right, right. But do you truly want every CNF to be able to reach every other CNF? And do you want to pay the performance penalty? Okay, I don't think that we should try to resolve all the problems. But the really fundamental thing is that the better analogy is not to services. The better analogy is saying what if every TCP connection that you needed had to be established in your pod spec at the time your pod started up. That's the analogy. Okay, maybe I mean, I don't know, maybe.  That's the analogy to the situation and just like anybody would obviously see that requiring static at your static and fixed in time at pod startup time TCP connections is insane. It's the same thing. So, we're pretty much out of time so people have to continue this discussion at a later time. So, there's surely a there's surely a problem needs to be solved. And, and I think we're on track regardless of the difference of opinion they think we're on track to, to solve such a such a range of problems. Is there anything else in your last in the comments that or announcements that anyone else has cool with that. Thank you, everyone and we will see you all next week at the same time. Take care. See you. Bye.


Welcome to ToffyTV. This is the start of the 11th show for the Merseyside Derby on Monday night at Anfield 8 p.m. I'm feeling more positive about this game than I would have maybe before the Arsenal game I don't think Liverpool are performing very well at the moment and You know, I know it's only one game, but it was a really good performance against Arsenal For Everton, so we've got to do the same again take the performance in and trying that best to get a result I think it's possible. I think it's unlikely. I don't think it's likely. I just think it's possible But I'm gonna pick my team started with Pickford and net The reverse fixture at Goodison Park in the Derby earlier on East it could have been 3-3 I think Alisson had a great game. I think Pickford had a great game. He saved some really good chances Everton haven't scored a lot of goals this season with Dominic Havett-Lewin not being fit or touching go as the manager said You know, it's not likely that Everton are gonna go and score three or four goals or whatever like that Even though Liverpool haven't been great at defense recently. So He's got to be on it because one thing One thing that goes for all season actually as well We're not going to score a lot of goals and don't concede a lot and if you can't score a goal then try and get Nicker they can nil nil or try and win Try and nick all of a set of p like we did against Arsenal when they get one nil We can't be going out and conceding two or three goals In any game because we can't outscore teams at the moment. So you've got to be on it this game Me full backs Coleman and Michel Encoe Don't the job against Arsenal and I think and I think that's do the same again Liverpool has seen that likes to have most of the ball So they've got to be defending Michel Encoe's gonna he's done it before with with Mo Salah He's just got to be on it switched on and do his job defensively. I'm not too worried about him getting forward at this moment in time Because I don't know whether he's being asked to I would like to see him get forward a bit more to be honest But for the main worries this game is just defend Salah and let's just try to try our best to give a clean sheet On the other side Coleman. Yeah, it is it's patterns back on the grass now But Coleman still there for now you just got to do your job Just defend get forward if you can Make no mistakes and do the best you can you're coming into just do a job.  I'm sticking with Cody and Tarkovsky Really well together against Arsenal. I think some issues in the past I think certainly against Southampton Tarkovsky was playing more aggressive and Cody was dropping off and he was leaving to a space in Now they're being both told to you know Press into the midfield and win the ball back there and be aggressive and get up people rather than dropping off If they do the same this game, I think it will work really well. They run against Arsenal So I just think they're a really solid partnership They've been playing together all season and when you when you do that when you have so many games in a partnership You build you build trust you build on an understanding and and they're getting used to playing each other now So they'll keep getting better and better in my opinion in midfield I've got a just a guy in a gay in the six and then do Corey and I went on in front of him I think that three men midfield just worked against Arsenal. I just think it was it was just It was just class to be honest, you know, you've got tons of energy to big bodies Ghana's not, you know, the biggest but he's got all the energy and he's really good on the ball Oh Nana was really good on the ball. Decorey just puts himself about and he's dangerous going forward He's really good with the press And I look at Liverpool's midfield and that's their problem this season. It's been pretty rubbish They've got I think the best performing hours an 18 year old kid better titches of whatever his name is It would be nowhere near as physically, you know, capable as owner in Decorey So we've got we've got a really physical midfield. We can make it really hard for them to follow their energy So I think that We're probably best sticking with that on the On the right. I've got on the great on the left. Sorry. I'm the great.  You get out your feet and get it into the box But this is the difference, right? I've done a couple news not on my team so The game plan for getting the ball into the box with a big tag. Yes, you've got Decorey getting there I don't know how they're getting in there. We've not got Don't have it losing so whether I said play to feet or play different style of football I don't know that's what we might have to do because obviously we're not gonna like for like backup for Domages on the board Different different problem, but yeah so I've gone for gray because he offers something different in terms you can pick the ball that I've cut inside and he's got Talents of shoot outside the box and score or get into the box make a really fast one into the box and get a ball into the Have her across or shoot at goal. It gives you a different option to try to try and score a goal Rather than just get the ball wide and get it into the box And he's actually that top scorer You know, you're missing your centre forward. You're probably gonna be mobile someone to come in I've chose mobile get into that in a bit But you need a different option to try and score a goal So I've gone with Demari gray and that's not me being tight on Dwight McNeil You need you need quality of the bench anyway, so if he's not working if McNeil comes off the bench And puts in performance like he did against Arsenal then you might make an impact But I've gone with them I agree and that's not to slander McNeil I just think it's just we've got to find a way to score a goal in this game And it's not gonna be a set piece or don't account Lou and then I think my money would be on Demari gray as he is our top scorer on the right I've got Alex Iwobi. I think he was good against Arsenal I think he was better when he went moved over to the right. He's just got a Try and create as much as you can because like I said, we all don't like Harvard Lewin try and create for great great create for the midfielders and I Pick my old pace or try and get the ball into my old pace feet and try and get a bit higher at the pitch Got a bit closer to the striker and offer something something in the final third It's costing it's got to be better in my opinion You keep trying to pick people out They just whip it as fast as you can we're gonna get a ball and get it in fast because it gives defenders Let's have a chance to deal with it If you put a floater in that they're slow have the keeper will come do that or the defenders can see it coming He's got like that if you just smash a ball in he could bounce off someone it could fall to someone You don't know what's gonna happen But I put it will be in all my sight. I miss the season because he's our biggest creative more most creative player So I think it'd be silly to come out. I'm not front I don't know who else I meant to play there to be honest. I think Sean that Sean Deitch is the sort of type of manager to this If you strike us into them play striker there I don't think he's one for like putting the DeMarie great in there or something like that Who knows we'll have to wait and see but what I would say is we're gonna have to change how we played So I played against us look because we don't have a target man Who was the focal point? You know you get the ball up you go along to Domicavit Lou and he'll hold it up for like four five seven six seconds Which allows your team to get up the pitch didn't sound like a lot of time, but it is if you if you're in the game Mopey can't do that Mopey wants to drop deep and and ask the ball to feet and then turn So maybe you might get your wingers higher up might get people closer to him. You might play someone next to him I have no idea. I think it'll be more Mopey than sims sims and do anything for me to prove that he should be playing the Premier League Maybe off the bench. Yeah, but no in terms of Being a vital point to how we play I don't think sims can I know a lot of people think you can do the same job Is because there's the same size as Domicavit Lou and I was a big lad, but he's not he's not a type of player He's not a target man. He's a poacher Domicavit Lou is a big target man Who's really important to the way you play so it is going to be interesting to see what Sean Deitch does But that's what I would do If he picks the same thing I mean win then great I'll feel great about myself, but You know, I think I've got a few points doing this so But who knows who knows what's gonna happen It's just hard because you're not gonna like like striker and this is what we said in January We need to get somebody because Domicavit Lou I think staying fit all season But yeah, there you go. That's my start 11. Let me know if you agree Let me know what changes you would make if any at all. Obviously we have to change the striker Because as Sean Deitch said Domicavit Lou is touch and go. We might be playing a mind game Who knows but yeah, let me know what you think make sure you like this video subscribe to this channel And if you've got more videos including live daily exclusive videos then join us on software TV premier There is an opposition preview with Chloe from the red men There is some other really cool content like business for playing stuff like that So make sure you go and check it out. Thanks for watching and we'll see you later


All right, we're gonna we'll give it a couple minutes as people still file in here, and then we'll get this going I don't know about that I'll just hold on to this All right, we're two minutes Yes, we're two No, not quite 30 seconds 30 seconds All right, good. That's the way to start. All right Okay, welcome. I'm Ed Baldoff with Solid Fire. I'm gonna be the moderator so I don't get an opinion here We'll start with a few stock questions. We'll have everybody introduce themselves We'll start with a few stock questions that we've written up or I've written up And then if you guys have questions They want us to use the mic in the aisle way there So just jump up and stand behind the mic and I'll call somebody So let's start with The illustrious John Griffith and then we'll work our way across everybody introduce yourself What company you're with where you reside like where you live kind of thing What's your background? Are you an ops person program or systems networking that kind of stuff? And then how'd you get involved with Cinder? John All right. My name is John Griffith. I work at Solid Fire I'm from Boulder, Colorado Well, kind of Boulder outside of Boulder So I've been working on Cinder since the beginning So about four years A little over four years ago I started on OpenStack and started this Cinder project with a number of folks that are on this stage Hi, my name is Zhiteng Huang So I'm from China I based in Shanghai. I work for eBay I work in the DevOps manner So we have a private call Patrick East from work with Peer Storage From Bellevue, Washington My background I guess is mostly like embedded applications But now I'm doing cloud applications I'm Sean McGinnis. I work for Dell in Minnesota I work in applications too Currently focusing on OpenStack Cinder and PTL for the Mintaka release Hi everyone Sorry, I break mics and sound systems I'm Mike Perez, former PTL for the Mintaka Sorry, don't quote me on that Sorry, Keela Liberty. I'm working on that So yeah, I've been involved with OpenStack since 2010 With NOVA Volume and all that stuff I had in previous working on stuff driver and things like that Currently I work for the OpenStack Foundation Focusing on cross-project I like cats Hi, I'm Jay Bryant I work with IBM, the liaison basically between Cinder and the IBM driver developers Based out of Rochester, Minnesota And been working on Cinder for about three years now How did I get into it? I got really lucky and my manager picked me to work on Cinder And I've got no work with these fine people So it's a good thing I'm Xin Yang. I'm a technologist from the office of the CTO from EMC I'm also a core reviewer in Cinder and Manila I live in Massachusetts in the US I wrote Cinder Driver for EMC Storage And contributed back in Grizzly That is how I got started And since then I shifted my focus to work on core contributions My work includes consistency group support Incremental backups and non-destructive backups Hello, my name is Walt Boring I work for HP I've been working on Cinder since the Grizzly timeframe Along with Kurt Martin We added fiber channel support to OpenStack Way back when I support the three-part drivers and left-hand drivers And I work on OS Brick Alright, cool I'm figuring out who I'm going to ask Who's the first victim here? And I think that's you, Patrick So the first question is what happened in Liberty So specifically what were you working on And how did it turn out? Sure So Liberty spent probably the first half of it Doing pure storage stuff We added fiber channel driver, things like that A lot of, you know, probably 50% of my time Throughout the whole release Working on CI system stuff So it's probably a common theme I'm not going to ask any of the guys Who have to maintain one of those And also adding in support for image caching To Cinder So we got that in for Liberty A new feature caching What does that feature mean exactly? Sure, yeah So it's Previously, or well, if you don't turn it on It's disabled by default Every time you want to create a volume from an image We download it over HTTP From glance, store it in a temporary file And do a DD copy the image data over the volume So if you want to spin up like 10, 20 VMs really quickly Backed on volumes You do that 10, 20 times As opposed to something smart Not very quickly Yeah, exactly Yeah, it's very slow Whereas pretty much every one of our storage solutions That, you know, the people who pay us To work on OpenStack They support doing really fast image clones Or, you know, some sort of Way to, on your On your storage array at the sort of back end You can clone those quickly So now we'll download it once Keep it in the cache Subsequent request, we'll just do the clone So, you know After you've done it once It's just Cinder database calls And, you know Whatever management APIs you have to do For the back end So this is not going to involve Going through glance Or at the glance back end store Or whatever It always stays within your actual back end storage That is a smart copy on right And it's a generic solution Which is why it's tremendous Walt, what did you What did you wind up working on in Liberty And give us a little Okay, so I focused primarily on Finishing the project of OS Brick Which is something that we all kind of Were involved with for a long time So what was left In Liberty was to Extract the Lever volume drivers That were in Nova Because a while back We created the OS Brick project That was part of Cinder And in Kila we extracted that From Cinder and created a Separate project called OS Brick Which is in GitHub And it's owned by the Cinder project team And so what I focused on Was extracting that same code That lived in Nova And made Nova Lever volume Which is used as Brick Thanks Walt Shing So I worked on Adding support for non-destructive backup So previously When you do a backup You have to detach the volume So now you can actually back up A volume when it's still attached And also I added An API to support clone CG Which allows you to create a new CG from an existing one Cool Jay you got it I helped Encourage John to work on Replication v2 I helped keep That going so for those of you that know the History there that was a lot of fun And then And then I've also worked with My team to kind of finish cleaning up Oslo libraries in Cinder Removing stuff that we didn't need anymore And getting Working on improving the way that we're handling Config generation in Cinder So those were a couple of the major work items We hit during Liberty John you want to Not give away our session on Thursday But talk about replication for a second Perfect Well for Liberty Specifically Liberty but what did you do Liberty I worked on a bunch of bugs I actually spent a lot of time And I worked on replication So we did v2 of the replication Feature in Cinder We'll talk, I won't go into detail But basically there was a replication Feature in v1 Or v1 replication in OpenStack Already But it was a little difficult To implement and stuff So we kind of regrouped as a team And I tried to lead an effort to help everybody Kind of get on the same page And agree on a core Implementation that everybody could adopt And we just focused on Doing the core piece of it for Liberty And then in Mataka Everybody's Almost everybody already has a driver So when you say core There's no drivers out there that support replication today So I was very unpopular Because my stance was That no driver should actually implement it yet Because it wasn't finished Until two weeks before we Were actually cutting the RC candidate Which In my opinion we're releasing a brand new feature Especially a major one, two weeks before RC Is stupid So we shouldn't do it because Customers will be very unhappy Cool, anything else? Alright Mike, I'm going to let you summarize Since you're the PTL for Liberty Any more? Yeah, all these people did all the work And then I Just looked pretty But no, seriously I mean sort of I guess my own work is just Making sure that the things work Bringing them into test environments Making sure that the documentation is there And I'm trying to think of anything else Was miss, oh yes We had some things that actually didn't land And that I was actually actively testing But wasn't involved to actually Allow them to be merged in time There are a variety of You probably hear me preaching a lot Especially on Twitter about rolling upgrades And being a boring thing to keep talking about But the great thing that This community is actually Working on right now is actual Solutions to solve that Not just within Cinder but within other projects It's not myself that's working on it I'm probably going to butcher the name But Thane Fam Is, yes, thank you Thane Fam for doing all the work On making the services actually Independent of The databases as well as The services themselves independent of each other By having RPC compatibility All these solutions are things that Are either eventually or have Already been brought into Oslo So that other projects could have a base foundation Because I do respect the fact that Every single project is going to be unique Thank you You did the V2, the API V2 stuff Right, that was kind of your That was my thing, yes So don't tell yourself short So let's talk about my talk While it worked in Dev Stack Shirt That is like my only good contribution To this community So we're going to talk about my talk And now a little bit, I'm going to turn Over to Sean because I'll let you give a Summary and call out whoever else Is going to be guilty John's always guilty Well a lot of it is continuing The work that's been mentioned here The replication piece because it came in So close to the end Now I'd like to see some drivers Some of the issues there, the spec is great It's when you get into the implementation That you find out what you're missing What you didn't think of Rolling upgrades There's some work being done Of actually being able to test that A little more thoroughly That'll be great to have in the community Actually being able to have some Backing data that will work For end users And like everything I expect there's probably some work The third party CI, we've come a long way On that, there's still Issues, I'd say Everyone's done a great job Implementing their third party CI Now we need to start refining that And making sure that The results that we get from that are consistent And at the point Where if someone submits A patch and there is a failure Out there with someone's third party Results That it's not just ignored Because that one always fails I'm not going to look at it And I guess I'll open it up to the team here As well We have a lot of different initiatives going on Anything we want to highlight One thing I forgot to mention That we worked on in Liberty Is getting migration and retype That's been a constant source Of bugs and concern We've been working on that You know, as IBM contributions Going into Mitaka To finish up that Reporting back to the users better How migrations and retypes are progressing And getting that code out there So that's a continued focus Okay Yeah, I have something to add about What's been done for Liberty It's a big step forward That we have support for generic Migration Which means for those search plugins To support iSCSI or fiber channel Now as long as they have a connector In our OS Brick library Then they can do the migration From one back end To another one Why exactly did we do the OS Brick library What benefit did that bring to other projects For example If we Before we have OS Brick We actually have code duplication In other projects like Nova So Nova actually has a copy Of our Cinder's code to do the volume attach Volume detach When we want to fix a bug In Nova project, it takes a very long time Much longer than What we can do in within Cinder project So now we Extract the power into the library Which means that we have that All holding on control So that's a big way Yes So we are talking about Things in Mataka So I'm going to steal your thunder Unless you are up to speed Well I have thunder So Michael Doolock has been Spearheading a lot of the HA Involvement inside of Cinder itself The idea to have a multiple volume Manager processes that can talk to The same back in storage without Clobbering each other There are other things that help with that initiative I understand There's a big piece that was actually solved Or I won't say solved But there was a really great consensus That came from the summit today That I wanted to highlight For the distributed lock manager There was a consensus that we will start Having abstract layers for the different Lock managers so that we can begin to Make progress on this sort of initiative With volume managers which is a huge Win, huge HA win and all that stuff In Cinder Anybody else want to comment on HA? I'll let that one lie A lot of people have been Spending a lot of effort on it as well Gorka has been doing a lot of work With that too so Scott DeAngelo The whole community is Really working on this effort And we hope to get something From a talk I think that would be a big win For Cinder I said And I'm going to propose A completely different solution Just one other thing I want to highlight As well Scott DeAngelo And others have been looking at Microversions in the API A lot of work has been done there I believe in NOVA Manila as well What does microversions get us? Microversions allows us To add Changes between API releases Without having to come out With an entirely new API We have an issue right now Something just to discuss today We have the v1 API There were some issues with that Mike did a lot of work on implementing the v2 API We'd really like everyone To use that v2 API But we have a lot of users out there On older releases still using v1 And for whatever reason They can't migrate off of that So we'd like to get rid of that code But we can't That's one piece of it And then moving forward And being able to add new features We need a way To have some flexibility In what we expose through the API Without having to Reimplement things and cause breaking changes So one of the things Microversions in the API Is add things Remove things with a With a little less burden On long term maintenance Kind of One thing to add to that Is if you look at So the way the API code used to be structured If you look at v1 And you look at v2 And you do an LS on the directories It's about 90% Code duplication across the two So it's kind of Silly the way we've done it in the past The thing with microversions Is when you implement a change To a specific API call You can just update That particular call version And still have the old call Available as well Without having two branches Or two copies of the same code For a lot of you from a user's perspective This may not seem like a big deal But actually At the end of the day it's actually For a user And that's what's driving this Is making an improvement for users Yeah, it also prevents us from The need of releasing a v3 Just because we make a change at the API Which means a lot of work for clients Which is kind of a nightmare For operators too So I think this is a good thing And a third copy of all of the code A third copy, yeah exactly So while you have the mic Anything about multi-attach Is that gonna hit my talk? Awesome So there was a mailing Us post last week Someone posted About trying to coordinate That effort In the NOVA He's project coordinator, right? No, it was someone else It wasn't Mike, and I just Wrote the guy today asking If he's here because I'm Very interested in that Multi-attach, we implemented That sender a couple releases ago And we've tried very hard To get the code in NOVA To land, especially In Liberty, but it The amount of work kind of Had scope creep quite a bit And it kind of blew up a little bit On us, so There's still an ongoing effort To try and get multi-attach Implemented on the NOVA side So we can complete the end to end capability And we're still very interested In making that happen Cool, thanks for the comment Attaching your volume to multiple VMs If you want to do a clustered file system Yeah I'll use this opportunity to point out You do need a clustered file system for this There's been some confusion on that You cannot just attach the same volume To multiple hosts and yeah The EXT4 will fall completely On its face if that's what you're putting On your volume Hand the microphone to Shing please Shing, you've been working on Backup a lot Is there anything for Mitaka that you're planning to do Or thinking about Or anything else you want to talk about in Mitaka Yeah, so I'm planning to Export for a backing up Snapshot, right now You can backup a volume But you can't really backup a Snapshot directly yet, so This just gives the user Another layer of data protection Thanks Patrick So A couple smaller things I guess on my plate So one thing we've Tried to add in L That didn't make it, which we're going to get This time is trim support That's, you know, those of us With Flash arrays really want that And users that pay for our Arrays probably also want that So that's Work in progress, we're getting there Is that in Cinder or is that Is it in Cinder? Yeah, so we landed the Cinder part in Liberty The Nova side is on its way in For M And the other thing, so We have a session sort of on top of Microversions Experimental APIs Maybe, you know I don't want to say it's happening, but If you guys have interest in that We're going to have a session on that So they may be there Which would be nice for things like When we merge giant new features Before we release You might want to explain a little bit more About what that means Yeah, sure, so Sort of modeling after what Manila's done On top of their Microversions They allow for experimental APIs Basically things that they've added They've merged into the Core service that If you really want to, you can start using these But they're kind of like big morning label Says, hey, we reserve the right to change these They're not totally finished It's a work in progress The alternative is Times like, you know, for replication Or other features like that If it's too close to the end of the cycle We don't have enough testing We don't have enough drivers We either merge it and you have this feature That somebody could potentially try and use But it may not work or you just hold the feature For another six months And slow down testing again So it's kind of the Goal with those Jay, do you have anything to say about my talk? I think I recovered My points of interest Thanks, ma'am And John, last, but not least You got it Yeah, I got a whole list of things Yeah, that's... The most interesting one for me Is something I wanted to do for a while Is I want to actually decouple Cinder A little bit more from OpenStack Make it more of a stand-alone service That's consumable for Clouds other than OpenStack Bare Metal Whatever it might be There's actually some work We've finally gotten started In the Liberty release to do that for Bare Metal I think we have Long ways to go And there's a lot of potential And a lot of cool things we can do And we're going to talk about those things this week Hopefully come up with some good ideas Hopefully it's finally going to happen And I'll just shame this plug Session we're having about replication If you want to see how it's supposed to work John and I are going to talk about that on Thursday It's going to be awesome Alright I'd love the heckler to show up And we'll put him to work Alright, so You have not said much So I'm going to ask you to throw you the hot potato What do you think Cinder needs For Docker support? We'll put the potato around if you're done So There are Several different solutions To incorporate Docker with to storage So some said you should use Share file system Like Manila or something Some said you can work with Cinder because you can attach The volume to something And then detach when it's finished Something like that Within eBay we're still evaluating Solutions like that So I'm open to Any solution that works For us It doesn't have to be specific You bring a unique perspective The panel here because you are an end user So that's why I'm asking you some of these questions So Yes, so I think myself is Different because I used to work for Intel is still not Storage manner And I mostly focus on Cinder and not that driver part So I tried to Contribute that Peace So one thing I wanted to add He actually Wrote our entire Scheduler code base Which was a pretty amazing feat And everybody should give him a hand for it But To the Docker question I kind of left that out That's actually one of the things about The handling services that I was talking about That's one of the big wins actually There's different ways Everybody knows containers are the hot new thing There's different theories on Whether containers live in open stack Whether Magnum manages them Whatever you might do My opinion is You should be able to do whatever you want One of those things Docker has the ability to do volume plugins now Having a Cinder plugin Talk to all the Cinder drivers Would allow all of us to just Continue on upstream And give a solution for that That's one of the impetuses for that whole effort Over on this side Anybody want to comment on Docker? That was a better answer than mine So cool I think it gets to I don't know a lot about the details They're going to need to be done But I think it gets to the discussion Of what do we have to do as a Cinder project To be relevant and continue to move forward And we need to be looking at these technologies Like Docker and what's coming down the road Making sure that we're able to work with them So that we continue to grow So that was a good answer, John Alright, let's change to a different Oh, if you got it No, go ahead I just just see that on the mailing list There were the ops They are trying to propose a tag Called containerization Just to see if a project Can be deployed In a container and In production So I don't know if that's going to be approved Or not, but if it is Then it's a good tag for us to have Yeah, for sure, and that's kind of a different That's the other side of it And that's one of the That's one of the things that's come up In HA solution ideas and things like that Is actually containerizing the service As opposed to building a bigger service And an HA service Actually break the service down And let another tool Whether it be a mesosphere Or a Kubernetes or something like that Let that actually take care of the HA pieces And the scaling pieces, right So, yeah Alright, so we'll Change directions here a little bit And talk about CI systems So you guys wrote up some questions here We got a lot of third-party CIs What's your opinion on Is everybody getting it right? Is it working? Because you're going to have to be the enforcer this time Alright Like I mentioned before I think everyone's done a lot of good work Getting their CI systems where they are today There's still definite Improvement That needs to be done I think The main thing being The results of them being useful Of being able to see a failure And not just blow it off Of actually looking at the failure You know Our third-party CI Just like a lot of other Ones still has random Failures Makes it a little less likely That someone's actually going to take a look at that When their patch is submitted And gets a failure result There's A lot of moving pieces I think there's just different areas We just need to work on refining how it works And I think Rami have done a lot of Great work Starting to standardize The CI solutions So that there's at least consistency Across everyone's CI's And hopefully that leads towards Bug fixes, improvements And making everybody's CI work better So one thing I want to add to that Just so people know And don't get the wrong idea The failures that some of these systems Are seeing and stuff like that The intermittent failures They're not actually that the back-end device Doesn't work and That that's failing They typically end up being things like The actual Jenkins deployment failed It's something in the automation Or in the CI itself That's the piece that's failing So if it was The driver or the back-end wasn't working We would have yanked them and thrown them out already So that's not the problem About this round I think the policy we had in the past It worked and we need to Enforce having Third-party CI running And if we can't get involvement From a vendor To make sure that they're at least working on it Then We have to take their driver Out of the code tree I think one of the good things Our team has been doing I look at Walt specifically Is Code is getting pushed up Now if your CI system doesn't run Against your own code We're not going to give it a plus one Or merge it at all So that's been Painful for some people Like me But it's given the right tool To go back and say look this is serious And we need to be doing this And run Tests running frequently So that we know that things are running For everybody else It's something to watch for Any questions from the audience I apologize if this Doesn't make sense in context But hopefully you guys can add the context My question is about the scheduler There's been some ongoing work So what's the current state Of the sender schedule You're talking about multiple volume managers One of the jobs of the volume manager I believe used to be picking the host Which who you're going to schedule the volume on Is there been any work going on that Or what's the current state of the schedule How does that work Or is that even still a thing in sender So I think that was done In keel That we added support for pools So the volume manager doesn't have to do Any scheduling Right now volume driver Will have to Tell how many pools That this driver has And then that scheduler made a decision So we have No So each pool will be treated like The whole storage backend So it's just like you have multiple Storage backend that single driver Can talk to It's just that you can even create a type That is specifically mapping to One of the pool not the entire Backend So our model is a little bit different At this point so all of our stuff Actually goes through the scheduler Before it gets to the manager The manager right now is actually Just kind of a shim Of common code that goes Into the drivers And as he was saying Basically each pool Looks like it's own independent backend And it's treated as if it was Was a backend by itself The problems that they're trying to solve On the Nova site are a little bit different Than what we've got And they're also looking at doing Things like more of a If I recall A common scheduler Like an Oslo version of scheduler code But they have some different problems That they're trying to solve that We don't really have We don't have the problem of scaling To a thousand backends We don't do that sort of thing in Cinder Most customers have one They might have three or four But Nova they're looking at I'm going to have a thousand compute nodes That I have to schedule across So that's a different problem Following on the aspect of scheduling On the storage pool as opposed to the host Or migration then So would we be migrating between storage pools then? Yes Because we support like Migrate from one storage backend Like solidifier to EMC So we definitely support the migrating From one pool to the other It's just a problem that If the storage backend is able to Create a shortcut for you Like if these are the two pools From the same backend That was created on the same They might have a chance to give you A shortcut to Accelerate the process Absolutely We're supposed to go to the tent after I see a clap starting so I might Be at the top of the beer hour I was going to throw out one more question Around testing if you guys didn't So what do you think the current state of testing is And I'll throw the hot potato like How are we going to test replication If it doesn't have replication Walt you're grinning Shing you're grinning You guys can start the comments down to that end I'm grinning just because It's a difficult problem We're still working on Actually implementing our replication Code and we're doing a lot of manual Testing right now But obviously we need to get to a point Where we can add that to the CI system That one is a very difficult problem Because we have multiple Arrays that we support And we need a lot of hardware To do the replication support Because we support ice guzzy and fiber channel For two different arrays And that requires a lot of A lot of hardware for us to do Yeah it's And we're struggling with that right now Because we have a lot of different scenarios That we want to test not just with replication We want to test multi Attach multi path We want to get it off for ice guzzy and fiber channel So our number of CI jobs Is kind of blowing up a little bit And it's a good problem to have But yeah Shing? So I think to test replication We need to provide API to do like a test Feel over because otherwise If you test it, feel it over We also want to feel the background We need to also be able to Have that support in the API as well I'll just go down the line I guess kind of a question Out to the rest of the core panel Is it something that we need to look at Each of the Each of the different Driver CI systems Being in charge of setting up Their own replication test Yes, yes But So here's the What's the but? I would say that since we've got ourselves Into this mess where there are Features that we are allowing Vendors to have In their drivers that aren't In the reference implementation Which I think is a horrible thing We should have never done When you're going to write the replication For the reference implementation But we're there, we've done that We broke our rules and we've done that So what that means now is That exactly what you guys are saying In my opinion If you want to Think about a feature like replication Or consistency groups or something like that That means that you Should tack on automation And testing to your CI That runs that, and that's an easy There's a process to do that, it exists So it's not hard to do So that'll be a fun future Discussion What do you think about test coverage? Mike, good today, needs more For what we have Needs more, sure John, more Definitely needs more But like the replication issue Being able to set up all the different Scenarios is a challenge That we just need to keep working on So I wanted to go back to the other thing Like exactly what John was saying Because I've actually Part of the cross-project work That I'm doing is also looking at Expanding out third-party CI Sender Like Neutron for example Neutron is actually more In my opinion more of a complicated Problem than it is inside of Sender Because there are Like with Sender the most that you can have In combinations is zone manager And said fiber channel driver And that's about it And then with Neutron though there could be A variety of switches, routers And then Whatever else that's involved There's a variety of plugins that are involved And so I think that's exactly What we've been looking into Is a way to actually Where in those CIs we can actually trigger Those particular tests For those certain kind of combinations Of different plugins Or drivers That are involved with that test If that makes sense Alright, we're near the end of time So we'll call it beer 30 A short one If not going once Going twice, thank you Thank you