I'm Alex Nisbet and this is the Service Design Show. In the Service Design Show we talk to people that are shaping the service design field about the current state of the industry, exciting new developments and challenges up ahead. Today my guest is Alex Nisbet. Some people may know Alex as he is the head of design at LiveWork and something that you might not know is that he almost became a potter but chose graphic design instead. So it's luckily that we have him in the service design field today. Welcome to the show Alex. Hi there. You're right now in the UK, right in London. Absolutely, London Town. London Town and I see the small LiveWork logo there. Yeah, awesome. Alex, you've been in the industry for 11 years. You just told me and I'm really curious what is actually your very first memory of service design? Mark, that's a great question. I think it actually goes back a little bit further than 11 years. So a friend and I, we were talking about cycling and we were talking about sharing bikes and things like that. We were both graphic designers so we were kind of creating people and we were just trying to figure out how on earth could you possibly design something around sharing bicycles? And this is a long time, this is before Velib in Paris and all that kind of stuff. And we just thought wouldn't it be great if you could but we just had no idea how you could do it. So I think that was probably without me realising it, my first thoughts about service design. Can you recall when you actually stumbled upon the term? Yeah, so I was working in interactive, I guess you could call it interactive marketing and we were already thinking about the user quite a lot and how to create messages for them and help them do what they wanted to do. And somebody got in touch with me and said hey, there's a position that a company called IDO and it's about service design. I think you'd be really interested in that. So that's where it all started. Alex, let's explain to the people that are listening or watching this episode how the format works, right? I've got a stack of papers with some topics and you've also got a stack of papers, right? Yeah, there we go. And your papers contain question starters. So I'll be holding up a topic and you'll be holding up a question starter and we'll co-create the question that we're going to discuss from there on. Cool. Sounds good, right? Let's just jump in and let me pick a topic that's... Well, I'm not sure quite what it is so I'm really curious what you'll make of this. It's called customer performance. Okay. Well, here we go. Do we get that? Yeah, absolutely. How can we... How can we design for customer performance? So first off, I think it's probably worth just thinking about what we mean by or what I mean by customer performance. So four years ago I worked at the London Olympics and one of the big challenges that we had was that the most spectators were only ever going to visit the Olympics one time. Okay? So they had no kind of previous experience and they probably weren't going to visit again. The Olympics and the way that we designed it had to be quite complex. You had to arrive very early. You had airport-style security. You had venues to navigate that you'd never ever been to before. And when you sat down, it was all going to be the most brilliant, you know, the greatest show on earth. So we realized that we really needed to help spectators to, if you like, get to the right place at the right time and in the right state of mind. And of course, if they did that, they were more likely to spend a little bit of money on the catering or the merchandise. But possibly more importantly, they provide that sort of perfect backdrop, if you like, for the broadcasters and the athletes. Now, in order to do that, it means that the organization had to be high performing, if you like. So not just the usual things like great signage and great information, but also thinking especially about ensuring that spectators knew that they had to arrive super early, that the games makers were on hand to help guide them to the right place, and that queues were minimized. And even if you did have a queue, it was actually a fairly okay experience. Now, this idea of customer performance or spectator performance that can also be applied to think about the airport. So oftentimes you're in a hurry, you don't know which way to go, the gate may be changed, but you really need to get to that gate on time. And if you can do that, then you're more likely to buy a coffee or maybe some last minute duty-free. But especially, you're going to be a happy passenger, if you like, a high-performance passenger. And that's what the airlines like. They like us when we're flying to be happy, to be punctual. But that means that the airport has got to help us get to the right place at the right time. And it works for hospitals. It works if you're visiting a city for the first time as a tourist. It works. So this idea of high performance for a customer. But the challenge there and how can we design for high performance is really to ensure that the organizations and businesses understand the role that they play in the service that they provide and in the customer or the spectator experience as well. So it's a very critical thing. It's a big challenge. But if you do it right, then you get satisfied customers. You get customers that are going to come back, customers that are going to advocate your service. I think this is really recognizable for everyone who's been a customer or a spectator or a patient or a client in a situation which is pretty complex and you're there for the first time. So we all know this. I've never heard of the term customer performance in this kind of sense. So it's really intriguing. What is your takeaway from the Olympics? What were the main ingredients that actually created for high performance customers? So I think there were two things. There was the fact that this was going to be a once in a lifetime opportunity for London but also for the city, for the spectators, if you like. And in that respect, at the same time, we knew that we wouldn't get it right first time. So it's going to be hugely impactful, hugely important, but we knew that the first day, the second day, the third day, it wouldn't be perfect. So I think it's acknowledging that you're not going to get it right first time. But secondly, I think my main takeaway probably from the Olympics was that this sense of continually improving the experience and for us at the Olympics, that was on a daily basis. Not just leaving it until next week or next month but actually refining the experience on a daily basis. And I think that's something that every service provider can take a little away from. So could you say that you actually designed more of a mechanism or a process to continuously improve the service instead of the service itself? Yeah, so that's something that I was very involved with at games time. So we were interviewing spectators as they were leaving Stadia. We were monitoring social media to hear what they were talking about, where their pain points were. We were observing their kind of behaviors at the venues. And overnight we would create, if you like, a set of scores for each venue on the how they were performing on, I don't know, speed of cues or how efficient security was or how spectators felt about the quality and the value for money of food and drink or things like that or the quality of the sports presentation. So all these things we were measuring on a daily basis. Real-time monitoring of service quality, service experience. And then feeding that back to the venue teams and the sport teams so they understood where they had to improve. And what was the biggest challenge in that if you look back on this experience? If you could pick one. Am I allowed to say lack of money? It's an interesting case where the, so London was the first host city, if you like, to really explicitly design with spectators in mind. Clearly there have been spectators a long time before that but we were the first city to actually have a spectator experience team. And that, but that meant that they didn't quite understand how to use us. We really had to build a very, very strong business case and fight for the funding to do what we were doing. Because back in 2012, you know, we were only just really coming out of quite a bad sort of financial crisis really here in London. So it was very much an austerity game. So as sort of creative thinkers, if you like, we had to be quite resourceful in how we used the funding that we needed. But once we did put mitigations in place, the organisers immediately saw the value that we were bringing. So once you make things tangible, often that's when organisations, that's their aha moment, I guess if you like. And have you seen any other field or sector taking learnings from what you've learned at the Olympics? Yes, so interestingly some of the airports, the train providers are beginning to think in that way. But they also, I'd say the city of London has taken a leaf out of the Olympic book and certainly the work that we've been doing with Transport for London in focusing on the visitor experience. So that, if you like, the experience of the Olympics began to cascade down to lots of other organisations at home or far away. I really like the topic because of course everybody's heard of staff performance, stuff like that, but customer performance, like I said, is quite new for me. Any tips or resources where people can learn more about this topic? Yeah, well, that's a good question actually, Monk, very good question. I think there are some people are beginning to write about it and I think partly out of my Olympic experience. So the new book by Live Work. We'll put a link up there. So we talk about that a little bit and I'm constantly talking about it. But it would be interesting to see of course at the Olympics in Rio later on this summer how that experience is. So I think the real proof there is how the spectator experience is there. Interesting. Alex, let's move on to a second topic because time is flying by. This is a topic that has been on the show a few times already in its capacity building. So there we go. Who are building capacity? Question mark, right? So I think over the sort of 10 or so years I've been in service design and I still feel myself as a relative newbie, if you like. Those kind of organisations that have been continually focused on building their capacity to design services have been those in the public sector. So if you know the story about teaching a man to fish and the public sector, especially here in the UK, have always been very focused on developing the skills internally to be able to design, deliver and operate services. So although they've been buying service design from design consultants, they've been equally focused on figuring out how to do it themselves. And this over the last few years we've seen an increase in commercial organisations wanting to do exactly that same thing. So maybe there might be a time when service design consultancies and maybe that time is now that if you look at the volume of services being designed globally, there is a time maybe it's now where service design consultancies are doing that much but then organisations are designing and delivering that much. And something about service designers, we're incredibly open. We want to share everything and we want everybody to succeed at this because as somebody once said to me, there can never be too many service designers. That's a nice statement, absolutely. I need to plug my computer into some power. Let's do it, yeah. Two seconds. So a lot of the work that we do here at LiveWork is not just about delivering content, if you like, actually supporting our clients in improving and innovating their services. But I'd say an equal amount of it is explicitly about helping them to build their internal capacities, their abilities, their skills to be able to, shall we say, not necessarily become a service designer. I don't think that that's necessarily the thing but certainly to understand what good looks like so that they can manage service design projects, if you like, but also how they in their own way can improve and innovate services using many of the kind of tools and techniques, the methods, the processes, if you like, that we are all familiar with. I've been saying this also and I see this development really strongly in our field. Maybe it's like you said, not becoming a service designer themselves but at least becoming better service design clients and being able to better articulate the challenges that they are having. Yeah, we're seeing a rise of the manager, if you like, the people that are actually employing us or buying our services. These design managers are experienced at design themselves and so they are very able and competent at helping us to co-create the brief, helping us to set up the project, helping us to work with their internal teams. I understand our language. There have been times, of course, where we've been working ahead of marketing. They kind of get it but they don't quite understand exactly all the detail, if you like, so having an advocate inside an organization like that is incredibly powerful. So my question also, Alex, would be what does this mean for service design agencies and the people that we employ are used to designing service and not per se being the people who actually train our clients? That's a good question, Mark. I think actually, you know what, a lot of service designers, by definition, if you like, we're very empathic, we're very curious, we're very open, we're very positive. I think those are all great attributes of somebody that would share, maybe teach, share their skills. So actually, I find that a lot of service designers are naturally very good at helping our clients to kind of do better, if you like. And in many respects, it's a very similar kind of skill set to design research and it's a very similar kind of skill set to workshop facilitation in many ways. I think we're on the whole, we're all very comfortable working with people and understanding their real needs and supporting them and in many ways that's what building capability is, a key part of building capabilities. So if you fast forward five years and think about capacity building, what do you think has changed in the next five years? Well, over the previous five years, we've seen a huge growth in the demand for capacity building inside our client organizations, right across every sector. The next five years, that's only going to increase still further. I think we're going to see more trained service designers working inside organizations permanently. So being members of staff, if you like, I'm sure there will be a time when we see designers at the very highest level of organizations, maybe on the board as well. So I can see those all being very positive things. I think we're also going to see a huge increase in the numbers of people that if they had to select a number of skills that they had, they would probably say that designing services or improving services was one of those. So I think there would be more people talking about it, more people doing it. And that's only going to be a good thing. Yeah, absolutely. So the field is definitely moving in a new phase. We're entering a new era. That's absolutely my gut feeling. I think you're absolutely right. And I think thinking about the UK and the government digital service, GDS, so we're putting service design on the agenda of central government. That's been, I think, a major turning point for us here in the UK, because for many organizations they could say, well, if it's good enough for central government, then it's going to be good enough for us. So I think that has been one of those turning points, especially here in the UK. Alex, we need to move on to the third topic. And you already touched upon it a few times. So we had a sneak preview. The third topic is about public services. Well, I picked this one up. So what if public services just kept on improving? And what if public services actually became better than commercially provided services? Now, wouldn't that be an interesting thing? I mean, I was thinking about this a little earlier today. I was thinking that the world doesn't actually need more chairs. It doesn't actually need more smartphones. What the world needs and what people, our citizens of the world, we need to feel safe. We need to feel valued. We need to feel healthy. And many of those provisions come from what we see as the public sector. Us as citizens have the right to feel protected by our country, to feel healthy by our health services, to feel valued and to have employment and all those kinds of things. And I think that the public sector has obviously has a huge role to play in all of that. And I think, yes, I touched on this before, the public sector is increasingly looking to design approaches and especially service design approaches to help them address the huge challenges that they face in doing all those things, in keeping us healthy, in keeping us kind of warm and keeping us clean and all of that kind of stuff, supplying those utilities, it's under increasing pressure to do that with certainly the huge cuts in funding that we've been seeing, with the aging population that we have, with the increasing movement of citizens across borders as well, which is adding huge complexity for those at local government level or under national government level. So I think that those in the public sector here in the UK and in many, many countries around the world are using service design approaches as part of their toolkit, admittedly, of course they're using lots of other techniques, approaches, but they're using service design approaches very successfully in order to deliver better services for less cash effectively. They're getting very clever doing that. And of course they're doing that by designing their services around real users, real needs. So it's kind of a basic for us, if you like, which is being adopted and adapted by probably, for citizens, the most significant provider of services that they're ever going to interact with. So what do you think is... What is on your mind if you think about public services compared to maybe the commercial services and I really like the statement that what if public services actually became better than commercial services? What are the questions that pop into your mind if you think about this topic? Well, I think that one positive is that I've always seen a lot of young designers, those that are graduating, if you like, with more of a passion for doing good, designing better, if you like, not designing more chairs, but actually trying to tackle some of the tough challenges around health, aging population. Well, I feel very confident, if you like, that there is a steady stream of very talented and creative designers who want to tackle those challenges. I'm also very happy to see that there's a steady stream of discourse, of conversation between the public and the private sector, sharing, if you like, and trying to understand how they can partner, how they can work better, how they can collaborate and learn from each other as well. This is what we're seeing, and there are some organizations in the UK, so I'm thinking about the UK Policy Lab, where they're reaching out to big business in the UK to get them to help address some of those challenges that young families have, for instance, when one of the partners is going back to work. So really trying to address some of the social questions that are around. I think it's a very exciting place to be. At LiveWork, we're very lucky because we work in both sectors, in both public and private. So that satisfies, I think, a lot of the designers because they feel very driven to address social design issues. But it also means that our clients benefit from the experience that we have working across the both, and if you like, sort of, cross-fertilizing those experiences. Right. I think we could do a whole episode on related to public services. Like you said, service design has a strong history in the public sector, so we should value that, cherish that, and embrace it and develop it even further. Alex, maybe you get this question a lot by young designers at LiveWork or outside. People that want to get into service design, people that are maybe just graduated, coming from a different sector, what would be your tip for them? Okay. My tip would be to... I've got lots of tips, lots and lots of tips. Like, never give up. Think positively. Try as hard as you can to get an internship somewhere.  I think there are a lot of very talented young designers that are going, you know what? I'm just going to go out there and do something myself. I'm going to collaborate with some like-minded young people. We're going to go and find an opportunity. We're going to build a pop-up studio somewhere on a high street in a store that's not used, and we're going to figure it out for ourselves. So I think one of the great things, possibly about design these days, if I can say that, is with the technology that's available to us, you don't need huge facilities. You don't even need huge funding, arguably. You can make a difference, I wouldn't say very easily, but you can make a difference on your own. And I think that's arguably... It's like musicians these days, they don't need record companies. So service design challenges don't necessarily need service design consultancies with a different mindset, thinking about the user, thinking positively, trying to understand the challenges in a kind of holistic kind of way. These are all things that we can do straight out of college as a young designer. We can do that. I'll summarize this as go out and make a difference. There we go. Perfect. Alex, this is your opportunity to actually ask a question to the people that are watching this video or listening to this episode. What would you like to ask them? So I'd be really interested in understanding... So here's the question, right? I think service design has always talked about an approach, a process if you like, and we're always talking about being user centered. We're inspired by our users. We validate our ideas with our users. Is there ever going to be a time, do you think, when we're actually going to know enough about some kind of situations? I mean, let's take an example, taking a train journey. Is there ever going to be a time when you know what we don't need to do user research? We don't need to speak to users because we have enough experience, enough intuition to do it ourselves. So that's my question. Will we ever know enough? Well, yeah, to do it ourselves. I don't think we'll ever know enough, but yeah. Here is what people will say about this. Alex, I know it's been earlier for you in the studio so thank you for being here, thank you for taking the time and giving us a bit of an insight on the topics that you are currently thinking of and working on. Mark, it's been a pleasure. What are your thoughts about the topics we've just discussed with Alex? And if you have any suggestions that you would like to see next on the show, be sure to let us know down below in the comments. If you enjoyed this interview and would like to see more service design pioneers, be sure to subscribe to our channel and check out some of the past episodes. With the service design show, we help you to stay one step ahead in service design. For now, thanks for watching.


Are we ready? Okay, let's start. Go ahead and start. Hello everybody, welcome here. It's glad to see so many people interested in the L2 and L3 agent. I'm Roselas Blandido, I work for SUSE, and I've been contributing to Neutron for almost two years now. And I'm Carl Baldwin. I've been working with HP, and I've been contributing to Neutron for about the same amount of time, about two years. And I'm a core viewer on Neutron since about a year ago, I guess, and I also run a meeting and lead a sub-team that works on, and focuses on L3 subjects within Neutron. So that's what I've been having fun doing for the last two years, and it has been a lot of fun. We wanted to talk about today some of the work that we've been busy doing, for the Kilo release, and also spilling over into the Liberty release. We've been heads down working on the agents, looking specifically at both performance, and also the scalability of the code base. We wanted a code base that we could take forward into Liberty, and even further, and more easily understand the code base, dive in, and actually implement some new features in it. And along the way, we added some excellent functional testing, and improved the unit test coverage of both agents as well. We'll start with a simple old typical open stack deployment diagram. We're going to focus today on the network node, and the compute node. And we're going to throw everything else into what we call a cloud controller node. You may have these split up. You may have HA clusters for every service that may be totally split up. We're going to focus on the other two. The network node and the compute node. Everyone knows what the compute node is. That's where the VM runs. That's where the computing happens. The network node, there are typically fewer of these in a deployment, but there can be multiple. And they're the node that get data moved around between networks and to the external world. And on the network node and the compute node are where the L3 agent, the L2 agent run. This is where they do their work. And this is where we're going to be talking about them. Rosella is going to start talking about the L2 agent and tell you how it works. Okay, so the L2 agent is the agent that runs on the compute node. And this is because its main responsibility is to configure the software bridges that are on that node. Usually there are two bridges, BR end and BR tunnel. BR end is the integration bridge. It's the bridge that takes care of tugging and untugging the traffic that it's coming from the VM and going to the VM. To tug it, it uses a local VLAN ID that it's assigned to the network. And the other bridge is BR tunnel. It's the tunneling bridge. And this bridge takes care of translating the tugged traffic. So translates the VLAN ID into the segmentation ID that it's used then for the tunneling. For example, if you're using great tunnels, the segmentation ID is used to specify the tunnel ID. The main task of the L2 agent is to wire new devices. By new devices, we mean the tab interfaces that are created by NOVA and are connected to VMs. We'll see more in detail what wiring means. And the L2 agent is also in charge of applying the security group rules that are firewall rules and they are implemented in Neutron using IP table and IP set. The L2 agent communicates constantly with the Neutron server and it uses RPC to do that. Now let's look at step by step what happens when a VM is created. I tried to highlight the role of the L2 agent in this graph. So when NOVA compute gets the request of creating a VM, it issues a call to the Neutron server allocate network that you see at point one. So the Neutron server will create the Neutron port and will write the proper data into the database. And at this point, the NOVA WIFT driver will issue the plug operation. The WIFT driver, in case you don't know, is the driver that NOVA uses to plug and unplug the virtual interfaces into the integration bridge. So it needs to configure properly. Like if you use a specific Neutron plugin, you need to configure the WIFT driver accordingly. Like if you're using OpenVswitch for Neutron, you need to use the OpenVswitch WIFT driver. So when the WIFT driver plugs the interface, the L2 agent will notice it because the L2 agent scans constantly to see if a new device appeared on the host. And you see that at point three. Since the L2 agent doesn't know anything about the device, it needs to request the details of the device to the Neutron server. And you see that at point four get device details. With this call, the L2 agent will also pass the host ID of the host where it's running. So the Neutron server can now bind the port point five. That means writing in the database the initiation between the port ID and the host ID. Okay, then the last step of the process is the L2 agent that notifies the Neutron server that the device is up with the UpdateDeviceUp message. Let's look now at the workflow of the L2 agent. So the L2 agent has a loop and some specific events trigger some reprocessing. So the loop is entered only when some conditions are met. The three kind of events are OVSDBMonitor has an update. OVSDBMonitor is the tool that the L2 agent uses to know if something has changed on the compute host. Specifically, if a port was added or deleted. So this is one event. Another event that triggers some processing is if the agent gets some message from Neutron, it can get two kind of messages. The messages that are related to some security group change could be rule updated, member updated, and a message that is related to a port update. And the last event is open this switch restarted. So this of course needs some reprocessing by the L2 agent. So let's see now how the L2 agent detects that there was a port change. OVSDBMonitor, before our work, didn't really pass any specific information to the agent. It just passed a flag to indicate if something has changed on the host or not. So the L2 agent needed to keep an internal dictionary registered port of the port that he has already processed. And then when has update for OVSDBMonitor is true, the obvious agent needs to do a diff between the registered port and the current ports on the host to infer which port was added or deleted. For example, if a port is on current port, but not in registered port, it means that the port was added. And the other way around, if it's not in the current port, but it's in the registered port, it means that the port was deleted. So now let's see more in detail what kind of processing is needed when there's a change in the host. So when a port is added, the first thing that the L2 agent does is request the device details, as you were seeing in the graph before. Once it gets the device details, it knows how to wire it. And by wiring, we mean assigning a local VLAN ID to the device if it's the first device of the network that the agent sees. And of course, install the proper flows, like install the flows for tagging and untagging the traffic. Then the other operation that it's performed is setting up the port filters. So install the proper IP tables rules. And then the last step is notify the neutron server that the device is up. For a port deleted, the operation are just the opposite. So filters are removed, the neutron server is notified that the device is down, and the local VLAN ID is claimed if it's the last device of the network that it's on the host. For neutron server messages, so as regard port update message, the processing is the same as for added ports. So updated ports are processed exactly the same way. For security group change, the filters are reapplied for all the devices involved in the change. So the last event is the case that OpenVswitch restarted. So how does the agent knows that OpenVswitch restarted? The agent at startup installs a flow that we call canary flow, and once in every loop it checks if the flow is still there. If it's there, OpenVswitch is fine. If it's not there, it means that it was restarted. And this, of course, requires to reconfigure all the virtual bridges and also to reprocess all the ports on the host. We need to reinstall all the flows again. So if an exception is drawn during the agent loop, the behavior of the agent before our work is simply to have a full rethink because the agent doesn't know which device has problems, so it needs to rethink everything. So registered port is cleared and all the ports are reprocessed again. No, Carl. Okay, thank you. While the L3 agent shares a lot of the same architectural aspects as the L2 agent, it's completely different. Where the L2 agent, it gets you attached to your networks. The L3 agent, it gets your routers attached to each other. It gets things moving from network to network and from network to the external world. Same deployment diagram here. I wanted to go through kind of the evolution of the neutron router. Before Juneau, there was only one kind of router in this implementation. The router, there was one router. It was hosted on one network host. And there is some bottleneck there. So if you... And there is a single point of failure, too. So to address both of those issues, in Juneau, we added HA and DVR modes. Now, this has been six months past, but it's still kind of new. The HA mode takes that one router, that one legacy router, and it replicates it across more than one, three, four different network nodes, possibly all of the network nodes that you run in your deployment. It uses the virtual router redundancy protocol to monitor the routers and name which one is the active one, and if it fails, it moves it. DVR took a little bit different take on addressing both the single point of failure problem and the bottleneck problem. What DVR does is it takes the router, it breaks it into pieces, and it pushes it out to the compute nodes so that as soon as a VM produces traffic that needs to be routed to a different network or to the external world, that traffic's routed right on the same compute host before it even leaves the NIC. But DVR did leave a centralized part. The centralized part takes care of giving access to the internet to instances that don't have a floating IP, that share that one address to the internet with other instances behind the router. There's also things like VPN as a service which are hosted in the router and also firewall as a service, which it wasn't either possible or it wasn't obvious how to distribute that out to the compute hosts. For the centralized part, there's an API for administrators to manage where these routers live so you can move them around. You can do maintenance on a node and evacuate a router. Now I wanted to dive a little more into the L3 agent. The function of the agent is really pretty simple. It listens for notifications. And the neutron server, when anything changes in the router, it sends a notification to the L3 agent. The notifications, they can be missed. They can go out of order. The L3 agent may not be there. But the notification kicks off the process. It tells the L3 agent, hey, there's some work to be done immediately. The agent takes that and it sticks it into a processing queue because it may actually have more work than it can get done at one time. So it sticks it in a processing queue. Normally it pops right up to the top of the processing queue and it's pulled off. The agent has the capacity to work on a few routers, four or eight, depending on how you configure that. It can work on a few routers at a time. But if it's working on every router at the same time, sometimes things actually get a little bit slower. So there's one more thing about the router processing queue. There are two types of events that get put into the router processing queue. There's the event that comes from something changing in the router. This ultimately is sourced from a user action. They do something or a script does something to change how the router is configured. And the other type is just like in the L2 agent. When something goes wrong or when the L3 agent restarts, it goes through a full sync. It's a little bit of an inefficient operation to just throw your hands up and say, well, everything could have changed so let's process everything. But that's what it does. And on a restart, that's kind of what it has to do because it may have missed any number of events while it was down. That's the other type of event. The kind of event that comes from user actions, those are prioritized over the more maintenance full sync type of events so that the L3 agent stays responsive to changes that come from users. The last thing it does, it also sends status updates so that when you plug ports or add floating IPs to the router, you can see the status of that. Now, diving even deeper, router internals, they can get a little complex. We use network namespaces in Linux, which if you want kind of an analogy, a network namespace is kind of like a VM or a container for a network device. It's not like a VM, it doesn't have file system virtualization or anything like that. It's really just virtualizing the network stack part of a VM. And the L3 agent relies on L2. Just like Nova relies on the L2 agent to plug the ports and do all the L2 wiring, the L3 agent relies on the L2 agent for that. But then once those ports exist, whether it's an OVS port or a VETH pair, which is basically like a wire with two ends that you can plug in that exists in the kernel, those can be moved around into a namespace, just like taking a wire and plugging it into your device. And then the L3 agent, it configures the IP addresses on the interfaces, it configures the routing, whether that's the simple network-to-network routing or if there are extra routes that have been configured on the router. It configures all that. It uses IP tables to implement the floating IP functionality.  Neutron figures out which router that instance needs to go through to get to the external network. And it uses NAT to implement that floating IP on the router. There's metadata access. There's the shared access for instances without a floating IP. And then some of the advanced services also get into the router. And they're integrated right in there. Firewall is a surface and VPN is a surface, most notably. I also wanted to go into, starting in Juno, we moved the L3 agent out to the compute host. And I wanted to kind of show you how that works, how it takes out the network node from the data flow. This is for DVR only. So it only makes sense to bind a DVR router to an L3 agent running on a compute host. But once it's there, it can host floating IPs for instances that reside on that compute host. And it can also do what we call east-west routing, which is routing from neutron network to neutron network through a router. I'll take questions later. So these two boxes, they're compute hosts. And we have a VM on what we call the red network and another VM on the green network. So let's say VM1 wants to talk to VM2 on the green network. It starts pumping out packets with VM2's address on it. Now, there's a little bit of L2 agent integration in here at BRN, where it says, okay, I know this packet needs to be routed. And I'm going to send it straight into the router that's running on the same compute host to be routed. There it goes through the routing tables, comes out. And then it comes out as if it... and travels from compute host to the other compute host, just as if the two VMs were on the same network. And it comes in, it goes straight to the VM. Now, the replies originate from VM2. They come out, the same logic in BRN sends them straight into the router on the compute host on the right. It gets routed and sent back the other way. So the return path is the same, but it's on a different host. And that's good to note. Now, we wanted to take some time to talk about how the work we've done during Kilo for restructuring. And Roselle is going to start by talking about the L2 agent restructuring. Thank you. So, the restructuring work for the L2 agent has three main areas. So I have to say that this work didn't make it into Kilo. So everything I'm saying now, it's like a tech preview for liberty. So the three areas are, get more information from OBSDB Monitor, improve the RPC calls that the L2 agent is using, and improve the RISC behavior. So as I was explaining before, OBSDB Monitor was only notified that an update has occurred on the host, but it didn't really specify what happened, which is not good since OBSDB Monitor has that information. So what we did was to create a new method for OBSDB Monitor, get events so that the L2 agent can get the events directly from OBSDB Monitor instead of scanning all the ports all the time. So it gets events like port x was added, port y was deleted, so it's much better. The L2 agent uses a few RPC calls, and we also wanted to improve them. For example, the update device up and update device down was only accepting one device. So if the agent had to send those messages for a list of devices, it had to send several calls, which is expensive. So we added a new RPC call that takes a list of devices. For get device details, update device up and down, we also added a new parameter, fail devices, and this is because the plugin before, when it was getting an exception, for example, when getting the device details, it was throwing this exception and this was causing a full rethink in the L2 agent. So to avoid that, now the behavior is different. If an exception is thrown, the plugin catch it and put the device that caused the exception in the list of fail devices. We also improved the security group's provider updated call. This call is sent when the provider rule needs to be updated. The provider rule is the rule that allows the traffic from the DHCP server. So when the DHCP server IP changes, for example, this message is sent. And before this was causing a full refresh of the firewall, which means all the filters of all the port were refreshed. And now we added a parameter to provide the list of the devices that need to be refreshed. That it's the list of the devices that belong to the network whose DHCP server changed IP. And so now we pass this list to the agent and the agent refreshes only those devices. And the last one is the port update message. We added the attribute that were modified during the update so that the agent can choose how to process the port. Not all the changes requires a full processing. For example, if the IP of the port is changed, then only the filter needs to be refreshed. So no full processing needed. Now the agent can make use of that. We also wanted to improve the rescink because it's quite expensive rescinking all the ports. So as I was saying before, we added this parameter of failed devices in the RPC calls, which are one of the main calls of an exception. So the agent now, instead of issuing a full rescink, takes the list of failed devices and it tries to rescind only those. And it can also simply ignore the failure in some cases if it's not important. So I've been talking a lot. I just wanted to provide you some real data to show that this really improved the situation. So I just ran a small test on a VM that was running desktop. I created a rally scenario, a very simple one that boots 20 VMs with a concurrency rate of two. So these are the results before our work and these are the results after our work. So I did the math for you. It really worked. The minimal time for spawning a VM is 0.6% better. The average time is 4% better and the 95th percent is 5.9% better. And so this test was run on a single host and the load was very low, like only 20 VMs because I was using a small VM. I'm pretty confident that in a multi-host scenario with higher load, these results are going to be much better. There's still a lot of work to do, so if any of you is interested, please ping me. I'd be very happy to work with somebody to improve all these items. So the first one would be to improve a VSDB monitor. So instead of using the CLI, to use the VS Python library and then to create a queue of events that the L2 agent can process so that we can use multiple workers. Also assign a priority to those events so that higher priority events can be processed earlier. And in general, we need to improve the state convergence and also the resilience in case of failure. And now Carr will talk about the restructuring of the L3 agent. Thank you. The restructuring here was less targeted at performance and more targeted at giving us a code base that we can move forward with. When I looked at the L3 agent thinking about some of the things I wanted to do with it, I saw it as a handyman. A handyman that does lots of things. This is after HA and DVR were added in Juneau. It wasn't a very good handyman. It was kind of forgetful. It would, all throughout the code, there were little cues about, if we're working on a DVR router, then do this. If we're working on a DVR router and we're on a compute host, then do this. Or if we're not doing an HA router, then do this. It got hard to read. It got long and it got hard to maintain, too. It was a jack of all trades that didn't really get good at any one of them. So I wanted to move to a different model that I call the contractor model. Now, this is the guy, the contractor. There's one for the network node, the network node contractor, and there's one for the compute host. And what they do is they pull out their plans. They say, okay, I'm on a network node and I need to build an HA router. I know who can do that.  So that was the model I went for. And I broke up the code so that we could compartmentalize and we have the agent, the L3 agent, which previous to Kilo was a very long file that did everything, 20 to 2,500 lines of Python code. We broke that down. We shrunk that down. We said, okay, the L3 agent's just going to have the L3 agent in it. It's the processing queue. It gets notifications. It's the contractor. It loads the right class and it runs it. And then it sends status updates. Very simple, very well-defined job. For the routers, actually there was one place I wanted to go before that. There was also in the L3 agent, there was VPN as a service and firewall as a service. One of them inherited from the L3 agent and one of them was a base class to the L3 agent. Don't ask me which one was which. I can never remember. But we started by pulling those out and creating a signal mechanism where the router could signal the VPN as a service agent or the firewall agent to do its job. But we took it out of the code so that there was some more generic interface there. And then with the router, what we did was we used inheritance. It's not a new technique. We created three kinds of routers. A legacy router, an HA router, and a DVR router. And there was some common code so we created a common base class. And this worked very well. This took all of those if-then-else statements, if DVR this, if compute-host this, that. It took that all out. And it made the code a lot more straightforward using polymorphism to get the job done. And I think of it as loading specialists that know how to do one job well and they do it well. There's also a few more that I... We didn't get to this in Kilo, but we also want to create what we call an HA DVR router. This is... It seems like an oxymoron, but it's not. The DVR router has that central component, and that is still a single point of failure for the use case that it solves. And combining that with the HA router functionality that we have could give us a fully redundant solution. And there's also still a little bit of code. There's DVR for the central part and there's DVR for the distributed part. And they actually don't mix that much, but currently they're in the same class. I think breaking those out would make it even more simple. And why did we do all this? Well, we looked at the L3 agent, and we looked at doing this stuff. We wanted L3 VPN. We still want that. We wanted to eliminate some of the wasted IPv4 addresses in cases where we can do that. You may know that a neutron router, as soon as you create it, it consumes an IPv4 address. That may be okay for you, but for a lot of people that's difficult to cope with. And we may want to look into doing DVR for IPv6. And even some things I didn't put on this slide, like we want to look at connecting neutron to BGP routed networks instead of just statically routed networks. And beyond that, possibly even other routing protocols. We also want to improve the routing situation with IPv6. And I'll be giving a talk here tomorrow about L3 and IPv6 with Sean Collins. Looking at doing that, it was very difficult to think about getting those done with the code base that we had. And splitting it out into smaller, simpler parts with a topology that we could understand made that look feasible. And in Liberty, we're going to get to work adding some of those features. They were also, just like in the L2 agent, we also wanted to eliminate the full sync. It's kind of a wasted operation. And we don't always have to sync everything. And I think that's it. Did you have anything to add? I guess we can take questions. Yeah, thank you. We have... I think we've got about five minutes for questions. Right here. L2 agent is running on the computer. L2 agent is kind of scanning. And then you have thousands of nodes which are overloaded. Yeah, so... Running on the computer... Yeah, so the question is, when you said the L2 agent is constantly scanning, you mean the L2 agent on the computer host? Yes, I have to say, so when I say constantly scanning, it's not that it's all the time scanning. It just starts the scanning when it gets the... has updates from OBS DB monitor. So when the L2 agent knows that something has changed on the host, it scans all the ports that are currently on the host. And this is done on the computer host. Okay. Yes, but I couldn't tell you where... off the top of my head. Yeah, there's a documentation on setting up the message queue. I think there's something in the dev ref for Neutron that was added recently. I mean, if you mean what kind of messages are exchanged, how to create a new one and raise the version, these kind of stuff, it's in the dev ref. Yeah, let's see. In Kilo, we can get metadata service high availability using an HA router. So with the VRRP, it will actually, as the active router switches between the multiple copies of the router, it will actually start and maintain that metadata service in that router. So the question is about port binding failure. So which rate of port creation would cause a failure? No? Okay, so the question is which log to look at in case of some bug related to the L2 agent. Okay, so the L2 agent also has a log and you can just check that. And usually when there's a failure you should see a warning or this kind. Yeah, it's in Varlog Neutron, OVS Neutron agent, it's in that folder anyway. Now it's in Varlog Neutron, there are several logs and one of them is for the OVS agent. I don't remember the exact name, but it's there. Yes? So the question is if we want to get our hands dirty playing around with this code, can we use a kilo environment or do we need something more recent? Kilo is pretty recent. You can start with kilo and get all of the L3 restructuring improvements that we've made. For L2 some of them did not merge for kilo. You may be looking for patches to pull down. Okay, so the question we stated is are there certain... he wants to use master for Neutron? Are there special versions of NOVA and the other services? No, I don't know of any requirement to use any special version. Yeah. Okay, the question is referring to the distributed routing slide and the routing between hosts. If the VMs are on the same host, it's routed on the same host and it never leaves the host. So the question is how is VRRP achieved? Does it use KeepLiveD? Yes, it does. It uses KeepLiveD. Are the IP addresses migrated inside of the router namespace? Yes, they are. And I think we're out of time, so if you have more questions, I'll be up here for a little while. Okay, thank you everyone.


Hello, this is Hans van der Kwaas senior lecturer at IG Delft Institute for Water Education. In this video, I'm going to show you how to find organizations where you can do a QGIS course and receive the official QGIS certificate after successful completion. I'll also show you where you can follow the next course that is organized by these organizations. And of course, this can be visualized in QGIS. Information about the QGIS certification problem can be found under Get Involved Certification. This page shows the information about the certification program. Organizations can apply for certification using a form on this page. The reviewers will check if the course materials are of adequate quality and if the organization has contributed to the QGIS community, which is possible in several ways mentioned here on the page. Following an initial review, the application will be sent to the local QGIS group for their opinion. The recommendations will go to the QGIS project steering committee who will make a final decision. After approval, the organization can issue the certificates. Each certificate will contribute 20 euros to the QGIS project, which is a great way to support the development of QGIS, but also for participants to have a paper to show that they have followed QGIS with an organization that gives the official certificate and for the organizations to advertise that they are certified for QGIS. When you click the here link, you will go to the project test site where the certification program is hosted. You can also find detailed information about the certification process. Now in the menu we choose certification, approved organizations to see the list of organizations that issue the official QGIS certificate for their courses. You can click on the name of an organization to find out more. Here we see IHE Delft Institute for Water Education and the courses that were offered in the past and that will be offered, including the amount of people that have received the certificate. We can view the details of the course. This is the next course that will be organized. But it would also be nice to see this on a map. Here at the top of the page we can find different feats to list all training centers, to list all upcoming courses or to list all past courses. If we click a feed, we get this page and if I then go to get and use the drop down, I can choose different format, I can choose JSON and I can copy the JSON link. In QGIS I can show a world map by typing world in the coordinate field and I can add a vector file in the data source manager, choose vector and I simply paste that JSON link and it will recognize it. Then I click add to add this layer to the map canvas. Now here with the default green dots I see here all the training centers that have the right to issue the QGIS certificate. Let's zoom in to a country and it comes with an attribute table so if I query I find here for this dot that there are two organizations here in Delft. It's Geo Academy Online and IHE Delft Institute for Water Education. I can also use the feed to list all upcoming courses in the same way. I also get here the JSON link, copy the link and then I add it in the same way. Click add and there's the other dot in a different color. I switch off the training centers, there's one dot left and I can identify the point and find them the next course that will be organized. So this is an easy way to see on the map when and where the courses will be organized near to you and then you can check the website of the organization to find more about this course that is organized or contact the course convener through the provided email address.


Dodgers getting the win and Manny Machado's debut two for two, two walks looking good. We know how the Dodgers been playing some good baseball first in the NL West, but what does this move for Machado do for this Dodgers team overall in the National League? Yeah, I definitely, I think it definitely puts them at the forefront of the National League. Definitely puts them as the best team, either them or the Chicago Cubs. I'm telling you, man, Magic Johnson's on fire. Either them or the Chicago, which team is the best team in the NL? I say one, two, Dodgers, then Cubs, right? And then the Phillies, right? So I just think you're bringing over a bona fide superstar, man. This kid's a young kid, but he's gonna play each and every day. And the one thing that I love most about that press conference was listening to him say, I'm down for whatever, because we heard just like everybody else did, said I only want to play shortstop. So a lot of teams might have been pushed away from that. Los Angeles Dodgers said, no way, man. If this guy comes over, he wants to win, we want to win. It gives them versatility. I think that's one of the great things that Dodgers have going for them. And Doc Roberts, Muncie can move all over the place. Bellinger can move all over the place. Taylor can move all over the place. Now you've got a guy like Manning Machado playing short and third, Turner's been down. Hey, Chris Taylor, same thing. Seger's been hurt all year. I'm telling you, man, in my opinion, this definitely puts that team at the front of the National League, best team in the division. No doubt. He's saying all the right things. And as you mentioned, he'll play both shortstop and third base. He's making an adjustment, coming from Baltimore, coming to a big city like L.A., which is an adjustment. But what other adjustments does he have to make, especially moving from another league, from the American League to the National League? Yeah, well, the one guy that I could kind of compare the situation to would be Giancarlo Stanton. I mean, obviously, being in Miami and Baltimore, kind of similar markets in the same sense. I mean, obviously, Miami's a little bit bigger. But all of a sudden, you're coming to the big bright lights. All of a sudden, you've got L.A. You've got New York. It took Giancarlo a little bit of time to get his feet wet and get into that rhythm. But man, he ain't got that much time, man. He's here for two months, and then all of a sudden, the playoffs begin. So I think the one thing it could affect is his switching divisions, too. He's been in the American League his whole career. I mean, I feel like he can hit everybody. But still, some of these National League pitchers that he hasn't seen before, may give him a little bit of trouble. He's a baller, but now comes the expectations, but I believe he'll give up, too. He's pretty good. He'll be all right. He'll be all right.