Good afternoon, everybody. Thank you for joining us for the last session of the day, I guess. This talk is about high availability in OpenStack. Yes, I know. Well, one more high availability talk, and Florian said everything, so we basically don't have anything to say anymore, I guess. Actually, yes, we have. Oh, we do. Okay. So, who's he? No, I'm just kidding. I think our talks are complimentary, so this should be fine. Thanks for setting the scene then. So, let's move ahead with our talk. I'm Sebastiano and I work for Innovance as a Cloud Engineer in Paris. I'm a DevOps and I love Ceph and OpenStack, of course. I work for the integration team with Emilia, and what we do is we basically build, design, and deploy really clever platforms for OpenStack. So, enough about me. Just give the floor to Emilia. So, welcome, everybody. So, we are working for Innovance, like he said. Innovance is a Cloud provider and also we are designing and building Cloud for customers. I'm like Sebastiano and working on deployments as a DevOps, and I am involved in the community since one year and a half as a contributor in the documentation, and also I am interested in quantum deployments. I'm not a developer, and that's all we can do. Yeah, we can move ahead with the agenda for today. So, today we are going to introduce a topic by our contribution related to 8 availability in OpenStack. After that, we will share with you our experiences that we had in Innovance to use cases. And finally, we will look further at the conclusion for the future of HR in OpenStack. Sounds great. Okay, so a little bit about our contributions within OpenStack. Eight months ago, we started a project with Stixel guys about the OpenStack resource agents. Back then, we just thought that OpenStack wasn't built for HA, and we had to bring HA for OpenStack. Nowadays, things change a little bit. But we had to make decisions, and so this is why we started with high availability stacks, base maker, core sync, these kind of players. So we basically wrote all the resource agents for every services and even stateless services, even if maybe today it doesn't make any sense anymore. So we have resource agents available for SX, Folsom, Grizzly, and quite recently we opened a new branch, the master branch with the Avanas support, because as OpenStack gets more mature, we always have to add new services, so we have to build resource agents. And future, no future because, well, OpenStack tends to be a little bit more HA aware with such things like, for example, RabbitMQ, with the MaruiQs. So let's move with the documentation now, the things we made. So as Florian said before, we actually missed some documentation at the beginning, and he started the work, and I continue to improve the documentation to explain to the community how to build HA using all resource agencies for base maker. And now the work is still in progress for active passive mode, and the next step for me, and for all people who want to contribute in the documentation, is to write a new guide for active active mode, how to deploy OpenStack for large scale cloud, and how to make OpenStack active active for scheduler and APIs and other components of OpenStack. So we need contributors, if you have any feedbacks in your company, if you have any feedbacks from deployments, of course it's a pleasure to welcome new contributors. So we use Puppet to deploy all our setups, our customer setup, and of course it's deployed in a fashion way, in a HA way. So we recently added the support for HA proxy for the stateless services and also some flags to enable the MaruiQs. So every development we made are available upstream in the Puppet Lab repositories. Some work is under review by Puppet Labs. So now about our experiences, it's what we made and what we are still doing, of course. In this second section what we try to explain is to expose two architecture references. For OpenStack, we're going to start with a hold one with active passive mode and the new one will be more active active with load balancing and stuff like that. So maybe as you know, Inovance has two public cloud, one in France, in Paris, and another one in Canada. Recently my team has set up a public cloud in Montreal. We have deployed Folsom release and we designed the HA in the active passive mode. So this is our architecture. So we are using OpenStack on Debian. We have the package working for Inovance and also publish the package on Debian. We are using Puppets. The thing that Folsom released, the Puppet modules was not ready for HA so we had to use a pacemaker and a DRBD stack. So RabbitMQ as well. RabbitMQ wasn't ready and the setup wasn't that big so we had to take decisions and we ended up with just a pacemaker stack, the same format as Grail. So it made sense to use two modules. That's why we have two controllers. We have one in active mode which is performing all the API services, schedulers, MySQL and RabbitMQ. And the other one is actually waiting for that the active is failing down. So for the compute node we don't provide high availability for now. And for storage, at this time we are not providing block storage as a service and Sebastien is going to work on deployment of Ceph. Maybe you want to... Yeah well, in the coming month we will provide a new feature in our cloud. So just the block device as a service. So you can either attach block devices and also boot from volume and it will be backed by Ceph. So Glance will be as well part of the setup. So Ceph will also back Glance and... Maybe you can talk about Ceph? Well, yeah. Yeah of course. Who knows a little bit about Ceph maybe already? That's not true. You. Okay, for those of you who don't know what Ceph is, it's a unified storage platform that can provide either well, object storage, block storage and distributed device system. So it's roughly and really quickly it's what Ceph does. In the block storage side there is a kernel module and there is also a QMU driver. So this is the one we use with an open stack. Okay thanks. So all feedback about this kind of architecture. The positive feedback is that we didn't lose time to make new modules for Puppets because we actually used the upstream modules we did some contribution to. For the stack of pacemaker resource agent we use the one that we did with Astexo and Sebastian. And so the negative feedback was that we cannot scale with this kind of architecture. We use active passive mode. So if we want to scale out the controller it's not possible as you know. We have also DRBD issues. It's not really an issue but well as soon as we can get rid of the DRBD for setups it's not like well. Everyone knows the overhead that it generates so well if we can build a setup without it that's better. And what we are going to move very soon is we are going to upgrade to Greasy release. And also to... You changed the slide. It's already in my part. And we are going to scale out the controller and to build a new architecture with active active mode. Actually the same kind of architecture that Sebastian is going to show you. So this is the CloudWatch project. So CloudWatch is one of our customers and we... The main goal of CloudWatch is to build the biggest cloud at least in France and maybe in Europe. Yeah so Innovance has been chosen to build this infrastructure so this is why we are working on that. The setup that I'm going to show you is not in production yet. It's something that we are working on but it's definitely the setup that's going to be in production. So no worries there. For stateless services like all the APIs we use HAProxy. So every API requires a load balance through this HAProxy node. HAProxy nodes are redundant so we have two. We use keepLiveD with a floating IP. One IP at a time is available in one node and we also perform some passive checks on the API. So if the API doesn't respond we just don't route the traffic on this node. Same for HAProxy. If HAProxy doesn't respond we just move the floating IP on this other side so we don't have any downtime. Thanks to this architecture. We have API clusters. It's more logical infrastructure because of course schedulers are part of the same machines as all the APIs, Keystone, Glance, Nova APIs. On the minuscule side we decided to use Galera. So while the setup is quite easy we have three nodes in cluster mode and our HAProxy load buttons all the requests from one database server to another. So this makes the setup quite highly available. On the rabidmq side we have two dedicated nodes with the built-in clustering mode of rabidmq and we enabled thanks to Grizzly the new Marriott queue feature and XHA policy. About a compute, well we use a distributed file system for all the Nova instances so this makes all the VMs highly available. And we have a really nice NFS cluster based on a NetApp array. So we also use this one with the sender driver to attach block devices. I haven't spoke about Horizon just simply because we don't use it. They have their own application to connect to the API but if you want to scale, or if you want how to scale Horizon, I think that the easiest way to do it is just simply to build nodes with Horizon, Apache, and one memcache instance so you can just scale out like this and use a router.  So even if you eat the same memcache cache it's perfectly fine. This is how you can scale Horizon if you want and this is pretty much it for this infrastructure. We forgot to mention that if you had any questions just right away after the picture you could ask this question so maybe we can go back to the previous slides if anyone has any questions about... Or feedback from your deployments. I'm sure you have feedbacks. So if someone in the audience has any questions about the setup that Emilia exposed, we can just go back or wait for the session. Do you have any questions? You do, okay. So let's move back to this one. Mine. Oh, yeah. They are available on the Forge for Puppet Lab. Yeah. Oh no, well just load balancer that supports sessions basically. So just that, well I tend to use LBS but well you can also use HAProxy it's perfectly fine. Any more questions on... Wow, let's move ahead then. Okay, some feedback around... Well it's not really feedback because it's not in production yet but it's just something that we thought about. The infrastructure is highly flexible, highly scalable. This infrastructure makes maintenance easier because we can basically shut down one node from the HAProxy and upgrade this node and well that's quite flexible for upgrade intervention, maintenance, kernel update, whatever. I will let Emilia speak about the networking. Thanks. About Quantum, I got networking in OpenStack. We have new features in Grizzly which are able to scale out Quantum L3 and DHCP agents but we don't still have HR for those components. That means that if we have the virtual router hosted on the L3 node, if we lose the node, the tenant loses all the router on this node. That's why we don't still have the HR. If you want to get HR from Quantum L3 agent and DHCP, you need to use our resource agents for now. We hope that in Havana we will have the same feature like we have with Nova Network, the multi-host feature which is able to have each compute as a L3 agent. The other issue that we have is about scaling the OpenVitch plugin in Quantum because maybe as you know, how many people here is working with Quantum software? Yeah, because it doesn't work. Just use Nova Network. Quantum today is supporting two kinds of isolation with the OpenVitch plugin. We can use VLAN or GRE. As you know, VLAN cannot scale up to 5,000 networks and GRE doesn't scale because of compute processors. Yeah, you're overrated. Yeah, overrated. So today we cannot scale in a big infrastructure with the OBS plugin in Quantum and if you want to build a full stack infrastructure with Quantum and for your networking, you need to buy a SDN solution and to set up the plugin with Quantum and make it working. VXLAN might be a solution or not? An option? VXLAN, yeah. Yeah, but VXLAN is not supported yet in the plugin. And it's actually there at the discussion yesterday with Edward Thuleau from CloudWatch about introducing new features in the Quantum plugin to be able to scale up the agent and to scale up the infrastructure, the isolation protocols and so on. At this time, I can continue. At this time, we are testing the Nova cells. Actually, this is a new feature in Grizzly which is able to build cells in your cloud in which you have Nova features, only Nova features and which are talking to the... with the RabbitMQ, to its parent cell and it's able you to scale your Nova infrastructure but not Cinder, not Quantum. So it's a way to segmentate, isolate logically your infrastructure to maybe make upgrades easier or increase your scalability. Speaking about that it's just assumptions and things we discussed while building this infrastructure because the main goal is to go with thousands and thousands compute nodes so we just have to think a little bit further about hyposcaling considerations. We can't really build just horizontal infrastructure so we should maybe try to segmentate through domain failure and build tiny, tiny clouds. It's just what we think. I haven't said that it has to be like this but it would be nice, for example, with the Nova cells to segment everything and try to isolate as much as you can and build really tiny clouds next to each other. So that's just an idea to make upgrades easier and things like that. New commerce? Yep. In the next series, also in the next series, LaVanna, we have a new component in OpenStack. The first one is Nova Conductor which allows us to... Nova Compute doesn't steal anymore. The DB access, the database access from compute nodes and these services can be scaled horizontally. So in our cluster, we didn't write any resource agent for pacemaker. We only put it in the cluster and run multiple Nova Conductor nodes. But there are still services in OpenStack in which you cannot run in multiple times like Cedometer Central Agents for people using Cedometer. And we have new services in OpenStack networking. We have the load balancing as a service which has an agent and also the metadata service which has an agent 2 working with L3 node. And that's this kind of service that you cannot run HR with multiple run. You need a standard cluster with pacemaker and cross sync. So you might always need nodes that can run pacemaker. So we work the resource agent and you can download them and try to... It's part of the LaVanna branch, the master branch and the git repository. So feel free to test it and return some feedback. And the documentation for it is coming in a few weeks. So that's pretty much it for today. We would like to thank you for your kind attention. And if you have any questions, feel free to ask questions. And yeah, this is French. Yeah. You have to. For the network node, you mean that L3 and DHCP server? So actually the better thing to do today is to set up multiple layer 3 and DHCP agents. So you have now a scheduler in Quantum. So if you need to scale out your cloud, you can build 10 nodes for layer 3. But you need to be aware that if you lose one node, you will lose all the rotors of this node. So that's something you should be aware. If you don't want to scale out the networking part in OpenStack, you can run the old architecture with active passive mode. That's the non-sware, yes? Okay, nice. Yeah, sorry, you have to, please? Yeah. Do you have any... Come again, sorry. For example, the network interface... Yeah, yeah, yeah. Oh, yeah, obviously we use bonding for this. Okay, I see the question. For all compute nodes, we have several accounts of network. We have the public network. We have the management network. And we have the VM network, inter-VM network. And all the networks use not one network card. We use at least two network cards for each network. How have you, perhaps, been doing things in the past and had changed the way you were doing operations for these deployments? Because all I've heard so far is just, like, we deployed OpenStack. Yes, that's what we did. How have you changed or how are you closer to, say, pulling from source from OpenStack? Or how are you getting more monitoring information to determine better operations procedures? What are you doing to make the life of an operator easier? Well, our priority is to upgrade our infrastructure and also to contribute to the greasy module in Puppet. You work on Puppet, maybe, and you know that all the Puppet modules are not ready for greasy release. So we are working on that to make deployment faster and easier. In the past, we didn't deploy all the infrastructure with Puppet, and now at this time we are working on that to make all the infrastructure able to be installed with Puppet in 100%. So if we need a full HSTAC, we just Puppet apply and it's working. Internally, we also use a continuous integration system. So everything we made has been integrated into Djinski and we run it and we can just be sure that everything works. Any more questions? Well, thank you very much everybody. The slides will be soon available. On the slideshow of Inovance and also on the OpenStack websites. Thank you. Have a nice day.