S volume. Why? Well, a colleague of ours created that volume and left the company. And we don't know why this volume exists in the first place. And you would say, yes, there is no communication with that volume, there is no bytes going to or from that volume for three months. So maybe it's safe to delete it or at least to snapshot it and delete it. But they would be reluctant because they would say, well, we've got contracts and we've got SLAs with customers and we would rather continue paying for that volume even though we don't use it than risking some outage of service or something like that. So again, these are real life stories, not something that we read in books, how we do stuff. One fun fact, the management didn't like this approach. They preferred this. This is the same stuff, but a little bit with colors and different shapes and it looks a little bit more modern and this resonated well with the management. When we had to present to teams what we do, we would use this model. And engineering teams responded well. This is simple, this is, you know, very simple design. But when we went to talk with management, this was the one because then we could say, yes, this is something magically that happens and then we process some data and then we generate some results and all that stuff. Apart from the first report, that was a recommendation for engineers. We generated a second report that went to the management. And what we did is we defended, well, we presented some recommendations on the company level. These recommendations would be like this. There is an opportunity to save X amount of dollars per month if you would be willing to perform these, these and these activities. We already conducted talks with the teams and we already got the estimations on how much time would they need to implement the changes. So that usually would be, you know, like open the Terraform file, change something, apply changes and that's it. But we went through the talk and we collected the data about how much would it take to perform these changes. So now we would prepare a report for the management and we would say, if you want to save this amount of money, it is possible, you need to come up with resources for engineering teams to spend this amount of time to execute these changes. And now management actually had something to, you know, make a decision. Some enough data to make a decision. It was, it was palpable. It was not tacit anymore. You could actually anticipate how much time you would need, how much effort you would need and what kind of money you would save. So again, this was, and this is, this is one of the, I love on these talks to present these guys. This is Niklaus Wirt. Niklaus Wirt is most famous by being the father of the Pascal programming language, but some of his work is much more than that. I personally like this plea for Lean Software paper that he wrote in 1995. It's a five pages paper that explains benefits of a simple design. You can find more modern books about this, like David Furley has this book, Modern Software Engineering, this great talk about how you simplify the design, how you start from a simple thing, how you add stuff into this simple thing and how it becomes more and more useful. The whole talk, the whole approach is, don't try to be modern, don't try to be, you know, tech for tech, but try to build tech that would be actually useful, that would sort out or deal with some problem. So again, this is the influence of the guy. This is how we build the models. After this initial success of the approach, the company said like, this is all one of my friends, a friend like to say, nice and dandy, but can you do it for some other resources? Like we've got DynamoDBs, we've got RDS instances running around, we've got EC2 instances running around and we have no idea how to save some money on these. And again, my apologies to all the people that are dealing with actual software that was built to generate metrics or to generate the observability in different definition scopes. This is a different kind of observability that we are doing. We are not getting so deep into how many processes are being run on certain instances and how much memory they use or how much processing power they use. This is a little bit different approach. We wanted to be in the middle between engineering and what can be good for the company. So we said like, okay, if we are going to do this for another EC2 or DynamoDB, let's say, we would like to build a framework. We would like to build something that can be reusable, that we can extend, that we can just add new stuff into the play. So we would have an engine and then we would just, well, you can say add plugins to this engine. So the engine would become aware of how to analyze DynamoDBs and all that stuff. So we started simple. One of the colleagues on the previous talked about ECS and it is the simplest form of running containers on AWS that you can actually orchestrate a little bit. What we wanted to achieve now into technical stuff, what we wanted to achieve here is try to minimize the need for orchestration and increase the choreography. So we wanted a lot of independent services running around not needed to be orchestrated because we wanted to avoid this. If it comes to orchestration, then we will deal with that. But until we have to, we would like to avoid it. So we said like, okay, let's treat this as a microservices architecture. Let's put the analyzing engine for EBS in one container. Then we put another one into a second container like that one would analyze DynamoDBs. The third one would analyze, I don't know, Aries instances and stuff like that. So we started adding more and more services following the same pattern. Every service would have a different model, but the model would contain same stuff. Like get me AWS pricing, get me some metrics that I need to assess whether this information is useful for me or not, and some costume parameters like the stuff that I talked about, whether some account is production account, so I cannot really apply some of the rules on the production account. Again, bear in mind that metrics and observability in terms of, let's say, how mid-CTC is explaining observability is different than this. This is like a high level of observability. Let's call it like that. We would rely on AWS to give us data. So what we did is we put these containers in place. Then we needed to somehow start them. We had an idea of running them every day, selecting the data, and once a month we would generate these reports that we would push to teams that are targeted for actual optimization and to the management so they could figure out whether they want to invest resources into optimizing the infrastructure and saving money eventually. So we said, like, since we don't have a lot of orchestration, let's do it simple. We created events that are timed. In the morning, we will schedule these containers. They will spin up, do some stuff. It was serverless. It was Fargate enabled, so all that is fine. After that, we figured out that a lot of these containers would actually go and fetch prices from AWS. We would need some secrets to store them somewhere in the AWS. And then somewhere around this point, we started thinking, like, okay, how about we make everything idempotent and serverless? That means that if we are going to build it, I can run it many times a day and generate the same results, so I wouldn't mess up with my previous data. That was idempotency. And then serverless, well, events, ECS Fargate containers and all that stuff, still is serverless, so that was cool. Then we proceeded and we said, like, okay, we want to go through accounts and we want to analyze what is in these accounts. We need some permissions. We need some, you know, account roaming stuff that will go around. And around this point, we started thinking, like, maybe this orchestration, microservices organization is not the most optimal one. Can we put aside the tool, the piece of code that would go through all these accounts and do this for us and just hand us the token and we continue doing it like that? And that is doable as well. In this case, when once we put it in motion, it wasn't really a good use of our time. We figured out that it was, even though it's elegant in the code way of things, in the architectural way of things, it wasn't really elegant in the ways of us having to maintain that. And we always had in mind that at some point we will hand over this work to the team that should contain to maintain this. So we wanted to keep it simple and follow this approach. So we said, like, no. Every container will have its own engine that will roam through accounts so we could separate them even more so we could avoid the orchestration. That was another reason why we avoided this because this would introduce some piece of orchestration and we couldn't scale them in the same way. So we kept the isolation thing. And then we said, like, maintain this serverless approach. Let's write some of the data to the Aurora serverless. Now, still idempotent, the data written in the database is not going to be there longer than a day because we don't need it longer than a day. So if I delete, if I rerun the task, it would clean the data for the task, it would regenerate the data, and the data would be fresh for that run. So this is why we could keep the serverless approach. We needn't keep this data for, I don't know, how long. And that was fine. Also, obviously, something on AWS, you need to log stuff. You need to send information. And this is the final stuff. This is where we, at the end of the day, the only piece of orchestration that we had to introduce is once all the tasks finish, we would push the data into the S3 bucket through some reporting mechanism and from that point on, the data would be picked up by some of the Power BI or Tableau tools. It would be invoked into the Snowflake. It would be reorganized or done something different with it. At that point, company said, okay, can this be used for something else? And this something else we didn't do because the contract was expiring and they didn't want to go through with the whole idea. But the question was, can we use this approach to make company more advanced on the market? Like, how do you do this? Well, you make your company serverless or use more serverless stuff. This company used a lot of EC2 instances, a lot of monoliths running around. So if you want to be more modern company or more competitive company on the marketplace, you should follow from the tech side of things how we see it. You should approach the way you build software a little bit different. So this is the slide that we prepared for the management, actually, when we went on a meeting and we said, okay, if the management would like to be innovative and competitive, and if the tech has a strategy how to support this. So that means, I as a company want to be competitive, cool. What the engineering stuff can do? Well, we can build serverless, cool. So we've got two ifs. Then, score teams by their current level of serverless adoption. And then finally, management needs to approve resources to re-engineer and re-architect and rewrite these applications. This is how it goes. And the final stuff is this. When we went on a meeting and we discussed with them and all that stuff, and you know what it turned out? The company was more keen saving money than becoming competitive because the company was already dominant on the market. And the idea of being competitive on the market was based on, okay, we can do it with not a lot of effort and create a bigger gap between us and the competitors. That was cool. But when you introduce them to the idea about how this works in engineering mind, like you tell us what you want, we propose how we can assist you in the journey. We give you the estimate how much time, resources, money, people we need to make this happen. And then you say, yes, we want to make it happen. And then you follow it through. And then you can support this approach using this tool to provide you with a proper analysis on how much teams are serverless, how much teams would be capable of follow through the whole stuff. And this is for us, for all of you who are my age appropriately, let's say, you remember Hitchhiker's Guide to the Galaxy and the Bubblefish, and it was all that in our heads. Like, you create a tool that is a good tool to communicate between engineers and management. They usually don't know how to communicate. I'm sure all of you have been in situations where you wanted to talk with the management, but it doesn't really go both ways. So this, us providing this tool, felt like providing the Bubblefish. On one side, we were able to explain to the management what can be achieved with certain amount of resources. And on the other side, we were able to provide the engineering teams with what needs to be done, what should be done, and why we need to do this. Unfortunately for us, they didn't go through with the second part of the idea of modernizing the approach and making something better with the whole stuff. But the first part was good enough. So the Bubblefish, in our case, really worked for the first piece. And it was very much thanks to Nicholas for the whole idea about how to keep it simple and how to be. There is one thing I want to mention at the end. It's really tough to build a software that you believe will create the benefit for the company and not introduce some nice and flirty technical things you can brag about because we as engineers like to be proud of what we do. And if you're not using the latest grade, you didn't see Kubernetes in this presentation. You didn't see any, well, not any actually fancy stuff that we did. It was simple, plain engineering that we did that did some good for the company. But I need to say it's not really easy to defend this approach when you go into the purely engineering audience. You would get all kinds of questions like why didn't you reorganize your microservices in a different way? Why didn't you do this AWS service or that AWS service? And it's really hard to defend when you have to offer an explanation like this. Okay. We knew which team is going to continue doing this after us. We were there for a limited amount of time. We wanted to build an application that will have a long activity, that will be able to continue operating after we are done and that will be able to be successfully managed by the team that we give it to. So that influenced a lot of decisions that we made during the architectural phase of the stuff and all that. If you're building a product, if you're building a service that you want to commercially offer to someone else, that's a different thing. You're building it for yourself with a different goal in mind. This was made especially for this company with the idea of then being able to take over the product for, well, the service from us and continue using it and it saved really a lot of money. Well, it's not that difficult to save money when you have a company that spends like five to seven million dollars per month, but it was fine. It was cool. In the end, I think the last year went through like the company said, if they don't save us as a team, Andy and I, if they don't save more money than they cost, you don't have to pay for those two guys at all. And that was really easy sell on the management panel. But I wanted to say, I wanted to re-emphasize this one more time and it is the end. If the company has a strategy and if the company is able to communicate through the strategy and if the company is able to stick with the strategy, then tech guys will follow. Otherwise, you will have all kinds of clashes between the management side of things and the engineering side of things. And this is it. And once again, thanks for Nicholas for his appearance. Okay, questions. Yes. Oh, to repeat the question. Okay. So the question was like I started as a DevOps and then I went into the FinOps and then I worked in establishing the communication between engineering side and the management side and was there enough management knowledge in the company to utilize, to use, to ride on this experience? I would say yes. The first piece of the puzzle that we put in place is a nice prove that it is. The company management really went along with the approach like okay, if you give us, if you give, I mean the ABS, the first thing was the key. This is the key that we use to unlock the company. So when we proved that we can give you the estimate and the amount of money that you can save if you follow the estimate, if you facilitate the needs for the estimate and when that happened and when the company had actually the results to look at, that was the moment when the company said like, okay, this can be done. But that was the piece of the work that was following the company's strategy. Like we want to save the money and that's cool. The moment we get to the second part of the puzzle when we said like okay, you want to innovate or you want to be more competitive, that was the moment when the company had to say, no, we are not really into that, which is cool. You know, which is fine. The bad thing is if you as a company want to propagate this idea, but in reality you don't really want to follow through really. You want to save the money and you want to look good so you would talk these things because they sound good. And a lot of people can align with the ideas, but not in reality. So my answer is yes, because it was, I think yes, because it was properly aligned with the current company strategy. And this is why it worked. It was still amazing how these two, I would say gangs in the company didn't communicate well. You know, everyone wanted their stuff. If you talk with engineers, they would moan about not being able to use the cloud freely to create new clusters to do stuff, you know. They would moan if someone would warn them that they need to clean up the stuff after themselves and all that things. On the other side, management also only wanted to control. You know what they did? They imposed strict policies on how much money you can spend. And you know how engineers got to this? They said like, okay, we will build something that is below that limit. And as long as we are below, no one will bother us. And then in this threshold, they could do all kinds of stuff and not a lot of them were good. So while analyzing stuff, we discovered that actually a lot of accounts that were below the budget, the limit, were very not optimized because they didn't care. They cared only on doing the daily stuff and be done with it because they were below the radar. The company didn't even notice these accounts because they were below the limit. Some manager that was leading these teams was good enough, I would say, fighter to go on a meeting with company management, upper management to fight for a good budget.  And these teams were really happy but still not optimized. This health company actually analyzed and isolated these accounts as well. So again, back to the question. I think yes because of this. The strategy in place was correct. It was about saving the money. The engineering was able to follow. The model was simple. It was easy to translate. We would invoke new... A lot of that was written in Python. Some of that in Node.js. So both very popular languages. It was really easy to invoke new engineers into the game and they picked up things pretty fast. Small isolated pieces of code. So relatively simple approach. Yes. Why we didn't? Accounts management. Cost management, yeah. So that is absolutely good question. The problem with this approach was that they were using this before us. And to properly investigate 400 different accounts using AWS Cost Management required them to spend a lot of time over and over looking through that. And you know, in that big company, things change on a daily level. Not on a weekly or monthly level. That is a drastic change is up and down in the use cases, in the usage, in the patterns, in all that stuff. They wanted something that would be automating the whole stuff. So cost management is a good approach. It also fails to fit in the... To fill in all the gap about custom stuff. Like I said, they would have production load running on a dev account. And you can't just undo that load. You can't just take it out and move it into the product account. So a lot of these goaches, an example, they wanted us to introduce something they called Big Red Button. You know what Big Red Button is? They wanted us to create a UI with a Big Red Button. They would click on the 15th of December. And they wanted us to shut down all kinds of resources throughout the company because they would go into the low usage mode until, I don't know, 10th of January. That was also part of the whole activity. But we started with one idea. 10 days later, we would have all kinds of exceptions from that rule. Because again, large company, large organization, a lot of managers were able to fight their way and get the exception from this rule. So we had to avoid them. And when you go, I mean, a lot of these AWS solutions were really good. And you can use it for that as well. But when you go into an enterprise, having so many different tweaks and things, if you are not willing to dig in and build something custom, not really going to work for them in a way they expected. In the end, we were using the same stuff as cost management. So we would execute API calls towards the AWS pricing API. We would execute CloudWatch calls to get the metrics about EBS volume. So it's underneath. It's all the same stuff. It's just that it was not really usable for them on the company level. They tried two or three different FinOps specialized products for that. So products that are fully baked with all kinds of reports and all kinds of things you can think of. The answer would always be, like, tag everything properly and we will handle something for you. That tag everything properly does not go well with that kind of a company. It's tag everything properly with two teams, three teams, five teams. That's really cool. But when you think about 400 teams and you have to go in them, we also had a situation when we would go to a team and we say, hey, you need to tag things properly so the tool would be able to pick stuff up from the account. And they would say to us this, and I quote, like, when you come to do our job every day and when you have to also manage the existing infrastructure and deliver new features in this amount of time, then you come and you say to us, implement this. And they would just refuse. And they knew they were below the budget limit. They were bringing money to the company and they would just say, now, not interested. And that's the reality. It's one thing that when we talk about it, but this is like the real stuff out of time. So thank you for your time. I hope you enjoyed and see you next year in Brno again about something else. Thank you. Test, test, test. Hello, can you hear me? Very good. So out of curiosity, not a full room, half, maybe quarter of the room, how many people here have used CEP before? Almost everybody. Okay, cool. How many of you are still using CEP today in production? About half of you, okay. And how many of you spent time benchmarking CEP in the past? Three, four people, five people. Okay. And how many of you enjoyed benchmarking CEP? One person, okay. Okay, cool. So today I'm going to be talking to you about SciBench, which is a new way to benchmark CEP. It's an open source project written in Go. And I'm going to give you a bit of background about the project and kind of why we did it. But first let me talk to you a bit about why benchmarking is painful. So it's painful for a number of reasons. You need to run tests many times. You need to run tests for a minimum length of time in order for it to be a good test. You need to, there's lots of variables, loads and loads of variables, especially with CEP. When you make changes to a workload, you need to be only changing one of these variables at a time as you go through the tests so you can kind of figure out and diagnose what changes are having, what impact. And then when deducing those results, even if you've only changed one variable at a time, there's a good chance that more than one thing has changed and you just didn't realize. So that's always fun. So with CEP specifically, benchmarking becomes even more complicated. The first reason is CEP has many interfaces. So we're not just interacting with CEP one way, we're interacting with it many ways. You have block, file, object, or you could talk directly to the RADOS API. With CEP, you have massively varying workloads. So the different CEP protocols all have different workload characteristics. And so you need to take different approaches and kind of measure each approach in CEP independently, figure out, okay, well, so for RBD, this actually works really well, but for S3, this is terrible, or whatever. So that's kind of another issue. Another issue is that distributed systems need distributed benchmarks. So because CEP's performance scales linearly as you add OSDs to the cluster, you also need to scale your benchmarking tooling in a very similar way to make sure that you're not limited by your benchmarking architecture. So the drivers or worker nodes that you're using to benchmark or client nodes that some people in benchmarking world call it need to not be the bottleneck. You need to have enough client nodes to fill up the CEP pipe. So another issue is that workloads can be invisible to you because many people that are operating CEP are like, oh, yeah, I'm operating this as a service, and they go and talk to their customers and they say, hey, what are you using CEP for? And their customers don't tell them, they just say, we want an object store, and they don't say what, because in many cases their customers don't know what they're going to be doing. So a lot of people that operate infrastructure as service providers, and so the eventual need is kind of opaque. So you kind of figure that out over time. That's a bit of a challenge. And then another issue with CEP, which is always fun, is that the background work in CEP can also get in the way. CEP's doing all this stuff in the background like opaque deletes. It's doing scrubbing. It's doing self-healing kind of checks. And those things are things you have to be aware of as well when benchmarking CEP to make sure that when you look at changes you've made, you haven't inadvertently introduced another issue. So first, I work for a company called Softline, and we use CEP as a core part of our product. And our last product, Hyperdrive, which was an appliance for CEP, we had this deployed in dozens of sites across a very large customer base, loads of different workloads, and we needed a way to figure out when we architect a new Hyperdrive cluster, what's that going to look like for a customer. And today we offer a similar product, which is a full end-to-end cloud, and we have the same problem, which is a customer is going to be using CEP in a certain way. How do we estimate what type of hardware to provide them? Are we providing NVMe? Are we providing SSD? Are we providing hard disk? And in some cases kind of what combinations. So we needed to figure this out as a critical, it was a critical blocker for our business. And so we started with this tool called Cosbench. Anybody heard of Cosbench? Also the same guy that loves benchmarking. A few more people. Okay, it was originally created by Intel, it's open source written in Java, and Cosbench was, for a while, the only tool you could do benchmarking of CEP with, it was targeted at S3 and Object, but it also did have some limited support for RADOS as well, and it was originally designed also to benchmark other Object stores so you could compare like your CEP deployment to an S3 deployment or your CEP deployment to Google Cloud or something like that. So we used this for a while, and we didn't really want to reinvent the wheel, but there's some problems with Cosbench. The first problem is that the JNI is expensive. If you want to use the Java native interface or use Cosbench to measure performance for anything other than S3, because Amazon does have a pure Java implementation of S3, for everything else, it has to... You have to traverse this native interface, and that's very expensive, so it completely defeats the point of doing benchmarking from Cosbench. There was a number of other problems with Cosbench, which is, first of all, it didn't really have a maintainer and didn't see any code contributions for years, so the Intel guys kind of abandoned it after a while, I guess they either moved on with their lives or stopped using it. There's also using this thing called OSGI, which was this horrid, kind of bundle-based Java kind of... It was very much built as a monolithic application, and so Cosbench was originally targeting lots of different object protocols, and that may have made sense for that, but it really was a very fragile structure and didn't really work for us. So there were a number of issues with Cosbench, the manual workflow, no-builder and sole system, and we kind of spent a bunch of time with Cosbench trying to figure out how can we make this better, so we tried to build it with Maven, how can we package it, document it, kind of use it in a sensible way, but at some point we kind of just abandoned the project because it was more effort than was necessary, so we ended up writing our own benchmarking tool and that's Sybench. So what was the goals of Sybench? Sybench, we wanted a tool that is simple and lightweight, easy to read, easy to run, easy to debug, so you know what's going on. We wanted it to be linearly scalable in the same way that Ceph is linearly scalable, so we didn't want it to get in the way, or to benchmark Sybench itself, we wanted to benchmark Ceph. We wanted to benchmark all different Ceph protocols, so this is a tool designed from the ground up for Ceph. It had to be efficient and low level, so we wanted something that lets us call out to C libraries like RADOS with our performance implications, and we wanted it to have similar performance to FIO because FIO is kind of an industry standard for benchmarking and it's pretty low level itself, and FIO is great frankly, and so we kind of wanted to be able to look at FIO and see some of the C7 numbers and look at Sybench and C7 numbers and see something that made sense as a result of doing that. So those main goals, and then the final thing is that we wanted to have some framework that gives us control over the data that we use to run the benchmarks, so basically we want to also control what data we're generating, not just using like DevRandom or something like that. So what's the architecture for this? It's written in Golang, so it's almost free to call out to C, it's both a daemon and a CLI tool, so you use it like a command line tool, but it's also running on your driver node as a daemon. It handles auth, so it takes the set keys where necessary and the S3 keys where necessary as arguments, and it passes them to the monitor the gateways as it needs to. It's multi-threaded, so because written in Golang it's pretty easy, but also by default every Sybench driver spins up a thread per CPU core on the worker node, and so you can control how many threads you want and play with that as a variable as well when you're doing benchmarking. And that could be useful if you have for specific workloads that do that. It does both bits or bytes networking people who seem to love measuring things in bits or people who seem to love measuring things in bytes. No one cares, divide by 8, whatever. So then it also has this control of ramp time, which many benchmarking tools do, so you specify a kind of uptime and a downtime, and then it won't really measure the first three seconds or the last three seconds. And then finally, Sybench also is not focusing on... It's only focusing on the benchmarking. It's not focusing on all the setup and talking to the monitors and capturing the data and figuring out... We figured we wanted something that just did the benchmarking and then we ended up writing another tool, which I'll talk about in a sec, called Benchmaster, and that was a tool that kind of helps you orchestrate your benchmark, run sweeps, run multiple things. So I'll talk about that in a sec, but Sybench is just about running one workload and doing it well. So what's the architecture of Sybench? You can see on the right here, you've got a self-cluster, you've got LibRatos, LibRbD, LibZephFs, Rados Gateway, LibRbD, and LibCephFs twice to demonstrate the file system. So there's... From a Sybench worker, you can either talk to Rados directly, you can use RBD images, you can use a mounted file system, you can use Rados Gateway, or you can talk the last two are kind of native. So this is like either a native file mount or a native block device. So you could theoretically use Sybench to benchmark stuff that isn't self. You could benchmark anything. So that's kind of what the architecture looks like. So for LibRatos, you just provide a self-pool, a self-key, and a monitor address. RBD, you just provide the same thing, and it handles the RBD images. So with Rados Gateway, and this was something that Cosbench did very well, is you don't have to worry about load balancing HTTP requests or figuring out how to do HAProxy or something similar in order to deal with benchmarking S3, because it kind of... Every worker can talk to a different Rados Gateway server. So there's kind of some inherent built-in load balancing just due to the fact that you have loads of workers and you can provide it loads of endpoints, and then it will do that for you. So that's pretty cool. So what some of the other stuff you can currently do, you can do bandwidth limiting. Some customers have requirements such as, hey, we want to have 100 millisecond response time when doing 30 gigabytes a second of traffic. So this is a good way of getting latency numbers at a specific bandwidth. When you max out bandwidth, that's not always a good indicative measure of latency. So it's a good way of limit workers from maxing out the pipe. So you get a really good view of what latency implications is this worker going to have. It has a slice generator, so by default it generates random data, but it's not very useful when you're trying to measure things like compression or deduplication, so you can put that random data into a buffer and then use the same data again for future workloads in order to see the impact. You can do read write mixes, so it's not just about doing a read workload, doing a write workload. You can also do like a 30, 70, 50, 50 split at the same time, because most workloads are going to be combined. Syringe also has support for individual statistics, so you can write out all the stats from a worker, and then you can go in and do statistical analysis and a whole bunch of other kind of investigations to figure out what actually really happened if the numbers don't make sense. This ends up being a really, really big file, so it's only worth doing if you're really confused. And then finally, Syringe doesn't delete anything. It doesn't clean up after itself by default. You can't tell it to, but it's because deletes instead have a huge impact on performance, and they're also silent. It's hard to tell when they're happening, so we don't do delete by default, but you can turn it on, because you can have a very representative view if you're doing loads of deleting. So Benchmaster. Benchmaster is a small wrapper for Sybench and also for Cosbench as well, because we kind of had a migration process moving from one to the other. It's for running a series of benchmarks rather than just a single one, so it allows us to kind of provide a set of options that we can sweep over, so I can say, hey, go and run a workload for 1k object size, 16k object size, and then come back and give me all the answers. And then finally, it also writes to Google Sheets. So with Sybench, I can generate a Google Sheet, and then all my kind of workload data ends up in there, and that's just a very easy way to draw graphs or use the output collected over time, so it's much more organized. So I want to show you guys what a Sybench worker looks like. Hopefully the conference... Can you still hear me? Hopefully the conference internet will allow me to do this, because I'm actually just doing it on a remote machine. Is it here somewhere? There it is. Cool. Cool. So, you can see I have this SAF cluster. It's just a small SAF cluster that we have in one of our Berlin labs. It's backed by... It's actually backing an open SAF cluster, so I could theoretically make lots of people's lives miserable if I do the wrong thing. That's fun. It has 36 OSDs. It's just three nodes. You can see... Yeah, they're all hard disk across three nodes, and they're all up. It's looking fairly healthy. There's some data on the cluster, not too much, about three terabytes used. So that'll all look pretty familiar. So, what's sidebench? The first thing is... I want to show you that sidebench is actually running. For this demo, we're not really interested in the actual numbers, like the benchmarking data, because it's a very small test cluster, so it's not really about hitting really big numbers. This is just about showing you how it works. So here, there's sidebench is running as a demon, and that's kind of how it works, and then we also have the command line tool, which has a fairly big help menu. But basically, for any one of the sidebench commands, I can then provide it a list of servers, and those servers will be the worker nodes, and you can run this command from any of the worker nodes. It doesn't really matter. As long as it has the demon there listening, it has an API that we'll just talk to, and it will send out the benchmark. Other things to point out... You can see the different sections. It has a man page as well, which is always nice. Some information about benchmarking with sidebench specifically, some guidance, more detail into every different command and option for the tool. There's also a website and some information on it. There's also a website which will give you detail on how to download it and package it and stuff. Is that better? Not big enough? I hear that a lot. Next, we're going to do a basic benchmark. I've got some historical commands, some of which worked, some of which didn't, so I'll run the ones hopefully to do. This is going to be uptime1, downtime1. It's just going to be a five-second workload, and we're not going to measure the first or last second. I'm providing it the SEF key, which is I'm just including the command to print the key, and then I'm giving it the monitor at the end. Let's see what happens. Very simple. We've got a write stage, prepare stage, and a read stage. The prepare stage was actually skipped because I didn't clean up the data, so it could just go back and read the written data. You can see that the reads were faster than the writes. Very simple. The next thing I'm going to do is I'm going to use Benchmaster to create a sheet. I've actually done this before. This is the Benchmaster help page. You can see here I have sheet create, and I'm going to give it a sheet name and an email address, and it will create a Google Sheet and share it with me. Let's try that. Benchmaster, oh, can't spell. Sheet create devconfc, and then I'll send it to myself. I just got an email, put it up, and that's a very small spreadsheet. Oops, that's a very big spreadsheet. There you can see the spreadsheet, and then let me go away and run a benchmark. I'm going to juggle seeing things on this screen and that screen. Now I'm going to run a benchmark, and the sheet I'm going to provide is the one we've just created. Devconfcz. You'll see here that I've added a new option, which is the object size, and I've added 128, 512, and 1 meg. We can see the different speeds at these different sizes. I'm going to make this a 5-second runtime as well so that it doesn't take absolutely ages, because it's going to do three. I'm going to leave that to work for a bit, and as it does that, I'm going to talk to you about the rest of... Let's go through the next slides. I can figure out how to manage this. Some of the ideas for the future with Sidebench. We thought about doing a workload generator. This is something that over time we realized it's quite hard to map a customer workload or a set user workload to something that we're going to be actually creating a benchmark that represents that. Also, the invisible workload problem, we had service providers that just like, we don't know what people are running on our system, sorry, so just give us something that works for everybody. It's like, well, but that's not really how benchmark or computers work. We wanted to have something that we could sit there, like a generator, and say, okay, here's the workload that's been on average over the last month, and here's a benchmark for Sidebench that you can run to test other potential clusters for this workload. It's something quite cool that we haven't got around to doing yet. We also wanted to do sweeps over OSD counts, so this is quite cool. Actually, I wrote a script that did this, but I never actually patched it into Benchmaster, and so the idea here was if you want to see, like, prove to yourself that the CIF that you're deploying is actually scaling linearly in performance, what you can do is remove a whole bunch of the OSDs from your CIF cluster and then kind of add the men gradually over time per node, so let's say you have 20 nodes, you just scale it back to three nodes, so you could just do a basic triple rep, run a benchmark, add in the fourth node, run a benchmark, add in the fifth node, run a benchmark, and then watch, and then if it's a linear graph, then you're happy because it's scaling linearly as was promised by Sage and the other gods, but if not, then you're in trouble, right? So I actually did this many times, and it did work really well, but just never got it into Benchmaster, it was really nice. So meta operations, so there's a whole bunch of stuff in CIF that isn't just like running like reads and writes, and there's other stuff that you might want to do, and so it would be cool to look at, like, how do you measure the impact of a snapshot or other things. There was support for Kubernetes, so obviously Kubernetes has a fairly mature CSI driver for CIF, it works really well, both file system and block, and so it would be cool to figure out, hey, can we have Benchmaster go away, spin up a Kubernetes cluster, spin up a bunch of pods, map the storage class or the persistent volumes into pods, run the benchmark, spin it down, and then spit out the results and make that repeatable process. So that was something we never really got around to either. And then, yeah, if you guys have any ideas, either leave an issue or some merge requests, patch is welcome, it would be cool to see what you guys have, if you have ideas as well. So while I've been talking, hopefully, if I can find my cursor again, we can see that this benchmark has completed, and remember I ran three benchmarks with Benchmaster, so it would have gone and told Sybench three times, go and do this, go and do that, go and do this, and then spat out the results in the spreadsheet, and there we go. So we have three results. We can see they're written with Sybench, the different sizes, the time, we can see that the right bandwidth differed massively between the different three object sizes. Same with the read bandwidth. It seems to have interestingly gone down between 1 to 8 and 512, which seems kind of dumb to me, but probably because I did a 5-second run or something. And yeah, that's the benchmark. So yeah, any questions? So is your question, the question is, can you take an FIO configuration file and use it with Sybench? There isn't, but I actually do really want that as well, because doing it on the command line is very annoying. So yeah, we can add, that would be a very trivial change to add as well. It's just about, yeah, exactly. But yeah, that's a very good question, and I think we should do that. So that's another idea for the future. I'll be at a minor one. It does, it does. Oh, yeah, the question was, is it also possible to count the IOPS? And over here you can see there is a latency value which translates to IOPS to multiply. That's basically IOPS, but it's just latency in many seconds. Cool. Any other questions? Pavar? Well, so CBT, so the question was, have you finally given up on CBT? And I never gave up on CBT. CBT is a wrapper for lots of tools. And so, in a way, it's kind of like the Benchmaster thing that we did. And actually, I gave this talk in one of the Seth days a few months ago, and the guy that wrote CBT was in the room. And he was like, hey, we should talk about integrating Sybench into CBT. So I think that would also be a very cool thing to do. But I haven't played enough with CBT to be able to say, the value of like comparing it to IO or I know it has support for Rados Bench, it has support for Cosbench. I think it has support for another one of the Go-based benchmarking tools. So yeah, I mean, I think that's a worthwhile thing to look at, but it wasn't useful for us, for our purposes. So we wrote this very much, kind of, for what we needed. But that's a good question. Cool. Any more questions? Well, thank you very much. Should we start? Okay. So, hello. Good afternoon, everyone. My name is Miguel Duarte. I'm here with my colleague, Daniel Meliado. We both work for Red Hat. I work in the OpenShift virtualization networking team. And he works for OpenShift monitoring. So we're here to present a talk titled Mayday, CNI Overboard. This will make sense in a few minutes, I hope. So, first thing we'll do is explain a little bit what is the current state of the CNI project and what actually led us to care about this. Then we'll introduce CNI and MULTUS for us to get, like, a whole understanding of, let's say, of the lingo and be able to properly specify the problem. From there, we'll enter a new CNI enhancement proposal about multi-networking and go to the upcoming CNI release. Well, this is not an upcoming CNI release because, right now, it's just like a set of requests for enhancements. Like, there's just a list of issues. That is something that is on scope to be worked on CNI 2.0. But it's very important. And then we'll finalize with what we've learned here, we see this going forward. So, the first thing we should spend a little time to discuss is the relationship between CNI and Kubernetes. So, for that, first thing is Kubernetes networking model is something extremely simple because it basically just says that every workload, every pod, gets a single interface in it with an IP address and every pod, where it is scheduled, we'll be able to communicate with any other pod in the system through that single IP on that interface. That interface is created and configured by CNI, which stands for Container Networking Interface. Now, you look at this like that, you get the impression that these two things are actually, like, bound together in a way. The reality is this thing in the middle actually does not exist. And what you do have is that Kubernetes understands something called CRI, which is the runtime interface, and this thing actually speaks to CNI. What this means is that there is no way for CNI to actually communicate to Kubernetes in any sort of way, nor Kubernetes knows anything about CNI. It really does not know. It even exists. On top of that, another thing that is missing from here is... So let's say that your workload, for whatever reason, it actually, like, requires more than one interface. You have one, Kubernetes gives you that, it manages it, but it just gives you one. What if you need more than one? So for that, well, we didn't create, but there's a project called MULTUS that is responsible for that. Its value proposition is that enables the pod multiple networking interfaces. On top of that, this MULTUS thing actually understands Kubernetes. It speaks its API, and it actually speaks also with the CNI API, which we will see later on exactly what it is. But yeah, so we have MULTUS that is responsible for these two things. It speaks Kubernetes, and it grants a pod the ability of having multiple interfaces. These multiple interfaces can be only a virtualized interface. Sometimes you actually need to tap into, like, a physical host interface, like, for instance, SROV, you might need that. And for that, you need to add more stuff into the picture. You need to use a device plugin, you need to use a thing, a SROV network operator, which requires MULTUS, by the way, to be able to get your pod, your workload, an SROV interface. Like, the picture is getting, like, more and more complex, depending on your use case. Like, the more things you need, the worse the picture becomes, and the complexity increases. And this is what we have today, like, all these things. And there are certain, there are new initiatives coming up, like, for instance, right now, this DRA, Dynamic Resource, it's assignment, this thing got merged on, like, I think it's alpha for Kubernetes 1.26 or 27. And now what we have is a new emerging community enhancement proposal, no Kubernetes enhancement proposal, for native multi-networking. And this is what we'll focus on later on this presentation. Let's set all this. So, I'll hand this over to... Can you hear me just fine? Okay. So, Miguel introduced us to a little bit about what's C&I, what's current status, what's evolution, but I just wanted to go ahead and go a little bit more on that. So, first of all, C&I, it's basically, everybody thinks that it's basically Kubernetes networking, so we've got all the plugins, but there's so much more to that. So, I would know if you have ever wondered why, if this is Kubernetes, why a C&I plugin doesn't have, let's say, a native config, in case you don't know, so you may be wondering why JSON and no Jamel and why not a CRD and even more, why there's no demon. So, currently, this is just a quick overview about what's going on in a Kubernetes node when we are just having all the components over there. So, currently, C&I plugin, in the end, it's just a binary, which speaks JSON, and it's a binary that's being run by the QLED or a container runtime CRI. That's what Miguel was saying before. What do we want out of that? In the end, we just want a network namespace with an interface or more, but we'll get to that. But in case you're familiar with any other projects of visualization infrastructure whatsoever, for instance, let's compare that to OpenStack. In case you're familiar with OpenStack, such as I know a lot of people here are, you can just get a VM with whatever amount of subnet interfaces you'd like to, but you can't do that natively because you don't want to do it on your own. So, yes, as we're calling that, this is a binary, which is installed by a demon set. So that means that you got a copy of the binary in every node of your Kubernetes cluster.  So when you get the binary, it's going to be looking for a config file, what is the C&I aspect? So far you know that C&I, okay, is it a plugin? It is a protocol, it's an API. So in the end, it's just a specification. It allows you to use four primitives that are C&I add, del, sec, and version. And even more, out of those four, most of the C&I plugins that you'll see around only implements two of those, which is C&I add, and that means, okay, give me a port, okay, delete that. And there's some complexity out of that, because as I was mentioning before, this is not a demon, so all the implementation gets down to the C&I plugin writer. So if you want to have a controller out of that, okay, go ahead and implement that. The C&I aspect, it's totally okay, and it only expects to read a case-on-config file, some m-bars from the system, and it will just execute the binary and give you a result. So I think it's worth just taking a quick look about this. So you've got a couple of environment bars here, C&I command.  I'm speaking about, like, these C&I add and del commands, and then I need, basically, which name I'm going to be using, which interface name, recall, just like the Highlanders, but, you know, there's more than that. And then the container ID, and if you see, again, here, I can point out, but it's over here. This is just some kind of case-on in which you have the C&I version. Depending on the version that you're using, you may have some limitations, but I won't get there unless somebody specifically asks for that.  Once it's there, it's going to be executed, and it'll give you an out. If you have been to some other sessions, you may have been hearing about the prep results, and that's because you can also, yeah, do C&I plugin training, which means that you can put several of those in a pipeline. And basically, if a thing goes well, you get an index code, like a zero, everything goes well, okay, go on. Then those multis. What's the problem here? I've been speaking about that we only got one single interface per pot. That's it in acute limitation, especially if you are working with some telco environments because the most basic telco workload is going to be a firewall, and how do you do a firewall if you only got one single interface, good luck.  In the end, this CRD, it uses an object on Kubernetes called network attachment definition, but hey, it's going to be just a wrapper for this old-style JSON. This is something that we would like to change in the upcoming C&I 2.0, but yeah, I'll get that for later. So for now, what I want for you if you get anything out of this is that, first of all, Kubernetes native only allows you a single interface per pod, use multis, you got several. So this is a quick assemble of this. So you got to use Flannel, but I don't really care about this, which plugin we are using, you can use any plugin for the community. Flannel, OVN Kubernetes, but most of the using multis, you get another, if you see here, another net zero interface, net one, net n, so you have several of those, and those we can start doing something which is pretty much more interesting. So what are the pros about the current approach in C&I? So first of all, it's super simple.  So I want to create a port, a completely port, and well, second version are interesting, are cool, so you know which version I'm using for the plugin, but if you see OVN, let's go save some fancy C&I plugin if I recall correctly, so if you go and do a git blame, you may get me wrong, but Cilium doesn't even have C&I second commented, so it's just nothing fancy. But again, so this is no demon, so any developer should have to reinvent their wheel and create their own reconcile loop for any C&I plugin, which kind of sucks because why would you like to reinvent the wheel for every C&I plugin?  You may be leaking resources, and you may even not know that unless you implement your own solution or your C&I plugin. And also, C&I, to be honest, although I think it's a super cool project, the community is getting somehow smaller, which is really bad, so feel free to go join app and contribute if you want to. We are totally happy to get patches for requests, docs, comments, whatever, and the thing is that its docs could be really enhanced, again, request accepted. And also, there's pod manager somehow replacing that with some newer implementation, and as you were seeing, and the original Diagrams Lameagle was showing, it's not really Q&A's native. You may say, why, but you are saying that it's a C&I container native network interface and what, and this is because original C&I was meant to be used with Rocket, which some of you may know, it was a container interface, we were able to, some of the other implementations that we have, so it's like it goes a long way backwards, and it's not really that native. Why haven't you just migrated that to Jamel and proper CRDs? Well, it just evolved that way, so we plan to change that, we are coming in new aspects and we do things that is super simple. What happened here? Okay. Thank you. So, what's the deal here? As I was saying, this is super simple, just four different primitives. It's still as simple to do a C&I plugin and a scale. There's even some examples on that. If you go to my GitHub and you say the KubeCon C&I, you may, you can just see that I wrote an example C&I plugin for the KubeCon in Valencia, and it does nothing, but just fork it and it'll allow you to get it fully working C&I plugin for your own. But it also has some limitations, and in the end it's just a trick, let's say, pony, which is add, del, and that's it. So far, and now I'm going to be getting back to Miguel because I want for him to highlight you about there's currently a cap, which is Kubernetes Enhancement Proposal, which isn't going about okay. Let's make this really Kubernetes native and how so we would like to use CRDs we would like to use objects. There's a caveat to this though because the current proposal aims to be implementation agnostic, so that means it could evolve in a way that would totally ignore the currency and I because it's not, let's say we would like at least C&I maintainers and C&I developers would like to avoid that because we would like to see that it's backwards compatible and it works fully, so C&I gets evolved rather than you know, substituted. And also this may lead to if some of you guys are working on support, if this goes on we'll like trying to do some migration path anyway. So I'll let Miguel go ahead and explain you a little bit about this proposal but we'll get back to you later with C&I. Okay, next slide. So I'm going to explain the current state of this Kubernetes enhancement proposal for multi-networking. So the first thing about it, it's being split into three different proposals actually, like the first focus exclusively on use cases. It's a quite far list of use cases that actually takes into account lots of things that we think are missing from the original specification like it has as use cases stuff like hardware devices, it's mentioned there a hot plug into the pod basically like this means that you'd have to react and actually introduce a new interface to a running pod. This is something that it's not very Kubernetes but there are some use cases that require it. Then the second one is about defining the API. It's like the one we are currently on and finally it would be a final cap that would basically introduce these code changes. As Anil said before this will be implementation agnostic. So every time in these discussions that you ask something specific like you want to for instance, plug some runtime information from a pod. So a pod request a particular IP or a pod request a particular MAC address. The reply usually is that is implementation agnostic. We might not care about that and we are providing one single implementation and it seems to be a little bit cloud focused. But on the good side, if you look at this there are a lot of things you'll get for free. You'll get a Kubernetes native way to interact with the network plugin. You'll get an entry Kubernetes native way to have multiple interfaces on your workloads. This means like all the entire ecosystem that I've shown in the beginning with multis and all that, like half of that is actually not required. You have stuff like dynamic interfaces are featured in the community enhancement proposal.  Which will simplify by a lot the complexity of the solution. And finally you get a native integration with things like network policies and services which are of course and native to Kubernetes. And I would just like for you to take a look at, so here in the left this is what the multi-CRD looks like. So you just you just you just give it a YAML with a packed JSON thing in it where you can put pretty much everything you want. So if you forget to put a comma here this thing will not work. Like it's unpacked within a YAML. It really looks bad. It's error prone and it's hard to get right. While to the right what you actually have is a YAML, simple like it's very easy to understand and obviously to get right. Now another thing that happens nowadays with CNI is whenever you have to address one of its shortcomings like for instance reconciler resource like if you are an IPM plugin you manage IP addresses you need to reconcile those IP addresses. This is done at the case by case that's done case by case. So every plugin that you have will need to be able to do that. If you want to have dynamic interfaces you will need to find another way for that. So for instance what we did was to create a controller that looks at the annotations of the pod, sees an annotation change, it computes like the delta between the interfaces it has now, the interfaces that you want to have and will then like either add or remove interfaces. So we had a new controller, we had to redesign multis in order to be able to receive more inputs so it's like a huge amount of work whenever you need to CNI to do something it was not built to do. Again, the one trick pony knows one thing. This one actually knows two but that's what it does. For instance for Slack there's no way for you to do this like natively and there are people that are after these things. Now not everything is bad, you have the upcoming CNI 2.0 like as I said in the beginning all you have is like a list of issues but you can opinionate on them, you can say what's your opinion, you can try to raise their priority but there are lots of things that are being considered right now that will make this easier and better so it is considering demonization so instead of it being like a binary file just on the host file system that will be invoked it will be run in a pod that is managed by a daemon set. That's a lot better it helps you to have another life cycle life cycle methods it will be easier to deploy and of course it's Kubernetes native you'll get an enhanced life cycle so there's a particular issue for this and the most important thing here is they're planning on adding garbage collection verb so instead of you having to write a controller to reconcile your IP addresses for instance you'll have a verb in CNI that will actually do that for you so it will simplify your life quite a lot again this is just like an idea right now and it's being planned another thing plug in events so this is actually a way for CNI to feedback information into the into Kubernetes so for instance let's say that CNI actually sees that your pod got a new IP because of Slack so it will introduce that new IP address to Kubelet and finally also device interactions okay as conclusions what we have right now is that this multi networking enhancement proposal is very strong in terms of use cases it basically addresses like all the current limitations of CNI at least as we see it and it kind of addresses the entire multi feature set so hardware devices it speaks it is native to Kubernetes and it allows you to add more than one interface come on it's multi network all of this put together makes that well CNI 2.0 better be good it has like one chance to do things right or it will actually probably become extinct or at least that's what we are concerned about and well you can help like you could give feedback to the existing issues like if you think that you as a CNI developer to get something like your life could be easier in any way just comment like ask for it and it will probably be like taken into account quite for sure and yeah we have no doubt this is where we are like Kubernetes users will go on and live a long happy life but CNI is freezing in the water and probably like it will probably die the fun thing is that probably there's room for everyone and they can still fit together and find a way to be happy and yeah thank you any questions oh that's very important so there's a weekly meeting about bi-weekly so like twice per month of this multi-network community and Kubernetes enhancement proposal and all the caps are online you can look at them comment so welcome to do that yeah if you have any questions like now is your time and if not well thank you for your time oh sorry no totally true like oh I need to repeat the question was very very very long and I'm really sorry I would really hope that we could kind of make this a little bit more interactive because there's one thing that I notice that if you ask me that question it's because I did not explain this properly I'm not saying that this kept has anything to do with CNI 2.0 those are two different things CNI 2.0 is on one track it is trying to do something and at the same time this thing is trying to do kind of what these guys are also doing but in a native way and they're trying to specify it so both of these entities are trying to address the shortcomings of what you have right now right and one of them is native and the other one is like an improvement over what you have nowadays so would you like to rephrase your question with this in mind like I'm not saying that the the cap is about CNI 2.0 because they're really not yeah the kind of the so the question is if we are scared of the multi-net cap as someone that looks like we are pro CNI 2.0 so the thing is I think there's room for everyone involved like first of I really think like all the advantages that you get on this cap like those are real and that's the real way to do things forward now what we want to have is a CNI 2.0 way of being relevant we still think it can still be relevant even if this thing enters because like part of its problem is also part of this multi-net cap power is also its weakness or the other way around like its representations agnostic like it can be just a wrapper over the existing CNI or it can be something totally different and replace it or ignore it somehow you can just combine them which is be a wrapper or combine what there's plenty of ways this could go second wow so the the question is this multi-net cap it sounds like it be ambitious enough and traumatic enough to the existing code base that you would actually need to bump a major release of Kubernetes and the question is I really don't know it might I don't in the meetings we've been that was never addressed that I remember not at all again like the as far as like the way that you hear the default network actually not I think a little bit about the answer is no like because human is only cares about one interface right that thing will be configured the same way this thing will just give you the ability of adding more things to it so you can preserve exactly what you have today I guess that eventually they will instead of have they will adopt the common or the pod default network as one of these multi-networks I guess that's the overall direction they're going and once that happens it might make sense to do what you're saying plugins so you want to take it I really don't know so the question is if in this cap does it still makes will the hardware devices still be will it still use the device plugin to actually grant the pod access to the physical network device so we really don't know but because that is only really listed on the on the use cases but I do think like the device plugin still has a role to play in it probably the thing that Malthus does nowadays of acting like a man in the middle to kind of instruct the CNI plugin of which device it allocated well probably this implementation agnostic thing will have to do something quite similar but we really do not know it's a use case it's listed as a use case so it'll probably address it when by whom we don't know thanks for your time okay so hello everyone good afternoon my name is Miguel Duarte I'm here with my colleague Kike we both work for Red Hat in the OpenShift virtualization networking team and we are here to present a talk about titled Qvert VMs all the way down custom sized networking solution for the cluster API provider Qvert okay so before all that clicks into place and you understand what we're talking about because we really don't know like how savvy you are in either Qvert cluster API provider and all that we're going to introduce these three projects Qvert cluster API provider Qvert and often Kubernetes once we have a common understanding about those three projects we can actually explain our motivation like why we care about this and what problem we're trying to solve the goals for the network plugin that we want to develop and after that Kike is going to walk us through the implementation details of this solution and show us a demo for it okay so the first thing we're going to introduce Qvert so Qvert first thing is a Kubernetes plugin it allows you to run virtual machines and pods in the same platform it essentially runs like live version QM process inside of the pod and that's pretty much what it does the tricky thing here is if you spend a few seconds to think about this you have like a virtual machine which is inherently a stateful entity that is scheduled and running inside of a pod which is essentially a stateless entity on the cluster those things will make some tricky things in the future and one last thing we should always have in mind in this scenario is that the networking requirements for virtual machines are a lot tougher than the ones for a pod mostly because of live migration that's kind of the feature we will live and die or live or die based on like live migration will be our bread and butter for this presentation so let me just introduce to you this project the cluster api provider cluster api is something that in their own words what it does is provide a declarative Kubernetes style api to cluster creation, configuration and management so all this means that the same thing you can do with I don't know Ansible or Terraform or whatever to provision a new cluster you can use this tool and it will deploy will end you a new Kubernetes cluster it has different types of providers AWS Google, Azure there's also a particular provider the one we care about that is Qvert so this means that the cluster that you get is implemented using Qvert virtual machines as Kubernetes nodes this begs the question of what would you want to do this one of the reasons like cluster scale you can have like one very dense huge cluster with thousands of nodes and tens of thousands of pods but I'd say that's really hard to manage and you won't see many of those it's a lot more common to have a lot of tinier clusters interconnected between themselves and our use case for this is like you have a cheap cluster provisioner that you can use for stuff like for instance you want to test your feature or you want to test how does I don't know your application survive a DNS upgrade or something you just create a cluster you run your test you tear down the cluster at the end and you're done finally let's introduce the Oven and Oven Kubernetes projects so Oven is essentially like an SDN control plane that orchestrates a bunch of open V switches that run on your worker nodes it's a value proposition is of allowing you to use like higher level abstraction that you get from open V switch so instead of you getting to manage like logical to manage open flow which you get manage our stuff like logical switches logical routers ACLs and these will afterwards be compiled into open flow and the nodes of your cluster so if this is often we then have Oven Kubernetes which is a CNI plugin that provides an opinionated topology and essentially translates Kubernetes objects to Oven logical entities so let's say that you provision a network policy on your cluster Oven Kubernetes will translate that into a set of ACLs and those ACLs will essentially be translated to open flow that will be installed on the nodes same thing with let's say services these kinds of things that's its task, translate from Kubernetes object to Oven logical entity okay with all these things in mind we can we're good to go on the motivation so our thing is we want to decouple the node updates from the tenant cluster VMs using live migration what I mean by this so remember that your cluster API provider Qvert it gives you Kubernetes clusters and it implements its nodes as Qvert VMs which is a Kubernetes plugin so you get essentially like Kubernetes inside of Kubernetes and we call the let's say the top most cluster it's the infra cluster the bottom ones the ones that are being provisioned by this tool are the tenant clusters so let's say that you want to upgrade your infrastructure cluster well that we don't want it to impact the workloads of your tenants underneath that cannot happen at all for that we will rely on live migration the entire time and essentially what we have today does not provide live migration it simply does not give us what we want and for that you see the wacky thing that Kike came up with why often in the middle of all this why should we go for often well other projects like OpenStack they're already using that technology and with really good results like you have like using some improvements you get like a migration downtime of around 100 milliseconds which is extremely good and we want to strive for those numbers so that's what we're trying to go for okay so we know what we want but now we have to set explicit goals on our network plugin so the first thing is the TCP connections that are established on the node basically for KubeNet and for the workloads of your tenants they must survive the migration like once the let's say the Kubernetes node that is essentially the KubeVM migrates from one place to another that it is running must survive moving to a different node another thing the IP and gateway configuration on that worker node it must remain the same it cannot be updated during the migration why well for instance KubeNet is bound to that IP address if it changes the IP KubeNet will basically go bananas and like your workloads will be impacted another goal that we have is that a tenant cluster cannot access anything on another tenant cluster unless that tenant expose it via services also a tenant cluster cannot access anything in the infrastructure cluster unless it is also exposed via a service and we need to do that for two types of services node port and load balancers and I'm now handing this to Kike so so hello my name is Kike I'm a software engineer at KubeNet working and we have tried to find multiple products to have some kind of like migration proof of concept so what we are going to see right now is like the big point that we need is implementation wise so what we need is to implement migration in the cluster the file network not doing it in a secondary network not doing it in the feature from KubeNet as multi homing in the file network we will see later on why is that why we want it in the file network we also don't want to set IP address in the pot we want to bypass as much possible the network inside the pot and pass all the information to the VMs so kind of KubeNet is not in the middle so for that we configure this HP options from OVN in the logical switch port which means like we kind of prepare a DHCP server so the VM can consume this IP configuration right also we copy some mechanism from Calico like they use something that is called layer piproxy what it means is like some parts of the topology you can configure some parameters that is able to answer from a foreign IP address that doesn't correspond to the subnet at this level this is how we implement ARPs for the file gateway so the VMs always the same the file gateway independently of the node where they are running and also the native or cache is exactly the same ok so now we are going to see the topology of the communication for the north-south communication right the most north part this is going to be exactly the same before and after the immigration but the important part here are this IP address here and here because this is the IP addresses that we use to redirect to do the point-to-point routing during the immigration right as we see here in these tags alright in the next we see the lower part of the topology like the slide before is like the upper part of the topology this is the down part of the topology and this is where the point-to-point routing is being done and for that we have two important OVN resources to configure egress and ingress we have something that in OVN is called policy for egress so we say ok for traffic going from this IP we want to go over this node with this IP right this IP is the same IP address as I have like pointed out in the slide before then we have like another thing which is the static route and we use it for the ingress communication so it's kind of the opposite if traffic is for this IP address you go over this port so also another thing that we configure in the topology is the IRP proxy and as you see and this is very important the IRP proxy is exactly the same for both nodes with this the VMs are going to have exactly the same default gateway independently of the node the neighbor cache is going to be the same so there is no need for updates in the neighbor cache so with the downtime after line migration this is lower so the idea is to maintain as much as possible the network configuration during line migration ok another important part here is as we say the configuration of the dhsp options so this is a kind of OVM terminology to start a kind of dhsp server right so what it does is it serves the ip configuration to VMs over dhsp and another important part is here the port the interface is not configured at all so this is just L2 communication so the integration doesn't make let's say noise during line migration and it's just L2 link between the VM and OVM and of course the VM is residing the IP address over dhsp ok then we are going to see how this bottom part of the topology looks after line migration as you see is kind of the mirror the only thing here is exactly the same the only thing that it changes is for the egress traffic this ip which means like ok now in this node so I want my egress traffic to go over this node that has this ip address and the same thing for the egress traffic right so what we do is the same ip but we say ok now I want to redirect this to the core of the nodes and this is the topology now we are going to do a real demo this is I'm going to explain with a pair of slides very quick the demo we are doing it's super simple but maybe it's good for illustrating things up so what we have to do is we have like a pair of nodes we have a hustler cluster and we have the worker pms so we have a client a client pod that opens yes a one tcp connections to a server with some super dummies somewhere that we have put there with the name tcp proof we just open one connection and if you get broken the server goes down so it's easy to see like the pod see if the pod is down the tcp connection is broken which is very important for us and then what we do the hustler cluster the pod goes from one node to the other and the tcp connection all right so let's go with the demo okay so before I start let's explain what we have here so the first part is the latency between request and respond in the client in the client what we have seen and in the bottom part of it what we see is like the the two pods that implements like immigration in cuber because in cuber as Miguel has say like all the vms are back up by a pod and during like immigration you have one pod in one node and another pod in another node and there is one moment where the will communicate state between one pod to another and then one of the pods dies and the migration that's end so we will see here the latency and here the pods of cuber and we see here one pod is in one node and the other is in the other node and we see here the state okay let's start okay now it has to start the latency in one moment we will migrate it all right and then you see here in the status that is the target pod in the target node is running now they are communicating the state between the two nodes using a liber mechanism like memory page and like they are communicating and now the old pod in the old node is going to be not ready and immigration is done so what is happening here is like this proof of concept we are doing is not perfect but what we want to accomplish is like the TCP connection is kept which is good enough for us it's not perfect but it's good enough for us and this is what it is there is nothing more here and what we have the rest of it is like we do again a live migration so we see also the same AP address all right and same happens but in the opposite direction it's like we are up here I know it's not perfect but it's good enough for us for now we have some ideas to improve it like in OpenStack they use something that they call multi-requested chassis and we will see what we do with that and that's it okay conclusions so now we explain what we are doing in the default need war instead of using a secondary need war where we have like more possible to change things up well in the world of tenant clusters we need to use a lot of Kubernetes mechanism to implement communication with the API server that is running in the infra cluster in the management cluster so for that using the full need war we have a lot of staff for free we have access to services also we have isolation we have like the network policies we have all of this then we have seen that using point to point routing in the primary interfaces so we can make the TCP connection survive and has a consistent IP address that follows the VM during the immigration and now with these points what we discovered is like we know how this this proof of concept works and we can start to implement it and improve it like little by little and that's it questions I mean you are going to have so the question is about what's going to happen if I understand correctly what's going to happen when you when they try to access the port during live migration right so it depends like it's already open before live migration and we have say the connection is not going to to broke but you are going to recite some latencies in the packages if you are going to establish a new connection during live migration it's possible that you are going to your client is going to retry until it can establish the connection so it's something like half a second now something like that I know it sounds like super bad but it's just a proof of concept so this is how it's going to behave ok thank you and no because in OVM node OVM sorry OVM Kubernetes you have different logical switches per node so all the L2 traffic doesn't escape the node so then the MAC address and the default gateway has the same IP address since L2 is cut in the node it's not going to traverse to the other node like you have different switches for each node so if it is the same the ARP you know when you use the full gateways what it does you just replace the MAC address with this thing doesn't go to the other node because L2 is going to be only in that node and you have the distributed router in top of it you know it makes sense yes we are we are not breaking it we are like the question is like this feels a little if I understand it correctly it feels a little like we are breaking what is expected for Kubernetes networking you have two different pods they should have different IP addresses that's why we are using a point-to-point routine because let's say the pod is in a very foreign node that only understands our subnets that's why we need these mechanisms we are kind of I don't know how to work to put it making it more flexible fighting it a little it's a very it's not a general purpose pod it's for a very specific thing like it's a backend for Kuber I don't know if we are going to use it for everything for something that is not Kuber but yes we know that people is happy about it because we prefer to have the immigration and maybe be strictly about this one let's see okay something else okay thank you I am an engineer working in the OpenShift networking team at Red Hat and this is Patrick my colleague he also works in the same team and we are both together here today to talk about OVN Kubernetes which is the default CNI in OpenShift from the 4.12 release this is the brief agenda that we'll be covering today we will first start with the basics of what is Kubernetes in OpenShift networking can we have a show of hands what is OpenShift before? yeah that makes my job easier then so once we finish the basics of that we'll go over to saying what is OVN and what is OVN Kubernetes and then why did we move to OVN Kubernetes and I realize I have not explained what the acronym OVN stands for so we will also look at that in a minute and what are the differences of OVN Kubernetes from its predecessors the legacy CNI plug-in STN software defined networking we will also look at what that plug-in is and finally we'll talk about how this works how does OVN work what are all the components under its hood we'll try to show a live demo of how everything works together and how do you get networking on your cluster so Kubernetes networking so the fundamentals are every pod must have a unique IP so you must allocate an IP to a pod that's unique across the entire cluster the pod must be able to talk to other pods in the cluster to other nodes in the cluster and even to external entities outside the cluster so basically Kubernetes networking model defines that you need to have networking set up as soon as the pod comes up and this is like the basic however we all know that Kubernetes is just a bunch of APIs so it does not really provide a networking solution by default and that is where we need portable plugins to come on top that can work with Kubernetes cluster and such plugins that provide networking to a Kubernetes cluster known as C&Is container network interface and this is an upstream community that defines a set of standards and how a C&I is supposed to be written and as per those C&I specification standards OVN Kubernetes has been written so it is one such C&I plugin that provides networking in a Kubernetes cluster there are plenty of others out there like Calico, Silium and Tria that you might have heard of but OVNK is just a counterpart or another yet C&I OpenShift networking so like I mentioned OVN Kubernetes is the default C&I in OpenShift so OpenShift is Kubernetes plus a bunch of operators and even for networking we use an operator it's called the cluster network operator and that is in charge of deploying all the demon sets and the deployments that are required for the network infrastructure to come up and C&I supports two plugins like I mentioned the legacy one which is OpenShift SDN and the new one the shiny one OVN Kubernetes and that is the default from the 412 release and between the C&O and the specific C&I plugins we have a jam which is Maltis and Maltis is a meta C&I we say what it does is it allows an inception so it invokes other C&I plugins so it is a C&I that invokes other C&Is so let's say for example in a pod you want to set up multiple interfaces on it so you can invoke the OVN Kubernetes or the SDN C&I to configure the primary interface for a pod and then you can have a secondary interface using SRIOV maybe you want to have a secured connection to a storage which is isolated so you can use another plugin and Maltis lets you call multiple plugins to get networking set up for a pod so these are some of the components in OpenShift that are involved in networking and we thought this slide is pretty cool because it shows all the C&Is that are supported officially by OpenShift but the first two are the plugins that Patrick and I, our team manages the rest of them are third party supported vendor C&I plugins so coming to OVN Kubernetes so what is OVN Kubernetes? let's start with defining the acronym OVN but before I move to OVN we need to look at SDN because I told you that we are moving from SDN to OVN in OpenShift and that is our story, we are here to see why that happened so let's take a look at SDN first and SDN stands for Software Defined Networking it's a legacy network provider in OpenShift it is also written as per this C&I standards and it uses two main networking technologies one of them is IP tables and the other one is OVS which is a virtualization technology which creates a multilayer distributed switch so you get a virtual switch on each node in your cluster that's how you can put it simply and using OVS SDN provides networking on a part right so it hooks up and creates the necessary entities, the floors, everything you need to get the part up and running in the networking arena but we also have services in Kubernetes right so we have cluster IPs node ports, load balancers, in Kubernetes clusters how are they implemented in SDN that's where IP tables come into play so SDN uses IP tables, a lot of nested IP tables for setting up services and OVS to set up the part networking and using these two technologies it's able to achieve the entire it's able to manage the entire networking life cycle of a part in a cluster another interesting point is when you have pods across the nodes you need them to be able to communicate to each other and pod-to-pod communication on a node often known on a cluster often known as overlay networking is achieved in SDN by using BXLAN the virtual extensible local area network protocol and this supports east-west traffic and now that we've seen what SDN is if you now move towards OVNK which is the star of our show it's an open source project which provides robust network solution to your clusters in Kubernetes clusters it has OVN open virtual networking stack on top of OVS so that's the extra piece here which is really what you should take away from here so SDN is also using OVS like I said but what does OVNK do extra it brings this new piece called OVN on top which is the abstraction layer on top of OVS and that lets you create simple logical network constructs so it maps things like nodes to routers and switches so that makes it easy for administrators and infrastructure developers and even users to be able to see those virtual entities that are very similar to your traditional networking stack like this is what you see day in day out as network administrators so that layer of OVNK abstraction gives you this huge advantage whether you're a developer whether you're an operator so that's the key and one of the main reasons why we moved in fact and another subtle difference is that we use Genif as a protocol because it also offers some other subtle advantages to VXLAN like in its header you're able to store more information for example so if you were to compare these two technologies on a high level right like the highlights of them both of them use OVS in their base both of them have an internet protocol address manager controller well don't get too worried it's just an IPAM controller that allocates an IP to each pod in your cluster so it's centralized pod IP allocation both of them use OVS flows at the end of the day to create their network policies or any other things that you want to implement they use either the VXLAN protocol to achieve communication across nodes but that's just a subtle detail STN uses IP table rules like I mentioned for implementing services OVNK is using OVN abstraction and constructs and I keep using the word OVN construct over and over we will see what that is in a moment and natting network address translation this is required for every Kubernetes cluster so that the pods can talk to outside the cluster so to internet and this is achieved again using OVN constructs in OVNK versus IP tables that's the SNAT and SDN now that we've seen the what let's look at the why why did we move to OVN Kubernetes OVN is the new abstraction layer I think we already mentioned this this allows for easier development so as developers if you want to add new features on top we have a whole new engine that can let you create such abstraction constructs on your network topology and that can help you easily add on new features versus in SDN you had to directly touch open floors and this is complicated and you had to have more understanding of another technology and that made things a bit convoluted the other thing is the flexible and expandable architecture that OVNK provides using OVN so you're just connecting nodes across the cluster using routers and switches we will show this in detail in a demo versus in SDN you were using IP tables for services and OVS floors but it was too restrictive you could not do many of the things that you would want to do with telco use cases the final thing open source so SDN was something specifically created for OpenShift and we wanted to take our project in an open source direction so OVNK has contributors who are not just from OpenShift or not just from Red Hat we have a vibrant upstream community so we wanted to intentionally move in that direction so these are the 3 wins that I would say that we get out of OVNK and finally coming to the how part of our story so how does all of this actually work under the hood with any component in Kubernetes cluster even the networking component has some parts running on the control plane nodes in your cluster and other parts that are running on the data plane the control plane parts the pods that are in violet or purple here the OVN cube master that's the OVN Kubernetes component right and the pink ones they are just details you can even ignore them but they are the core OVN pieces so they are the bits that provide that abstraction that I mentioned so what OVN Kubernetes master does is mainly 2 things one it gives out IPs to pods IPs across all the pods in your cluster the second thing is it's continuously watching for all the Kubernetes API objects that are created it translates each of them into OVN logical entities and it stores these entities in a database so this is happening across in a centralized manner in the 3 control planes and once these entities are created let's say OVN and OVS are doing some magic right so they are creating they are taking all these entities and they are doing the necessary plumbing and creating the flows that are required for networking to function on a cluster and we talked about OVN construct so this is where they come in like the logical entities so if you take a node in a Kubernetes cluster as an example and when you create a node a corresponding thing that OVN Kubernetes master does is it translates this node into a specific switch, a router some policies so it does that abstraction for you so you have a switch for this node I need to now create a router for this node and then imagine multiple nodes in the cluster so you are going to have multiple routers and all of these are connected by switches so you get this whole network topology, logical apps traction at the end of the day and this is very similar to how you see in traditional on-prem networking and another thing like let's say namespace it's a very common entity that we create namespace is just a bunch of pods so if you take pods in a namespace as an example we might want to track all the IPs of those pods so we create a logical entity called an address set which is a collection of IPs and that's used to provide a specific functionality for features if you take a pod as an example a pod is translated into a logical switch port so it's a port on a switch so you have a switch creator for each node all the pods on that node will be a port that are hooked to the same switch and the switches amongst different nodes will talk through a router so it's basic networking and the same for services and endpoints so each of them are translated into entities called loadbalancers and Patrick will talk about them in detail so those are the things happening on the control plane and once the control plane aspects are dealt with on the data plane that is running on each worker node in the cluster we also have some a lot of things actually going on so like I mentioned the master has now allocated an IP to the pod in the cluster once the IP is allocated the CNI aspect the actual binary that is doing the CNI add event is located on the node so the OVNCube node pod is continuously also watching for these IP annotations that the master does and once master's finished its job the node takes over and the node is literally creating a node that is required for the pod it's actually setting up the plumbing inside the OVS that is required for the pod to have its networking up and so on so the CNI executable commands like we mentioned which is done as per the CNI specifications is all executed by the node component and in the node also we have these pink magic containers that are translating these entities which we created in the database in your control plane for simplicity and now that I've covered all the fundamentals in the theory aspect of it we go to the core part which is the demo aspect where Patrick is going to show everything that I said but on an actual live cluster so over to you Patrick Hi everyone, my name is Patrick I'm from Red Hat and I work with Surya on OVN Kubernetes and general OpenShift networking so what we have prepared here is how all of what Surya of us ties into an actual cluster so for our general upstream development we are using kind so in the recorded demo I'll first clone our upstream repository which all of you can easily do and then we'll move to actually starting up a cluster so as I said we are using kind kind is Kubernetes in docker so so for every container that docker creates you will see a Kubernetes node so we've prepared in our upstream repo a very neat script that allows you to with one command to create a kind cluster that actually uses our CMI of Kubernetes as it's networking as you can see there is a ton of options that you can choose from for scenarios like IPv4 IPv6, dual stack and all of different features that are available in OVN Kubernetes as I said to set up the cluster you just need to run one command it will take around 5 minutes to set it up I'll fast forward here so we don't waste too much time as you can see we build all of our containers we've set up the Kubernetes cluster at the end we apply simple YAML files that take care of deploying OVN Kubernetes into a cluster that initially doesn't have any CMI on it ok, just tying it back to kind if any one of you is not aware as you can see we have three nodes OVN control plane OVN worker and OVN worker 2 these are of course represented as docker containers and we have a little bit of an exception here to exit into any of those docker containers you can see that there is cryo and you can see that there are additional containers running on our worker node now moving on to OVN Kubernetes as Surya said we have a control plane and the data plane the two top bots contain our control plane components that contains OVN databases that we use to store the logical entities that's the guy here and the second one, OVN Cubemaster that's the main piece that does the translation between Kubernetes and OVN so this is the component that will create logical entities in our NVDB that I'll talk about more in the next slide of course we also need to provision the data plane components as you can see the next piece is OVN cube node this is a Damian set that runs on every node in the cluster and you can see that in the nodes section and it contains OVN cube node pod that pod runs the actual CNI so when a pod gets created we call the container runtime which will then make a call to the CNI and we'll take that call and provision pod networking for every pod that's clustered additionally and that's something that's upstream specific you can see that there are OVS nodes pods these are running OVS upstream so these are responsible for setting up actual flows and applying them to actual packets so after all of these layers of abstractions coming from Kubernetes to OVN we configure flows that get applied to each packet that flows through our cluster this is not something that we have downstream and in downstream we have a host network service sorry we have a system D service that runs on the host and it's compatible with a simple kind cluster since we have just Docker containers additionally please note that we do not have HA here I tried to keep it simple for the demo purposes but of course you can enable it so all of the control plate components would be multiplied to provide the HA functionalities the key in the eye of you can see that all of these that are actually from the host network this is because we are the CNI there is no CNI to provide us with any cluster network level networking so that's something that you will often see in OVN Kubernetes that our data plane pods and control plane pods are host network and yeah you can see that they share the same IPs as the nodes let's get back to the slides for a second and talk about how we represent a cluster in OVN ok yeah as Seria said the cluster is represented with a bunch of switches and routers and that's it it's very simple to grasp if you know what a switch and a router is so as you can see the dotted boxes represent every node in a cluster so here we only have two but you would get everything that's in the dotted box you would see on every node of your cluster up top you can see the underlay network this is the network that allows us to connect the nodes and to provide external connectivity to the cluster and this is not something that we configure and it comes as underlay network so it's provided to us we can use it to join the nodes you can see here that every node in the cluster gets a host subnet this is a subset of the whole whole cluster network it's specific per node and all of the pods in that specific node will only have IPs from that host subnet starting at the bottom as Sria mentioned every pod will get its own logical switch port every pod will have it created by us so during the pod creation will create a VIF pair and one of the ends will get connected to the pod net namespace and the other would get plugged in into an OVS switch that would then be represented as a logical switch in OVN so this gives us the advantage that if there are two pods on the same node they will use that logical switch and it's local so there is no need for the traffic to traverse node boundaries now if the pods are located on different nodes and they need to communicate with each other they need to have something that connects them so we use centralized cluster router for that this is a singular router that will be only one per cluster so this gives us an opportunity to connect logical switches from different nodes and additionally it gives us a path to north south connectivity so if a pod wants to talk to the internet it will go through the gateway router so the gateway router is a node-specific router and it is responsible for providing the ingress and egress functionality out of and through the cluster as you can see at the top here it actually takes over the default interface that's on every node so when you provision OVN Kubernetes that interface that was the default interface connecting the nodes will get taken over by us and will manage it and that thanks to this we can provide external connectivity and direct traffic as we need to and one note I wanted to make here if your pod is host network which means that it doesn't need the cluster network IP it's not going to be represented in OVN it's not going to be represented in OVN and most of the traffic flows that's to external entities will not use OVN so thanks to it our CNI uses that approach and we can set up everything before we start handling pods let's see how all of that looks in an actual cluster so I've prepared the trivial YAML file that contains a client pod and a service backed by a deployment of three replicas so I did that to showcase the very minimal approach that you could take to see all the components that OVN configures to provide service and pod to pod connectivity so how do you actually see what's there so we applied the pods we see them here and they all are running and right now to look at the logical representation in OVN what I do is I enter one of our pods so we can directly interact with the database so for the remaining part of the demo you can treat it as the left side contains what's in Kubernetes and the right side represents what's in OVN and so we configured that allow the connectivity between pods and services as I said in my previous slide we have three nodes and all of them get a node specific gateway router that we use to provide external connectivity basically and additionally as you can see there is just one central OVN cluster router that gives us the possibility to connect pods and services that run on pods there are also logical switches so every node gets its own logical switch and additionally we use external switches to take over the interface from the host and we use one joint switch that allows us to basically connect gateway routers into the cluster router okay moving forward on the left here you can see pods with OVN that you can additionally look at the IP addresses that we've allocated and the nodes that the pod is running on so here I show the OVN worker switch and on that switch the first port that you see is one of our packing pods for the example service I created additionally you can also see that on the same switch we have more ports that are not related to that deployment so all of the cluster network pods that are on a particular node would show up here we of course have a port that allows us to connect to the router and a port the worker port that allows us to provide a cluster network to host network connectivity same goes for the other switch that hosts the default client and additionally one of the packing pods for our service so you can compare that to what we see in the Kubernetes world and it matches and additionally we see that we have the IP addresses and MAC addresses that we've allocated in OVNK let's take one of the logical switch ports as an example and this is the object that we store in the database that represents one of the logical switch ports and one interesting thing I would like to point out here we are using OVNK port security which is a feature that allows us to restrict the traffic that's allowed in the switch so anything other from this IP coming from this particular pod won't be allowed the same goes for the MAC address so if some rogue user enters the pod net namespace and changes the IP address on the vif pair from that namespace we won't allow that traffic it will get dropped so that's a neat feature additionally we also see this this is useful if there are any issues with that port but here you can see that it's up so it's functioning correctly okay so we've talked about port connectivity and how those ports are connected to a switch but what happens if a pod wants to connect to the internet right what we do in the gateway router we create an S NAT rule so we're going to translate the source IP of that particular pod into the IP of the node so to the outside world the IP that's egressing the node will be the one of the nodes so that's the IP address we take from the default networking interface and then we'll an S NAT so we'll translate it back when the result reply from the external line so as I said at the beginning of this part we've also created a simple service and you can see the details of it here so it's a service with that IP address listening on port 80 and these are the backing ports that are the endpoints for the service and let's see how that behaves in Oven so in Oven we have a construct called load balancer and it does basically load balancer traffic from this IP address on that port into all of our endpoints that we've defined in the service the name of this entity is matching the service that we've created so as you can see you can easily get the networking representation of what's actually in a Kubernetes cluster you don't need to look at particular OVS flows you don't need to dig through layers of IP tables and it gives you an insight view of how the networking is configured in OVM of course all of this logical representation of the network it's not something that will actually handle your packets so what you can see here is a dump of flows that we configure in OVS so every logical switch every router and every nut will eventually get translated into an open flow rule that will be applied by OVS and its module in the kernel so it's extremely valuable when you are deploying features and when you are doing it fast to have an abstraction layer that allows you to not deal with all of that because it is really difficult to dig through all of the layers and steps that the packet takes when it traverses your cluster and that concludes the demo yeah I'll just briefly go over that okay we do have some additional features in OVNK but we unfortunately don't have time to get over them sorry but you can check them out on the slides or you can check out our OpenShift or OVNK documentation for that and that would be it thank you all and do you have any questions so the question is is OVN better at throughput compared to OpenShift SDM that's an extremely difficult question because to get the answer for it you would have to specify the exact traffic flow that you use to pull that throughput because it's different when there are multiple hops between pods when there are different traffic paths and especially if you consider packet sizes protocols and all that it's very difficult at the lowest level when everything is configured it's OVS so it will traverse the OVS data path and then based on the connectivity between the nodes this will be the engine that handles the packet so this OVN as itself is an abstraction that in the end will still configure OVS flows so I would consider them but to get any detailed results it's not really easy to answer that we use Geneve so with all of that with either VXLAN or Geneve there is an overhead and it's something that you can alleviate by using something like hardware offload and as far as I know it's more difficult to get hardware offload for Geneve but yeah, there is an overhead and I wouldn't think that Geneve is much better than VXLAN in terms of the overhead especially that we pump more data through Geneve because we use the optional header options in Geneve that will increase the packet size yeah, you just need to use JumboFrames and one traffic path and you'll be good it seems we are out of time