The Carnegie Mellon Vaccination Database Talks are made possible by Autotune. Learn how to automatically optimize your MySeed call and post-grace configuration at autotune.com. And by the Steven Moy Foundation for Keeping It Real, find out how best to keep it real at stevenmoyfoundation.org. Alright guys, let's get started. Welcome to another Vaccination Day this summer series. We're excited today to have Michael Runstrom. He is the head of product at Hotworks, where he's been working on RodDB. Michael got his PhD from the University of Lipnitch. I'm butchering that one. But in Sweden, he's been working on NDB, which is the backbone of MySQL cluster for over 20 years now. And since then, he's been spun NDB off as RodDB as part of this larger Hotworks platform that they're building. So, as always, if you have any questions for Michael, please give him this talk. Please unmute yourself, say who you are, and ask a question anytime, that way he feels like it's a conversation. And we thank Michael for being here because he's in Sweden and it's 10.30 at night. So we appreciate him staying up late to talk about his database. Michael, the floor is yours, go for it. Okay, thank you very much Andy. So, RodDB is, as mentioned, is a fork of something called MySQL NDB cluster. Which originated within Ericsson in the 90s and has been developed a long time since then, and is still actually developed inside of Oracle. And so I'm going to go through today a little bit more about the requirements that started the project and continue to lead the project, so to say. I'm going to talk a little bit about LATs, what that is. The architecture of RodDB, use cases, how do we do high availability, some of the basics about data distribution and a lot of other things. And then at the end, I'm going to do some deep diving, particularly I'm planning to do a deep dive into how we do checkpoints in RodDB. So that's more of where we go into details. So a little bit about NDB in the beginning, I mean, I've always been very interested in numbers, still am. And when you do benchmark, you get numbers. So I always have this keen interest in getting things to work faster and faster and faster. So that's always been an interest for me. And so one of the things about RodDB is that we try to always make it very fast, both with throughput, low latency, high availability, and now also scalable storage. So we have a lot of interesting benchmarks, competitions in the past. And the most lucky one was one where we had the competitor was organizing the benchmark. And we still won the benchmark. That was a pretty nice one when they actually changed the rules three times just to try to win it. But we still won three times. Can you name names or bleep it out? Well, the other ones were internal databases inside of Ericsson. And then there was a database called Clastra, which was bought by a son in the 20 years ago. And nowadays it's gone from the earth. And ESA was one of them. Okie doke. So let's move into requirements. So the original requirements came from the telecom systems. And the idea here was that you had tens of interactions. When you have, for example, you started a mobile telephone call. So you get a number of interactions, 10, 20, 30 interactions with a database. You have to do all of those in 10 milliseconds, even at high load. That's the requirement. Because people, they are waiting to connect their telephone calls. So they don't appreciate waiting too much. Obviously extreme availability. Networking applications is another category. For example, the DNS server, DHCP server, AAA servers, you name it. They all have extreme availability, but they don't have the same requirements on throughput and latency, but availability always on. Gaming applications, they are more about complex interactions and high availability. And the latest thing that Rondi B is focusing on now is machine learning applications. And they can have tens to hundreds of interactions within a few tens of milliseconds. And again, there is a user waiting. So you have to complete those even at high loads. It has to complete within a few tens of milliseconds. And you can have tens of thousands of such interactions. So there was a benchmark which was performed by Spotify, where they tried this out. And they had 40,000 interactions per second. And they had 200 records that they retrieved in each interaction. And another thing about machine learning applications is that here the size of the storage can grow to all the way up to the gigabytes. So if we put that together in a technical level, we have the latency requirements. We have to basically, it's driven by users waiting for the database transactions. And it's mobiles, finance, gaming, availability, downtime equals no phone calls, no social media interaction, no internet, no financial transactions, no business. So obviously, downtime is completely forbidden. Throughput is driven by the many tens of thousands of interactions leading to many millions of key lookups per second. And scalable storage is driven by, well, it's driven by data and many, many millions of users. Okay, so that was the last property. So what do we do with RunDB with that? We have a key value lookup latency, which is around 100 to 200 microseconds. And that's at high load. So that's not sort of at low load. Even at high load, we have these latency numbers. And then we have to get down to about 30 seconds of downtime per year, at least in some systems. I'm going to talk a little bit more about what you actually need to do in order to get there. And then each CPU in the data nodes have to handle about a couple of hundred thousand lookups per second. So that leads that each server can do many millions of lookups per server. And that leads to that the cluster should be able to do at the level of billions of lookups per second. So we're talking per second here all the time. And when it comes to storage, RunDB scales to 16 terabytes of in-memory storage per data node. And you can at the same time have hundreds of terabytes of disk storage per data node as well. So obviously you can scale to fairly large sizes because we can have up to 64 data nodes in RunDB. So then we go into the architecture. This slide is something that anyone that wants to build a database has to understand fully and completely. And I could probably give a lecture on this for one hour, but I'm not going to do that. But essentially, you have to define where do we want to put the data server APIs? Where do you want to put the query server APIs? Do you want to put them inside the database? Do you want to have a distributed architecture? So if we look at what we have in RunDB, you can see here that we have the data in the data nodes. We have the actual application code or server code is in another layer. So we have, for example, a MySQL server that can access the data. You can access the data from a C++ application. You can access the data from a Java application. You can access the data from a Node.js application. You can also write an LDAP server on top of it. And in June, we're also going to provide a REST API towards RunDB. So as you can see, it's an architecture that is not just meant for an SQL database. It's meant for SQL. It's meant for LDAP. It basically has a generic data server, not so much as a database. It's more of a data server. So it can serve many purposes and it can actually serve those purposes at the same time. Okay, so use cases. Here's a good example of how you can combine all those things. In the telco system, you can have LDAP servers and at the same time have SQL statements being executed. And you can have special telco applications that takes care of HTTPS, CLI, radios and so forth. So as you can see here, you have many ways to access your data. And this is what kind of tradition from the telco arena that NDB took and has then developed further into RunDB today. This is a very simple system, a DNS server. So looking up, translating internet addresses to IP addresses, obviously you need to have very high availability. But this application is written directly towards the C++ API. So here we only have one API. Here's another application and financial application, the stock exchange. So you have all the changes on the coming in from real-time. You have the stock order feed where the actual updates come from. And then you can get the real-time stock quotes from the Java application. So this is a pretty typical application of NDB and could of course also be a typical application for RunDB. This is an example of where we actually had some researchers at the KTH that they started with a Hadoop file system, a distributed file system which actually had a bottleneck in that there was only one metadata server. And then you could use journal notes and ZooKeeper to make it highly available. But they wanted to have a more scalable architecture, so then they brought in RunDB for that. So after RunDB was added, they had a completely scalable architecture with any number of name notes, any number of RunDB data notes, and any number of file server data notes where the actual large files were stored. You could even store some of the files in the RunDB data notes. So this is a file system that we're actually using in Hopsworks. So everything we do in machine learning actually depends on this file system. So we have a back-end file system and this also is working on top of RunDB. So when we are selling to customers a feature store, they can store their machine learning features for online usage directly in RunDB, but they can also store things for offline features. They can store that in this distributed file system so we can have really massive amounts of data in Hopsworks. So I mentioned that we wanted to support 30 seconds of downtime per year less than that. So we actually had, for many years, we even had down times on a pretty high level that was down to less than one second per year. But of course the humans program system, so that meant that eventually you always get this large crash. So after a while you always get to around 30 seconds of downtime per year because humans cannot really make things better than that, I would say. So that means that in order to get to that you actually have to have replication both in a cluster, but also between clusters so that you can survive earthquakes and other things. And we have a pretty advanced system in handling this. And the author that I noticed that he's here today, Frazier Clemens. So he's actually one of the main authors behind this part. But I'm not going to talk so much more about that today. So just want to mention that as well as... You said it's quite dedicated. Is there like a one that says what makes it special? And is it good at airing or is there something more fundamental like theoretical? It has conflict detection, which is pretty advanced actually. I don't really know of any other system that has such an advanced conflict detection system. So you can more or less run transactions and have basically discovery exactly which conflicts that you interfere with and based on that you can write an application that resolves those conflicts. So quite advanced I would say, but it probably too advanced for normal users. But from a research point of view it's probably very interesting to study it. But again it would take hours to explain all of that. And if you're really interested, I wrote a few chapters about it in a book, which is called MySQL Cluster 75 Inside and Out. That's some chapters on that. Okie doke. So in the cloud you have to sort of, I mean, you have virtual machines in a cloud. In order to get the highly available database cluster in the cloud you actually have to make sure that you put the virtual machines in the correct availability zones. Actually I mentioned here failure zones as well. So some clouds can also have within an availability zone. Oracle Cloud can do that. I'm pretty sure that Azure can do that. Not 100% sure about all the others. So what is needed here is that you need to make sure that, I haven't really introduced no groups yet, but think about no group as a shard. And so basically that means that you have to make sure that all the nodes within a shard are placed on different availability zones. And then the MySQL service you can actually scale them any way you want. As long as you can access the MySQL server it can access the rest of the cluster. So we also provide managed run-to-beat. So this is for when our customers want to use run-to-beat. The only thing they need to do is to create a cluster. They need to specify the number of replicas. They need to specify the VM type of the data nodes. They need to specify the number of node groups, so shards. They need to specify the block storage size or basically the file system size. And the number of MySQL servers and the virtual machine type for that. And also they can add API nodes if they like to have some special applications. So it's really simple to create the cluster. You just specify those things and then say start. And even more interesting is what happens when you discover that your cluster is actually too small. Let's say that you wanted to start really small. You started with one replica and four gigabytes of memory. And now you want to grow to two replicas and 16 gigs of memory instead. So the only thing you need to do is to change the number of replicas. Specify the new VM type and then basically just say submit. And then the software will make sure that this change happens and it happens online. So you can continue sending transactions to the cluster and it will still operate. And if you want to go down from two replicas to one replica that can be done. You can go up to three replicas you can change. It could be a bit to go down is always more difficult than to go up. But because you could actually end up in a situation where you don't have enough memory. But if you have enough memory you can go down as well. So this is the interface and how it looks not going to go into any details. But it's as you can see it's not you don't need to be a database expert to start a cluster. You only need to know some something about what you what you want to do. And this is the interface when you want to change the cluster. And then you get the list of the changes to be submitted. And then our software will make sure that you that these changes happen in an online fashion. And it will actually not do all of them at once. It will do one thing at a time to make sure that the cluster stays online all the time. Okay, so let's go into some basics about about things. Let me just I just need to keep track of time so that I don't you're going to talk too much. So I've already talked a little bit about node groups. So at the very beginning when NDB was designed you there was a choice actually how how do you distribute data. I mean there's one way of distributing data where you optimize on quick recovery. And you can also choose to optimize on surviving the maximum number of failures. So I decided to go for the variant where we survive as many failures as possible. So this means that if I have two node groups for example with two nodes in each that means that I can survive up to two node failures actually at the same time even. If I have one node group with three replicas, I can survive one two failures as well, but they have to happen one at a time. So I cannot survive two at the same time. So, so we can actually survive quite a few failures at the same time, given that we have focused on high availability and not done. So that means that when you when you recover you are always using nodes in the same node group. So that means that when we distribute the data we split the data into partitions in horizontal partitioning. And then we decide that this partition goes into this node group this partition goes into the second node group. The next one goes into no groups here again the next one goes into no group one again. But we also distribute the primary partition in an efficient way. So that we have a fair load on the data nodes and on the CPUs. So this is perfectly classic data distribution using hashing. So we have a hash distributed hash, which we use to perform the partitioning. Very short about the commit protocols so wrong to be also supports transaction so actually everything that happens in wrong to be is performed by transactions. So if you have studied to face commit, you might be aware of that there is one variant to face commit which is called linear to face commit. And what you can see here is that we do linear commit for each row. So that means that we first send the commit to the primary and then to the backup and then sort of back to the transaction coordinator and then we go back again so that means that we actually save some communication. But if we have a transaction that involves multiple rows, then those happens in parallel. So so we do sort of have an optimization but we don't do linear commit completely we do sort of a mix of normal to face commit and linear commit. And one problem that you have probably heard about is that to face commit is a blocking protocol. We actually solve that problem in the way that whenever a node fails whenever that TC transaction coordinator fails. We will take that transaction coordinator and rebuild it so we will basically rebuild the state of that transaction coordinator so that we can abort or commit all transaction that it was that it was handling at the moment when it crashed. And we can do that because all of the nodes, the participants are still operating or not all of them but enough of them are operating so that we can still get the state. So that's why we actually do have to face commit, but it's non blocking.  We don't log the actual to face commit we log later on when we do a group commit so we essentially have two faces where we, the first phase is what we call network durable. So that means that it will survive crashes, but it won't survive a cluster crash. And then every second we make sure that that all those transaction that committed the last second are also forced to disk. So that means that we are this durable after one second but we are network durable as you commit. Okay, and now we go a little bit into the road structure so we don't really have any advanced data structures. I wouldn't say I mean it's a fairly standard so we have a fixed size columns which have obviously for fixed size columns for in memory. And the that's obviously the very fastest that you can read and write. We also have a variable size in memory part. It also has a dynamic part. The dynamic part is very important because that means that you can add columns without sort of any, you can do that as an online operation. And one of the things that I'm adding right now which is almost ready is that we also will have variable size on this columns. That means that you will be able to add also this columns dynamically without any downtime for the system or for the tables. We have an index state we have two indexes we have a distributed hash table. If you're interested in in learning more about that you can search for my PhD thesis it's called LH race to three because it's hashing in three steps. It's very very much intertwined with CPU caches. And it's been very very successful all the years so that's at least one of the reasons why Ron the B has a good performance. We also implemented an order index which is an in memory order index which is using a tea tree. You can read about that there's an article in 1990s on tea trees. It's nothing I mean it's basically not that different from other binary indexes like the trees and so forth but it's specialized for in memory. So if you look here. Have you done have you done I mean I don't know if you've done debate with a tea tree.  It's a mental like on the super scalar architectures. We did benchmarks when we implemented tea trees and compared it to a B tree. And actually the result was that they had the same performance. So, I wouldn't say how long ago was this benchmark. That was 20 years ago, now 15 years ago. Okay. So there was super scalar CPUs but but not very much more than that. I guess you guys have known what's using tea trees and I know there's extreme TV. Because times 10 is using to teach teachers to they got rid of it. Or like it's, it's, you can get people's people's trees by default. So this is this is rare. This is awesome. Yeah, that's probably true. I mean we were more or less sort of we had a choice. Do we want to go with the tea trees or do we want to go with the trees. I mean they had the same performance and so we just decided we go for the tea trees and and I wouldn't say that the tea trees sort of has been the focus. I mean it's really key look up through the hash key that has been is really, really the focus. But nowadays I would say that we also do scanning very, very efficiently. So, so I'm pretty sure that tea trees is not the optimal solution, but it's also implemented fairly efficiently. So it's definitely not a sort of significant problem for us. Be quite like Ron DB has it and then therefore NDP has it. Correct. Oh yeah, this is like implementing tea tree is definitely a few years of work. So you don't sneeze it out of your nose like that. So, okay. So this now I'm going into the what we call the virtual machine architecture, but I actually remove the name virtual machine here because it might be a bit you're probably thinking about virtual machines as virtual machines in clouds and operating systems but what virtual machine here is more actually the term virtual machine. It's not. So basically, but here we are just having a layer that takes care of sending messages between. So we have three layers here. We have blocks, which is essentially modules of software. We have threads that contains a number of those modules. And we have nodes, which are essentially processes. So whenever you send the message in Ron DB, you send it to a node, a thread and a block. And we have this architecture that make sure that you can sort of direct it so you can go internally in the thread, you can go to another thread in the same node, and you can do a send receive over to another node. And the nice thing is that from a program from your medical point of view, you don't really see what you're doing. It's only the address that tells you where it's going. So that's an interesting part of the rendezvous implementation. Another thing that we've added lately too, and this is actually that started in NDB but completed in Ron DB. And that's the memory architecture where we now have completely taken care of all the data so that we sort of manage the entire data memory inside something we call shared global memory. So effectively when you start a data node, you more or less the idea is that you take care of all the memory in the machine, except what operating system needs and so forth. And then we automatically decide how much memory is going to be used by the different parts, but there is also flexibility. So if if for some reason you need more transaction memory at the moment, then you can steal that from some other parts. And if you need a lot of query memory, which is used for complex queries, well, then you can use that. So we manage data much more efficiently nowadays. Okay, and then we have the threat architecture. So this is an interesting topic that again could be discussed for and this is actually one of the most important reasons why Rhonda B is fast.  So as you can see here, we actually when when we handle a key look up, it's not handled by one thread. It can handle by up to four threads. So you get you come in to the receipt thread that takes care of the IP sockets. You send it over to a TC thread, which takes care of transaction handling. You go into a data manager or query thread to take care of the actual database operation. And then you go into send thread to actually send the response back or to another node to handle update of the other replicas. So we had some experiments done to see sort of how does this architecture compare to a single thread architecture where you do everything in one thread. And it turns out that this is much more efficient than but read the blog. If you're interested, there's a lot of data about why it's faster and and so forth as well. This is basically data. This is basically what data the stage event driven architecture from like Matt Walsh from the early 2000s. Yeah, I mean it's it's it's a message message passing architecture where we actually have message pausing even between threads internally. I'll post it. I put the link in the chat here. We can cover this later. Yep. Okay, another thing we added lately. I mean, most databases that do key value operations. They are often designed in such a way that they partition the data based on on the partition. And there's one partition taken care is taken care of by one CPU. And we had that architecture in MDV and in rendezvous for a long time. But in lately, I've actually added the possibility that read reads key reads are actually possible to do also by what I call here query threads. So query threads, they can actually execute operations for any partition. And obviously that means that there has to be some new fixes and things like that. So that means that we have to have a scheduler for those read operations. And that scheduler actually takes care to make sure that that we only use query threads that share every cash with the actual data owning thread. Because rights, they are still performed by the data owning thread. So we have to make sure that query threads that reads the data that they don't go and read data from the wrong CPUs. It's mostly in AMD CPUs that this matters. So because they have this. Well, they basically develop the CPUs in a different way compared to Intel and ARM. So we'll see what happens with Intel and ARM CPUs if they go the same route. We go back to the previous slide. Sorry. What? Go back to the previous slide. Yeah. So this is what there's operations and you have this LDM thread. And you're basically saying when the LDM thread has nothing to do, you're pushing on to it's like work queue to go do something. Right? Is that what this is? Yeah. So actually it's a kind of, I mean, again, this is something you could talk about for around an hour because it's really philosophical because. You spend, when the TC received threads, they don't really know on a detailed level about how loaded those threads are. So what we do is that every 50 milliseconds we distribute data about the load level. But that's a fairly long time in a CPU. So we also take care of looking at queue levels. So, and that means that we at least know what has happened one millisecond ago. So based on that information, we choose whether we go to the LDM thread or whether we go to a query thread. And so we have very, very, very detailed information about the load level and the queue, the queue levels on each other thread that can handle the query. And then we put it into the queue. And then once it's in the queue, then it will be handled by that thread. Sort of like co-routines, right? Sort of like everything's saying, all right, the TC received thread. The idea isn't work, but let me go, if I can, let me show it down to the LDM thread. If I think it's idle, I can take care of it. That's the gist of it. Yeah, more or less that if the LDM thread is sufficiently idle, go there. And then we have a special query thread that if you look at this picture, you can see that there is a query thread that operates on the same core. And if that's available, then that will be used. If both of those CPUs are very heavily used and there is another one that's not so heavily used, then we can put it on a different CPU core, but still on the same entry cache. I understand. Yep. So that was kind of interesting. It's the first time I've been reading a physical book about time and philosophy and apply that within a computer server. Kind of interesting. At least I thought so. So some rendezvous resources before I dive into the advanced stuff about checkpoints. So we rendezvous.com obviously. So if I have time, I will also show you a little bit about this YCSB benchmark. It's kind of cool because it's the first time in as far as I know that somebody has published numbers on a benchmark when you're doing recovery at the same time as you're running the benchmark. So we provide numbers both on throughput and on latency while the recovery is ongoing and also when the crash is happening. So go there and have a look and see if you have some get some interesting ideas. Okay. Now I'm going to go into the rest of the time into a topic. I guess it's about 10 minutes left or something like that for. Yeah, like 15, 20 minutes. Okay. So partial checkpoints is something that I actually spent almost two and a half years on that project. So it's recovery architecture is inherently complex, not so much because it's more complex than other code, but mostly because the crash happens not when you did the problem when you had the bug. It happens much later. So the only way to find the bug is to do extremely extensive logging. And that of course means that you're changing the timing and so forth. So it's sometimes pretty hard to find the problem. But the reason for changing the checkpoint in scheme is pretty easy. So we used to have a full checkpoint and I looked on other databases that are in memory databases and I so as far as I've seen, most of them still use full checkpoints. And if you look at what that means it means that if you have an in memory data size of for example 16 terabytes. It's pretty obvious that that it's going to be hard to do a full checkpoint. Because that takes ages to write to this. So we wanted to have an architecture that supports in memory sizes up to 16 terabytes we want to be able to survive with a redo log of 64 gigabytes whatever the workload. We want to be able to have a checkpoint on this which size which is no more than 60% more than the actual data size and with compression it should be even smaller than the data size. And this is obviously driven by memory sizes growing in the introduction of SSDs. And 64 gigs log size that's just like that's a number you came up with right that's not something specific to around me. I came up with that simply because that's what was required when when I was running at the very very high load. Okay, with 32 gigabytes I wasn't really able to to get it completely stable in all workloads. Most workloads even four gigabytes would be okay. But I always test with very, very, very tough workloads so and with 64 gigabytes, more or less everything works. It's not the right time for this conversation but like, is why to be a really tough workload if you configure a certain way or that like that's not even like in the ballpark you guys are considering as a tough workload. So the workload that I used for testing was low, I can show that. I think it's here so the test case that we used was actually loading data into tpcc with dbt2 so we had 53,000 warehouses. So basically five terabytes of in-memory data that we load at about one million records per second or something like that. Yeah, so that was the test case. So we were fortunate enough to have in the Lona's persistent memory server. Otherwise it's kind of hard to get hold of a machine with five terabytes of data in memory. So what we concluded is that the full checkpoint isn't really practical beyond 200 gigabytes of memory. And when we were thinking about how to solve this, obviously one solution would be to simply use page-based checkpoints the same way as this database does. And I was thinking, well, why not? But then I sort of made some calculations. How would it happen if I have a database of 16 terabytes or one terabyte even here only? And I do a page-based checkpoint with one million rows committed per second. And it turns out that the page-based checkpoint has to write 320 gigabytes. Whereas the row-based checkpoint only has to write four gigabytes. So that's 80 times more efficiency with row-based checkpoints. So obviously row-based checkpoints is a lot more complex. But if it's 80 times better, well, you go down the drain and do the complex stuff. So a row-based partial checkpoint means that you write all the changed rows since the last checkpoint. So basically, in a sense, the checkpoint is sort of an extra redo log. And you could probably, if you really spend time on it, you could probably combine it with a redo log, but we didn't do that. But you also have to write a subset of all rows in each checkpoint. And what we did was that we divided from the B uses a row ID, which consists of a page ID and a page index. And we basically divided the page IDs into 2048 parts. And from zero to 2048 parts were checkpoints in each checkpoint. And each table partition is handled independently of others. Recovery applies checkpoints based on a control file per table partition. And you always start with all these data files and go to the last one. So that means that some rows will actually be encircled more than once. And I'm not going to go into details, but there are some interesting mathematical intricacies in that. That was kind of interesting. This I already mentioned that we need to avoid running out the redo log. So again, we have in Rhonda B, we have quite a few adaptive algorithms. And the checkpoint speed is one of those adaptive algorithms where we keep track of CPU usage, we keep track of redo site, how much we do we have used, how much writing we have actually done to the database. And all of those things are put into a sort of into some computational efforts.  So that means that if you write faster, we will checkpoint faster. If you have problems with the IO, we will have to slow down for a while. If you use CPUs quite heavily, we will also slow down temporarily the checkpoint thing. So we always try to sort of to provide something which is good. But obviously if the redo log, if you're almost running out the redo log, we will start spinning up the speed quite heavily. And the test case, as mentioned, was dbt2 with 63,000 warehouses where each warehouse is about 100 megabytes of data. There is some interesting problems. So checkpoint is complicated since pages can actually be dropped off during scans and between checkpoints. So in the middle of this long development, I actually had some doubts whether this algorithm would actually work. So I had to write a proof, not so much for everybody else, but even for myself, I have to prove to myself that this algorithm actually works. So if you look into the code, you can actually find a 10 to 30 page long proof of the algorithm. So if anybody's interested in seeing if it actually works, then feel free to do it. So basically this is just to give you an idea about the complexity that you can find. Let's see. So when we're scanning a page, we have on each row, we have a timestamp. And the timestamp is implemented by what the thing I talked about previously, the global checkpoint, which is the sort of group commit executed once per second. So that's the level of if something has happened the last second, we will write that row. If not, we will ignore it. And also there's some some work to do when you're writing a row. You have to sort of decide whether the checkpoint has actually already passed it or if it's still to do the checkpoint thing because checkpoints have to be there. The checkpoints are fussy, but they are actually transaction consistent at the same time. Probably doesn't explain enough, but Yeah, what do you mean by that? But they're fussy, but they're also consistent? Well, fussy is means that you don't have to lock anything when you're doing the checkpoint. Yes. Transaction consistent and actually even action consistent means that the checkpoint actually restores a very, very specific point in time. So you have to restore exactly what happened when you started the checkpoint. And the reason for that is that we have to actually have we have pointers that points from in memory data to disk data. So if we don't have a complete synchronization of that checkpoint, we wouldn't those pointers would be able to point to the wrong place. You I think are you switching to like a multi version mode when you start the checkpoints or do you have a consistent snapshot. Yeah, basically, we're doing a consistent snapshot and that means that every time that you make a right, you actually have to check whether a checkpoint is ongoing. So it's a sort of sort of similar to this. When you have a page space, then you basically do a consistent snapshot of pages, but here it's for rows instead. When you check to see is a checkpoint occurring. What is that writing thread do just not overwrite and make a copy or does it like wait, what does it do. And that's what it says here it basically writes the row ID into checkpoint key list. So that means that that when you commit the transaction, you actually save the old version of the row of the row. Okay, until you have time to actually write it to this. Okay, I want to be both immediate something similar. This makes sense. Yeah, yeah, I think this is. It's definitely not unique for one to be but it's it's a kind of problem that if you make the solution in a certain way you have to solve this problem. And I think probably skip this. So this is also an important point that if we have one terabyte of 300 bite rows that's about three billion rows. So if we need to check each of those rows that will take several seconds of CPU time just to do that. So we had to do a little bit of optimization for that. And so each page has a two layer two level change map. So that means that we're using 136 bits in the page header to optimize the checkpoints. So that means that if you only have one right in a page, it's sufficient to check those 136 bits and check one part out of 128 of the rows in that page. So this means that we more or less can skip 99% of the rows we can skip them when sometimes even more than that. So that this is important to sort of to make sure that the checkpointing itself doesn't consume all CPU power. And we could what it could be necessary to go even further and have change maps on megabytes and so forth. But this seemed to be sufficient for us. Okay, and let's see. I think that this is a good time to start seeing if there are questions that anybody else wants to dive into. Or have something specifically that they want to know a little bit more about. Are you putting something to talk or like what? Sorry. Is this the end of the other presentation of the slide? No, I mean I actually have. I can talk a little bit more about one thing more at least that's quick and so this one I think is kind of cool as well. We have something called adaptive CPU spinning. So again, this is adaptive so CPU spinning means more or less that you have a thread. You don't have anything to do. So what do you do?  And in the past we had a fixed static CPU spinning. So basically we said that let's spin for 100 microseconds independent of what happened. But what we did now is that we have adaptive CPU spinning. So we measure how much activity that we have. So this means that we have sort of a guesstimate on how long time it will take until we will be woken up again.  And then we have three different levels of how active you want to be. Do you want to be so active that you only say that you are more or less efficient? Or do you want to focus on low latency but at a reasonable level? Or do you want to be a database machine and more or less keep awake as much as possible to give the best latency whatsoever, even if it costs a bit? Another nice thing about spinning is that if you're in a hyperthreaded CPU, you actually use special machine or use special CPU instructions. And when those are executed, the other CPU in the same core will be much more efficient. So that means that CPU spinning in a hyperthreaded CPU almost comes for free. So that's another nice thing about it. I have also lots of slides on how we do networking. I think that that would take us pretty far into the night if I started using that. But maybe one thing I could say is about this. So when we are communicating inside a node, we are communicating to memory buffers. And so we implemented that in such a way that each input buffer has a single reader and a single writer concept. So that actually means that we don't have to do any mutexes in order to send a message from one thread to another thread. It does, however, require the use of memory barriers. So memory barriers is something that's required in order to make sure that the other side actually see the operations in the same order as we wrote them. And so there are two types of memory barriers. The right memory barriers that ensures that the receiving CPU sees all writes up to a certain point. And we have a read memory barrier that ensures that the receiving CPU sees all writes written by the sending CPU. And actually in order to have a proper thing here, you have to follow this protocol that you see here. So that you have to use a write memory barrier on the sending side and you have to use a read memory barrier on the receiving side. And you have to make sure that you update the head pointer before the memory barrier or after the memory barrier. I mean, and that you have to write the message before the memory barrier. So you have to be a bit careful when you're writing, but in this way you can actually communicate without locks. And actually you can even implement this using distributed memory. We use that in a previous version of NDB, but we nowadays that happens in the device driver instead, so we don't support it internally in NDB anymore. I guess. And take your time and wrap up. Yeah, I think that's, I mean, I could, I could talk for five, ten minutes more, but it's up to you if you want to. I have a two year old downstairs. I can't, sorry. I mean, I was, I'm already into the section that I didn't really think I would get into so. Okay. All right, do you have like a last slide? We're hires like that, or is it just ends? No, I don't really have, let's see if I had something at the end. There's CEOs on the call. You got, you know, you got to push it. Well, we definitely hired that in my last long. Got it. Okay. Yep. I'll clap my hands for everyone else. So we'll open up to the audience with time for one or two questions if everyone wants to support. Yep. So my question would be, we run to be, or sorry, well, any of you and now run to be, this was your PhD dissertation, right? From the nineties. You've been working on this for 26 years, which is rare, right? There's an older than single light, which is like, there's one guy doing single light. Monty has my single, but it's been a lot of people touching that, touching that code. It could have been your baby. So what's the vision? Like, like, what's the next 20, 20, 60 years look like for, for Ronnie for you? Like, what's like an ideal scenario? So I think that the, the feature store is obviously the first couple of years. And what that means is that we want to be able to also handle much more data on this. We want to be able to, to scale to more notes. We want to be able to scale down from notes. We want to be able to do even more operations online. Another thing that I'm hoping that this is something that is mostly work upon inside of Oracle. So, but I'm, I'm hoping that the Oracle team will deliver also some work on, on complex queries. Actually, when I was at Oracle, we did quite extensive work on Optima to also make sure that the rendezvous can work efficiently pushing down joints. And so at the moment we can do, we can do 12 way joins inside of the data notes, but we, we still haven't got support for aggregation. We haven't support for max and means and things like that. So, so that I'm hoping that it will come up and then obviously the next step would be to be able to do a little bit more of a hybrid architecture, where we also move into a sort of, we still want to be a sort of online database, but it would be nice to also be able to do some more analytical queries and more efficiently. So that might mean that we have to implement some columnar level optimizations. Compression is obviously something else. I have satisfied myself with thinking 10, 15 more years. I'm not thinking that I'm going to work more than 15 years at most more. So, I mean, I don't think 25 years ahead, but somebody else has to implement the last 10 years at least. Yeah, actually, that's one aspect of Brown to be reminded me that I didn't quite, I didn't realize that like, there is actually execution engine logic. Like it's not just getting fetch by getting delete get set and delete right the other you can there is an execution engine down there that you could do some joins. It's quite quite advanced thing actually. So it's called linked joins I think it's called so. So we have some queries can be quite efficient actually. But unfortunately, not all queries.


Good morning. Good afternoon. Good evening. Wherever you're hailing from welcome to another Stack Rocks community office hours. This is episode two. We're gonna be talking about ebpf 101 today And I'm joined by the one and only Robbie Cochran. How are you Robbie? I'm great Glad to be here. You want to introduce yourself to the audience just to let them know who you are what you do around these parts? Yeah, absolutely. So I am an engineer at Red Hat formerly was an engineer at Stack Rocks And I've been working in Kubernetes security for a few years now and in particular, I've taken a focus and interest in Kubernetes security and ebpf together and Yeah, cool. So we have A doc for some questions in it that I'm trying to pull up First question. What are the use cases for ebpf? So ebpf you can think about ebpf is a better way to extend the kernel And the use cases are pretty wide open Observability security Networking features Ultimately, you can think about, you know, I like to use the analogy and it's I know this has been used before from the perspective of web browser without JavaScript is kind of like a Linux kernel without ebpf and you if you if you thought if you think about all the things you can build with JavaScript in a web browser and The the way that Linux kernel development used to go is that you would have to if you wanted some new feature in the Linux kernel you'd have to Go talk to kernel maintainers or write your own kernel module and what ebpf provides is a way to get data from the kernel the Linux kernel as well as Take actions on events and that is a seem may seem like kind of a Not that big of a deal, but it allows for a whole nother layer of tools and techniques and things to be built on top of The kernel and another way to think about the use cases of ebpf are, you know I don't know your persona or you know, what is your role and you know, what problem you're trying to solve The the kind of the first way and I'm gonna give a demo in a few minutes That I think about it is in terms of just debugging So I want to know is this you know, like think about a breakpoint in your code I want to know what's happening when this function within the kernels being called or Or you know, how many times it's being called that sort of thing so performance counters things like that There have been subsystems built into the Linux kernel, but ultimately you run into, you know, well This feature isn't available in the subsystem like the perf subsystem the audit subsystem These things are not extensible and what ebpf allows you to do is attach small programs on these events and then report data back into User space so you can have a user space program that understands what's going on in addition you can actually write new networking stacks within the kernel in the ebpf and then if you need to update them or change them or configure them You're able to do that nice So, you know, if I were a new person Using ebpf inside my cluster like what would that look like from a pod perspective? Yeah, so there's there's several ways. So let's say we have a kates cluster up and running an openshift cluster and we want to Understand what's going on within the cluster. So we have a performance issue For example, and maybe it's related to disk activity. You could use ebpf scripts and tools and pieced together kind of An idea of what's going on by just extracting the data and getting performance metrics. So this is no observable from the observability standpoint Another use case is if you want to actually Use ebpf to help with security. So visibility Less less about getting sampling and aggregate information from the kernel but more I want to see an audit of all of these events and Have it correlated with additional pieces of data like the container ID or the network connection that a process made or something like that And so that's that's one way to get use ebpf in your cluster. I think from the individual perspective The debugging is kind of the debugging and performance analysis is the first entry point It you know, it actually requires just like building a large web application on JavaScript a lot of effort Building a platform that's built on ebpf is you know a significant effort And so there's a lot of existing platforms and tools and things that are that are available so for example the Stack Rocks product now ACS is uses ebpf to extract process Executions as well as network connections from the kernel and what's great about ebpf is it allows us to operate on any kernel version kind of going back to that challenge of You know any recent kernel version That challenge of you know We're not able to add stack rock specific features into the Linux kernel. We don't want the customer to have to have a special kernel We can just insert an ebpf program within the kernel and then get this data out That sounds way easier than running custom kernels everywhere Yeah, totally um, yeah So like I think of the ebpf and I think of like tracing Kind of the same vein right you said debugging right like there's nothing better than running the s trace on a process, right? Really seeing where it's breaking right does ebpf give us More insight less insight. I mean, what is that? What is it like on yeah, so s s trace if for folks aren't familiar s trace is kind of an old-school system call tracing approach and it uses It's kind of ping-ponged back in between user space and kernel space So you you can print every system call that an application is running and that's super useful for determining You know what actually is happening when I'm reading a file or you know and so The the problem is that s trace has a huge performance over it. Oh, yeah It's it's essentially, you know adding an extra round trip between user space and kernel space the Cool thing about ebpf is that you can you know, of course ebpf can introduce overhead because you're doing work You know some sort of additional compute within the kernel, but you can be selective about it You can say, you know, I want to record all the system calls and then report them out. That's a good thing So Yeah, you could easily build an s trace on top of ebpf And getting those system calls like very telling very helpful How would you install it right like how would you get ebpf in your cluster? so I will let me do a demo and Let me show you guys share my screen and Before I do that, I'm gonna go through just a little intro on some slides here Is that visible Chris, yes Cool so kind of as I was saying, you know We have the Linux kernel which is hard to update and then we have our application stack on the right That's just a set of processes and you know These could be running within containers and our cluster but ultimately the processes are sending information on the network reading and writing files or launching other processes and you know since the The UNIX API the post-ex API's were created. It's it's made a It's been a stable place to run programs on Linux. So you can run exec you can run open It doesn't matter which version of Linux you're on or you know, which flavor However, within the Linux kernel, there's all these different subsystems the memory manager process scheduler disk network If you want to get data about those internal components, it's difficult. So you can either write a kernel module or Actually submit a patch to the to the kernel maintainers and that takes a lot of effort So we have this stable API for our user space programs But we don't necessarily have a good way to extend the kernel about, you know, a lot of effort so what eBPF let's us do is add hooks within the Linux kernel and You know, whatever we can add the hooks at the syscall level So that's thinking about strace where we want to see all the interactions between a given process and the kernel via syscalls or we can attach eBPF Programs to any any function really within the Linux kernel. Of course The syscall layer is going to be more stable So I think the you know the easiest way to get up and running if you have a Kubernetes cluster And run some eBPF programs is to use a really nice tool called kubectl trace And that's what I was going to give you the demo of here. So this is a available here at this github URL and the prerequisites are really just you need a Kubernetes cluster up and running So let's see kind of share my terminal so I have a cluster a kates cluster running and I will show the No, it's so let's see we've got Make sure we can connect to the cluster there we go. So we got three nodes in this cluster. In fact, let's look at what what the host OS is So these are running Ubuntu 20.4 Linux kernel 5.4. So that's You know, you're normally when you have a kates cluster, you're not really thinking about the node level nodes are cattle they come and go and What eBPF allows us to do is Kind of get into the the Linux kernel on the host. So we're kind of getting underneath our containers and seeing what's happening Right now. This is a completely empty cluster. So there's no pods. Of course if I Check the cube system namespace which we'll see the We'll see that there are Here we go. So we've got the you know the base Containers that are actually running in Kubernetes. So even though there's no applications running in this cluster. There's still stuff going on So what we can do is use kubectl trace and kubectl trace is a kubectl plugin There is Essentially it allows us to still use a kubectl command, but then it calls out to another process And so we can run kubectl. We can install kubectl trace using crew which is a kubectl plugin manager or just downloading the binaries directly and kubectl trace allows us to run BPF programs and particularly a particular type of BPF program called a BPF trace program Which is a higher level scripting language that's built on top of BPF And as I said, this is I think the easiest way to get started with running EBPF in your in your case clusters. So what we can do is let's let's see what Processes are running on a particular node. So of course I could exec into the node and do look at you know There's a lot of different ways to get that information, but let's see how we would do that with EBPF So I'm going to get nodes again and then I'm going to run a BPF trace program Let's start with exec So what this is going to do is it's going to run a kates pod on this node G4x2 is the the particular node and it's going to run this execs new program So we start that it creates a pod and This pod has to be configured a certain way so we can actually look at it and let's Describe this pod actually Edit it so we can see all the different configurations So it's key about this is it's running a container that is actually Just to cuddle trace BPF trace and it the other kind of key thing here is that it's running In with admin privileges so the security context here is privileged and if we're going to execute BPF programs within a container we need to be privileged because The BPF system call Which is you know, we're monitoring system calls or observing system calls. We also use a system call to insert the the program It requires admin privileges So I've run this trace program now. I want to see what What what what's actually running So we can tail a the pod itself, which is actually going to be just printing the data it's getting from the BPF program and so this is just tracking the It's adding a BPF program to the exec system call Looking at the arguments and sending them back to user space And so from a single container we can see all of the process executions that are happening within on the host system within each container and on the host system itself And so this is actually kind of a cool way to see what's you know, what's going on what kind of activities happening on a on a Kate's cluster. There's a lot of just, you know, Version checks and sanity checks some health checks that sort of thing That's cool. Yeah, so let's so we ran this program called exec snoop. So let's look at that program And this is kind of when I when I was saying earlier about how you know one of the first places I think people you end up using BPF is to diagnose performance or Other kind of issues in their cluster. So, you know, you might want to see what's running in a pod or that sort of thing And this BPF program is very simple. I mean, here's the here's the code right here. It attaches a The hook that we're attaching to is the sys call for exact the and we are just Combining the arguments and then printing them out. So an exact system call has a path associated with it and we're also printing the the process ID Okay. These programs can be run using on any Linux system. I think all distributions have these tools available in their package repositories. This is I'm talking like, if you're if you're not on a Kubernetes customers on the Linux box, you'll install BCC tools or you'll install app get BCC tools. You'll actually be provided with a lot of these programs. So, yeah, so there are this is just exact. We could also look at networking need to meet this trace and look at you in hygiene your wonderful pristine cluster. I know right this cluster has Has never seen any any high CPU all the CPUs are just ready to go. So what I'm going to do here is Let's see what we wanted to Actually, sir, what do you want to see Chris? Well, what do you got. I mean, well, you get there's a directory of BPF trace programs. I'm curious what's in there. Yeah. So this is a this is the kind of default programs. And of course, all of these are, you know, I would say not it's not too difficult, not too much effort to extend them for what you need. And this the snoop programs are particularly interesting. And you can imagine there's a couple different models of getting data out of the the kernel. You can just stream all the events or you can also aggregate within the kernel and then So let's do just Let's do connections. So yeah, yeah, what's something that's actually going to give us a bunch of results. So So we'll see let's get the logs for that. Now this is probably going to be really active. There's going to be a lot of connections and this will show us the Just stream this into a Okay, so this is just a snapshot of connections that were being streamed Via this BPF program and we're getting the the PID the command of the program. So this is the command is going to be, you know, stored in the kernel and just reading that data on the source address the destination address and so we can see all of the Connections that are being made in which program is making those connections. Let's say we there's a, you know, curl that's probably a health check just checking for readiness. We've got kublitz talking to the API server, most likely and so on. Very handy. So, yeah, you talk about performance metrics right like And can you show us anything around that. Sure. Yeah, let's take a look at the performance of a trace. That'd be super meta. Yeah, you know, and I think it's that's worth mentioning. You know, what's the overhead of running eBPF anything based on eBPF within your infrastructure and Like, I think s trace is a nice example because I, you know, we were describing s trace for every system call you're adding an additional roundtrip because it has to go in and out of the kernel and So you want to make sure that whatever events you're attaching to aren't too frequent. And that the data you're extracting is reasonable. Of course, you know, you could you could attach to every function in the kernel and just completely, you know, doster system, or you can have, you know, kind of targeted Events that you want to get information about. And there are a lot of ways to Aggregate data so eBPF provides different techniques for for getting data out of the kernel one one approaches histograms and maps and so on. You know, I don't know if there's a good way to do that with These programs, oftentimes you kind of want to like, you know, build a flame graph or some kind of More visual view of this data. This is kind of the raw data coming out. But let's say Just for example, you might want to see like, I think what's really nice is looking at TCP Issues like drops or, you know, TCP lifecycle that can be very helpful for networking. Oh, yeah, debugging everything. Yeah. Yeah. So question came up in chat. Where did these programs come from? Yeah, that's a great question. So these are BCC programs. So let me go back to my slides and I'll kind of give you a better idea of The ecosystem of eBPF. Okay, cool. So let me actually go to switch down a few slides. Okay, so this is a, this is from the eBPF.io website. This is a nice, excellent resource for eBPF and it's Recently forming, you know, kind of an open source consortium of developers and companies and everyone that's interested in eBPF. And these, what I just showed you with kubectl trace is A BCC program. Those are BCC programs. So this is the logo for BCC. And these are available on, you know, get up so you can download these. And of course, there's our, there's a lot of Oops. So yeah, there's a, I was going to try to show you the website with all this. So yeah, BCC tools are what those scripts were and BPF trace programs. And then we were running those using the kubectl trace. It's a lot of BPF and trace and tools and that sort of thing. You know, I think that it can be overwhelming when you're looking at all the different ways that BPF can interact with all these different kind of subsystems. The key thing is that, you know, the use cases of networking security and observability and tracing are kind of where you want to start with. I would say BCC tools are ideal for observability and tracing. And these are historically based on the detrace approach. Nice. Solaris. Yeah. So is Iovizer BCC the right repeater to get these from? Yeah. Yeah. Cool. I'll drop that chat folks. Yeah. Yeah. The BCC compiler collection or BPF compiler collection is what these are toolkits and wrappers for writing eBPF scripts. You still need to write in kind of a pseudo C and then the BPF trace programs which I showed you guys are built on top of that. Cool. Cube Cuddle Trace, this was from chat. Cube Cuddle Trace makes things easier. How do we use it to generate flying graphs? So earlier. Yeah, I think that the best way probably the one of the most full featured toolkits for generating data like data, like, that is the vector subsystem or vector system from a performance monitoring system that's Netflix has open sourced. And also, you know, these, you can generate flame graphs in different ways. You can just have, you know, text histograms, but I think the vector. Performance management system is ideal for flame graphs. Okay. Cool. For folks. Drop them a link. I'm going to see if. I see if web pages right now. So what has been done to harden the eBPF subsystem, right? Like, how do I know this isn't. A back door. Right. Like, like, is this something I need to watch? Right. Like are people making the. Yeah. Yeah. So, um, you know, the.  So you know, the. Kind of going back to the, you know, what are the alternatives to running, you know, this, this container that is inserting these bpf programs is running and is privileged. So, um, of course, you want to make sure that your user space code that's managing and running your edbf programs is, is secure and managed properly. The, uh, um, Verifier. you know, there's kind of two angles to that question. So one is, what's this, you know, is this gonna crash my kernel? Is this gonna crash my host? And then the other angles, you know, is this, is this secure? And I would say that, you know, there have been some, there haven't been an EVPF attack, but there has been some data leaks and things that enabled with EVPF. But really the, any user of EVPF has super user privileges. And so that's something to watch out for. Now, of course, the verifier essentially, that your program must pass is built into the Linux kernel. And it will, the verifier will ensure that the code terminates and isn't gonna get into any infinite loops. In fact, unbounded loops aren't allowed in BPF programs. The BPF instruction set is actually quite, quite constrained in what you can do. And that actually, that helps a lot with security. Null dereferences, a lot of the kind of, you know, caveats that come with writing in C that you would have if you were writing a kernel module are not there with BPF because it's got more safety features and it's a more constrained language. Okay. So very tightly scoped, that's good. Very, you know, sanitized for lack of a better term. Yeah. And I would say that, you know, if you're using an BPF, the, you know, it's essentially a sandbox within the kernel. And so you get the security guarantees of that. Nice. Awesome. So what about, are there any downsides to the BPF, right? Like we talked about overhead a little bit. What other downsides are there potentially? You know, I think that there really are no downsides because it's, and, you know, of course that's a very blanket statement. But, you know, maybe what are the, you know, what are the issues that it can bring? I guess it's maybe a different way to think about it. And so the, I think one challenge is making sure that you're running on a kernel that has, you know you really need to be on a more recent kernel. There's been a lot of effort and changes that have been added to the links from recently. And so if you're running on in the cloud you're going to be using a more recent kernel. But if you're, you know if you're in an air gapped environment or running older distribution you're not going to have all the most recent BPF features. RHEL, Red Hat has backported a lot of BPF to older versions of RHEL. And so usually I think if you're on RHEL you're going to have all the features you need. But making sure that your host can support EVPF is key. The other challenge is, so I mentioned that we're essentially extending the Linux kernel. And in order to do that you need to know what is the layout of a particular struct that I want to get data from. And across Linux kernel versions that those struct layouts will change. And so in the kubectl example demo I just showed the what actually is happening is we need the Linux kernel headers in order to run the BPF program. So when we compiled that script into BPF bytecode before it's inserted into the kernel the headers are needed. And that can be very challenging because especially if you're trying to support you're developing software that you want to run on a variety of different Linux kernels. And coming from someone who's worked on a security product that you want to run on a lot of Linux kernels that's a challenge. And that's something that the community the BPF community is very well aware of and they've been adding features to export the struct formats and the data types within the kernel so that you can just subscribe to that. It's called BTF or BPF typed format. And there's another initiative called that uses BPF called core or compile once run everywhere. And that's kind of the ideal in-state for these tools is that you're able to write a BPF program and run it anywhere. But right now that can be challenging because only the most like recent kernels support that. Depending on if you're on a cloud if you're in the cloud it's gonna be supported but otherwise maybe not. That's just something I'll have to look into, yeah. You know, the other challenge I would say with BPF is it's, you know, still relatively new and you're running a code in the kernel. And even though this is like a great tool to get information out of the kernel and you said something earlier about profiling the meta of BPF that can be difficult. You know, anytime you're adding overhead within the kernel if your application's slowing down maybe that can be difficult to diagnose. There are tools and techniques being kind of developed for performance counters and measuring.  For that example, I was only attaching to the exec system call or the TCP connect system call. But if you've got a lot of places that you're attaching you're gonna wanna profile these and see make sure that especially in production if you're running a BPF in production. Nice, good call. Is there anything else you wanna just show off with the BPF, is there anything you think people's minds would just be blown by it? Yeah, let me, let me go back to this slide here. And I think that, so we have this bottom layer of the Linux kernel. And you know, this is the, I would consider this slide from the kind of the roadmap of the BPF. There's, you know, BPF has been added to Linux kernel. Microsoft has been adding an BPF layer to Windows. Yep. There are several SDKs for writing BPF. Because not only do you have to write BPF code you also need the framework to help you inject the program and kind of manage it and get the data out of the kernel. And so there's Python and Go and C++ and Rust libraries for doing so. And then on top of those are these tools and platforms that are being built. And I think that's probably the most interesting thing going on with BPF is, you know, it's just like it's good to know about assembly code but no one's actually gonna be writing assembly code usually. And so it's really cool to see what people are building on top of the BPF from the perspective of security. The coolest thing is to be able to have a way to not only get data from the kernel but actually take action within the BPF. And so that's where I think when you start writing when people start writing and this is already begun writing BPF programs that actually can take action within the kernel then there's a lot of really interesting tools and platforms that can be built. So from the perspective of security there is a new Linux security module based on BPF called, I'm not sure if I get this right but kernel security, it's KSRI. Oh, okay. Kernel or maybe I got the SNR slot. Kernel Runtime Security Instrumentation. There you go. And what that allows you to do is actually have checks. So, you know, is this process authorized to take this action? And you can just deny it. And so those types of rules can be updated and tied into the higher application or authorization frameworks. And what's great about that is it's, you know anytime you're taking action within the kernel it's performant because it's not requiring you to go back and forth between user space and kernel space. The other place that's really interesting is just observability. So there's a lot of observability frameworks Pixie is one observability framework. And I think in general if you're doing any kind of application tracing or monitoring, it's gonna ultimately be based on EBBF. And I haven't really specified this or kind of gone into detail about it but not only are you able to trace events in kernel space, you can also do that in user space. So you can attach to essentially use EBBF to check break points in user space or to look at events in user space. And basically you can imagine that just opens up a whole world of observability and just flame graphs are kind of like the beginning of all the different ways you can like dive down into what's happening. So security and actually doing enforcement that's one area that's I think super cool. The observability frameworks that are being built on top of EBBF. And then finally service meshes and networking. So Cilium and Cilium is a Kate's CNI plugin. So it's an EBBF based networking plugin for Kubernetes. And I'm not an expert on Cilium but my kind of my understanding of what the benefits it provides is again, you're able to not have to use IP tables to manage the connections between deployments or even between clusters. This can all be configured within EBBF. And there's a component of the Linux kernel called XDP, the Express Data Path. And that is used often by hardware manufacturers to tie as a way to integrate their NICs with the Linux kernel. But then also you can have essentially a software layer built in there. And so that's what Cilium is. A kind of software EBBF based networking plugin. K-Tran or K-A-T-R-A-N is a load bouncer based on EBBF. And even Calico has added EBBF features. So I think moving forward, if you're running a service mesh, it's also gonna kind of probably be built on EBBF. Yeah, it'll have some EBBF components to it, it sounds like for sure. So quasi-question slash you might not know the answer to, but wouldn't EBBF effectively introduce a stable kernel interface and permit proprietary programs for the kernel? Yes, so like, yeah, I'm sure that's a contentious thing. So... So I'm actually not sure I completely understand what the questioner is asking in terms of where they think, where they're going with that. Of course, if you are writing a BPF program and it's inserted into the kernel, there actually is a GPL license check in the verifier.  So, I think that's an interesting idea. I'm thinking from the perspective of hardware, like a device driver or something like that. I would say that this is already feasible with kernel modules and other things. The kernel is already extensible. This is just an easier and safer way to do it. I think safer is kind of the, right, like... Yeah. Easiness aside, this is just safer to do than like building kernel modules left and right. That's right, yeah. Yeah, and to be honest with you, I don't know where eighth doctor was going with the question, but if that didn't answer your question, eighth doctor, please let us know. Khaled Raid says, the most interesting thing to me is how you can control network traffic on layer seven with EVPF. That is super awesome. The fact there's a load balancer that uses EVPF find the scenes. I don't know how you could extract any more performance out of the kernel than that. I mean, yeah, I've never used Patron before, but it's gotta be fast and performant in all kinds of different ways. Yeah, absolutely. I mean, that is, you know, essentially you're giving these kind of core data paths context. And you know, user space can provide the context through EVPF, and then you're able to build these performant networking tools.  Taking action on things, you know, I haven't seen anyone do like an IDS and EVPF, but those types of things are... That would be cool. Yeah. Yeah. You know, I think, and maybe going back to the challenges, you are limited to how complex these programs can be. I'm not sure off the top of my head, how many, there's an instruction limit, how many instructions can be in the program.  So, you can't build, you know, you can't have a machine learning engine running in your kernel and using EVPF. Right. So, you are limited to also the just, the limitations of the EVPF verifier, as I was saying earlier about, you know, bounded loops and what memory you can access and that sort of thing. Yeah, so. Yeah. Detective Conan Kudo, he basically, the issue here is that proprietary programs for the kernel that are tied to specific kernel versions bring us back to the problem that prevent people from upgrading kernels in the first place. I don't think EVPF does that here. You have that instruction set limit, it doesn't live in the kernel, it calls the kernel, right? Like so, it's one of those things where it's like, if people are concerned about this causing you to like stick to a certain kernel version, I don't think that's the case. This is going to be persistent and kernels going forward. You're able to run the same, you know, traces as before. Right, yeah, I mean it's important to, we're not hot patching the kernel. We're kind of, it's event driven hooks that we're taking action when that event happens. And so that does have certain kind of limitations. You can't necessarily, you know, of like change code paths or that sort of thing. You're adding on additional, you're extracting data or, now for example, if you do the thing, so I described from the perspective of attaching to system calls, but psyllium, which is attaching to the XDP subsystem, you're extending that. So rather than going back and forth between user space, you're going between your EBBF program and that subsystem within the kernel, if that makes sense. And the same with Linux security modules, you know, you're not necessarily, you need to have some code within the kernel that allows you to then do those actions. And so I think it does from, you know, from the perspective of Linux kernel features being added, knowing that you can build a subsystem with the idea of users using EBBF to then make it what they need to be. You know, it's like to extend it the way they would like. You know, the Linux kernel, if you're previously without EBBF, you had to be very opinionated on what things were, how things were gonna operate. But now it can have a, it can have the framework and then users can write their security rules or audit rules, for example, for the kernel security instrumentation, runtime instrumentation, and, you know, use them as they see fit. Yeah. Cool. So we've got about 15 minutes left. Is there anything that we haven't touched on that you want to? We talked about the cons. We talked about, oh, what has been done to harden the EBBF subsystem? We touched on that a little bit. Yeah. So, you know, I- Instruction limits and stuff like that. Instruction call limits, the, you know, I think from the, from the beginning, you know, EBBF is one of those acronyms that doesn't really have any meaning anymore because it kind of, we're talking about the whole EBBF subsystem and all these platforms and things when we say EBBF, but ultimately it's extended Berkeley packet filters. And so Berkeley packet filters and IP tables have been around for a while. And so, you know, it's, I think it's, it is secured through the simplicity in that, in that you can, you need to have a constrained programming language or instruction sets. And once you have that, then it allows for the verifier and static analysis to be much more effective because we're not exploring, you know, a million different paths. And all the different paths in your EBBF program are essentially walked to look for null dereferences to make sure that you're, you know, it's, there's no way that you're gonna dereference a null pointer. And also the kernel provides a lot of EBBF helper functions and that's usually what's used to get data. So you're using the helper functions to then interact with the rest of the kernel within EBBF. That's cool. Okay. Yeah. Awesome. Folks have already asked about Cilium. I've dropped the link to the Cilium page for that. So I would definitely want to mention that the EBBF.io organization just, actually, I think it just finished like an hour ago. Had a summit yesterday and today is the second summit they had it last year for the first time. And for people that want to know more particularly about, you know, the lower level details of EBBF, but also just different use cases. Definitely check out the EBBF summit. It's free. All the videos are online. That sort of thing. There's a lot more information there. And it's... Find a link. Go ahead. It's, yeah. And, you know, there's an active community and kind of like I said before, that, you know, depending on what persona or what role you have in your organization, EBBF may, your interaction with EBBF may be either debugging or you're just using a platform that happens to be built on EBBF. Right. Yeah, that's a good point. The, I'm looking at some of this schedule. It's intense. But, you know, typical suspects, Liz Rice, Yana Dugan, all the fun people to have at the sale. Sable. Yeah. All the low level folks I know that know everything about SysCalls are speaking at EBBF summit. So that's cool. So we also have, we're a diamond sponsor for the Cloud Native Security Conference coming up at CubeCon. Are you part of that or? I'm not playing in any way. I'm not playing on presenting, but yeah. Yeah. I'm sure I will be involved one way or another. Yeah. There's just to read a little bit. There's a hosted capture the flag learning experience that will run in parallel to the event for attendees to learn firsthand practical attack defenses and techniques. Yeah, I think those capture the flags and those hands-on labs and demos are such awesome ways to learn. Yeah, the capture flag demo that was done during QCon EU was just fascinating to me. And like the second we found the problem because we were running out of time and I say we because it felt like we were all working on it together in the live stream. It was just like, oh, there's the problem. That's the naughty bits right there. Let's get rid of those and off we went. But yeah, it was a nice. Yeah, you have a lot of those aha moments. Yeah, like you can really get an eye-opening kind of mind-blowing experience from some of those sometimes. Because yeah, you can easily stick something on a Lexbox and make it do bad things. Yeah, and I think there's something really exciting and cool to be able to get information from the Linux kernel and get into these low-level details. You really kind of opens up your eyes into like what's going on. So yeah, those are great learning tools. Absolutely. Anything else for the audience? Do you think they should know? So I can share some resources. Yeah, that'd be great. So, you know, I've kind of... Slide her up, folks. I'll give you a link. Yeah, no need to get your pin and paper out for this one. I kind of broke down some of these resources based on, you know, if you're going to want to do EVPF development. I think EVPF development is going to be, you know, most likely associated with some sort of platform, right? You're working using EVPF within Cilium or that sort of thing. But for scripting and debugging, and that sort of thing, the KubeCuttle Trace is a great place to start. And then there are books. Brendan Gregg is the resident expert at Netflix on EVPF. He's got a great book on performance and observability with EVPF. And yeah, there is a lot of platforms. Red Hat Advanced Cluster Security for Kubernetes. We use EVPF, Falco, Aqua has some EVPF tools, Cilium and so on. Yeah, now there's a wealth of information available online. And like I said before, I think it's it can be really overwhelming when you're getting started, like looking at these things and, you know, just to start with something simple, like BPF Trace. And I think that's a good, good starting point. Yeah, looking at questions here. Are there any chances Red Hat will become a member of the EVPF Foundation? I do not know, I have no idea. Have you seen any discussion on additional stack unwind support for EVPF? No, I haven't.  I know that Pixie has done some work on performance, you know, kind of continual performance monitoring with EVPF. And I believe they've opened, I don't think they've yet open sourced there. It's they have a community edition, but they have some interesting blogs about performance or profiling with EVPF. I can share that after the talk, yeah. Yeah, that'd be great. Yeah, that's so, you know, Conan Kudo says, I hope they don't do too much more for EVPF if the goal is to not be turned complete, then it needs to be fairly restrictive. I mean, do you know of anything on the horizon for EVPF that would give folks reason for concern? So no, I mean, I think the language is going to stay relatively stable. The instruction said, I mean, there are, I think there's been recent work to add atomics and also additional like Linux features to get data out of the kernel. And so there's several different ways to, you know, you can insert a BPF program just using the BPF system call, but then to get data out, the program can write to a Perf buffer or also there's BPF ring buffers which have more performant characteristics. And so I think really the language, the instruction set will stay relatively stable, but then it's all of these kind of tools and other ways to interact with it that are going to change. That makes sense. Yeah, and grow. Awesome. Well, cool. I don't have anything else for us unless you do or the audience might toss in something here at the last second, who knows. But this has been awesome. This has been very insightful. I'm glad you came on today. Thank you so much. Thank you to the audience for your questions and participation. This was awesome. What's next? I mean, you know, where do we go from here?  I dropped the projects link and I noticed that like some of those are newer projects that have come along in the past few months. So I expect to see a lot going on in this space as more people become aware of it, right? Like having the second to VPF summit. Yeah. That's a big deal. And it just really fits well within like, you know, the microservices architecture and like where you have, we have all these ephemeral containers and clusters in our environments and you need a kind of a way to slice through, you know, not horizontally, but vertically. What's the user space program doing? What's the kernel doing? And UBPF is kind of like the best way to do that right now and so the, I think, like I said, there's gonna be more and more things built on in VPF. Awesome. Well, Robbie, it was great having you on. Thank you. Yeah, thanks for having me, Chris. Hey, no problem. And we do these, how, I forget how often we do these. Hang on a second, let me double-check. Please, like, obviously please come back for the next one, but the next one is, I forget. I believe these are monthly. So confirm, not, that is incorrect. But so here's a better idea. Subscribe to our calendar and you will get all the fun links to all the programs that are coming up on the show or the channel. And yeah, highly encourage you to do that so you can keep up to date with all the latest and greatest cloud native fun things we're doing here at Red Hat. Yeah, so Robbie, without further ado, thank you very much and y'all stay safe out there. See you soon.