On today's Visual Studio Toolbox, Phil shows us how to use computed columns in Entity Framework Core. Hi, welcome to Visual Studio Toolbox. I'm your host, Robert Green, and joining me is Phil Jepixi. What's up, Phil? Hey, I'm doing great. Get ready to talk about some new cool EF. Well, not new, but some cool EF core stuff. I know. We're continuing our series here. What's up today? Today, we are going to talk about computed columns in SQL Server and how to use them within the EF core world. Okay. Is that difficult? It's not difficult. There is a trick to it if you want to do it through migration. There's two things. First of all, in the 1.0, 1.1 versions, which is a long time ago, they're out of support. Computed columns based on formulas, GDFs were not supported. They didn't work very well. They've worked fine since 2.0, so they still work great. The problem is that people will not always want to use them, or they think there's this imaginary wall between EF core and the database. I'm either going to use EF core or I'm doing everything in store procedures and straight ADO. It's probably more of a training thing than a technical problem of just getting people to say, hey, use the right tool for the right job. For example, let's say we have an e-commerce site. We've got an orders table and order details table. If I want to get the order total to display in a SQL Server query, I can certainly do that by doing select star from orders and then do a sub-select on order details where I'm doing a sum and putting it back. But that also makes it really hard to query. What if I want to get all orders that are greater than $50? Now I've overcomplicated my query. The one watch out I will say is that you have to be careful with using functions in SQL Server as computer columns. They can become a performance problem. It's always the right tool for the right job and as I'll always test, test, test to make sure it works. Let me share my screen and we'll jump into this. What I want to do, the final result is I want to create a function, a scalar function, and it's going to be a really simple function. We'll just create to a new query edit window. Here I have my function, get order total, pass in an order ID, returns money, and then it's going to select sum from order details where order ID equals order ID return result. Pretty simple. Here is where the, I won't say tricky part, but let me keep SQL Server over here. The gymnastics comes in. I'm building up my database using code first and migrations. I don't have any tables in there. Then I want to build my order details table, and I want to create my function to get to sum. Well, the limitations are going to SQL Server side. If I try and create this function before my tables are in place, it won't work because the tables don't exist. If I try and create the computer column before the function exists, well, that also doesn't work either. The fields have to be in place. Then tables and fields, then I create my function, and then I assign it to the column. It's really a three-step process. If you create a column and want to change it to a computer column, you can't just change it. You have to drop the column and re-add it. If a function is referenced by a computer column, you can't change it, SQL Server prevents you. It's not a complex problem, it's just a timing thing. It's a three-step process. Let's walk through that. Here I have my model, I've got my order and my order details. Order details is fine. I can build this up just as we've talked about all along. This I can do, I'm going to show you how we configure that without having to drop and create, because this is just an inline computed column. Let's look at this, this is inline. We go into our store context, we go down to our order detail entity, and we just say has computed column SQL, quantity times unit cost. I can build this as I'm building the table because there aren't any external dependencies. But to create my first migration, I actually have to comment out this part, and go into my order table, and comment out this, and then run my migration. When I run this migration, it creates the order table without the order total column, it creates the order details table. Now I've executed this, but now I need to get my function into the database. What I do is I then create a new migration that's empty, so I don't make any changes. I've applied this initial migration, which creates those tables. We talked about migrations pretty in-depth earlier in the earlier episode. Now I created migration that initially defaults to empty because there haven't been any table changes. A convention that I've put in place with my team is, we always have the letter SQL in the name somehow. We know that this is a hand-rolled migration as opposed to an auto-generated migration. In the up method, I create my function. In the down method, I drop it. If I want to roll back to initial, and have this be applied, then it acts like any other migration with an up and a down. The only difference is I created it instead of entity framework core creating it. Now I execute this migration, and I have my function in SQL Server. I go back into my order or add it in either way. A lot of times, I will put on these fields the attributes for computed, but we don't need to because it's actually handled in here. Here is my property order total, has column type money, has computed column SQL, stored I get over a total. Like I said, it's not a complex thing, it's just a timing thing. I have to have the tables in place so I can make the function. I can't have the field in place that will get the function until the function is ready. When you're using a computed column like this, it's a three-step process. It's a three-migration process. Now let's say you're going the other direction. You have the database and you scaffolded the context based on the database. You're not using code first and migrations. Now you go back to the database and add that computed column. So you're going to have to rescapable it. It will pick up. No, it's fine. It'll pick up this computed. Whatever the computed definition is, and if we look at the actual column definition. So we go here, columns. I think we have to go into design mode, we do. So order total, it's computed and there's a computed. It'll pick that up. So if I scaffold the database, I'm doing database first, and scaffold it'll be there. I don't have to worry about creating a migration for the UDF because UDF's already in the database. Now if you want to, if you rescaffold entirely, then you lose any changes you made such as changing collection names to plural, which we looked at in our very first episode. Can you just add the context code for that computed column to the context class itself? I haven't tried. I don't know. I don't know the answer to that. I'm going to say yes, based on all my experience working with EF Core. Because if you're in a scaffolding situation, then your context and your models, your entities have to match the database. If the database changes, you don't have to rescaffold. You can just add those changes as long as they match. So I guess you could scaffold from scratch in a different location and then just compare the difference and add the missing code. Yep, you can. Presumably. So there's another nice little benefit that comes out of UDF's. That's the fact that I can map a C-sharp function to a scalar function in SQL Server. So it's got to be on the store context. It's got to be static. What's interesting is it never gets called. I have a throw new exception here just to show that it never gets called from within C-sharp. But we then map it to our schema and function name. They happen to match names here. But the benefit of doing this is now I can use that function in my link statement, and it happens server side. So the link translation engine in the SQL Server provider will actually say, select all from orders where store.getOrderTotal of ID is greater than the amount. So this happens server side. So this looks like a C-sharp function, but it's not. I'm cool. So I can build and this is more often where we use UDFs today. I was super excited with the computed column, but honestly, the number of computed columns we write are pretty small. But this is something we're using more and more because you'll have a series of, especially if you're thinking, my gosh, how do I get this link query to work? Well, if you've got some really complex link query, create a user defined function, put it in there, map your C-sharp function to it. You can use it in the link. You're not doing the context translation back and forth between client and server like you would in 2.0 and 2.1, because that's fixed in 3.0 and 3.1. It's still all server side. Cool. So that's what I got. I mean, we can run this and we'll just do what we expected. We'll just run it. Sure. Prove that it works. Well, it was your Timex commercial. It worked to rehearsal. The elephant stepping on the watch. Timex. I guess folks can bing that, right? Yeah. There's a commercial where they had the elephant step on the watch, and the guy says, it's so tough, even the elephant can step on it. Then he goes, it worked to rehearsal. I guess it was effective at advertising because I remembered Timex from it. Yeah. So order total is 118. If we look at, let me scroll down here so we see what the code is doing. Do it like this. Right. So it just creates some orders and then it's going to use eager fetching to get the data in, and then it writes out the total for the order. There's the order total column right there, and then each line item detail. Then the final thing that it does is, says give me all of the orders where the order total is greater than 50. There's only one order in a database because it's just silly little data. But then it pulls it up. So that's using the UDF to refine the where clause. Right. Very cool. All right. Another little hidden gem. Excellent. All right. Thanks. Hope you guys enjoyed that, and we will see you next time on Visual Studio Toolbox. See you soon.


Hello, I'm Dr. Travis Hoffman, North Dakota State University and the University of Minnesota Extension Chief Specialist. We're providing a video series sponsored in part by the American Lamb Board and premier one supplies. With this, we're going to talk about American lamb quality. We have behind me a representation of the lamb produced at North Dakota State University. One of the things that we work on is providing a consistent product for our consumers here at our university. And in fact, as we discuss the American sheep industry, we realize that there's a diversity of production management systems that range from our western ranges to our Midwest or Eastern farm flocks. With those, a lot of production decisions impact what we produce for American lamb quality. Consequently, there's differences in value as well and some differences in terms of consumer expectations. We'll talk a little bit about American lamb quality and how you can help us and help our consumers to feed their adventurous side.


Amis Panjuguna The sales manager at Amco Properties Limited So we welcome you to Precious Gardens Phase 2 As you can see, it's a very beautiful place Ready for residential, you can do farming So we have all the social amenities on site As you can see electricity, water, neighbors on site So we welcome you for free site visit teaching every single day From 8 a.m. to 5 p.m. The plots are going for $89,000 So we welcome you to Precious Gardens Phase 2 We welcome you to Precious Gardens Phase 3 07-01-293-199 Hello guys, my name is Kahehu from Amco Properties I'd love to invite you guys for a free site visit We sell prime property, not just property In Kikuyu We sell land for speculation, residential Commercial If you are looking for a place to build your home This is the place to be We just don't sell property We sell prime property The views are to die for The ambience, there is water, electricity All on site All you have to do is just call us on 07-24-960330 You can also check our website www.amcoproperties.co.ke And you will get to see everything that we are selling Thank you so much for being part of us And we appreciate SPM bars for joining us As we sell Kenyans prime property in Kikuyu Thank you so much Don't just ask, ask Kahehu What's up everybody Welcome to Precious Gardens Phase 2 In Kikuyu Guys this is amazing 899,000 Kenyan shillings You are a land owner And what else do we have Liz? We do have daily site visits So this is phase 2 Phase 1 If you want to come for a site visit That daily site visit So make sure you contact Amco properties And come for a site visit Such a great deal And guys the proximity is just amazing It's 35 kilometers from Nairobi Imagine only So all you have to do is call Amco properties today And book yourself a site visit


Thank you very much. And thank you everyone for coming to this session. Hopefully this is going to be quite interactive. I've got some like tutorial labs that are running online. You can do them on your laptop. You can probably even do them on your phone, I'm told. So hopefully you'll be able to actually get hands on this afternoon, which will be excellent. I work for a company called Isovalent, which is the company that originally created the Cilium project. Does anybody here a Cilium user or familiar with Cilium? Yeah, a few of you. So, Cilium is a networking project that uses EBPF very heavily, and we have a ton of people that I'm really fortunate to work with who are extremely experts in EBPF and actually develop it within the kernel. In fact, some of my colleagues are somewhere upstairs having a kind of private discussion about the next things that are coming in EBPF. So, I'm in this really great position where I get to learn a lot about EBPF from my colleagues, and I hope to share some of that with you today. I wrote a book, this book. I've got a few copies of it here. So, people who ask the best questions, basically, I'm going to give you copies. So, that's your motivation to ask really good questions, all right? But you can also download it for the price of your contact details from isovalent.com slash learning EBPF. The interactive labs that I just mentioned, they're all basically taken from the examples from this book. So, you'll be able to kind of use that lab. If you want to, after this talk, you'll be able to sort of go away, use those labs, build on them, and do a ton more with the examples from the book. We're not going to cover everything in the book in this afternoon's session in an hour and a half. I think that would be a bit ambitious, but hopefully it will be enough to give you a sense of what EBPF is to get a start on EBPF programming and to get a feel for why it's so powerful. Now, those interactive labs take a couple of minutes to warm up. So, if you want to get your phone out or your laptop, go to iso go.to slash EBPF dash Vancouver and just kind of click on the kind of get started button. And as I say, it just takes a couple of minutes to spin up the instance. And while those instances are spinning up, we'll just have a little bit of a discussion about what EBPF is and why it's so exciting. The way I've tried to structure this afternoon's thing is I think half a dozen different interactive labs. I'm going to talk a bit, then give you a bit of a time to just sort of work through those labs and talk a bit more, you know, take questions as we go along and so on. I will put that link up again shortly. Hopefully, it's pretty easy to remember iso go.to slash EBPF dash Vancouver. All right. Is anybody having any trouble following that link or is it? Is it all? Somebody's giving me a thumbs up. OK, awesome. So, what is EBPF? It stands for Extended Berkeley Packet Filter. And you can pretty much put that out of your mind now because that acronym is pretty meaningless. It does so much more than packet filtering. Just don't even worry about it. So, what EBPF is, is an ability to run custom programmes within the kernel. We can change the behaviour of the kernel by writing EBPF programmes. I should quickly just get a show of hands. Has anybody here actually written any EBPF code before? OK, yeah, got a couple of experts I'll be calling on you for help when it all goes wrong. No, it won't go wrong. OK. Why is dynamically changing the kernel a new thing? Why is this interesting? So, the kernel is involved in everything that we do that's interesting on a computer. Every time we do anything that involves hardware, whether it's writing to a file or reading from the network, or allocating memory, all of these things require assistance from the kernel. And the kernel, it's under constant development. The kernel is constantly evolving. But if you want to change the kernel, first of all, you've got to convince the whole kernel community that your change is a good idea. And then you've got to make the change, which, you know, there's millions of lines of code in the kernel. It is not a simple thing to make a change to the kernel. You've got to convince everybody that your change is a good idea. And even if you get a patch accepted into the kernel, it will be literally years before people are using that change in production. Because people don't run the latest and greatest kernel, mostly in their production environments. They're usually running a kernel that's three, four, five years old. So if you want to change the behavior of the kernel, you've either got to wait for a really long time before that's something that you and your colleagues can run in production, or you can run a very bleeding edge kernel, or you can compile your own version of the kernel. But if you want to distribute that change to other people, that's pretty hard work and you're in it for the long haul. With EBPF, we can just write a program, load it into the program. We don't even need to reboot the machine, and we can change the way that the kernel is behaving. So that is fundamentally why this is a game-changing technology, because the kernel is involved in everything. So usually when we write application code, we're writing in user space. And our application is making system calls to ask the kernel for assistance. As I mentioned, anything to do with hardware, it's going to require the kernel to do something. Most application developers don't really think about system calls, that it's all abstracted away. You usually have some kind of open or something for opening a file or opening a socket or writing something to the screen. There's probably some kind of print command. But all of those things underneath at a lower level are making system calls. Within the kernel, we can attach EBPF programs to pretty much any event. We can actually attach to events in user space as well, but mostly I'm really interested in what we can do by attaching them to the kernel. Anything, literally any function call, any event, any perf event, any network packet, network packets arriving at various points in the stack, can be used as events that we can attach to an EBPF program. So whenever that event happens, and we'll see some examples of this very soon, it will run our EBPF program in the kernel. And yeah, there are lots of different types of event that we can attach our programs to. And that slightly influences what kind of things we can do with those programs. So K probes and U probes are essentially functions within kernel or user space. Trace points, there are trace points in the kernel. You can also attach to trace points in your application. You can attach to network packets, as I say, arriving at different points in the stack. You can attach to the Linux security modules. Anybody used things like APARMA or SE Linux? Anybody enjoyed using APARMA or SE Linux? OK. Those security modules use this Linux security module API, aren't they? And we can use that to trigger EBPF programs. Perfect events that we might use for measuring, well, performance really, all sorts of things. Anybody here using Kubernetes? Yeah, quite a lot of people. So I just want to quickly talk about how using EBPF is really important and influential in the world of Kubernetes. So in Kubernetes, we run our application code in containers, and those containers run inside pods. But however many pods we're running on a virtual machine or on a bare metal machine, there is only one kernel. The containers all share the same kernel. And they're all using that same kernel whenever they want to do anything interesting, whether it's networking or accessing files, or if Kubernetes wants to create a new container, that's going to request assistance from the kernel. If anything wants to change privileges, that requires assistance from the kernel. So the kernel is involved in everything that all of our application pods are doing. And that means if we can instrument the kernel, we can use that to see what's happening in all of our pods. We can use that to potentially change the way that the kernel is behaving in relation to those pods. Maybe we're going to prevent some things from happening for a security reason, or we're going to influence the way that network packets are delivered around the system. So I think that's enough talk and time for you to do some action here. Hello world in eBPF. So in that online tutorial, you should find some instructions on one side that will show you what it is you need to do, and some terminals that you can use to actually run those commands. Basically going to just give you a little bit of time to go through that hello world. Any questions at this point? Okay, I see a few people. Okay, the question is, is it a shared environment? So we're using a thing called instruct, which gives us all these different sandboxes. So you've all got individual sandboxes. So hopefully you will not be able to interfere with each other's instances, but it's quite nice. I mean they are using containers to provide this kind of little sandbox. And if we have questions, just wait until I give you the microphone so that people at home can hear it and for the recording, please, thank you. Yeah, we're using a platform called instruct, which is really great for selling up this kind of thing. All of the labs that we're going to use today, they're all going to have the same kind of file structure. The examples from the whole book are there in all of the labs.  I see a question here. Hi, so do you have to be an expert in kernel events to write these? That is a really great question. Yeah, so everything that we'll do today is pretty basic. At some point, actually if you look at that hello world function, you get past a context parameter, and that context depends on what kind of event has triggered the program, what kind of program type it is. And at some point you might want to interact with that context. You might want to interact with the network packet, for example, and you need to know the structure of the network packet. Or if you were, let's say you were attaching to the Linux security module API, you kind of have to know what the consequence of allowing or failing that call is going to be. You can get to kernel data structures, so you might need to understand how those kernel data structures work. Yeah, it is true that at some point this is pretty much kernel programming. It's safer for reasons that will come to you later than writing your own kernel module. And you get some helper functions to kind of assist with a lot of this contextual information that you might use. But yeah, it is basically kernel programming, so it can get pretty detailed pretty quick. But what I hope you'll get out of even just doing hello world type exercises is a sense for why EBPF is really interesting. So even if you go away and you never write another EBPF program again, you'll have an understanding of why it's such a good platform for building things like observability and security tools and networking functionality. I see another question. Thank you. I have a question about the user area, because when we are writing programs in kernel level, there are some different performance speeds in kernel level and application layer. Is there any best practices or design patterns that help us to achieve some goals? Because I have some conflicts and some issues on managing getting data from the kernel because of the difference of the speeds. How can you help me with that? Yeah, so I think what we're talking about there is the kind of cost of context switching between kernel and user space. So every time you do that transition between user space and kernel or back again, it takes some time. EBPF gives us some maps which we're going to see in the next section which helps us pass information between kernel and user space. And it allows us to do stuff in the kernel, put information into a map and then deal with it in user space asynchronously. So we don't have to just keep doing this transition between kernel and user space. It's also the case that the more you can do inside the kernel, the higher performance your tool will probably be. So a lot of what we do in, for example, there's an EBPF security project that we have as part of Selim called Tetragon. And one of the reasons why it's super high performance is because we filter loads of events within the kernel. So minimising the amount of information we're sending to user space. Did that kind of answer that? The next thing with maps might help with that. Has everybody pretty much been able to do Hello World at this point? Yeah? I see a few thumbs up. Anybody want me to sort of stop? I'll move on. Let's move on. So what you've seen there is Hello World, a very simple programme that's just generating some trace. And every time you triggered that programme, in that case it was through the execve system call, it should have generated a line of tracing. I already just briefly mentioned this context parameter. So depending where you've attached your programme, you might get different contextual information. The context might include things like what is the user space process that caused this event in the first place. So for a system call, a user space programme process had to make that system call. And you'd be able to get hold of the information about that process with a helper function. In fact, the kernel tracing gives us some of that information. So what is the executable, what's the process ID, timestamps and all kind of other information that gets included as part of that tracing message. This is just Hello World, right? And it's not terribly useful using this kind of tracing mechanism because you can run multiple EBPF programmes at the same time and typically an EBPF-based tool will run lots, like for Cillian we have an EBPF programme for every network interface and there's one for every container. So there will be lots of EBPF programmes. And if they all wrote information just to the same tracing location, which they do, that's not terribly helpful. So this is just a kind of illustration of the fact that you can trigger something from and it will cause your EBPF programme to run.  This is not an efficient way to get information out of an EBPF programme. So instead, we're going to use these things called maps. A map is a data structure that you can access from an EBPF programme. You can also access it from user space. You can write information depending on the type of map. You can write information in user space and read it out from an EBPF programme or you could pass it from one EBPF programme or use it in multiple EBPF programmes to share information between them. And there are lots of different types of maps. A lot of them are hash tables that will have a key value pair and that's the example that we're going to see in the lab this afternoon. There are also some Perth event and there are actually some more modern ones that I don't think are in this list that allow us to stream event information into user space and get information out in a very efficient fashion. There are also some kind of special purpose maps that we can use for things like having a map that lists other EBPF programmes. That's a whole other topic that's probably not worth going into at this point. So the next lab, which is also in the Chapter 2 directory, it's going to show you an EBPF map so it's going to give you an illustration of using a map to pass information from kernel space to user space. So feel free to go ahead and do that second lab and if anybody's got any questions, I'm sure Eric will come. For the second one, when I had it running and then I was running commands on the other side, I wasn't seeing a one-to-one increase in the counters. Is there a reason for that? So there's a few things that can be going on there. So one is we're still attached to the exec v system call. In the second terminal, you're triggering that system call by running things but there could also be other things that happen to be running on that virtual machine so maybe there's some kind of system d demon doing something and if it happens to call that system call, then you might see that event triggered. And actually that's a really good illustration of why EBPF is so powerful because you didn't have to change anything about that terminal, the shell it was running, what commands it's running, the system d thing that might be triggering exactly, all of those things, they are instantly instrumented as soon as you load that EBPF program and attach it to the event. This is one of the really cool things about EBPF. We don't have to change anything about our applications to get visibility of them from within the kernel with EBPF. See another question? So I see all these programs are basically C programs which are strings and then gets passed via Python. Is there a reason for that? We could probably go directly and I guess the other question is what happens if my code, I'm sure all this code runs but I make a mistake in the C code. I'm assuming it gets compiled somewhere and it blows up. That is actually a really good question. So for these first couple of examples I've used a framework called BCC and BCC is essentially a Python library and it does a lot of heavy lifting for us. So it's taking that C program. It's running the compiler. It's loading the program into the kernel. There is some Python code there to tell it to do it but a lot of the detail is taken away from you. We'll see some examples later on where we'll just have the C code and we'll compile it directly and load it into the kernel separately. BCC I think is a really great beginner tool. And if you want to just write a simple script to instrument something, BCC can make that really straightforward for you. I would say that in this day and age there are more efficient and better ways to create code that's more portable. One of the interesting things is that it's compiling that code for you like on the fly. The reason why it does that is if you compile on the system that you're going to run that EBPF program in, you can have a lot of confidence that the code is compiled with data structure information that matches the kernel where it's running. If you then take that EBPF program and send it to somebody else who's running maybe a different version of the kernel, data structures could have changed and sort of in the olden days or in the past that was a problem. Your EBPF program might not be compatible with different kernels. There is now a thing called compile once run everywhere which I haven't covered in the lab today but there is a chapter in the book that is all about making your programs run on different kernel versions. Not only do you compile the code but it comes with information about the data structure layout where it's compiled and some really complicated stuff has to happen to adjust the code when it's loaded into the kernel. This is one of the things that BCC just kind of takes out of the equation. We don't have to think about it at all and I kind of think it's a bit complicated for day one. So that's why I haven't covered compile once run everywhere in this lab but if you want to dive into it, it is in the book. Can you make another question right here? In a co-bonada scenario there are multiple nodes and you have kernels in each of the nodes so you kind of load EBPF into it so my thinking is can EBPF potentially replace or play the role of a service mesh? Yeah, it's like you're some kind of sillion plant. Yes, so every node is running its own kernel. I showed a picture before with containers or sharing a kernel but you're absolutely right every node has its own kernel but it means we only have to instrument the kernel once. We will see an example later of just some simple things that we can do with network packets. We can change the way that networking behaves on a node. This is essentially the basis of sillion networking but we can intercept network packets, we can understand, sillion can understand the Kubernetes identities that are involved in that traffic so rather than dealing with things like IP addresses your pods, every pod has its own IP address but as pods come and go their IP addresses change dynamically so it's pretty hard to debug a networking problem using IP addresses in Kubernetes. What you want is what's the name of the pod, what's the node it was running on, what name space was it in and all that information we can be aware of in sillion. The basis of sillion is Kubernetes-aware networking. Service mesh, there's a lot of what do you mean when you mean service mesh? Do you mean MTLS? Do you mean load balancing? Do you mean canary rollouts? There's all sorts of different things that people imply. Layer 7, sillion uses the Envoy proxy to handle layer 7 termination which is required for things like layer 7 load balancing understanding what service I'm going to send different requests to. Today there is a little bit of layer 7 passing that's possible within EBPF but mostly we call out to Envoy proxy to do that for us in user space. It's a pretty complex behaviour. The answer to can you implement a service in EBPF is kind of yes and no because probably 80% of service mesh functionality is handled in EBPF in sillion but for the bits where we need to actually pass a layer 7 packet we mostly call out to the Envoy proxy to do that today. That was probably quite a complex answer to a... How's everybody getting on with the EBPF maths? Yeah? Thumbs up? All right. Okay, so this kind of comes back to that question we just talked about around compiling code so under the covers BCC is compiling that program string which is C code is compiling it into an object file and that object consists of EBPF bytecode so this kind of gets into a little bit of like what is happening when we run an EBPF program? It's essentially a virtual machine so there's an instruction set that looks a little bit like machine code that object file is a series of bytecode instructions that look pretty similar to machine code if you're familiar with that and EBPF implements a software virtual machine so there are 10 general purpose registers numbered registers 0 to 9 this is an example of an instruction if you're familiar with machine code of any type this is kind of pretty straightforward it's much simpler than looking at, I don't know x86 machine code but you have these pretty similar like assigning a value to a register or assigning jump instructions comparison instructions, that kind of thing so the compiler creates an object file that defines this series of EBPF instructions that make up the program and it also defines any maps that the program is going to use and there's a BPF system call that's used to load that program and any maps into the kernel goes through a process called verification which we're going to dig into a little bit later on but verification is what makes EBPF safe to run it's what differentiates EBPF programs from kernel modules because EBPF programs can't crash because the verifier makes sure they don't so once your program is loaded into the kernel it gets attached to an event you have to say what is the event that's going to trigger this program to run and then we can use system calls to get information out into user space from any maps alright so the next couple of labs are all about a tool called BPF tool which is like a kind of Swiss Army knife for manipulating EBPF programs and EBPF maps and it's also going to show you an example of bytecode for those of you who are interested to see it I don't feel like you necessarily need to understand every single instruction but it's there if you want to so that next labs are going to show you how some of the things that you can do with BPF tool BPF tool can perform a lot of those system calls for things like loading programs into the kernel reading information out with maps BPF tool can do this kind of stuff for you and it's a very useful tool for seeing what BPF programs are on your system what the contents of maps are you can manipulate maps all sorts of things with BPF tool so yeah go ahead and try that out my colleague Quentin Monet did a really really good Twitter thread that was I don't know 50 messages long or something with all sorts of different things that you can do with BPF tool see a question again Do you have a good reference for is there like a place that we can go to if we're not familiar with kernel programming to see all of the different hooks and like specifically what they referred to kind of like a jump start to pour the hook in and what to like the pieces that went into that so there is some rudimentary documentation in the kernel docs and I say rudimentary I think it's improving all the time and in the book there are links to where the best places are so for example there is some kernel documentation about the instruction set but I think there's actually an unofficial documentation that's part of BCC project that I think is actually a bit easier to read and better there's a site called ebpf.io which is a kind of community site for lots of ebpf things and you'll find lots of great links on that I think it would be fair to say that partly because ebpf is continuing to evolve it's not brilliantly and perfectly documented anywhere completely comprehensively but I think over time the kernel documentation will get better and better it's also slightly complicated because the set of things that you can attach to depends on the version of the kernel that you're in because it's still continuing to evolve so it can be a bit like here is a list but it only applies to kernels up to 5.18 or something and then you might be interested the source code is the really kind of comprehensive documentation I see another question here this might be related to a previous question that was asked but I see a number of see group events on this terminal is that because of the environment like if I ran the same thing on a laptop I'd just get the program that's attached to the K probe partly I think it depends on the kernel version but I think in modern environments you pretty much will see a program attached to every or you will see a series of see group programs I'm not 100% sure what those programs are and it's something I keep meaning to look up but yeah they it may be related to the I don't know network environment I should really know the answer to that but I don't one of the reasons to maybe force yourself to think a little bit about the bytecode instructions is that when we get to verification the verifier runs on bytecode instructions it doesn't run on the C code and I think that's the source of a lot of you hear people talking about how fiddly is to get ebpf code verified and the messages that you get back from the verifier can be a little bit critic but I think one of the reasons for that is because the verifier is looking at bytecode and you have to do the job of relating that to the source code that you gave that got compiled into that bytecode so that can be one reason to kind of get to grips with the fact that there are these registers and these instructions that are acting on those registers I see people still working their way through by the looks of that Any other questions at this point? So I'll just follow up on what you just said so it looked like we were compiling with Clang so are we able to generate a debug mapping to map the bytecode back to the C code that generated them? If you compile with the debug flag then when you inspect the bytecode you will get the source line code information as well and you'll see that in the verifier output too So it's not a complete like try to figure out what the compiler did on her own? It's not complete but I feel like there's a hint about this is the line of source code and this is the bytecode that was generated from it but the yeah it's sometimes it's clearer than other times I think particularly because of the way the verifier is keeping track of the type of information that it thinks every register is holding and sometimes there's a point at which the verifier says no that's not what I was expecting you kind of have to go back and look through the bytecode for where was that value set and that might be in a previous source code line that doesn't necessarily seem to relate to where there are it does but it doesn't at first glance always look like it How are people getting on with that? Go on to the next one EBPF and networking So far you've seen EBPF programs essentially just attached to K-Probes but as we said we can attach them to points in the networking stack and there are some really great very simple reasons why we might want to do this this example I think is particularly great so this is the concept of a kernel vulnerability where a particularly crafted network packet can crash the kernel Hopefully there aren't any of these live today but there have been a couple of cases in the past where let's say the network packet has some kind of length field but if you set that length to the wrong value the kernel and it passes the packet and maybe tries to access some memory invalidly or something and then crashes and if you were an attacker knowing how to craft these packets that the kernel will crash on would be what's called a packet of death and when these kind of vulnerabilities have occurred this is a kernel vulnerability so in the olden days without EBPF if you wanted to patch your kernel that's going to involve installing a new version of the kernel and rebooting your machine which could be pretty painful if you have a large fleet but with EBPF you can just load a program dynamically just like you've been doing already and that EBPF program can check the format of the network packet see if it has whatever the packet of death sort of format is and then just drop those packets and that means you can just mitigate this kind of vulnerability without disrupting anything that's running on your machines you can literally just go around all your machines install an EBPF program and it is no longer vulnerable you didn't have to upgrade the kernel I feel like this is a really powerful illustration of why EBPF is such a great platform that has all these really interesting mitigation properties of course we can do other things we don't just have to discard packets we can modify packets, we can redirect them we can create things like Cillium that do really interesting things with network packets but what we're going to see in the example in this lab is the ability to drop a packet and essentially there's some code that looks for ping packets if the packet has the ICMP which is the proper name for ping then we can drop that packet and what you're going to see in the lab is you can dynamically change that program so while your ping is running you can change whether or not those packets are going to be dropped or passed through I mentioned Cillium a few times I want to just talk a little bit about why EBPF is so great for container networking and Cillium in particular so typically containers get their own network namespace and that is typically true I'm saying typically because it is possible to run containers on the host namespace almost nobody does that in real-world scenarios so mostly your containers and your pods in Kubernetes have their own network namespace and the container is connected to the host network namespace through a virtual ethernet connection so if you're not familiar with the idea of a network namespace it's like a network stack so from the perspective of the pod to the IP address and if it wants to talk over the network to something outside of its own pod it has to use this network connection to do that that network connection happens to be virtual rather than physical so a packet that for example comes from the outside world is coming through the physical connection into this machine and it can be pretty convoluted there's quite a lot of things that the kernel does with network packets and then it goes to the virtual ethernet connection to then get sent into the pods network namespace and vice versa for packets coming out of the pod they've got to traverse these two network stacks and that can be pretty inefficient particularly in the world of Kubernetes where as we've mentioned before these IP addresses are coming and going and changing dynamically IP tables are used to create rules how are we going to handle these network packets what are we going to do with a network packet where are we going to send it and IP tables rules whenever you make a change they actually the whole set of rules has to get rewritten the more rules there are the slower this can get and the fact that IP addresses can be changing and pods can be coming and going can make it quite noisy the way these IP tables have to get updated so much more efficiently we can use EBPF to intercept packets as they arrive on that physical ethernet connection and essentially insert them straight into the pods networking namespace without going through the entirety of this host networking stack because we know here's a packet it's destined for this IP address we can pass it to that pod directly there are actually lots of different places in the networking stack where we can attach programs this includes there's BPFC groups the programs that you sort of see in that long list of pre-existing programs we can attach to sockets we can attach to the sort of the top of the stack sort of as a packets received from an application into the networking stack or just before it's passed to the application we can intercept at various points within the stack they have different qualities and different reasons why you might want to do that the example we're going to look at today is my favourite which is XDP XDP stands for Express Data Path and this is imagine a packet arrives on a physical network connection and before the kernel has had a chance to do anything with it this is where the XDP hook is the reason why it's before the kernel has a chance to do anything with it is because it's actually possible to run XDP programs on network cards or inside network device drivers so this is called XDP offload and if you imagine you can have a network card that can run these EBPF programs that can handle things like packet of death mitigation or firewalling or all sorts of load balancing for example and those packets can be handled on the network card and never even have to hit the CPU they might never have to be handled by the kernel so this can be really like high performance networking not all network cards support this XDP offload but you can think of it conceptually this way don't necessarily have to get all the way to the kernel's networking stack that can be manipulated by these XDP EBPF programs and that program can do what it wants to do to the packet it can look at the contents of the packet it can make decisions about whether to pass it on or drop it or send it to a different address it can change the contents of that packet so if we wanted to send a packet to a different address we'd probably have to change the addressing inside the network packet we don't have to go into the details of that today but it's just kind of cool to know that you can what we're going to see in this example in the lab is the idea that either the XDP program is going to return the value XDP pass which says to the kernel carry on as you were with this packet I'm finished with it, pass it on as you would have done or XDP drop which is just get rid of the packet just drop it throw it away and we're going to use that to intercept some ping packets in the next lab this one is in a different directory you need to go into chapter 3 this time rather than chapter 2 I think it's pretty cool that Instruct lets us run these kind of things on a phone quite a good way of getting to see these things without having to set up the whole environment well spotted, thank you sorry, I can't remember how long they're going to be up today there are some more labs we're going to make this available in the full set of ISOvalent Labs, I'm not quite sure whether that will be straight away but it's not later this week very soon there's a ton of other labs so it's ISOvalent.com slash labs but yeah, I'll try and keep these up at least for this week so from an enterprise adoption perspective currently we deal with images and then we have some enterprise related stuff on top of the image that we have like peter redhat or ubuntu for example and then we ship out these images for the this is machine images, virtual images so if we talk about bringing an ebpf into an enterprise adoption kind of a thing then there's obviously a lot of things that's possible and how do we ensure what's the code quality of these ebpf programs that's going into these images is there anything available so the verifier which we'll come to in the next lab answers some of that question it ensures that unlike a kernel module ebpf isn't going to be able to crash the system but I mean I've shown you ebpf doing some pretty powerful things like intercepting network packets and changing them potentially the ability to load an ebpf program is it's superpowers it's not something that you should give away lightly it's certainly like root access plus plus so you do need to not give ebpf privileges to just anything be very careful about what and who can load ebpf programs and you have to be sure of the provenance of those programs I mean some of this it's like any software do you trust your vendor to have given you appropriate code open source project you can read the code there is it kind of begs the question of signing like how do you know that your ebpf program is the program that you expect it to be and that is actually a much more complicated problem than it is in most software because I mentioned compile once run everywhere and this is one of the reasons why it's kind of complicated it's even without compile once run everywhere you kind of end up with some relocation of the code as it's being loaded into the kernel that means that you can't just run a hash over all of the instructions and say is this exactly the code I expected because it might have changed by the time the kernel actually gets to see it some of that relocation is done in user space so there is a there is worker foot in the ebpf kernel community to come up with a way of signing ebpf programs so that you can have this kind of validation of where it came from and that it is the exact program that you expect it to be but it's quite a lot more complicated than it is for user space programs so today there is not a fantastic answer to that question other than the fact that you're going to get an ebpf program delivered to you probably as part of a project that involves user space code as well and that can be signed ebpf program itself it's hard to sign it's also still quite common to compile programs in place and if you're compiling it in place you can't sign it question there hello how would a distro maintainer use ebpf during a root of s creation let's say we want to generate a root of s for a distro and we want to integrate some program with ebpf how will we go around that? Will we be using a system d service or whatever the unit system is to run it at each boot? I think the answer to that is probably yes I think there's certainly some people who've been doing some interesting work and even boot loaders I'm not super familiar with exactly how they've done that I think system d would be the sort of that would be my natural way of doing it or some kind of in it so there's no unified configuration file that it can be added to so that it can get loaded on boot I don't think there's a standard no it's typically after the fact I guess part of the point is that you can change it after boot and that you can dynamically change the set of ebpf programs rather than fixing it at the start of the day but I don't see any reason why you can't do it as part of any kind of in it script or system d how are you all getting on with the xdp? some thumbs up I don't know if you might be doing your email for all I know so we mentioned the verifier a few times and this is basically the last of the labs for today is to look at some of the things that the verifier will do for you so the verifier runs over those ebpf bytecode instructions analyzes every possible path through that ebpf program keeps track of what the possible values could be in every register and then uses that to assess whether or not it's safe so for example making sure that you're not going to dereference a null pointer and that means that in order to get code to pass through the verifier you have to explicitly check that any pointer is not null before you dereference it so just because code might be valid C you know you can dereference a pointer in C and it will compile and you can generate the bytecode but at the point where you verify it the verifier will object because it will say I needed you to have checked this pointer before you dereferenced it and there are several things that the verifier is doing another thing that it's doing is checking that your program will run to completion so analysing all these possible paths through the program and it essentially keeps track of how many instructions it's processed the instruction is called the complexity limit essentially saying how many different instructions have I passed through on this analysis and if it reaches a million then it's going to say I don't have confidence this program is going to actually run to completion so I'm going to reject it that complexity limit used to be 4000 but in more recent 5 point pretty much anything kernels a million instructions so that you can run some pretty complex code in EBPF programs these days but that's one of the things the verifier is looking for it's also checking that you're using appropriate helper functions so you saw the helper function for printing out trace messages there are helper functions for getting hold of information about the currently running process I think I mentioned that before there are helper functions for helping you access network packets but you can't use a helper function for accessing network packet from a program attached to a K probe because there is no network packet involved you can't use a helper function for finding out what the current process ID is or what the current running command is you can't use that in the context of processing a network packet because there is no user space process involved so the set of helper functions that it makes sense to call depends on the context and where you've attached that program and the verifier will check that kind of thing for you so the next lab just gives you a few examples of things that will break the verifier I'm going to warn you that even when you run this lab the first time it's going to generate the verifier output for you and it might look a bit this is a bit overwhelming but if the program doesn't stop if it just sits there then the verifier has passed if the verifier fails the program will stop and you'll see your prompt and then you can read back and look for whatever the error message is so the idea in this lab is you start with some code that does pass the verifier and you modify it in a few different ways that give you some examples of breaking verification there's a ton more examples in the book and actually you might see in the file that you're editing that there are some comments that can give you some clues for other ways that you can break that or cause this example to not pass the verifier I see a question Just more tactical things since this lab is actually asking us to use an editor I could just use a basic text editor because I can't figure out how do you save with this editor Towards the top of the screen there's where the file name is it's kind of like a tab there's a little disk symbol and if you click on that it should save but yes you can use any editor I'm not sure what's installed probably by another question I may have missed this but how do you figure out how do you pass the information about the process that triggered an event So it's kind of done automatically for you by as a result of the type of program it is where you've attached it So let me see if I can find So this example where it was an XDP program the context variable by convention we always call it CTX but it's this I can't quite see from here but it's this sort of network packet metadata structure that because we're attached to XDP that's what you get if we are attached to a K-Pribe I'm actually called it void start here because I wasn't using it but you would get a different structure that represents information about the process and it's a kernel data structure so you can access it directly or using the helper function so a lot of the helper functions will take this context parameter as an argument so the answer is it depends another question over there Right now with the programs we are playing we are getting the output in the terminal but how do we go about using this in the real world like is it like a visualisation on top of it because so that we can make more sense out of it or just it's always going to be a terminal based technology Yeah, no it can be anything so in that the second lab the one where it was the eBPF maps depending on how familiar you are with Python code you can see the Python code kind of accessing the map and printing it out to the screen but you wouldn't have to do that because it's much more complex that extracted that information and sent it to Prometheus you can do anything you like with that information once you've got it into user space it's like if you read information out of a file you could print it to the screen or you could do anything else you like with it I'm tracking off his question for like practical application of this so let's say I have like a 100 node Kubernetes cluster and I want to actually deploy a program across all the nodes like is there an ecosystem that exists for the management and deployment of these programs to more than one kernel keeping track of all of them The short answer to that is no there isn't but there is a good reason for that is the examples that you've seen here are all kind of standalone here's just one EBPF program that does a thing and there are lots of really good use cases that behave like that actually as part of the BCC project there's a ton of observability tools that you can use for kind of measuring or observing all kinds of different events Brendan Gregg was a real pioneer in that and just this huge family of observability tools and those typically are one or two or maybe four EBPF programs something that quite often happens particularly if you're looking at things that involve system calls is you'd have one EBPF program attached to the entry to a system call and a different one attached to the exit of the system call because at the entry point you've got information about what was part of the arguments part of the system call and the exit you've got the results and those EBPF programs maybe use a map to coordinate and kind of combine the two because by the time you've exited from a system call the arguments are no longer present so you kind of have these two different EBPF programs so even with those pretty straightforward examples we're already talking about two EBPF programs if you want to do something a bit more complex you might have many different EBPF programs silly you're going to have it depends what functionality you're running and how many containers you're running and that means that there's no kind of really standard way to manage those EBPF programs and you typically going to have, I mean you saw a EBPF tool EBPF tool is a way that you could manage but it's pretty low level with a more sophisticated I'm going to say silly but anything, you know Falco anything Parker, anything that's you know a sort of production ready EBPF tool will come with a user space agent that handles the life cycle of the kernel programs for you another bit of complexity here for example so with Cillium there's a user space agent that in a Kubernetes environment it's going to run as a demon set so there's an instance on every node and that's going to take care of loading those EBPF programs into the kernel and updating them and what have you but by design if the agent stops running for any reason the programs remain in place and the corresponding maps that they have they remain in place in the kernel and that means your data plane still exists it might not update the new pods until the agent is back in place but it means that if for any reason the agent gets disrupted your data plane is still in place all this being a very sort of long and complicated answer to say there is no really standard way of approaching managing that kind of life cycle because it really depends on the application of those EBPF programs I hope that made sense another question there following up on your comments about there are going to be separate EBPF programs one for the entry one for the exit and when you talk about multiple nodes there are going to be many more things so how do you stitch the things together like how do you stitch all the calls in particular operation or transaction together when it's being emitted by multiple of these programs which are being batched and sent to the agent which we might be running in that particular node which kind of listens to those things the mechanism is going to be maps because you can access the same map from multiple different EBPF programs the sort of design patterns around that very much depend on the application so I wouldn't say there's like a standard way of doing that also because EBPF is evolving some of this is kind of continues to develop things like how we get a stream of events out of the kernel there are newer ways of getting perf events that become more and more efficient depending on the version of the kernel that you're using you might see code that was written three years ago that uses one type of map and that if you were re-implementing it today you'd use a different type of map just because it's more efficient there's no kind of standard for that and it very much depends on the application unfortunately you do once you get past some pretty packet drop hello world things do start getting quite complicated quite fast was that another question? as things are getting more complicated what does the developer workflow look like in terms of do you have IDs that already know how to interpret EBPF C codes so you get nice syntax highlighting and warnings do you have CI pipelines that are spinning up a dozen different kernels against different environments? in the Cillian project we do have a number of those things I'm going to say pretty new frameworks for things like getting code coverage on BPF programs there's a couple of different things for that that I'm aware of which is called something like BPF Cove and another one is I think part of the Cillian put it into the Cillian maybe BPF actually I think maybe it's its own repo on the Cillian but yeah so there's a few different tools for and again this is something that's kind of developing those code coverage tools are being created within the last year things like CI CD again it's going to depend on the life cycle but I would definitely say projects like like Cillian, like Falco, like Parker probably Facebook have a project called Catran that's a very mature networking project load balancing project so I think all of those are probably good places to go and look at examples of how things are handled but yeah there isn't like a kind of like sort of commonly accepted practices I think yet it's still a bit early for that A really question here so question is regarding can we use Rust for example to write this code in that case the verifier can be we can make verifier simpler it's a really great question yes so the answer is yes you can the BPF virtual machine is still going to operate on bytecode but your compiler Clang, GCC and the Rust compiler are all today capable of compiling to EBPF bytecode Other compiled languages it maybe doesn't make a lot of sense so anything that has garbage collection that wouldn't be compatible with the EBPF virtual machine so yes Rust is a actually pretty common language today I think for some projects to write their EBPF code and if you're a Rust programmer that has the advantage that you can write your kernel code and your user space code in the same language you can write it in C I think today C programmers you know I don't know C programmers are really happy with writing in C but a lot of user space programmers will want to write in something else so there are things like Go libraries that are good at the user space part that kind of user loader interface Python as well you've seen BCC as an example but that can only handle the user space part in terms of improving the verification I don't actually know if there's been any kind of research into that because I think you'd still have the requirement to do things like checking explicit checks on pointers it wouldn't make any difference to the helper functions and so on The reason I ask is I look at this the reference example because I think in my opinion in Rust you don't have that kind of problems that's why I'm saying maybe it's still got to go via the bytecode though so I think this kind of comes back to when the verifier is running over the bytecode it doesn't know what the compiler was it doesn't know what the original language was So I see the book is available in a PDF is there ebook friendly formats like EPUB or something like that I'm going to say no I mean it's it's available on Kindle for example but that's really kind of a question for the publisher so I don't get to control that unfortunately which does remind me that I have five copies of these I've been paying attention to people who've been asking questions so I'm going to bring out a few of these Would you like a book? Yes, absolutely One over here I'm afraid I don't have one quite for everybody who asked a question That was a really good Rust question and apologies that I didn't have enough for everybody who asked a question What did you all think of the labs? Was that useful? I'll try and keep those up this week and we'll probably try and keep them actually put them on to the labs page for longer and if you enjoyed that there's a ton more detail in the book and we still have about ten minutes if you have any more questions Thank you Liz


Hey everybody, it's Christian Buckley doing another MVP buzz chat and I'm talking today with Maddie. Hello. Hello. How are you? I'm doing well and for folks that don't know you, who are you, where are you and what do you do? Okay. So my name is Mattipakran. I'm from Finland. So it's December. It's very dark here. So dark, more stiff. It led me on. I'm a fourth year Microsoft MVP, Microsoft 365 apps and services. Working as a modern workplace lead, I guess might title a little bit changed on a small company called Tahto Group in Finland. Working with small and medium sized enterprises and nonprofit organizations. So doing a lot of things around Microsoft 365, helping customers to migrate to the product, use the product, improve creating internets. Doing a lot of stuff around Microsoft 365, security, device management, whatever is there. What do you see with your deployments? I'm always interested to know, since we work in this similar space, where do you see the biggest gaps with customers you're talking to? Do they have trouble with adoption or they have trouble with governance compliance and security? Where do you see the greatest need is or the most asks for help? Yeah, I would say the security and compliance side is one of their missing. And we are in Finland, kind of in a Northern Europe and everybody thinks that we are in this kind of safe place. We're not. So there's a lot of things happening around security space, but they still need someone to translate whatever is happening. What is new? How should I do this? How should I manage my devices? How should I manage my data? Those kind of things. And then the adoption side, of course, as well. Still, we have been seeing teams over five years or over six years now already. Like six years now. Yeah. And I think still a lot of users don't know how to use teams, for example. And they still struggle. I have teams, but I have SharePoint. I have OneDrive. So they don't really know what to use when. And trying to clarify that with customers as well. You have seen that some Microsoft senior leaders there, they even expressed that their frustration was like, why is that question still out there? The which tool to use when I'm like, you know, and then after this, it was kind of a banner year for Microsoft creating new logos for products, like all these different things, all the co-pilots, all the Viva stuff, like all of that out there, which kind of increased the number of times I hear the, well, which tool am I supposed to use when for each of those? Yeah, exactly. But that's the opportunity for the MVPs to write about, talk about. So what kind of stuff? Like, what are your topics? What are you writing about? What are you speaking on right now? Well, of course, teams is one of my interests. All the collaborations tools I have been working with, collaboration tools, 15 years maybe SharePoint started on 2003, actually was the first version. Also writing a book about SharePoint currently. Oh, wow. Hopefully, getting it done on January. Not making any promises, but let's put it out there. So I would say not writing books. It's the most work you could do for the least amount of money. It's a big effort. So I, yeah, yeah, probably, yeah, I respect you for, for, for venturing down that path. Yeah, it was more of a learning experience. So learning to write for technical English and because it's my, not my native language, so learning experience. What? Yeah, no, no, that's, that's great. So who's your publisher? Who are you doing that through? Packed. Okay. Excellent. So we'll watch for that for sure. Yeah. I guess it's out there already, but not making any promises on the launch date. But the book is not out yet. Yeah. Yeah, I'm sure they're doing some promotion out there of that. So yeah, well, excellent. Well, what, so let's talk about your, your journey to becoming an MVP. Like what, what was that like? How did you find out about the program? How did you get involved? Yeah, I had a couple of Finnish, Finnish friends and the Finnish community, a couple of people there. And, and then I, I had a good mentor. So Vesku was my mentor actually and kicking me in the butt. We work at the same company. So, so I get good mentoring about how to become an MVP and started writing blogs, started kind of speaking at the events and also started hosting the Finnish teams community. We are having a five year celebration on April. So we have been actually having quite good community inside Finland around, around teams. That's great. Not so many speakers. I think we have the same familiar faces already all the time speaking but still, well, so I always tell people that are interested in, you know, going down that path and maybe, you know, working on becoming an MVP is like one of the easiest things to do to get speaking, you know, experience is reaching out to various user groups and saying, Hey, I'd love to speak on these topics. And if you get picked up, it's relatively easy to do in the virtual world. Yeah, nowadays it's so, so simple. And there is a lot of those user groups. And I guess all the user groups are missing new speakers. All the familiar faces are actually there all the time. So yeah, so that's kind of a problem when on the user groups. Yeah, there's a lot of opportunities just just to look for and look at and follow the community. So well, having gone through and helped, this is another great topic is you're starting the user group. What advice do you have for people that because there's a lot of, you know, look, there's a lot of virtual user groups you can go and join that are regional or, you know, around the world. I mean, I've got friends that are down in South Africa.  But if you don't have a group meeting in person locally, what's your advice for starting up? What should people keep in mind if they're looking to start up a user group? Yeah, that's a tough one, because we actually started. We had the first local event. We have food and everything gathering, opportunity, sauna and everything. So we had a lot of happening around that and got a lot of people there. And but I guess it was the teams were so new, so brand new. And everybody said, hey, we should go to team. So it was a lot of interest around that. And of course, bringing in customers and try to get them to speak and join in. So but any good tips? No, just just maybe just. Trial and error. Yeah. Yeah. Just do it. Yeah. And see, does it fly or not or not? So and maybe we actually had a. SharePoint community in Finland, it have been. I guess you see the road and I started that that community. Yeah, I've actually I came over. I presented to the user group in Helsinki years ago. Yeah, I traveled over, did a number of events and swung by.  There's a pub is kind of down around the corner from where the user group met. But yeah, I think that was the one that you see started. Yeah. And we actually started as a side group there. So so. So already had some some kind of a base base of base of community users there. So yeah, well, that that is that's another way to start a user group. Like if like here where I live and we have our what used to be the SharePoint user group and it's kind of morphed and changed. And there's other smaller user groups. There's other constituencies. There was for a while like an AI focused user group. I don't know that they've done anything in the last year or so. But there's a dynamics group that's pretty healthy.  And, you know, that kind of stuff. But you can always go and build off of an existing if you have a if there's an impetus, if there's enough interest to go and create something. But yeah, I always tell people is like, look, even if you don't get masses showing up, but there's four or five people sitting around sharing experiences, you're going to get value out of that. Yeah. Sharing sometimes pains, painful stories. Yeah, true. As well as your wins. Yeah. Yeah. I think it's just that we are going to have kind of have a monthly event, of course, not under summertime or during the Christmas time, but still figuring out the new fresh content if you have don't have new speakers. So that that needs some effort. Yeah. But of course, Microsoft is publishing a lot of stuff which you can talk about. And of course, you can always take co-pilot and do that. Of course. And as much as we're all like, I don't know, getting a little burned out on, you know, generic co-pilot, like people are asking questions. People are interested in the topic. So yeah, sometimes it's like, pull your audience. And if, you know, even if you're seeing repeats of the topic, if that's what people want to hear about, give the people what they want. Yeah. Actually, last season was pretty good. We got a couple of tips and tricks around teams from the audience. So that was good. Yeah. Of course, those familiar faces will be speaking before, but still someone from the audience as well. Yeah, that was good. Well, that's getting the audience interaction is always the best thing. It's as a presenter. It's one of the worst things when nobody's asking questions, nobody's providing feedback, nobody's sharing stories. Nobody wants to present to a wall, you know, you want interaction. Yeah. And that's kind of a bad thing about those virtual events that you cannot actually see who is there. And people are so shy opening cameras and everything. So yeah. And and just presenting using is there someone actually listening other than maybe the host? Yeah. Well, that's why like our in-person we're doing our collab days, Utah event in April and it it's in-person only. We're not doing the virtual like this will be we did the virtual the last two times and we're like, no, we're going to do. We're going to focus on the people that are in the room. Yeah. Yeah, we have been doing some hybrids, but it's still kind of a pain to do. You need to take care of the kind of the hybrid version as well and see who is on the line and and that needs some effort. And of course, it's it's needs some more practicality. So it's not just opening a laptop and shooting it. No, in fact, for the for some people that know, there's the the European Cloud Collaboration Summit. Yeah. And there's the North American Collaboration and Cloud Summit next, which is moving from Branson, Missouri to Dallas this next year. And so I've helped run the hybrid portion. So organizing volunteers because where it worked for hybrid is where you have a dedicated person that's in the virtual room and making sure that it's good to have somebody that's managing that. But is there physically so if there's a question online, they can represent ask the question to the speaker that kind of that. That's where it works the best. I but we've I believe we've decided for this this year to not do virtual to virtual to do in person only. So yeah, I think you're going to start to see a shift in events back to yeah, only I guess fine. I already know like stream a keynote. But for the sessions, my preference is in person. Yeah, already seeing that a lot because because nowadays there isn't so many options for virtual virtual events to attend as a speaker. So yeah.  What what what announcement was Microsoft made that you're like, hey, I'm I'm going to I'm on that. I'm going to be keeping up with that. Should I say something else? Is there another answer? Yeah, yeah, actually have been have been testing it for for a couple of weeks and and impressive. But then you need to have you need to prepare everything should be prepared. Your data should be good and and permissions, everything. So preparation work is really important. I see a lot of a lot of, for example, in my customers, there's a lot of outdated, unnecessary data available for the co-pilot to crunch on and getting bad answers. Isn't that funny? We've been saying that for years about search. And it's still true. It's even more so that for co-pilot. Yeah, archiving. That's something. Discussions with every customer. No one is archiving, really archiving, doing the things, of course, in Finland, less institution, and then you have a real archiving and separate system for kind of having. Kind of a set metadata and everything, but it's. Not so heavily in use in, for example, in small and medium size enterprises. So for folks that I've been paying attention to, Microsoft made investments in archiving and backup in archiving that they finally recognize that, you know, hey, this is essential. There's a number of the big consulting companies and, you know. You have like Gartner and Forrester that have been recommending having third party backup in addition to what's out of the box, but having more granular control over your backup and archival. But Microsoft finally went and invested in that space for native solutions as well as third party investments. That's an indicator, folks, that you need to be looking at your archiving. Yeah, and the archiving solution already looks quite good. I actually tested it and, well, still probably deleting something is also good that you don't need to have that data, but people like to spare everything over 15 years or something, right? Well, it's the because if you delete something and that and then, you know, find, hey, that had usefulness and I've already taken steps. Well, that's why having a process for the archival in staging it and how it's moved is preferable so that you don't just delete things that you might need, but again, stage that content so that you have time to decide, hey, like if it's if it's available on tape and I realize I've made a mistake, I can go back to it. Yeah, but yeah, life cycle, data life cycle, data life cycle from start to finish or the recycle bin, maybe. Yep. So well, Maddie, really appreciate your time getting to know you for folks that want to reach out to you, connect with you. Where are you most active on social? How can people find you? LinkedIn, Excel, trying on Blue Sky, but it's quite silent. But maybe LinkedIn and Excel are most active. Yeah. Well, awesome. We'll make sure to have all of your social links, of course, in the profile, add on the blog on a buck the planet and add on YouTube. So really appreciate your time. Yeah, thank you.