You just pull the nose hair out that way. No, it's the size of it. I thought you were just like, da, da, da. Welcome to your 30s. I'm quite proud of this one. So, keep out. So? So, it's this time of year. It is. You get the hat. You went all out and didn't tell me. I was going Christmas shopping. And I. And this is what you bought. I walked past this. And look, look, look. Oh, because Chrome. It's what Chrome says when it crashes. And I saw that and it's like, that is my Christmas sitting here for a while. You were wearing that and it did not make the connection. Well, that's what I saw on the shelf. But then when I walked to buy it, I was like, I would like this T-shirt. And the person who was saying, oh, it comes as a pair. And I was like, two T-shirts. And they went, no, they're pajamas. And they're like, oh. And they are. And of course, I've got the full set going on. There we go. Which is not going to be visible for any of these episodes. But you're wearing them for consistency sake because they do come as a pair. Do you realize that the actual thing says or snap, not oh snap, right? Oh, so it's not correct. It's not correct. You're technically incorrect. The worst kind of correct. I'm so sad, actually. I'm genuinely sad about this, because I thought it was. Right, well, that's ruined. That's Christmas ruined. Summer's ruined Christmas yet again. As every year, as is tradition. So last year, we talked about the features that were released in whatever the last year was, which was 2017. 12? 18. 2018, sure. So we thought, we didn't have any better ideas. So we thought, let's do it again. Do it again. Yeah, let's do it again. By China. But this time, we have the features from 2019. Oh, wait. I prepared 2018.  But I'll improvise. But this is one of the things. When we do Google I.O. or big Google events, the Android folks or the Cloud folks, they come and talk about the new stuff they're about to release. They use the events to make announcements, to unveil their new things. Yes, whereas the web just releases stuff every couple of weeks, or like six weeks in Chrome's case. So we thought this is our opportunity to celebrate some of the features. So we're going to do. What kind of time to celebrate in Christmas? In Christmas time. We've got 16 features that we're going to go through in a series of four episodes. And at the end, we will bang for the buck. And the best thing is, we get to decide the winner. Because we ran a similar thing at Chrome Dev Summit. We did not agree. Disagreed with some of the choices. So we get to just. So this is the canonical outcome. Yes, this is our opinion. Yeah. All right, so we're going to each pull a feature out of. It used to be a head. Now it's just the line on the table. We've got some pieces of paper on the table here. Grab on each in turn. So this is it. This is our first round. Yes, first round. All right, go for it. Ow. I'm going to take this one. This one looks nice. What you got? Numeric separators. Brilliant. Excellent. Boom. So we'll assign that to you. Do you want to go through the slides now before we pick the next one? Oh, interesting. Yes, I can absolutely do that. So this is a feature that was landed in Chrome Stable in 2019. Yes, as all of these features. Yes. And so numeric separators is actually one of my favorite features that landed. Right. Because every one of us has been in a situation where we go through code and we see a number like this. Yes. And we're like, what is this? And why is it so big? What number is this even? And with numeric separators, you're now able to write the number more like this. And you can see at a glance that this is 10 subtillion. And I think this will fundamentally allow us to write better code. That's cool. So when you're doing calculations with this, how does this make a difference? What's the impact of this on code? It makes it more readable. Right, yeah. But I mean, in terms of how JavaScript's going to execute this, what's the difference? None. Right. So I can just put those underscores wherever I want. Yes. Can I put two underscores in a row? Good question. I just thought of that, so I don't. OK. Or actually, a bad question. Why would you? Why would you want to do that? I mean, something actually is, yes, you can. I would assume you can. I'm not sure. I might be wrong in this. That's exciting. But nobody is forcing you to put it into groups of three. Because on a hexadecimal, you want to do it in group of soft. Two, usually. Oh, that's kind of nice. In octal, you shouldn't be doing it because why would you be using octal? Yeah, yeah, yeah. OK. Binary could actually make sense to put it in a group of eight or four. That's true. And we do have binary numbers in JavaScript now. So, OK. Well, it's very good. It's very good. So, what is that going to go up against? I'm going to pick this one, which is Intel stuff. Intel stuff. Intel stuff. There we go. Good one. So now, obviously, it is your turn to do them slides. Yes, it is. OK. So here we go. So this is a few features combined into one. We're grouping them because it all to do with internationalization. You've seen the stuff in. So the official API name is not Intel stuff. It's just many APIs. Stuff in Intel. And Intel? Intel there. So Intel. Internationalization. Intel. Yes, yes. Good. I mean, that should become obvious by the time I finish these slides. Oh. So we've got Intel relative time format. And you give, you know, like a country code there. And then you can format a number and say this is going to be seconds and it will tell you, look, put it like that. Or you can say, you know, minus 15 minutes. So let's say 15 minutes ago. This is something that we see, like, libraries like moment. Moment is like you have a canonical example that is quite big. Yes, exactly. And now it's in the browser. And you can do, yeah, different languages as well there. And that would be the entire point. Yeah, cool. Another feature of internationalization, lists. So you can put two things in there and it will say, you know, that thing and the other thing. That's how you end two things in English. Yep. And there you go. Big fan of the Oxford comma? Oxford comma, also a big fan. But again, you can say it's a disjunction rather than a re-junction. Conjunction. Conjunction, thank you. I like re-junction though. Yeah, I don't know. I knew it had junction in it. And so it's all rather than and. And again, different languages. Even speaks German. Amazing. Even speaks German, just like you. There you go. So you can already, like, you know. I can relate. I can relate to this, exactly. I can't speak German. At school, the only bit I can remember is Ich havers bei Pony. Which is a lie. Which is a lie, so it's completely useless to me. Because you have two cats, not two ponies. Yeah, but I don't know what cat is, so. I'll teach you after. Excellent, brilliant. Are there even more? There's more. Look, number four, Matting.  And we're saying, look, there you go, 2,000, sec. I wouldn't actually write it like that, but fair enough. And again, look at that, different languages. Everyone's happy. So this is like a three for one API. Yes, it is. And we need to decide which one is the best. I mean, you know where I'm standing. Numeric separators is core, especially because. You're really fighting for. I mean, especially because it only really makes sense with big ends. Without big end, numbers can't really go that big anyway, so. You can still go in the millions, right? Yeah, but do you need numeric separators? Yes, you do. And that's why it should advance to the next round. Like, yeah, Intel, it's an absolute big hitter. Like, we don't have to load libraries for these tasks anymore. You get pretty much all the languages that Chrome provides, which is quite a bit, it turns out. Yeah.  I would say there's still work to do here. Like, there's no, as far as I know, I could be wrong. Like, there's no way to give it a number and for it to split it as saying, like, this is. For the last task, if you want to convert it to, like, a language or something, like, these are 2,000 seconds. You might want to say it instead, six minutes and eight seconds, whatever it is. Yes, and the same with the relative date time. Oh, the relative as well, yeah, of course. It's the same problem. But, yeah, OK, so we're saying Intel stuff. Fine, fine, numeric separators are out. Well, that one, done. Next round. Now, Intel stuff has advanced to the next round. So now we need to find, and that's how brackets work, the opponent for Intel stuff, which means we have to go to the next matchup. I will go with this. And I have match all. Match all, OK, a strong one there. So, over to you to tell us what that is and why. Match. This is how it used to work before. You can put an array X and then you will get the matches for it. But now, with match all, you can actually get access to the capture groups that were in your regular expression. Because before you get like, this is the string that matches your regular expression. But if you had parentheses in it, you usually wanted to grab these subgroups out and reuse them. I see. So with the plain.match, you're not getting this bit here. You're not getting this. This is great. OK, I want a word that sentence us off with hello and then followed by a word. And I want to know what that word is. Right, so you have to. But match wouldn't give you that. So previously, you would have to use Reg X, EXCC or whatever to go in. But now you can just iterate over them. Oh, it's an iterator. It is an iterator. All right. I mean, at least iterable. I actually don't know. It could just be an array, couldn't it? Good point. I think it's an iterator, you know? OK, I think it's an iterator. Fine, fine. All right, excellent. And this is like, it reminds me of the string replace where it only replaces the first incident. What we really want is replace all. This is the first step in the right direction. Hopefully replace will be addressed at some point. Yeah, replace all is coming as well. Good. It's just not this year. So watch out next year. 2020 when it will win. The winning feature. Yes, excellent. All right, up against match all. Up against match all is. I'm going to phone down for this. Class. Oh, class fields. That is not weak. Let's take a look at class fields. So you've written code like this before, where you've got this.foo, whatever. And so in the constructor here, we're just setting it as a property. It seems a shame to have to use. It always weirded me out that suddenly you just start using props and don't. That you haven't declared. Yes. Start existing. And in other languages, you wouldn't see it like this. You'd see it declared somewhere else. Well, lucky for you, this is now a thing. Great. And it does exactly the same thing. It is just syntactic sugar in the end, isn't it? Yes, it is. I'm guessing this kind of assignment runs after the super constructor, but before your own constructor? Yes. Yeah, so there's a little bit of kind of magic sort of stuff going on there. But yes, like it will run, I believe. I think it actually might run at the end of the constructor. No, in between. There's a thing to look up, but we're really showing each other up in this episode of stuff we can't remember. It's either magic or not. That is the other two choices. Excellent, yes. So if you're using TypeScript, you're already doing stuff like that. Yeah, because it compiles down. Yeah, and I think there's a babble thing for this as well. But that's not all. Oh dear, Jake. I've got another one, which is just a mixture of a load of features. You've done this before. Well, because we don't have private, let's just use conventions. Yes, underscore is basically saying, don't touch this. But you really can. So you can reassign it here. Some people file issues, and they're like, well, you used a private prop, but they were like, eh. Exactly. Well, what if? What if syntax hiding completely breaks? Yes, this is so new that a syntax highlighting stops working. And yet, it has landed in Chrome in 2019. And it has landed in Chrome in 2019, so you're using this hashtag counter syntax. But it means you can refer to it like that. That's actually a separate proposal. The private class fields, isn't it? Yes. But it is compatible with the class fields proposals, and then you can declare public and private class fields without having them inside the constructor. And they both landed this year. And that means that you can only access these things within the scope of the class declaration. Which isn't ability that you couldn't really do within the JavaScript. That's a new JavaScript capability. You couldn't prevent access to properties on your object. Absolutely. Yeah, you would have to do sort of weird hacks and stuff, whereas this just makes it really simple. So as you can see here. Also, when you say no, does that mean throw? Like it's just like, if you access an undefined property, or does it actively throw? I think trying to access a private property outside of a class context will throw, that will be to that. This, if I'm going to throw, I'll see these two identifiers as being different things. I think the idea of this might be seen as different things. I wouldn't be asking if I knew the answer to that. Yeah, we'll put something in the comments that lets us play as enough. Fact check center. Yeah, exactly. So yeah, this will work fine because it's the public part. In fact, I guess that's sort of suggesting, I think this will error if not. I mean, I know. Oh, I'll tell you why this will error in this case. Yeah, because there's no setter. Because there's a getter, but no setter. So yes, it treats them as separate things. So this is it. See, I figured it out. You still didn't tell me if accessing. So ignore that one. I figured this one out. You see this, well done you. Exactly. You write JavaScript. This will error because there's a getter and no setter. They are different identifiers, which is like, actually solves this problem as well of having to have a different identifier of the setter. These are named the same. It all ties up nicely. So yeah, once again, I get two features in one. I mean, in terms of how most of us write code, class fields is going to be more interesting. Yeah, it's going to be that, hasn't it? It's going to be that. So yeah. I mean, the right thing's good. There was a hole in JavaScript there that we've now sorted. But yes, we're through to the next round. It's me versus me. Which is always a good one. Do you want to take? Should we do the slides again? Just for lolls, because we've gotten better now. No. No. OK. OK, so we're going to decide who wins this. Yeah. It's this is hard because this is, for me, it's a DX versus UX feature almost. Intel stuff is a win for the user. Less code to load, arguably better language support to be determined. While this is, the user doesn't care. So if you wanted to do the private field thing today, you would have to use weak maps to have a mapping between the class and this value. Or closures, but then you're creating so many issues. But no user cares. You shouldn't use private class as a security feature, right? No. It's supposed to catch mistakes of other developers. Yeah. Where something like TypeScript would be enough. Or a convention might also be enough where you could just punt it to a linter. Oh, that's interesting. I like your point about things like TypeScript and linters. Yeah, OK. And whereas Intel is going to make the bundle smaller because you don't have to include. Potentially faster. Arguably a bit less flexible than user-land libraries right now. I'm torn I'm leaning towards Intel because it's an actual useful capability. While we've solved the private class fields problem or the class fields problem, but now it's part of the language. Which is great. Like I'm not saying it's worthless. I'm just saying in terms of value Intel adds more. So I think you're right. The class fields thing, OK, if you're compiling that down, it is slightly more code. Fair. But only a little bit. Whereas with moment that you can have 300k or whatever, especially if you include all the languages. I actually was going to say class fields, but you've convinced me. Yeah, all right. That's that is lucky you have won this round. Yeah, me versus me. It wasn't going to go badly. But yeah, and so that is our first semi-finals. Our first semi-finals. It's a good point, I think, to cut the video. To chill. Take a breath. Take a break. And the next episode will be out soon. Join us. Join us. Please join us. We need the views. It's our job. Excellent. Shall we do another? No, that's it. That's going in. We can't edit this out. This is it now. Just all through the credits. There's going to be two overlay. The thing to click on next is going to be here or here. This is the end. This is the end of the episode. The episode's full news anchor. I'm going to keep talking until you press stop.


Shaved with a proper blade. Here's my story So I went down to jack of all fades because I needed my chin to be tidied and yeah They gave me some drank and they started the clipping after the initial long hairs were trimmed It's time to open up those pores and put a big hot towel on my head. There we go. All right Let's lather up that chin. Yeah, that feels so nice that blade though.  Yeah Baby bottom style I began feeling thirsty again, but I was preoccupied Anyway, we got there in the end. Let's see the reveal. Oh What do you reckon jack of all fades good. Thanks


Awesome. Thank you. I am unmuting myself and moving the doc around. So, hey everybody, thank you for joining. My name is Everett Pompey, and yeah, thank you for joining me for Run Fast, Catch Performance Gressions, and EVPF with Rust. This is going to be a little bit more of an extended session from my Linux con talk that I gave back at the Linux conference. So, all righty. As I said, I am Ever Pompey, and that's my email if you guys have any questions or anything after the talk, and that is the repository for Venture, which is a tool we'll be talking about here in a bit. All righty. So, what is EVPF? Kind of to start off with all the buzzwords in the title there. EVPF, if you kind of think about it conceptually, you have user space and kernel space, and EVPF, you take your source code and then you compile it down to EVPF bytecode. That then gets handed over to the kernel side and put into the EVPF verifier, which basically makes sure that it can kind of solve the halting problem essentially on it. And then as long as that checks and it does that, then it's good to go. And then it moves it over to the EVPF VM again inside the kernel, and it gets rough. And so there's a lot of restrictions on that VM and things like that and how that verifier works, and we'll talk about that a bit more as we go through. All right. So, kind of different kind of EVPF things that you can instrument and do with it. There is this thing called trace points, which means you can trace point any syscall within the Linux kernel. There's also a U probes, which are user space probes, which means if you kind of know function definition, you can watch that, which is pretty powerful. There's also K probes, which are allowing you to watch an instrument kernel functions. And then there are is XTP packet filtering, which is kind of one of the original uses for EVPF and even BPF before that, which was before it even gets into the kernel, being able to filter things really fast. And then there's LSM modules, so you can instrument any Linux security module hook and see what's going on there. And then what is Rust? That is a programming language. As you write that source code that we're talking about. And it has a focus on performance, reliability and productivity. I used to write a lot of C++ and now have kind of converted over to Rust and very much enjoy it. So, very excited that, you know, in Linux 6.1, it is now part of the build chain for the kernel. And so, but that is not what this talk is about. This talk is about writing Rust for the EVPF part of things and not the actual kernel itself. So, that is where we're kind of writing our Rust code to. So then what is a performance regression? Right on our last part here of what we're trying to prevent. It's the idea here is that performance bugs are bugs. A lot of times they get overlooked, right? Like folks might have lots of unit tests, but absolutely no benchmarks. And it just kind of depends on the application you're building on whether that's important to you or not. EVPF can add up to 100 milliseconds of latency if you really try. And so it is important there that like, you know, on every single syscall if you're adding that, that's quite a huge overhead. So, I would argue that it's pretty important here. And then production is also the most costly place to find bugs. So preferably we would shift that left as far as possible and find our performance regressions as early as possible. And generally performance regressions, like when do they get detected, right? They can happen in development and you can catch them there if you happen to be looking for them and you're like doing performance tuning and things like that. But a lot of the time it doesn't happen. And so it's possible to catch them in CI and automate it just like we do unit tests and the whole original idea of continuous immigration is that those tests get run automatically so you don't even have to think about it, but most folks don't have anything set up there. And so that means it ends up getting all the way production when things are on fire and users are upset. That's when you tend to find those issues. So kind of an overview of where we're going to go from here. But we're going to look at a basic eBPF program written in Rust. We're then going to evolve that eBPF program. And then we're going to benchmark that eBPF program. And finally look at continuous benchmarking for that eBPF program. So just an overview here of the possible eBPF tooling that could use. There's essentially tools to write it just in C. And then there's BCC, which allows you to use both C, Python and Lua. And that just interacts with that lib eBPF that's written in C over there. Or you could use Go, which is kind of a full stack Go solution. And that is possible, but we are going to be going with Rust here. Maybe by the talk you can maybe have imagined. So looking at the Rust tooling, just kind of the ecosystem here. There's a wrapper around the lib eBPF or lib bpf, which is written in Rust, which means you can write your user space code in Rust. But you still have to write your eBPF code that gets compiled down into that byte code in C, and the syscalls there are also in C. And then there was another library that came along called redbpf that allows you to actually write your eBPF code in Rust, which is super cool, but it was still using libbpf as that interface there with the syscalls. And then AIA, a really great library, came out and that allows you to kind of do full stack Rust here. So we're swallowing the pill here and going full stack Rust. So then now on to our first part that we are going to talk about, which is a basic eBPF program written in Rust. So this is going to be an XDP program just to keep it nice and simple. All that this is going to do is log the IPv4 source address of a packet that it receives. So in order to create this, again, we've got our user space and kernel space side of things. This logo, this little robot here is going to indicate user space, and this b here is going to indicate kernel space. And you'll see it up in the top right hand corner there to help us kind of know where we are at any time. So we're going to start off in the kernel space here in eBPF land. And so we've got this Rust function called fun XDP, which takes in a context that is a instrumenting an XDP call, and it returns a unsized 32 bit integer as its result. And so using AIA, we say this is an XDP, this is a macro that says we're going to be using XDP instrumentation here. And then based off of the result from a helper function we're going to look at in a second, we either return the value we get, or if there's an error, then we just sell XDP to abort that packet. And taking a look at that helper function, we get the Ethernet header. And then see if it's an IPv4 source address, otherwise we just move along because we only care IPv4. And then we then read that header. And then get the source address from the header. And then we log it. And then we pass. And that's it. Pretty simple. We're going to take a quick quick look here at the pointer at helper function, we're getting the source address, the header. And so this pointer at function, it's kind of an important thing to understand with EVPF here is that we're trying to really make that that verifier happy here and make sure that we're not doing reading any memory that we're not allowed to. So we're essentially passing in this offset here, and the context we take the start and the end. And then we take the length of the thing that we're trying to read. And then we make sure that our start plus the offset plus the length is, you know, not greater than the end and if it is we're going to error out because that tells the verifier that we are, you know, being good stewards here of where we're going in memory. And then we return the start policy offset for getting the pointer address, if it's good, and then balance. And then we also, again, to please the verifier rust has the possibility of panicking. So we have to tell it like we are not going to panic here that this is unreachable. So with that complete, we are going to move over to the user space side of things. And on the user space, we have a main function here, and we are going to parse some command line arguments and a net logging. We also have a path to our ebpf binary that we have created. And then we're going to load that, or bytecode and load that into the, the kernel. And then for that program that we knit, we're going to get a handle to it. And yeah, load that load that on in. Sorry, we've loaded the logging there. This is getting the program there and then initializing it. And then we're going to attach to it for the interface that gets passed in as one of the arguments to our function here. And then we'll wait until someone hits control C. And once they do, we'll exit, but in the meantime, we just kind of run our function. And so that is the very simple user space side of things. And so with that, we're done with our very, very simple ebpf xdp application. So let's kind of, you know, actually have it do something other than, you know, essentially hello world here, and of all of our ebpf program. So we're going to introduce the concept of maps. And so maps allow you to communicate between user space and ebpf land and said data in between them. So we're going to add a fizz feature here, which is we're going to push fizz into a queue, if the IPv source stress is divisible by three, and otherwise return xdp paths. And so let's go ahead and do that. And again, we're going to start out, or just take a look here, I guess, on our mental model of where the kind of map, map fits into things. So from ebpf, we're going to pass over using that map into user space. So that's essentially what we're doing. Cool. So we'll start out taking a, taking a look at the map here. The code for that kind of map logic that's shared between both user space and the kernel is pretty simple. It's just the message that will be passing. It needs to be representable as C. We need to be able to clone and copy it. And this is a AS sort of thing, which is on the user space side, right, we need to be able to like know that this is something that we can kind of deserialize from the kernel. That's what that there is with the pod. And then also on the user space side, just for being able to see things nice and pretty, we can implement debug, but in the kernel and ebpf land, we can't do that because there's no real concept of standard out and printing there. So that is the map side of things. And now we're going to go and look at the kernel. So with those kernel changes, the first thing we do is declare a map, which is saying that we have this queue, and we're going to, you know, have up to 1024 messages on that queue, and those will be source address messages. And we're going to update that try find xdp helper function that we looked at just a minute ago. And instead of just logging, what we're going to do is check to see if the source address is divisible by three. And then, if so, we go ahead and push it onto that queue. And then we pass. That's, that's basically it. And then on the user space side, we also need to make some changes in that main function. We're going to create a spawn agent helper function. And then kind of just wait the same as before. And so if we look at spawn agent helper function, we have to create the user space side of this map, which is again that queue. And then once we have that, we can just loop over reading off of it, as long as there's items in the queue, and then we print those messages from the user space site. So, with that in place, that's kind of it for our v one of really our fizz feature here. And then let's look at adding a simple update to that right. So this is our fizz buzz feature. I'm going to please you guys kind of know where this is going here, you push vision to the queue if the IPv source address is divisible by three, buzz of divisible five, or fizz buzz of both. Otherwise just return XTP pass. So, then we're going to go and take a look now at the map, and see what needs to get updated there. It's not that much. It's just these additional message types so we had fizz before. And then on the kernel side of things. Again, we have to update our logic, but it's just implementing fizz buzz here, based off of that is what message we send. And if so, it gets sent across just like before. And then other, you know, at the end we just return XTP pass. There's really no changes to make on the user space side of things. So we're done. So that is our next iteration of the code. And now, you know, let's, you know, kind of make it even better, right. Let's make another seemingly simple update, which is fizz buzz fibidachi. Okay. So, push fizz, you know, divisible by three, buzz of divisible by five, fizz buzz of both, except if the remainder, if divided by 256 of the IPV source address is part of this fibidachi sequence, then return fibidachi. Otherwise return XTP pass. So, we're going to go in there into our map, right, we have to add an additional message type which is fibidachi. And then from there we got to go over to the kernel and update our logic again. So, change the logic there. And now we're going to be calling this is fibidachi helper function before, you know, checking to see what message we need to send. And that is fibidachi function, just checks to see if the number is part of the fibidachi sequence. Nice and easy. And then the user space side also stays the same. So we're good to go. And everything's totally fine until three weeks later when our customers are very upset and things are on fire. And we spend the whole day tracing our tails trying to figure out what happened. And that's when we finally get back to this, you know, chunk of code here and we're taking a look at it.   You know, this could be drastically improved. So we're going to, you know, make a fourth version of our app here memoized version, in which we, you know, simply check whether it's against the because there's not really that many numbers in the fibidachi sequence below 256. Right. So we can just check this. Nice and simple. Now, I don't know if you guys notice anything here. But I asked chat GBT to give me all the Fibidachi numbers below 256. And this is what it gave me. But I don't know if you notice. Yeah, are indeed missing. So robot overlords aren't quite taken over just yet. So with that in place, we are now able to go play firefighter, put out the fire and production and solve everything. But like, why, why did I have to get this far right why don't have to get all the way production before we could fix things. Why can't we shift this left. Right, so we're going to take a look at trying to catch these performance regressions in development and what we can do there. And now let's look at benchmarking and even if you have program and rust, because you can't prove, but you don't measure. So, before we dig in too much, we kind of have to understand micro versus macro benchmarking micro I think it was like unit tests. So this would be checking our is Fibidachi function directly. So micro would be doing something kind of more the integration level equivalent to integration tests. And this would be that spawn agent function that we had over on the user space side of things. So first we're going to take a look at the micro benchmarks. There are some options for doing rust micro benchmarking. There's lib test bench, which is built into the Rust standard library, but it's nightly only, and it's currently in the process of some pretty drastic revamping. You have to use an unrelated crate, also called venture, if you want to use it on stable rust, and it's not actively, you know, being developed for like, keeping with what it's doing right now. And as I said, it's going to be, there's a lot of churn, there's a lot of things that are about to happen there. So, keep a lookout, but probably not the best choice. But there's criterion, which is a available available on both stable and nightly. And it has kind of become a de facto standard in the community here. And it is much more feature rich and allows for a lot better analysis comparison of your benchmarks. And then the final option we're going to look at here is AI, which also available on both stable and nightly but it's considered experimental. And it's set from the same crater as criterion. And it does things a little bit differently to single shot benchmarking using cash grind so it counts instruction counts, I want accesses L2 access Ram and estimated cycles. So, this can be really great for kind of single shot benchmarking, as opposed to using wall clock time. Just to kind of keep things simple here, we're going to use criterion. And so yeah, so let's take a look at using criteria and then for micro benchmarks. We're going to go over to this map section. And what we're going to be doing is actually refactoring code from the kernel side into this map section since we're able to kind of access it a little bit easier. And so in our dev dependencies, we're going to add criterion. And then we're going to add our benchmark called source address, and we're going to say harness is false since we're using criterion and not the default built in benchmark runner. And so we're going to implement for that source address kind of message type that we created. We're going to implement a function for it where we just pass in the source address. And here is where we calculate what message should be sent back based off of the source address. And so this is the same exact code as our last version of the application. So if we take a look at kind of the, this benchmark here, we taking a criterion object, or us, or source address benchmark, and we're going to run it for every number from zero to 256 trying to create anyone. And then this is kind of set up here for criterion is creating a benchmark group, and then our main function to run this benchmark. All we then have to do is run cargo bench, and our benchmark to one. And so we will be able to benchmark the source address there. So that's micro benchmarking. Now all done. Nice and easy. So micro benchmarking down here, this is going to get a little bit more complicated. Because we're having to test end to end with the ebpf side of things which means we, you know, can't just move things over from the kernel side we really need to instrument that. And so there's a couple of different ways to instrument kind of looking at ebpf. There's kernel bpf stats enabled. That's both the runtime nanoseconds and the total run count all ebpf programs so it gives you the total amount of time it took and the number of times it ran. It was added in kernel version 5.1. It's off by default. And it's off by default. Okay. Then the next one is BPF tool program profile. And that is sort of like III but on the the kernel side of things so it collects perf counts so there's instructions, loads and misses and cycles and things like that. And so if we were to be using III this might actually be a good compliment on the integration side of things. It was added in 5.7, but it does require BPF tool to be built with Clang greater than version 10. And another one in BPF tool is program run, which is pretty neat. It lets you run a specific ebpf program and then like provide its input and context there, and it will return its output in the data context as well. It was added in 4.12, but only for specific ebpf program types, which I've listed here. And even though XDP is on that list, like most of the ones folks are going to use aren't here and so instead we're going to go with the BPF stats enabled for our example. Alrighty. So in order to make this work we're going to have to make quite a few changes here on the user space side. So in this function we're still going to parse arguments, but now we're going to add this shutdown Boolean, and we will pass that into this helper ebpf run function. And then we just wait this before. And then if we get an exit, then we tell the shutdown that we are indeed shutting down, and then we exit. And so let's take a look at that ebpf run function there. And this is going to return this process structure, which is the process ID, the program FD and a handle to a running process. And so this will become important for being able to both run this as an application but also cleanly started up and shut it down when we are benchmarking. So we get the process ID of what is running here. And then we get the program and load it just as we did before. And then we're going to spawn it as an async sort of process so it's kind of running, you can think on another thread or green thread sort of thing, and using the spawn agent helper function. And then we return the process information. So, taking a look at that spawn agent helper function, it takes in the bpf that we're running in that shutdown pool. And so we create the map, like we did with our previous helper function, and then loop over and get the source dresses and print them out, like we did before. But the only real difference here is that we're checking every so often that we should indeed exit. And that's it. And then on to the custom harness. So, this is basically kind of recreating what criteria did for us and the micro benchmarking. And so we just have a benchmark in the name and a function. And we use this great called inventory to help kind of collect those up. We don't have a macro like on criterion to make our main function. So we're going to do that ourselves. We have results are back of results. And then so for each of those benchmarks that we register with inventory, we're going to go through parse the benchmark name, and then we're going to run that benchmark function. And then we're going to take the output. And we're going to assume that it's in the JSON format that we would like. And then we'll kind of push it onto our results here. And then we're going to parse those results as JSON and then put it into a string, save it to a file. And we're going to go. So, then that's us adding a specific fun XTP benchmark to those that we're going to use. And so taking a look at that there, you're essentially just going to return a 64 bit floating point as the results of however long, you know, it takes for this benchmark. And so we are going to need to spawn a new runtime, create a new shutdown. And then with that, create a copy of it and spawn our process very similar to what we did on the like user space, kind of main program side of things, right, using that run helper function. And then we're going to do some work, like go into a cool website like venture.dev. And then we're going to get the stats. And then we'll finally shut down and return those stats. And so taking a look at the get PPF stats. We are going to go into the PROC FD info file for our specific process and the file descriptor that were given for the ebpf program. And we're going to read that file and then just basically loop through until we find those runtime nanoseconds and the run count that we wanted. And then if they're both there, we basically just calculate the average runtime. And otherwise, we return zero. So either we get the time and return it turn the average or or we don't. So there we go. So now we're actually can run this and, like I said, BPF stats enabled is disabled by default. So we have to set that to one. If we want to be able to do this. And then once that is enabled, we can build our ebpf binary, or it's not quite binary, but code that we need to be able to hand the kernel. And we CD into our user space side of the code. And because this has to run ebpf, we need to run it with elevated privileges, which is why we're sudoing here before we basically run what's equivalent to cargo bench. And then the output of that is going to be this. And that results here from our fun xdp function in JSON format. And we basically have that value there for the latency. All right, so that is the user space side of things. And that's it. And those are all the changes that we need for the micro benchmarking so a lot more work here on the user space side for creating that that custom harness here. And maybe kind of a lot that is a how to build a REST custom benchmark harness hidden here inside the talk as well. So that is the macro benchmarks, all taken care of. And so now that we've benchmarked our ebpf program, we're going to kind of look at continuous benchmarking, which is sort of comes to this question like, does your software performance matter right. Well, which I think in most cases with ebpf. And where are use cases is the answer that's yes. And like, but looking at more generally like software where performance matters there's kind of a scale. Right. I like to think about it. You've got kind of performance criticality here. Ascending going down. And there's enterprise applications where, you know, the buyer is not generally the user. So the performance of it doesn't really have to be all that great. Then you got kind of more business level applications where, you know, it's sort of nice to have and consumer applications where, you know, folks make churn. It kind of doesn't work but it can be, you know, like a React webpage. It's fine. It's kind of farther down here and kind of a bit of a gap away. You can have database software and kind of library software and system software, which is definitely where we are with the kernel and ebpf kind of sits somewhere in between here between the library and system software. And so that's definitely us. And then how to track the software performance. Right, because not all software has access to production. And again, you got another axis here. And if you think about this is like access to production at the very far end down here, you have Dynatrace, Sentry, New Relic, all these like Datadog, these APM tools that do give if you have access to production and let you instrument everything and see all that you want observability and things like that. But for the folks who either have very limited or no access to production, there's not really all that many good tools for doing this. And that's kind of where better fits in. So the idea here is that instead of waiting for things to get all the way to production, which is too late. We've shifted as far left, but then running things in development. The problem is that that's local only in manual, and for the same reasons that we started running integration tests and CI. We, in like unit tests. We also want to run our benchmarks there. And so that's the idea behind continuous. So venture is tool open source tool that I built in order to be able to do this. And it's both self hostable and there's a SAS version at Bencher.dev. It has multi-tenancy, multi-language support through adapters. So you can use things other than REST with it. And then it has statistical thresholds and alerts.  And then it also integrates with GitHub actions to make that really easy. Kind of the idea is that to track your benchmarks as we went through those versions of our application, right, the first couple were kind of fine, maybe add a little bit more latency. And that final version is really what, you know, blew the top off and should have generated nowhere for us. And that's about detecting those performance revisions. And so, like, this is actual dog fooding here with Bencher being able to kind of go in and detect performance anomalies here. And so, yeah, this is an example of this statistical threshold set there between the main branch and a merge request branch. And the way this works with detecting those anomalies is again using kind of statistical thresholds. So as you're kind of like get those first couple data points, they're going to be here near the mean, essentially. And then that last point is going to be out here farther down the distribution and essentially where that green section ends is going to kind of be the cutoff of what is acceptable in the distribution function. And then you generate alert when you exceed it. So with continuous benchmarking with criterion, trying to do this with Bencher. It's pretty simple. You would just do Bencher run and you get the use the adapter for us criterion.  And then you can pass in the air flag if you want, which is the thing that will make it fail if a threshold is exceeded and alert is generated. And then on the macro benchmarking. You can do Bencher run again. And then this adapter is going to be JSON because we created that JSON output. And you're going to read in the file, and then run that command. And again, you can have it fail in CI, if there's a problem. So that is what kind of venture allows you to do to catch performance regressions in CI. And in order to add it to GitHub actions nice and easily, there's actually a built in GitHub action to download it here. And, you know, if that branch exists, it'll it'll run there's a bunch of documentation on the venture web site about how to set this up and how to get things integrated with GitHub and be able to comment on your pull requests. And you can go into that more, if you're interested. So, in summary, continuous benchmarking like detection leads to the prevention here. Production is too late. Development is local only and continued benchmarking can save us a lot of pain. And also don't reinvent the wheel again. This is a prior prior section of folks who have had to reinvent the wheel here because there hasn't been a like easy to use readily available tool to do this before. And so everyone just kind of has rolled around. And so yeah, so I'm hoping to save you a lot of work and make that easy going forward. This is super high dimensionality, which is kind of one of the things that allows it. I feel to like offer to you be able to use it for just about any project that cares about this, because it compares things both based off branch. So such as your main branch, the test bed so if you're running on like GitHub actions that'd be like Ubuntu latest. Specific benchmark in our case like fun XDP. And then also the specific metric kind so like latency. So here is an example of running four different benchmarks and comparing them all. They're running on the same branch test bed, but they're different benchmarks and the same metric. And then if we wanted to kind of compare instead to something like the develop branch versus the main branch. Then we can also slice and dice that way again so this is the same benchmark but now we're comparing the differences in in the branch that we're running this on. So this is helpful both for kind of looking at things like this with comparisons but also when you're creating those thresholds, a lot of the other tools that people tend to make are a little simpler, and that they can't handle this difference and so you say can't like separate out and say we're running on a Windows box or Linux box on or like a Mac here for the test bed, or even if it's both on Linux, you know this one's arm and this one's x86 and things like that. And so for a lot of folks they do want to be able to do those comparisons and so this high dimensionality is super important within venture. This is a public perf page. So this is a for public projects, you're able to kind of be able to direct people and when it comes up comments on the PR this is easy way pulls up for you to be able to see exactly what's going on and track things over time, and kind of do those slice and dice comparisons based off the dimensions that I was talking about. This is publicly available you don't have to sign in kind of thing. These also the ability to embed your perf blot so if you want to kind of stick it in a bigger site or somewhere in your documentation or things like that. You can also, if you just hit the share button. It's, it's there to be able to embed it. And there's also static images versions of these so if you want to be able to embed them to your read me, which I think is pretty nifty. You can't have dynamic content and GitHub read me, but this all the dynamic stuff is done on the back end. So it's a static image with a very low refresh rate, basically zero from GitHub point of view so every time they pull it, we regenerate this image for you of exactly what your, your plots look like. So, so that's kind of continuous benchmarking there with venture. We kind of our overview here we went over creating a basic ebpf program and rust. And then we evolve that ebpf program, right, add fizz fizzboss and then fizzbuzz Fibonacci. And then we finally fixed our performance regression there with fizzbuzz Fibonacci. So how to benchmark our programs, both on the micro and macro benchmarking to try and prevent, figure out how we can prevent those performance regressions in the future. And then finally continues benchmarking and when it's applicable when we care about performance. And if so, what do we do about it, as opposed to just kind of rolling around solution there. That's been run fast catch performance in ebpf with rust. Again that's the repo link and if you don't feel like typing that much you can just go to venture.dev slash repo, and that will just forward you there. And yeah, if you guys like the project. Please do give it a star it actually helps. People use it to, you know, kind of think about projects so yeah. Everybody said you can use this to benchmark any project. What is a project largest part project you are others used to benchmark. Yeah, I think the largest public one would probably be Hydra, which is a database is using it. There are a few others who have picked it up and they're using it there. And so, I guess the key question is like, are you, is the concern in size there like number of benchmarks that are being tracked or. I was just looking to see the kind of projects or software that could benefit from venture and how it's wanting to learn more about how it's being currently used. Yeah, who is using it. Yeah, yeah. I'm, I am using it. So yeah, there's quite a few teachers who have signed up and used it public projects are probably the best place to kind of go and look for that. But yeah, I know for sure like the Hydra folks might be kind of a good example to talk about and using that to track their database performance. Yeah. So it tends to be kind of like library code and those sorts of things where there's no production environment for them to kind of instrument into. So if you were to, if somebody were to look into how they could use it to benchmark kernel per se. So any of that would be the libraries. Is that correct or. Yeah, so the key, the key part is like venture is benchmark harness diagnostic. So there's these adapters that are there to make it easy for folks who kind of use pretty standard benchmarking harnesses to be able to they don't even have to actually in the docs here I said that we're specifying the harness. You don't even have to specify it venture can just know it. It kind of rocks which harness you're using without you telling it. And. And so, there's that, but then if you say we're doing kernel development and you have a special benchmark runner or harness that, as long as you can output that Jason, that venture understands. You're able to use it. So it's kind of completely agnostic to exactly how the benchmarks will run. So, yeah, so like the way that the project works and things like that. The way I designed it specifically is that this could be used across basically any project. Pick a benchmark that kernel benchmark and then hook it up with venture and make sure that it venture can understand the output Jason. Yeah, it's a, it's like, it's a pretty simple. Format like, let's go to the docs. So this is basically the format here, like as long as it's the benchmark name. This is the metric kind so latency. And then these values, the lower and upper value that's, if it output something in that format venture can can take it. Do you all of its magic. That sounds good. So let's, to tie this kind of back to the talk here that custom. So, like venture obviously wouldn't have a adapter for a custom benchmark harness that we did for macro benchmarks. And so that's why when we had this here, we output it. This is the benchmark format here, which is the BMF the venture metric format. Does that answer your question. Yeah, yeah. So we, we have some benchmarks we run in a kernel performance benchmarks per tool runs part of the kernel. Yeah. And that outputs the, so if we wanted to use venture to do we automate some of that benchmarking, and then, then we have to hook it up with the output to be Jason. So, you know, bench is what I'm thinking about when I'm thinking about this is a bench that in the corner that we use. Okay. It should be, you know, like fairly simple to get it to play nice together. So, and then yeah, it generates, if it generates that then that could be made the benchmarks could be run on the kernel we do run them. And people do run the benchmarks on as kernel new releases are coming out. Yeah, it tends to be folks like right now, it's either it's manual, or it's kind of like a homegrown solution. You know, the kind of like, like Russ perf is an example of one where they've, they've kind of. I've taken a lot of inspiration from them, but like, you know, it's just for the Russ project. You can look using it for anything else kind of thing. So, do you guys have, like, Derek, is it time to just be someone like running it on a box they have at their house to compare it or do you guys have like a hardware setup where Well, there is the kernel CI that we have, they run primarily tells, and I don't know that I don't know if they run performance benchmarks, I tend to run them manually. At the moment, you know, when I am looking at a performance problem, or something looks off or I want to get a generally see what's happening and per developers kernel developers they run them. Of course, they're continuously using them and running them, but and then various users run them as well. There is a user base that's on some but most of all of it is manual. Okay, it would be manual. Okay, who would be the like rate rate folks, albeit plumbers. Next week. Would there be any folks that would be good to chat with about that. So there is, if you're going to be at the plumbers, I'm going to be there as well. So that there is the performance stocks that are happening. Yeah, for either I don't think there is a micro conference, but there is a per. Talk, either at the Colonel summit or referee track, check those out. I don't remember I do remember seeing a perf talk in the list, I am on the program come to you and I'm trying to remember if I have seen any I think I have seen a one or two perf talks. Okay, that would be on the performance like I mentioned the perfect bench tool that we have. Yeah, perfect. The perfect tool does a lot more than benchmarking. It does a lot of other things but benchmarking is one of its things. You can just pick up the. If you have a kernel repo, and you're running the latest kernel you can you can just go just say perfect bench CPU for example it'll run benchmarks on the CPU or perfect bench mem or perfect bench all it'll by default run CPU memory and I think it'll run it on the schedule. So, that sounds really good. Okay. So, yeah, look, look those up on LPC. I think it's scheduled out you can take a look and see if. Yeah, I think the schedule is there. I started taking notes on on what I might go to so. That sounds good and any questions from anybody has been a very silent group today. Any questions on the importance of getting led not letting these escape to the product and any questions on how expensive could it be if it were to escape to the to release after release or finding benefits of finding it now. Yeah, it's like 10x the cost or something. Yeah, the cost. Yeah, depending on depending on where the problem is really in some cases the problem could be depending on what you're testing right if you if there is a testing off you have a product that has the vertical of hardware software hardware and Cardinal and all of those and the problem could be somewhere in the summer in in the place you can't really easily fix once it's deployed. Yeah. 10x. Sweet. Well, the folks kind of want to want to take a look at venture here. Yeah, go ahead. We have time. Okay. So, yeah, if we go back here. This is me dog fooding venture with venture. So, this is the. Okay. And so this is the latest report, which is from when I pushed things up on Halloween to release here and these are just kind of those adapters that I was talking about. We track those. And so these alerts here are me playing with how tight can I get the threshold on GitHub actions just using the default runners.   And, you know, and somewhere from there to kind of what I'm doing here which is just running with the default runners on GitHub actions. So if you kind of look at this boundary, you can see me sort of playing with it around over time and still trying to figure out where, you know, even though this code hasn't changed it, you see kind of the variance here in the runs. What, what's kind of the best way to handle that statistically. And so like what, what's the value and how much, what's the noise threshold there. So yeah, that that's kind of a thing. I'm also looking at with venture long term to make it easier to do those dedicated runners that might be a little less noisy than CI. So if anyone is interested in that please reach out. I've been working with some of the kind of higher profile rust projects to that are also looking at this to kind of see what they want to do there. And so there's that. And so yeah, if we kind of look at just take a look at one of these alerts, even though this is a false positive right now because of the, you know, playing here with the constraints but if it wasn't if we wanted to kind of look at it, we can come in here, and it will tell us kind of where it occurred, and the kind of upper boundary that was calculated and what our actual value was. And so that we exceeded it. And we can also dismiss that alert. So then if we go back to the per plot. The alert, I guess we'll, we'll refresh here, get easier that alert goes away. So you can dismiss alerts based off of how you want to handle them. And as you can see, I have a bunch of alerts because I've been playing around with it here, as we kind of set things up. The metric kinds are essentially the units. So these are reports which is the running of all those benchmarks together, which is what we were just looking at in the plot. So metric kinds, by default, each project starts with latency and throughput. But you can add additional ones. So if you're using something like that I I adapter, there's actually a built in adapter for that and venture, and it'll create the four metric kinds, the kind of, you know, instruction counts and things like that. And then before you, or like with the kernel, if you're using perf bench, then this will be, you can add whatever your kind of standard units are that you you want to care to calculate there, if it's something different than these. And then with the branches as you can see, these are for merge requests, I did not create these these are the way I configured things in CI these are created from my. branch that they're branched off of so this happens to be develop. And so the, the branch here basically does a shallow copy of that branch and then any new metrics that are added on top of it are just for that PR branch. So that allows you to try and catch those performance regressions in CI by, and like on PRs before they actually merge into your major branch, main branch. When those branches get cloned, it also clones the threshold, which will take a look in a second. So there's test buds, which by default you just start with local host. This is my bun two latest test bed, which is used for running things on GitHub actions. And then the benchmarks. These are the benchmarks that I have set up so far with venture, which is just for those kind of, there's more adapters but this is just kind of showing. The Azure adapter, which allows you to not, it's the default. And so if you don't specify an adapter. The parsing is smart enough to figure out which adapter should be used and use that. But it means it's a little bit slower. And then these are the thresholds that I was talking about. So, essentially, if something is a merge request running against a file or main, it will essentially clone the threshold that is there. And then use that in its processing. So this is the example of that cloning students T test with a point nine percent point nine five percent cut off there, which means like below in that cumulative distribution function with a maximum sample size of 30. So that's the thresholds. We took a look at the alerts. Here. And so these are the alerts for the different benchmarks that we hit that we kind of looked at here a minute ago. So yeah, that is the And like being able to use statistical thresholds, you can also visualize, visualize the lower and upper boundaries here. It's probably easier to see if we're only looking at the single plot. So if you do have those lower and upper boundaries and much variance, you can also visualize those there's ability to right now this is based off of date. So this is days in October here, but you can also change the x axis to be based off of the version number. And so there's just kind of increments with each version you push up. So if you just kind of want a more normalized view of things as they kind of go through here. That is available there. So, yeah. Also for folks who are non native English speakers. I have translated the docs using GPT for into eight different languages. So, yeah, hopefully one of those might might be more useful for me. So, I'm just looking at the venture dev. You have a self hosted and venture cloud. So what does venture cloud entail if a project wants to use venture venture cloud. Yeah, great question. So on the venture cloud is essentially the same thing you can, it's the venture self hosted, like I'm just running it for you on venture dev. And so the, the just thing is you don't have to worry about running it. And, yeah, that's, that's kind of the key pieces. I am trying to, you know, pay, pay and maintain it by charging for private projects. But if you're a public projects, then it's completely free to use. So, so that is the cloud one that you're talking about. Um, that's both it's that's both. So like, but I mean if you put it behind your VPN, and no one can access it, you know, like, as long as you're okay with folks who have access within your VPN, being able to come to this projects page and take a look at things, then, you know, that's, you're fine. Okay. That's all good. Thank you. Any questions. Whatever it today. What would you would do if there are developers that want to get involved in your project, either by improving venture and such. What would they do. Yeah, great question. Um, so I have a couple of good first issues tagged here for different adapters. If any of those are kind of pertinent to folks, probably within this community, the like hyper fine adapter might be a good one which is like CLI tool for like benchmarking binaries and things like that. Which might be a good one to start with. And it should hopefully be very easy to get started contributing. I have a dev. You can use both a local dev container or a GitHub code space to start developing. So if you don't want to have to kind of download and install everything, you basically just click those. And I have script setup that will auto set up everything you need inside of there. So that dev container will just create a that works on any operating system.  But but if you just general venture things than yes, this this dev container should spin up and it also when it spins up, it also spins up the development back end server and UI. And so you're kind of like ready to go out of the box with with an example of the kind of PR comment that you'll get with venture over here. Yeah, so like if you have an alert, this will pop up and you can use a plot and view the alert, things like that. Yeah. Thank you very much. Awesome. Well, thank you guys so much. Yeah, I can wrap us up. Thank you ever and show up for your time today and thank you everyone for joining us as a reminder this recording will be on the Linux foundation YouTube page later today and a copy of the presentation slides will be added to the Linux foundation website. We hope you're able to join us for future mentorship sessions. Have a wonderful day.


Russia is arming civilian self-defense groups in Belgorod Oblast. The Russian government is continuing to arm the so-called self-defense groups in Belgorod Oblast, which borders Ukraine. Belgorod Oblast, Governor Vyacheslav Gladkov, said that civilians were issued with cars, quadcopters and anti-drone guns. Now each of the eight battalions has five off-road vehicles, additional digital car radios, quadcopters and anti-drone guns. We gave the first weapons to the most prepared fighters, he added. Gladkov said that the use of weapons will be agreed by regiment and battalion commanders together with the Russian Defense Ministry, the Russian Guard and the Federal Security Service. He added that the Russian state Duma is currently considering measures that would allow it to arm the entire regiment with firearms. Russian media reported earlier that members of the so-called territorial defense in Russia's Belgorod and Kursk Oblasts were issued with weapons and military equipment by the local governments. Subsequently, the governor of the Kursk Oblast, Roman Starovoy, also said that local authorities began to distribute weapons to the so-called voluntary people's squads because of the alleged threat from Ukraine. He assured civilians that all the conditions for storing weapons have been created. The first batch is already in the base. In the near future, the number of distributed weapons is to be increased to 300 units. Dmitry Peskov, the press secretary of the president of Russia assured that the gun control will be maintained. Weapons are never distributed in an uncontrolled manner. All control mechanisms, of course, must be very carefully implemented and foreseen. Actually, there is no doubt that this is the case, Peskov said.


Hello everybody, my name is Fausto from Cybertrain University and I want to invite you all to come today, which is Wednesday at 8pm Eastern Standard Time and tomorrow if you miss it on Thursday at 12pm Eastern Standard Time, how to follow the big money. This is a great opportunity for you to kind of see what it's like to trade today's volatile markets. You've seen how volatile the market is, especially with COVID and it's been a lot of great activity in certain stocks. Stock's going up 200, 300% in one day, but there's only one way to know that. You need to follow the big money and I'm going to show you how to use something called Level 3 and Level 4. These are some of the great tools that we use that help us watch and monitor these ECNs and how these brokerage firms hide their orders. But if you want to know how to do it, just click on the bottom, like it, friend us and register and we'll see you there. Enjoy the rest of the presentation.