Thank you for this. Yeah. Hello guys. This is Vandana. I'm a Linux kernel engineer. I've been working with Linux kernel like more than two decades to really 24 years and I've worked on various subsystems as a more and not just limited to device drivers. In case of device drivers, I've worked with embedded systems, SOC, on-base developments, as well as networking device drivers, network subsystem storage, as well as the other subsystems related to memory management and security as such. So here in today's session, what we are going to do is going to look at the kernel tracing functionality, some of the, how to look into the kernel activities that are happening, but then using the EPP framework as such. So to get started with, let me give you an oral idea of what this today's topic is going to contain and how we are going to proceed with this. So the agenda is to start with the basics of the EPP framework, understand the architecture and then look at how this framework is more valuable or useful compared to the existing architecture, existing tracing functionality as such. And the way it has been used for doing the dynamic and the static instrumentation, particularly as the session is more for the kernel processing, we'll look at the kernel instrumentation techniques and then we'll look at from time to time. I try it and to use the tools that are part of this BPF tool chain as such that help to demonstrate some of the features and functionalities that can be traced or the data can be collected from the kernel using these tools and tenderly use some VCC tools or BPF Trace tool as well as the programming using the BPF programs as such. So the oral scope and expectation you can have is the high level introduction to this architecture and all the processing can be done with the help of this set of tools, as well as from the programming as such. And also provide the useful pointers for the information that has been referenced as such, okay. So to start with, let's have a bit of background of BPF that is the Berkeley Pocket Filter Framework, which was originally being created to speed up the filtering of the network packets as such. And eventually, in 2014, Alex designed the whole of this BPF frame with the language of the BPF framework, adding extensible functionality to it, so that it turned into a more generic general purpose existing engine, and which was not limited only to packet filtering but more functionality was being added to perform variety of operations, such as profiling or debugging or enforcing the security policies as such. So some of the features that got included with the efforts that has gone into 3.8 in kernel included like an introduction of this BPF system call that would be used to load the various BPF programs that would potentially do different functionality profiling debugging and looking up at the various kernel functionality as such. And based on those different hookups, there is the set that would be added for different types of programs, not limiting to the packet filtering as such. And that's how the BPF framework evolved what we call as a, now we call as extended BPF functionality, EBBF as such. So basically, BPF or EBBF, they are like interchange in the being used as such. So, EBBF is basically a revolutionary kernel technology that allows the developer to customize or customize the code that once when it is loaded dynamically into the kernel, and that would help, that would help. That would potentially change the way the kernel behaves, means if you want to perform something additional as to the kernel code that is there loaded dynamically, then that can be achieved through the BPF programming as such. So basically it provides a more flexibility to add to build up the tools or to customize the policies as such. And that is dynamic runtime as such. So, basically the tools that are based on APBF, they are, they are being developed and they have been evolved to do the, potentially observe the events, any kind of events that are happening across the kernel, as well as applications, either on the virtual hardware or physical hardware or even in the patronized environment. Basically, in a short, what we can say that BPF is a technology, the compose of the instruction set that is the BPF program that is run. In the storage objects, how the information, if it is collected information, and if it is shared between the user space and that information, and the set of helper functions that helps collect this information are to write the policies or how that behavior kernel functionality can be changed. And there are the set of library functions that has been defined in define as such that means that those things can be taken help by this life and functions. So basically, it can be considered as a virtual machine running in a totally isolated environment as such. That's why this eBPF is important. For that, let's look at the difference between the basic behavior kernel and user space in Linux. As we are aware that the application they run in the un-privileged layer called user space means which does not have to direct access to the hardware as such. And in that case, the applications make use of system called to make the request kernel to do its work on its behalf as such. So really that might be like accessing the hardware or changing some policies as such so like reading writing file or sending receiving packets or accessing memory or allocating memory and several other activity. So that application from the user space that is done by invoking the system calls. Whereas like the kernel runs in the privileged layer and acts as a bridge between the application hardware basically it provides the abstraction of the hardware, physical or virtual as such. And that is done through the set of the system subsets or layers handling all those different resources. The use of this subsets typically allows for some level of configuration, level of configuration based on the needs of the users as such. So, like, if the desired behavior has not been configured in the kernel as such, then or if it is not present as such or it is like some changes are needed to the current functionality as such. So the weight can be done and the kernel change would be needed to add that support. And it has been particularly defaulted to the two approaches that was adding the native support that is adding the change into the kernel, or like adding the change in such a way that it can be dynamically loaded, thereby taking the functionality of kernel modules as such. So each of each of each of this method has its course pros and cons as such, like if there's some change has to be added into the kernel that is an eight make it as a native support as such. Then the change has to be implemented and then it goes through these iterative cycles of reviews with the kernel community means you have to convince the current community what that change is necessary as such. If there is several iteration and waiting, then that change might get would get reviewed and accepted and then available in the upstream. So there is like the waiting cycle, if the change has to win as a native as a message. Another option is the kernel module where you make the changes as form of formal module. The downside is like you need to maintain maintain that change and fix like as in when the kernel changes or the apis that have been used that changes as such. And also there is a risk of correcting the kernel your system, if it is not been implemented properly and having some security issues as such. So in that case, that option might not be always workable as such. So that's where we have this another functionality that comes to the BPF rainbow, which allows the behavior of the kernel of which allows to add this functionality without making changes to the kernel sources or loading the kernel module. And taking the help of this functionality, we would be going using this functionality to do some tracing of the subsystems as such. So here, this is a block diagram or simplified way of projecting how this architecture can be viewed as such BPF architecture so here. The board that you want to load load and running the kernel they they have been implemented as BPF programs and this BPF programs can be loaded dynamically and executed in the kernel as such. So when this programs are done loaded and executed there are there are the other components of this framework that ensures that ensure the security and safety of the program that is to be loaded and runtime as such. So that's where this BPF verifier component it does the work of ensuring the programs are safe and cash free. And it does does some work of checking whether it's breaking the security policy in some way like the program would be rejected if it is deemed to be unsafe. Such as like some of the features that is considered as unsafe is not as having the program has in for like a loop or loop as such are backward branches and those kind of function to the program has that kind of functionality and the verifier would reject it as such. Once the verifier ensures that the program can be loaded and that they have the other component is the engine compile just in time compiler basically does the work of translating the micode into the machine level instruction that would be executed as such. And once that is done, the program is attached to that particular function or particular event as such and as and when that up the particular event offers them that program would be triggered as such. And like when this programs are being designed, they will be designed to provide some functionality as well as collect some data as such. And so for collecting the data or this one of the cons the constraint construct been developed is called as the BPF maps as such, which basically provides the facility ability to store the collected information and share between the user space and the common space. And based on the requirement, like there are different types of maps like hash tables, arrays, buffers, stack, and stack, trace as such and of course there are a lot more. So you are in this case this example is this but particular by a program. It is to be attached for but this network related system calls as such. So here in this case the programs get attached to send message and receive message will see how this has been done with the examples aside. And whenever the applications makes the socket program, so network programs that calls the second socket system balls in that case that particular BP program would be executed as such. So, yeah, so here we talk about loading the BP program which is done dynamically as such. And this program can be loaded and removed from the kernel dynamically. And once they're attached to the event, the they are triggered regardless of the cause of the event as such. So it can be caused by any application or maybe a process getting executed or maybe process communicating with each other. But with that particular event has been triggered that BP program would be executed. And the programs, they can be attached to at various places at various hooks as such to give some of the example is like the system calls are any kernel functions as such. It can be a particularly if you see system call like the entries system call entry or system call exit. Similarly, for the other kernel function entry exit can be the disprograms can be attached. Again, another is these trace trace points, the various trace points that are there. Then the different subsistence particular network events or other events of the subsistence events. This BP programs can be attached as such. So to give an example like the BP program can be attached to the open system call edge point entry of the open system. And whenever any application tries to open a file onto the system, then this BP program would be executed as such. The example as for this diagram to say that the BP program is attached to the exact system call. Like whenever a new application is executed, wherever it is, then this program would be executed and based on what functionality has been implemented. And this BP program that would be executed so basically like one way is like using the health function and the various constructs. The arguments passed to those system calls can be collected on the credential process credentials can be collected. And from the availability point of view, the system observatory can be done like what's happening on the system and who is doing what and how the information, how the parameters are being passed. So all that thing can be collected in the collected through this BP program on as needed the changes could be made as such. So basically, this functionality. This, this leads to one of the greatest trend like observable observing our observatory or security tools that the tooling that can be done using the BP framework as I so basically what it does like it gives you the instant visibility or everything what is happening on to the system as such. So the BP framework is one of the latest framework that has been used for this observability and the security tools as such that you can enforce the security policies using this BP framework. So, oh yeah so this is the way the dynamic loading and programming can help to get you the get you into the system into the system know what is the health of the system or what exactly the system is up to as such also like fine figuring out some debugging issues or performance issues that can very well be are done with the help of this BP of programming as such. So here in this case before getting into the details what we would we could come run back and forth with the examples as such. At this point you might not to get all this. Each of the function that has been used or it might be like you are already using, but like to start with what we'll do is like just go through a program or hello world program hello world. We will return in a python python code as such and how with the help of this python code how the BP program can be attached to some system call and then see how the system call can be invoked as such again. So basically, the python program can be considered as, let me say it consists of two parts as such, like, yeah, like it contains the actual BP of program and then the, then the python code that does the work of loading the loading the BP program into the kernel as such, compiling and loading as such. Okay. So here in this case. So just cover slide and then go back to the code as such. So, as we said that it consists of two parts that is the BP program that will run in the kernel. So it's just basically nothing but a C file a C function as such and it has a simple help function that does the work of printing just similar to print K as such so here it is the BP trace brain key doing hello world print as such. Okay, so this, all this all together is your BP of program so which is a C like functionality as such and the rest of the part is the python code. So the python program that does the work of loading the BP program. And basically the BP of tools tool chain that is BCC one of the two that does the work of for compiling the C code before executing it and then loading it into the kernel. And then it attaches to a particular event as such so you're in this example taken exact system call as such a proof function. Okay, so you're looking at this diagram like the year this is for a program python code which and execute that what it does it's does the work of loading the BP program on to into the kernel. And that loading and attaching it to a particular even that would be triggered as such. And whenever that even is triggered whenever the applications as such makes a call as such like exact system call application new application is executed. The exact system call gets executed and every execution exact system called the BP program will be executed. And whatever the functionality it has done now here in our case it is nothing but the print statements, it prints this string as such, so which has been collected in some kind of offer, and which is passed back to the application and it prints that on the screen as such to hold. Okay, let me go back to the pool as such. Okay, so here this is the be a redefined the BP program and then the rest of the Python code does the work of initializing this BP program by creating the BP object as such. And then we choose to attach this BP program to the exact system called event as such. So here these are the set of helpful function that have been used as such. Here the exact system call is to be used. So here what we do is we tend to make use of the health function to get the function corresponding to the exact system call as such. So, and then this exact system call function name. We try to create, we try to attach it to the attach this attach the function, the function that we program that has been defined to this exact system call event as such. And that has been done using the cable cable functionality as such. And once that is done then the print print trace print does the work of outputting the new cross trace put alphabetic kernel as such and we could eventually go and show up on this screen as such. So basically, to go and say, basically the output is stored into a seed of wireless. So this the running running this program. Depending on what the load is there what is happening on the system what the different types of application that are running. And whenever they make the exact system call them that is this event will be triggered as such and then you will see this hello world string as such. So here in this case, what we'll do is just execute some command as such BS command and then we see if we are getting anything printed. So here we see that there is one more print as such. And let's do this. Again, you say so you're in this case when we are on the another terminal, some application is being a process is been executing so that execution happens to the exact system call as such so whenever that has been tracked through this Python code as a Python application which loaded the program to trigger the events whenever the exact system call is executed. So when we are doing that on another terminal, we get to see a log be generated that through the printer statement as such and you get to see this hello world string as such as part of that. And also you get to see it's not just the string that is coming but there is on a other contextual information as such particularly like the PID process name and the process ID as such. So, basically, this output that is coming that is actually coming from the from that file or from the terminal log that is the that that basically what it does the Python that goes and reads this trace pipe. So the file as such the all those traces are then fetch from that as such and another phone information we can note down as the other contaction information so that is basically that is basically that is added by the current tracing framework as such infrastructure as such and giving the information process ID and process name and other information so it's not very specific to BPF as such but it's a part of tracing tracing framework functionality as such. So this is like a very simple hello world kind of application we just control see and terminate it as such so you define and so yet the programs are defined as a C function and it has been passed to be a baby of object as a string as such so you just pass the whole of this function creating the baby of initialize the baby of and then that was a function of functions, you would go and attach the BPF program to this particular event as such that might be set of system calls, then trace points for events and other events as such. And basically, yes, particularly using the key proof functionality, which is like, which is used for doing the dynamic instrumentation as such, and then there are other ways, the events can be attached as such. So this, like, getting started helps with this. Now, one more thing that this trace pipe, here actually you can open this file and see that you would be able to see the trace output as such so basically this does the information that is a string based on simple strings as such. But at times, the based on the functionality and how the BPF programming has to be done to do a lot of other operations as such. Now you're what we are doing anything as such but in real use case, a lot of data data is being shared with the from the kernel to the application and based on that data, the corresponding actions are to be taken so in that case the stress pipe might not be a sufficient mechanism for getting the data as such. So therefore this BPF maps are being used as such, which are, which is a data structure that is being accessed from the user space as well as kernel space from the programs and as well as from the user space and basically provides the key values tools as such. So, if the data has been shared has to be shared it's like in some structure data structure as such, maybe like a socket socket structure or network packet as such so in that case, you cannot use this trace pipe it would be an efficient for tracing or scanning that output as such so by making use of the BPF maps. And that that information maybe a whole structure or nested structures and if related information. They, they are being shared that keys this BPF map is a useful constraint to be used and based on the requirements and the definition there are different types of maybe maps that are implemented. You can figure out that in the BPF dot hr file, and it was commonly that has been used are the hash tables suing buffers areas, various others as such. And also like there are other types, the different types to be defined based on the type of operation like five foot five foot use tags and let's reuse the LRU data source as such and also like based on the different subsystems or information specific like particularly for the networking as such. There are different BPF maps that are constructed for doing this sort of a storing and sharing the socket and network device information so this is just an example of a socket and that will do the sock maps and the dev maps as such. So we'll see some of the examples how this maps are being used as such. One minute. Yeah, is there are several questions in the chat. Okay, probably would be better answered. If you can take a break. If not, let me know we'll we'll get to the question later. Okay, okay. Yeah. So you will, I think the first question is I think that a lot of conversation happening here. Some of the questions are answered. Okay. I did not look at them. No, that's okay. That's okay. I am watching but I thought that I would wait for a good stopping point and kind of found one. I thought but okay the first question is from Praveen. I would like to know how if EBPF can help when applications need DPDK networking like some user plane might need DPDK. Can you repeat I often do. Okay, so if I would like to know how if EBPF can help when applications need DPDK. It would have been great I think Praveen could expand what DPDK is but I'm thinking that it's device, some kind of API. DPDK networking, like some user plane might need DPDK. Yeah, DPDK is provides the user space networking programming as I told together so basically I think for the DPDK we have for that from the networking point of view there is the XDP interface is there in part of the BPF where the network traffic acceleration parts are data parts are accelerations are being done. So, BPF has the circle for that as well. So, most of the things that is related to networking, I think that that goes to the XDP interface as such, and that can be leveraged in the DPDK stack as such. Hope that answered your question Praveen if not, please post a question in Q&A or chat. So, it's next question is does EBPF assume a physical machine or can it run well on virtual machine too I think this question has been answered. Yeah, yeah, back and said it can be done on both. As long as Cardinal supports it. So the other question is, does someone know the difference between Dtrace and EBPF, I think this is also answered that Dtrace is only available on Solaris and Oracle's and it's not in the main line Cardinal. I think that's been answered. Let me see. It looks like a hook environment between user mode and Cardinal mode, how does this work with LSM, for example, Cell Linux. So I think this probably in reference to the Python program that you're doing hooking it to the XDP. So that is the question there about specific to that. So, particularly for LSM hooks, there is the new constraints that has been added in the BPF framework that is a BPF or LSM, which can help you to add the hooks into the LSM, which can help you to attach the programs to the LSM. Hooks as such. And that is one of the sub component of the BPF. This is I think the latest governance support start. BPF LSM hook. Yeah. Okay, so that kind of makes it more secure, especially Cell Linux kind of use BPF methods to trace and then attach hooks. Okay. That sounds good. Would Slice be shared? Yes, they will be shared after the session. And we run a fleet of how are you doing on time? Do you want to continue and take the questions later? One enough? There seem to be more questions. Yeah, okay, actually I've lost the track of the questions as such. Should I stop by and look at the questions as such? No, I can keep going reading questions, but I just wanted to see if you want to. I think if you can read out the question, then I would try to elaborate it now, otherwise it will like the list might grow. Absolutely. Okay, thank you for that. Let me read the next one then. We run a fleet of Linux VMs, thinking of building a health monitoring service based on EBPF, complete with EBPF and dashboards. General idea is to provide a periodic pulse of interesting metrics, Brandon Greg's checklist, plus on demand deep dive on interesting processes. Any thoughts on that idea? Yeah, actually, that is like the current requirement of the current trend that I would say that has been used for the security based product assets, where the security big process where the previous traditional security products based products were like using the functionality that's doing the collecting the security infrastructure, the user space applications as such using the oddity framework and other things as they do help to get the information, but they have some drawbacks or downside that not all the information is available. And another reason for that you have to get into the have to be in a kernel space to get all that information. Thanks to this BPO framework, which helps you to run the application from the user space and collect the kernel information as I can go and look at any kernel function and get that information. So, leveraging this functionality is like a key milestone for the security security solutions as so like giving a simple example like a security application like file monitoring through the security solution is basically like I had worked on one of the previous security file system, sorry, file monitoring system, like the application was user space, adding the events, trigger the events for system calls, the distressive system calls and for checking the check, keeping the check on the configuration files as such and all that. And that oddity framework did help to get the information but had some limitations and at that time not all the information was available. But I think with the EPP a framework for you can get a very detailed information about the system calls as well as the whole of the system or the process execution with along with the credentials and the all that information by using this various interfaces and collect huge rich environment data as such and then that can be interpolated and that can be interpolated as well to check not just the security or also like if there are any vulnerabilities or some attack or compromise has been happening as well. So definitely, it is the very right answer for that solution as such. All right. Thank you. So I think it this regard it regards to sell the next LSTM and so on I think it works. The question is, I think it works with selling X as far as I know, unless you hit something that is blocked by the policy, the default selling X is targeted. So, with the help of a sale next week and enforce the different security policies as such. So, so yes, like if the sale Linux is been unable as such and the sale and it's folks they are at the point. The point of system call entries as such so there is a flow of like where the sale Linux goes and checks as with the, with the particular operation is permanent permitted or not based on the policies that have been loaded. The mandatory action access control has been enforced as such. So here in this case, the BP program has to be aligned with the SLA policies that are there running this like suppose you for running a baby program. So, as a, what do you say at the administrator as a conflict of enforcement that you're doing that you would be doing in compliance with what are the configuration that is there as such. What is the significance of our on the program line in the Python program that you showed. Can you repeat the question. Yes, the question is, what is the significance of our quote quotes on the program equal to line. Okay, yeah, basically it is like it is passing the whole of the program to the Python Python basically the Python program at the back end the by the BP of framework tool chain, it takes this code to see code, and it does the work of converting into the byte code and then which would be going through the BP of framework as such to be loaded as such and this like if you go back to this tag. Here in this the Python code basically to load that code BP of program, it goes through the system called BP of system call as I so that code has to be converted has to be passed which would be would be converted into a byte code from the white code into machine learning machine language as such, which would be attached to the event as such so that for that, that is the convention to specify. This is basically a specifying that this is the seat of the program as such, and then we pass it that whole program as a string as such to the BP of object. And then I think this is more of a comment from somebody responding to another question about hooking. If you using ebpf to look at system calls then it looks like a hook environment between user and kernel. Sure. However, you be a ebpf can also be attached to purely kernel level functions is there anything that you would like to add to that one. Other than the system calls sorry. Let me read it again. If using the ebpf to look at the system calls then it looks like a hook environment between user and kernel mode. However, ebpf can also be attached to purely kernel level functions. Yes, yes, it can be attached to not just not limited to the system calls, but to any kind of functions. Okay, so that's that's correct. And this tracing is helpful as such like, we will not only be tracing system calls as such, but any kind of function that like, it might be developing some kernel module some kind of driver as such and you might want to have looking to the functionality maybe some performance issues or some other debugging issues in that case, that can be done through this bpf programs as such. Or maybe you are debugging a problem and you want to see a particular system call is hit. Yes, or not system call particular kernel interfaces hit. Basically, there are like a lot of a lot of help functions are being defined as part of this bpf framework as such that caters to the different system subsystems as such networking block layer volume volume your memory management. We will see that in the following slides as such. So, one more question here. One concern could be performance impacts if used for regular monitoring, instead of occasional debugging, perhaps that will be covered in the stock. I think you probably will touch on that later one then also do you want to address this question now or you have slides on that. Yeah, or you're like bpf programs like they are been loaded means converted from whiteboard to this assembly or machine level. So, so they are the overhead is significantly very less as such. That means not very a lot of things about like if it has been added into the places which is like been triggered like suppose you're adding the bpf program to a schedule or as such are monitoring which would be which would trigger, which would be called consistently based on the workload as such if the if the work load is heavy and potentially the points have been given some intricate from that particular workload related information done in that case there would be some overheads. So there are some books that are generating that there are some books that are potentially having a more overheads but overall the, the way the bpf programs are loaded and kind of done. What do you say in just in time conversion and machine machine instructions are executed so there are minimal minimal overhead in general. There's another question about the security to have any negative impact to bpf program and also any current lockdowns. So, yeah, so one thing is like this programs they are running from the user space that is loaded so as per the. What do you say the architecture bpf architecture it has this very fire that would do the work of a very fine that the court that is going to be run in the kernel that would not put the kernel in the in the infinite looping or lock state as such one of the way that would be triggered is by having the loop as such so if the program has been implemented that is going and taking a loop as such. Then that kind of food is not to be allowed as such to those some some of those checks are being done taken care by the very fire. Okay, I think one more and after that maybe we can hold off on the questions and let you complete. Could any ABI comparable language be used since he seems to be the de facto language to use like they got even see Python. Yeah, we are coming to that slide. Next. All right. So, at this point, I do we do have questions in the chat and then also a few and a, but I would like to have one and a continue representation at this time. So we will get your questions at the next stopping point. Go ahead. Yeah, with having a basic understanding of the BPF program what we would do now is look at the how the that program how that BPF framework can be used for doing the dressing of the current functionality as such. So basically the broader basic broad understanding that the type of tracing it can be used for debugging and troubleshooting the various issues as such and the traditionally there are there are the various tools that are available as such, like the after a during the function tracing of tools as such, and the user space as trace of the system called race and trace as such. So basically these are function these tools are also helpful I based on the use cases as such. But if you want to dive into every aspect of the current system that is going on that's where we make use of the BPF functionality as such. And these tools can be used along with the BPF framework to explore more information as such. So here in this case, so we will look into how the tracing can be done with the help of a BPF framework. And as we know that the tracing gives us the full visibility of the across the full software stack, and it allows to create the data for profiling for debugging and troubleshooting as such. So this is the slide diagram from the Greg Reagan's blogs as such. So here in this case, this notes down the different top systems, any kernel, any kernel, and I also the various layers and application level. So the information from the phrasing can be done as such. So if you talk about the kernel subsystem and the information visibility can be. We can get more visibility from the system called the interface layer as well as file system and the virtual file system layer business pertaining to whole of the file system on block layer as such. Networking at the network layer in socket layer, TCP IP layer, as well as the device driver layer like related to interrupts and as such, and then the other subsystems like the schedule or memory management virtual memory. So, so to get to know what is happening at each of the subsystems in the BPF events, the BPF programs can be attached to those this is subsystems as such and with the help of this support BPF tracing support multiple means which which provides the multiple sources of events as such, and that will provide the visibility to the entire software staff. And this visibility, we can get in by making use of the different instrumentation ways as such. So basically the instrumentation that's a dynamic instrumentation and static instrumentation and that to as a kernel level and at the user level as such. So basically when instrumentation is happening at the user space level, it makes use of this you prove functionality. And that's a dynamic instrumentation dynamic instrumentation means instrumentation instrumentation instrumentation is done at the runtime as such on the live code as such. So with the dynamic instrumentation it provides the ability to to answer the instrumentation points into the top and running code as such, and for the kernel, the cable functionality is been used. And for the user space. And that is done in the with the help of this you, you proves functionality. The dynamic instrumentation that can be used to instrument the, at the starting or end of the kernel function, or the application function as such. You can add the proofs at these different points as such as of course not only start and or you can also add this before pro at some offset of the function as well as such. And as this dynamic as dynamic instrumentation you would go and attach to some particular function as such. And this kernel function API is maybe I might change from version to version as such so that's why we. That's why it's been say about the current forms does not does not provide quite a stable ABM it would change as per the kernel version if the API definite declaration definition changes as such. Whereas in case of static instrumentation, the instrumentation points are added in the code itself, and they're made in by the developers as such, and for for the kernel. It's been implemented using the trace points, which the code has been added since 2006 32 colonel. And for the user space, another, that is the form that is provided through this user statically defined trace points as like. Again, the instrumentation can be done in the kernel and the users applications and as they are been implemented are added by the developers, they are kind of provide a stable API interface as such. So, what we asked for our topic what we are going to look at is the kernel instrumentation that is dynamic as well as static that is through the Cape ropes and the phrase points as my focus is around this. framework as such functionality as such. So what we will do is we will first look at the Cape ropes Cape ropes is the framework functionality that allows to set the folks at any or almost any function or instruction in the current. And basically, and basically the eb programs they are attached to using the Cape ropes that is particularly for the entry of the functions are at the specific offset of the function after the entry. And there are different ways that can be done through the BPF for BCC will come to this for this BCC and BPF tools are there. And these are just to give an idea of what are the functions that have been used and basically also that our instrumentation can be done through the carry. The return proofs that is that is that is attached at the exit of the function. So, when you attach the program to that particular event and reassert exit exit function so through this case written process can collect the information and then exiting from the function as such. Now you're in this example like to take an example like a VFS read or the read read system call like in the case of read what we expect is the output of the read is the number of bytes that has been read so in particular analysis as such you might want to know that application, what kind of read load is happening as such or overall system load that is happening through the read operation so in that case. And return prove a return prove can be used collect and collect the red number of bytes read that is the exit that which at the exit point the kernel code path is aware of that the how much read has happened as such. Particularly this example, it is doing the work of collecting the VFS read function that is executing executed throughout the system and this display displaying the histogram for the read operations that are happening as such. So we'll come to the syntax, how, what is that in a later examples, but this is just to give an idea like at the basically k proofs at the entry is like another example here example I've taken is the open system call. So open system call like if some security has to be enforced in this case, some security tool has been developed and you're trying to monitor or enforce the security as such in that case. So you can add the hope at this open system call and if a particular file has not to be opened by a particular set of users, then in that case, in that BPO program you could have a functionality of this is the file which comes as the first parameter to the open system call corresponding. As you are in the current you have the point to do the task that is running. So from that you can get the credential of that particular task and the process and have some based on that policy decide if that open has to succeed or not. Or you can just do the work of collecting that information for further analysis as long above us like monitor what the things happening around on the system at that particular time. So that's where this probe probe can be used for collecting the information at the start of the function so you're in this case function now you're in this case you will have this do says open is the function particularly corresponding to the open system call and here we are not using the sis open asset but the inside function of the system all assets so you're in this case it's not just restricted to the system files but any function kernel function can be can be proved as such as the BPO program can that as to any function. We'll come come to this what is a BPF trace in a couple of slides next as such. So basically, using this key proofs means for debugging and for developers can write a kernel models and attach the function to the key proofs are use make use of the tools as such or write the BPO programs all together. In the operation. So this key proofs is for dynamic instrumentation. What we'll look as the static instrumentation kernel that is done through the trace points. So trace point is the functionality kernel functionality that allows you to insert the code at a logical places, while implementing that functionality may be or implementing the driver as such. In that case, you, the developer as a developer you would might use the trace point for some of the functionality as such so that case the developer is adding the, adding the static markers into the current path as such. So you can list list on this what are the different trace points that are there on your system by looking into this file as such available events as such. Okay, so if you look at that, then that is trace available events as such. So if you look at that there are, there are, there are the phrase points being added by the different subsystems. Now you're in this case you see this NFS and related to a subsystem and this is the RFS. This, this code has been added by the different, different, different for maintainers or the device drivers as such. And again, you get to see the trace, trace points being added for the system. I just feel it taking the snapshot of some of the system calls that has the trace points with the trace from this like the trace points are there at the exit entry as a, this is about the, this is trace point at the exit of the clone system call entry the clone system call and similar for the fork and fork as such. So basically like if you do, you can count on how many kinds we trace points are there available on the system as such. So you're in this case. I'm having a bit older kernel that is 5.4 kernel. So on 5.4 kernel, currently there are like 15, 1500 plus trace points present in that kernel as such. So basically, what we can do is like use the trace points static trace points to collect the information as such. So you're in this example, we'll go through the example later as such, but here I've just pointed out the example for the read system call as such. So whenever the read system call entry the read system call happens. So you can go and train the count that is the input parameter for the read system call. So here in this case it would print the number of bytes that has to be read. And here in this case the constraint that the process should be SSDSR. So here in this case we get the information about the process, the SHD process doing performing different kinds of doing the reads with the n number of bytes that has to be. So, so to get started with how to use that interfaces will look at this BPF tool chain as such. So basically, in the, in a lot of scenarios or in the current scenarios as the EPBF framework or the programs they are not directly used as such, but they had indirectly been used to via projects of the DCC BPF trace or CDM project as such. So this project provides more abstraction, provides an abstraction on top of the four BPF framework and abstraction in form of tools and script as such so you don't need to write the program, programs directly as such. So to start with is like you can use this program scripts and tools provided by this project. And if they satisfy the requirement, then that is good, but if you have more customized requirement then you should go and write your own program fast. And this is not limited to these two frameworks, these are the most prominently used front ends for the BPF tracing that is a BPF compiler collection, DCC and BPF trace framework as such. And the BPF programs, they can be written either in the pseudo C code, go line, go language or Python as such, and the set of helper functions they are provided for the BPF library. And this code is compiled to bytecode using the LLBM and the C line compiler. And the way you can start is like if you're looking for the tools to run then you start with the DCC tools. And then go for the go with the BPF tool as such if you have great customization and if you want to write your own customized evaluation or adding code or customizing based on the requirement. And in that case, start with the BPF trace, which provides the high level language kind of interface, and then write your own DCC programs as so most of the programs that are there like CDM or the other tools over so tools are there. So, some of them, they use a combination of BPF trace and they own DCC programs as such. And most of the hardware acceleration or network acceleration with the customized requirement that would be done by writing their own DCC programs, BPF programs as such. So this is a collection or a framework that enables to write the program, write a Python program and embed the BPF program inside them the example that we saw as such. And the running the Python program basically generates the bytecode. And that's loads into the column is basically the DCC framework handles the work of converting the program see see like program or pseudo C program that into bytecode and load into the kernel with that the back end. The BPF system call is executed which would load the load this BPF program under the kernel as such. So basically it is useful for some complex tooling or the given kind of application. And BPF trace is a high level tracing language with which is like, I will inspire by a can see language interface. So it provides that kind of interface and then it uses LLVM as a background to compile the scripts into bytecode as such and load and the for loading and interacting with the kernel. The DCC framework has been used as such. And it's the, like one liner kind of interfaces, something that we have seen in the previous example so this is, and this is the BPF trace tool that this whole of this syntax is in its, it's been converted. It does the work of converting into a port that would be loaded into the kernel and then executing as such. To the help of this one liner syntax as such. Even I can write a small syntax short script as such. So there are some syntax syntax and primitive status to based on some pertaining to DCC and some syntax formation for BPF trace as such. Again a slide from Greg, guys, talking showcasing the different tools that are part of this DCC project as such. Okay. So your DCC, let me say, yeah, yeah. These are the, these, these, these are the reports that are from which we get the DCC and BPF trace as such. And all these tools they are, yeah, go ahead. I don't think there was a question, is somebody this question for Vemina? It doesn't look like it, Vemina. So you're like, if you look at this thing, so you would say that we can go and look at each of this subsystem and this information from each of the subsystem is collected by making use of this different tools as such. So as a beginner, what you can do is you can start with using this tools that help you to get the understanding of what is happening at each of the subsystem as I and then as you get comfortable with that then go and implement through them. Programming interface as such. So what I'll do is I plan to take some of the, this tool to demonstrate and then see what are, what are we getting on top of this as such. So here for, let's look at some of the system called interface as such. So here what the system called interface in this case this is some of the tools that are come for that particularly interacts with the system called interface and here the example I would take is the open snow which works the tool that that shows the process. Basically it tries to show which process are opening which files as such. And this tool can be useful for like if some application is failing or if there is some misconfiguration, then in that case this could provide some pointers as such. Similarly, there is a stacked snow. Does the work of snooping on tracing the different does traces stats system calls as the variants of stats system and which would help you to understand like which process trying to read some information about which files as such. And again, you can do the snooping with the help of the exact snoop which does the work of monitoring or snooping the exact system calls. So, like, if you run the application, this tool opens snow. So it trains the information like process ID command with the name file descriptor if there is some error in that and the part which file is been opened so you're in this case this is generally the same rule for the whole of the system as such. So one of the things that you can look up is this the identity just for explanation powers. So Iq balance is a is a is a demon that does the work of the interrupt interrupt load balancing as such so and basically does that goes and checks into the SMB affinity file for balancing the load across multiple CPUs as such. So the activity that is happening in the background as such done by the Iq balance so we just goes and open this files and does whatever the operation so right now we are looking with the open snow is just what are the files that are getting opened by what which process this process named the ID on the path and again like if there are any other errors that would be shown under this column as such and this is the file descriptor ID as such. Similarly, like, we have the start snoopers in the case. Now you're this happen be a BFC that particularly means that it's a VCC tool as such. So you're in this case like you're now like my mind system is just a virtual machine as such and nothing great or anything workload is been generated but based on the application, whatever the testing or some debugging a performance analysis you're doing. Then in that case, while that application is working with this generated these tools can help understand the some of the dynamics going on with that application as such. And similarly we have this exact exact smoke will give you like, is it just not like a performance delivered also like from the security, if the security appliances tends to understand that which process is trying to execute the binaries and all this so from from a security point of view this kind of this interface is quite helpful to track all the know you're in this case there's some of the demos they might be running in the background. So that is printed but some are some some application is running particularly that can be monitored like in that case, or from the security security tools are there. So if there's a lot of things going on, then all that can be taken, looked at a lot that when the exact system call is going as such. So here I am what I'm doing is first just going with the understanding of tools and then we will do the same thing with the examples as such with the programming example. Particularly for the file system, particularly like there are other other tools like file top is particularly used to trace the file read writes as such and it brings the summary and basically the tool what it does is like, it is particularly the VFS related functions as such, particularly the actual kernel function that is a VFS read and we have sprite that they have been using dynamic tracing as such. And it, what it does it brings the reads and writes as such so basically it is the VFS read and VFS write and the read and write workload as such. Basically it does the sort sorting based on the amount of data that has been read as such. And it's been in the default interval of one second that you can change it as such. Okay, so that helps to give gives an understanding like where actually the system is getting utilized and what kind of operation is happening. Similarly, we have this VF, which is that VFS count and VFS that it basically used to for getting an idea of general workload characterization that is happening. And if you want to know, like what all the VFS really functionalities for getting executed in this VFS count basically traces all the kernel function that are starting beginning with VFS underscore so basically all the functions that are part of the VFS of system as such. So all this thing is which brings this VFS count to as such so here in this case it brings this information as such the that this particular VFS function is called so many times so this would give us the idea. Which means under load as such or what what is really causing a blockage and what which are the, which of the kernel code has been more used or executing that can be traced by making use of things if it is particularly a file system load or a block by a load going to the file system and then this is one of the interface that provides this information as such. And VFS start is like it traces some common functions like read write sync and as such. Another that we can look at from the by using the tool is this scheduler, of course like from the performance point of when the system at higher load and is there and schedule is one of the key component as such that helps you to know, like exactly what is the process that are getting that are executing what they are been getting the time slot and who are been starved or as such so here in this case the one of the tool in the CPU distribution to say that does the work of measuring the time that you ask the process spends on the CPU before getting the schedule and basically internet uses the some type of BPF maps, which throws that information and it brings the information in form of histogram as such. So this with the help of this tool basically it helps the like to find a suppose like a task is spending short time on the CPU. So that kind of information is an indication of like, there might be multiple excessive context switching has happening, or the workload distribution is not. Surely poorly been done as such that gives that can that can be estimated by looking at the information that as such are possibly that can be the reason that the locks are being held assess the possibility of the contention of the resources. Like the some process has acquired the mutex and then another process trying to acquire the same mutex and as the lock is not been available there, there's chance of the application application is just context switching a lot many times as such. So that's why the application is not getting the CPU so information can be derived based on the stats collected from this. The task is not spending off time more that before it is getting scheduled as such. So that can happen that the task might be waiting for the IU to completion, or a long IU is happening as such. On this we can derive the information to know the health of the system as such, and all this information for collecting this information storing it uses this internal baths or storing the timestamp and the histograms as such. And this year, this is one of the examples like where there can be more may incur the overheads for some kind of rules because here these traces it goes and traces the scheduler trace point so there can be extensive load. This is the overhead created by this kind of tracing. Tracing events generated in the scheduler as such. So it runs in the background and creates a histogram as such. So here in this case, it is displaying in the information. This is happening through this timestamp. Another one is that we can look at as the virtual memory as such. One of the major concerns with that to have utilization of memory, like the memory is available or not and then that can be traced with the memory leak. The allocation location request and what it does it collects the stack traces stacks for the allocation that happens, and then brings a summary like which stack stack perform the allocation that has not yet been for it from that information and get to know the health of the memory allocation that is happening. You can trace the memory leaks for a particular process by specifying the input parameters that is a specific ID. In that case, the, at least the function from the leapsie that is malloc a lock and the related memory management functions and you can use it for tracing the full processes. So in that case, it goes into more of a kernel function that came along for kernel allocation API as such, along with the page allocations itself. So you're like this list is quite extensive and actually like if you get to a study each of this it's like you get to get into the details. And that's really as part of the kernel subsystems outside extensive very incredible resource information we get from this. So, when then you have about five minutes left and then I'm not sure how many slides you have. And do you want to. Okay, I'll. Okay, should I take questions. Yeah. Okay. So, the question in the q amp a box is, can ebbf be used in in a term of us. In a term of us. Yeah, I think that should be able to be used because BPO framework. And that's part of the kernel. So in the any drama, you tend to load the kernel modules and do some other basic functionality at that time of minute. But I think that by that time because BPO framework should be available.  So, the user process. Not big kick no in it. So, through the internet, you've been able to load the module module that is you do you do through in small amount of product model probe as such. So, right, right. So here basically we're using BPF trace tool as such like, if you do, if we tend to use BPF trace then I think we should be able to do that functionality, or even BPF application programs been written to be this is the interface I think that should be able to be done. We are at a point that the kernel is up and running, and then you're loading the file system or any such components, but the functionality is up and running. So through BPF trace or BCC tools, I think that would be free that should be able to be done executed at the in it. I think some of the application might have that need as well. And you'll have to make that part of the in it. Right, yes, not not at the command line of course. Right, right. Yes. Okay, so that is great. That's one. Okay, let me see answered live let me get rid of that. Since it has ability to monitor the processes and virtual memory, I think it has the ability to tamper the memory data. Yes, you should be able to do that means if you're attaching those kernel functions as such. So memory dumping would not be a very good way through the system through the BPF but the information related to memory functionality would be good as such but of course you can dump that. There are other interfaces provided for dumping the memory as such because like it would be how much amount of memory one to dump us because the BPF map or the trace pipe, it would have limited size limitation as such. But if you define if you define if you divine your own BPF map functionality. And I think in that case, you should be able to do that. It's not a raw memory number some way to pass on that memory memory contents. There is one question that might be a good one to answer. Can the BBB a program store state or does it simply pass data back, we need to build a state in use of space. So the data that is available like by default we're using the kernel buffer, that is the buff. Then in that case, once it is passed it is emptied but other case it is there in the maps as such. But once it is access I think it gets flushed from the kernel store as such. So user space has to once they obtain the data user space has to keep track of the state and yes, yes, that would make sense, because these are these are. These are running buffers go anywhere. Yeah, or flow buffers. Yes. Okay, I think there are other questions. However, we are out of time. Candice would. I think that yes we are kind of it's where at time now 1030. I mean, 30 minutes. So do you have any more slides. I think I had some examples as such but it's there in the slides like it can be referred on the government, the main concepts and these things I've covered as such. Candice, I'll throw it back to you. Thank you everyone for joining us as a reminder this recording will be on the Linux Foundation's YouTube page later today and a copy of the presentation slides will be added to the Linux Foundation's website. We hope you are able to join us for future mentorship sessions. Have a wonderful day. Thank you guys.


Hi everybody, my name is Bobby Eschelman and I'm a current engineer for the systems technology to the engineering team at by dance and I'm going to present to you. VD use supporting VDPA devices in user space. Now, here's the agenda for the talk, we're going to first look at the defining the problem and the solution. And they give a little intro to VDPA and VD use. Thank you for the explanation of why user space simulation matters. And then a design and implementation deep dive into VD use. And then finally, we're going to look at a use case. So first defining the problem. As we all know, modern systems today are deployed typically in hybrid environments of both containers and VMs. Many applications also have specific IO requirements. For example, this could be direct access to block devices or NICs, or hardware accelerators, GPUs, ASICs, FPGAs, but yet container IO interfaces and VMI interfaces are not the same. So for example, it's applying block devices to VM. So we're just going to simply use for a guy out. You're going to provision your VM with a bird eye or block device, the guest kernel will detect it and present a block device to user space via the bird eye or block device driver. And then the back end will just be emulated transparently for containers a story is different right. So we have block devices, but they're just real block devices. The kernel doesn't have any capabilities for virtualizing a block device. The container or bare metal really just has direct access to any block devices on the system. You can kind of work around this use NBD where the you have your container accessing a network block device and then the network block device server is just a user space. And then you can modify that to kind of act similarly as you would like a virtual back end. But it's not really designed for that. And so there's a major performance and kind of system complexity costs. So the main point to what I'm trying to drive here is that non generic interfaces with non generic back ends come at a development and maintenance cost. So now let's take a look at the proposed solution. First, we want uniform interfaces for both the ends and containers. We don't want applications to have to interact with different interfaces. And we don't want to have to provision systems in any specific specifically different way to to provide the services that we wanted to provide to our applications. Well, VDPA does this VDPA provides a way to give vertio devices to both VMs, which is not necessarily dependent on VDPA that you can go without VDPA. But the real value and a piece of VDPA allows you to provide vertio devices to applications running in your host user space. And then we also want to provide vertio device back ends to be uniform. And we also want them to be in user space, which is where VDUs comes in. So when you use VDUs and VDPA, you get a couple of benefits. One is greater portability for both applications and back ends. Another is easier maintenance of the plate systems, which we'll talk about a little bit. And then better performance compared to some alternatives. VDUs is very, very, very performant between zero copy pulling and just like minimal context switching VDUs is very, very, very performant. And we have compared it to some alternative setups. So let's look at a VDPA specifically. VDPA stands for vertio data path acceleration. For those not familiar with virtualization vertio specification for para virtual devices. What is it? It describes how virtual machines drivers are to talk to a emulated backing device. So VDPA specifies hardware devices that understand the data path of vertio. So first, we have in VDPA the hardware devices themselves, we have the VDPA bus that's actually, you know, it's kernel now. And we also have the VDPA vertio drivers. The benefits to VDPA are obviously the performance gains because the data path now completely bypasses the host. You don't have to trap in the host user space and have your device emulator and then make system calls into those kernel to, for example, persist a block or use a file backed scheme, writing through a file system. Or you don't have to track back into the host kernel. If you're using V host, for example, with VDPA, the virtual machines, virtues are shared directly with the VDPA device. So there is no need to involve the host. It also seeks to minimize development and complexity for hardware vendors by limiting the amount of extra work that they need to do to make a VDPA compliant device. This is mostly because of the form of only requiring implementation of the control. I mean, I'm sorry, the data path in vertio and not the control path. So private implementers have VPA bus functions that they need to implement in order to plug their device into the VDPA bus, but the actual control portion of the devices is vendor specific. But importantly for VD use is that it provides a uniform interface to bare metal container applications and to applications in the end user space. You can have a vertio device being provided to your host user space as well as a vertio device being presented to your VM user space. And that's very important. It simplifies. So here's a really, really simplified look at the VDPA data path. So first looking at host user space applications and these can be container or bare metal. What they see is normal vertio device and this vertio device is actually presented by the vertio VDPA bus driver. And the virtue that's owned by that driver is shared directly with the VPA device. So for guest applications in your VM, they also see a normal vertio device. And the back end is a VOS device that has been presented by the VOS VDPA driver. And in that, the virtue is also again shared with the VDPA device directly. That's where the performance benefits is being derived from the VDPA in hardware. And just a kind of double click on this point that control path is vendor specific. So the way VD use connects to VDPA is we're essentially just taking the hardware device and bringing it up into user space. The device is exported into the host user space. And the guest applications still see a normal device, your guest applications still see a normal device. But the VD use and VDPA device may be emulated with all the flexibility and utility of user space applications. And to point out that emulator still has full access to the vertio virtues. This is accessed via shared memory or a bounce buffer. And we'll go into that a little bit later. So why do we care about user space emulation? Why not just use like an internal emulator like the host or go down that path. One is faster building cycles compared to internal. Back to this point, your project trajectory can be pretty quick when you have an aggressive schedule. And yeah, just user space applications can be developed very quickly. There's easier maintenance and this is really in reference to live upgrades. They're just easier to do for user space application than for kernel module. And also more flexibility you can use you user space libraries and importantly you can reuse QMU code and rest VMM code. So now let's look into the design and implementation of VD use. Here's a high level high level overview of the VD use architecture. VD use consists of two components. One is the VD use daemon, which is in user space. And the other is the kernel module, which is obviously in kernel space. The daemon has two primary responsibilities. It's really device emulation and implementing the vertio data plane. One of the mechanisms that it uses to achieve those goals is one using iOctl on the character device that the kernel module provides to user space. Octl is used for device initialization and configuration. And the third is the reading and writing to that character device, which is for passing and receiving control messages. And then the third is a map for mapping in shared memory for from VMs to have access to the DMA buffers for the devices that are in VM. So I'm going to look a little bit more deeper into the data path and the control path here. So vertio vdpa, which is really referring to the bus driver that is used when your container or bare metal host user space application is accessing a vertio vdpa device. This is a context mountain DMs just just to clarify. That uses a bounce buffer so what the bounce buffer does is it performs a copy of the DMA buffer into a buffer that is mapped into the daemon. And one of the reasons that that was done is because the buffers, the buffer rights may not be page aligned and so just simply like mapping in a DMA buffer page directly into user space may be exposing kernel data that the application doesn't have any business having access to. And so this bounce buffer mechanism avoids that vulnerability by only copying the data that should actually be accessed by the daemon and that accessible by the daemon. And that's actually implemented using an MOU based software IO tail be for VMs except last example is for containers for VMs via as vdpa is used. And that's just straight up shared memory for the control path. As I mentioned before, I opt all is for things like vertio feature initialization update the config space of the device querying vert queue information. The forwarding control messages from the vdpa bus to user space. This is the reads and writes to that character device. This is used for setting device status getting vert queue state and so on and so forth. So now just a look at a specific use case for vd use. This is a distributed storage use case. We have a storage service. Internally there's a storage SDK that hooks up to all of our remote storage services that we we provide to our applications. So the storage daemon. First of all it needs to emulate vdpa block devices just as any video use daemon does. I request coming in from the various kernels. In this case, we have both our host kernel. You can think of it as being routed from the container accesses through the vertio and vdpa infrastructure through the vert queue, the software iotlv mechanism, and then up into being handled by the storage daemon. Or the shared memory with the VM. So the block I request are coming in from from those directions. And then I request being fulfilled using the remote storage back end, which in this case is a well maintained and documented well used storage SDK. And that can easily be used because the daemon exists in user user space. From the container perspective, it just sees a plane for a block device. And from the VM perspective, the same. The result of this from a performance perspective is that we see a one and a half times improved performance in both IOPS and average latency over alternative setup that we had prior using mbd which we provided to our containers access to a network block device, which was being serviced by a server application user space. This is one and a half times more performance than that. And then again we still have that flexibility and maintainability that we were looking for. Okay, thanks that that is my talk. I appreciate the time.


In this course, we will talk about performance metric, detailed second part of performance metrics. So we saw performance metric like recall, precision and accuracy in the last video. Let us look at more detailed performance metrics which come as different classifiers. Consider there is a binary classification problem and the classes are 0, if student will not pass the course 1, the student will pass the course, the only 2 classes so it is a binary classification problem. What do you have the table like this, yi is the actual dump value and y predict is the output of the classifier. And we computed the confusion matrix and we computed precision recall and accuracy, right. The focus was on predicting students who will pass the exam. So I want to tell that. So the true positive is focus is on the students who will pass the exam. If I have focus on 0 that is who will not pass the exam, what will be the change? I am not going to discuss what will happen if you focus on 0 in this video, but this is for you to think take it as extra work. So please check Wikipedia, they are very good resources. How to check Wikipedia? Just find this particular page, just type, precision, recall, accuracy, Wikipedia and Google you will get the page. Let us check the next problem in classification. Consider your binary classification problem that is 2 classes, the performance of predicting which students will get more than 90 marks in final exam and which students will not get more than 90 marks in final exam. And you have n equal to 1000 samples that is 1000 students data from historical semesters and courses. As you know, there are very few number of students will get more than 90 marks, they will get university ranks or something like that. So the number will be very less. So consider there are only 20 students who got more than 90 and other 980 students got less than 90 marks. This data set is imbalanced. For example, in a true value, there are only 20 positive classes and 980 negative class that is they will not score more than 90 marks. So if we have these kind of in balance data set, what will happen? So let us compute precision, accuracy and recall. Accuracy is 980 correct plus 9 correct, 989 by 1000, 98.9% is the very highest accuracy you can get. And precision is it predicted all the 9, 9 divided by 9 plus 0 is 900% precisely predicting. Recall rate is 9 by 11. So what do you think about this? Is it good? This value is interesting. Let us see. Consider you have two classifiers. The results are not same as what we discussed in the last slide. This is a bit different. Consider there are two classifiers which use the same 1000 data set and which give the results like this. Accuracy in percentage that 98.9% and precision 34 recall 45%. Classifier here you might have used decision tree or here you would have used some naive base classifier. Classifier 1, classifier 2 results on the same data set is given. Why? It has a very high accuracy but very poor recall and precision rate. And which classifier is better? Please pause the video, think about it. Take a minute, think about it. Why this two classifiers give really, very accuracy and very less poor regression and recall. Think about it and which classifier is better? After you list down your answer, you can resume to continue. So, the reason for very high accuracy is imbalanced data set. Given that the data is 980 is positive or negative. So, if a system which classifies everything as negative, still will get a 98% accuracy. No need to even try to create a new logic, new rules, nothing. The simple rule can be classify all the classes into majority class. So, you will get high accuracy if it is imbalanced data set. And which classifier is better? Given the data set, it is not enough to say which classifier is good because it depends on the research goal. If my interest is on precision, my interest on recall, based on that, which classifier is better can be told. So, in order to make a decision which classifier is better or which performance is doing good, we need a score, we need a metric which combines precision and recall or some other kind of metrics. Let us look at those metrics in this video. The one of those metrics is F score or F1 score. It actually the harmonic mean of precision and recall. What is harmonic mean? Almonic mean is simple, it is a one kind of averaging technique. It is there are like a two values since precision and recall. So, harmonic mean is simply it is a 2 divided by 1 by precision plus 1 by, if you simplify this, you will get the results for harmonic mean that will be like a 2 into precision into recall divided by precision plus recall. So, it gives importance to both precision and recall. So, is it good? Should we give importance to both precision and recall? In the last slide, I mentioned there are some research questions which will need better precision compared to recall, some research questions which need better recall than precision. Can you guess, can you think of one such a research problem? This is not the activity, but you can pass it and think about that. So, we will talk about that such a research questions later, but please think about it. In order to avoid this challenging that the F score is giving importance to both precision and recall, what we can do, we can have a variation of F1 score computation methods that is it gives a more importance to precision, you can add a weight to it when you are doing the precision and recall computation. So, there are variation of F1 score that can be considered if you give weights to precision, but we are not discussing that in this video. You can check the Wikipedia page on what is the formulations to do that. So, there is other metric which is developed by Jacob Cohen and Oscar Kappa. In Kappa, it is developed to measure the inter-rater agreement of two-raters. What is inter-rater agreement of two-raters? Let us take an example. So, there are two-raters, two researchers are watching the students facial expressions. There are say 10 students in the class are attending a class. Two-raters are looking at the students facial expressions and body gesture, the tone, everything, classifying them as one of the affected states, both confused or engaged, something like that. So, there are two-raters, we cannot have two-raters for our complete research, you want to use one-rater for five students and other-rater for five students. But how do we avoid the bias between these raters? So, that is called inter-rater agreement. Initially, we have to ask two-raters to observe the same set of students and check whether the two-raters agree on their classification. That is, if you have a class, items to classify say boredom, confusion, engaged and you want to classify into two or three categories and how these two categories are accepted by both raters. So, in order to measure whether there is agreement between these two-raters, Corgan's kappa is used. So, Corgan's kappa, the formulation is k equal to Po minus Pe, data by 1 minus Pe. Let us see what is Po, Pe and all. Po is the accuracy. From the confusion table, we can compute the accuracy which we computed in the last class. And Pe is the hypothetical probability of chance agreement. What is the hypothetical probability that both raters will agree? What is the minimum value they can agree? So, how to compute Pe? It is a sum of estimated probability that both raters agree for a k number of items. We will see example how to compute a kappa score. Let us take this table. Let us understand the table first. There are two-raters looking at students facial expressions. They looked at, so, 40 plus 30 plus 30. So, they looked at around 100 instances of facial expression. I am not telling 100 students. There might be two students, there will be one student, there might be like 50 student. But they have 100 instances of facial observations. Both observed all the 100 observations. So, 100 instances. Rater 1 said, at 40 times frustrated, same as the Rater 2, but Rater 1 said that 20 more times students frustrated, but Rater 2 did not say that. He might have said not frustrated. So, this is a simple confusion table, similar to what we saw in the classification problem. So, Rater 1 and Rater 2 agree, they are frustrated. Rater 1 and Rater 2 not agreeing their frustration. This is the cross values that is Rater 1, not agreeing it is frustration, but Rater 2 marks as a frustration. This is the wrongly classified problems. There is no agreement between these two-raters. There is high agreement between these two-raters. What is the accuracy? Simple to compute, there is 40 plus 30, thereby total number of observations that is 70 by 100. Very simple to compute. What is Rater 1, yes agreement? Let us compute P E now. What is the Rater 1 agreement? Is Rater 1 says yes for a 60 percent of time compared to all 100 samples, 60 percent of time he says yes, that is 40 plus 20, 60. This is 60. This is 40. This is a Rater 1's agreement. This is 50. And this is 50. If you just add these two values. So, Rater 1 says 60 percent of time yes and Rater 2 says, Rater 1 not says yes for 40 percent of time, that is Rater 1 saying no is 0.4. What it says that Rater 1 has a bias of telling yes more compared to no, which means when you see a small slight expression in the face, it might mark it as a frustrated, that is the Rater 1 bias. In a Rater 2 it is equally biased. Say for example, it says 0.5 percent of time, say 0.5 probability, a Rater 2 says the students are frustrated and another 0.5 probability that is Rater 2 or students are not frustrated. So, this is 0.5 and the Rater 2 saying no is also 0.5. This is from this value 50 by 100 and 50 by 100. So, what is the probability that both Raters will say yes, that is simply multiplying the Raters 1s value into Raters 2s value 0.6 from here, 0.5 from here. So, if you compute there will be 0.3. Similarly, for Raters saying no will be that is 0.4 into 0.5 that will be 0.2. So, what is the observation probability is like 0.3, that is yes value plus no value that is 0.5. This is the, we saw that sum of hypothetical probability of chance agreement that is P as a observation accuracy we say. So, if you use the 0.7 value in the formula you get the cap as core equal to 0.4, you can compute that. Please apply that in the formula we gave it in the previous slide and do that. Is K equal to 0.4 good is the question. Think about it, if you want can pass and go and search in internet and see is Kappa score 0.4 is good. There is no answer, no exact answer to this, but it depends on domain the K value good or bad can be inferred. So, in this scenario that is the both Raters will agree will be 0.4, that is not so good score for, which is not a so good score for inter-rater agreement reliability. So, do you compute Kappa? Do you need to do it every time like this? It is a simple website which uses like a 2 cross 2 table confession table. Just enter the values in the table and say calculate, it will calculate until the Kappa score, this is the website. And lot of tools like script languages have a library and the machine learning tools have the library to compute Kappa score easily. So, in this video we saw what is imbalanced dataset, that is in a dataset there are too many positive cases or too many negative cases. And we in order to pick up the better performance or better performed classifier, we have to combine come to the new score which combines accuracy, precision and recall. The one can be F score simple one to start with or the Kogan s Kappa. Kogan s Kappa is used widely to pick the right classifier. We will look at more better classifiers or better metrics. In next video we will check more metrics on picking the right classifier. Thank you.


Hey what's up guys welcome to our trending reviews, so super excited today to introduce the brand new gimbal from Feiyu Tech which is a more updated, feature packed and advanced version to this AK2000, it is called the AK2000S. One of the biggest differences is the visual one, so as you can see here from the box it comes with a detachable versatile arm to give you a more comfortable grip when shooting low angle videos. On the topic of grips the main handle actually comes in this very nice looking light rosewood wooden design grip, which you can't really find on any other gimbal. So pretty excited with this, let's go ahead and give you an unboxing. Now I love it when they come with these really solidly built hard cases, they look like a nice little briefcase, so they have a little clip here, let me just unclip that, there we go, so this is everything that comes in the box, there's a lot of stuff here, so let me just go through it, you have the actual tripod stand, which is metallic, has rubber grips on the sides, so I'll be mounting this at the bottom. You also have the main gimbal here, which I'll go through in a second, this is the detachable arm which I'm going to be connecting shortly, you also have a release plate, which is a longer one than the usual one, you also have the smaller mounting plate as well, so this is to give you quick access to remove and attach your camera, you also have this, which is the lens mount, so if you wanted to use the automatic motor system to change the zoom levels, then you can rest your lens on this. So this is the follow focus motor, now it is a follow focus and zoom motor hybrid, so it comes with this little extension here in the box, there's a little extension tube and also you have the tube mount here to make sure it all connects together and this is for your lens, so you can zoom in and out using the knob on the actual gimbal. Now elsewhere in the box, you have the various different accessories that allow you to use this zoom motor here, so if you wanted to follow focus and zoom using the knob, then you have to put all of these pieces together, connect the actual strap to your lens as well, and then you have USB cable, Allen key, some screws, etc. And then lastly you have this nice little pouch, let me go ahead and show you what's in here, so a whole bunch of things, now you have various different USB connectors, as you can see here there's two USB ports that come on the gimbal, so here you have USB connectors for the following different cameras, you have one for the Canon 5D Mark IV camera, you have the Sony ones, a default Canon one, a Nikon and EOS one, you also have a 3.5 mm microphone adapter as well, and then you also have one for the USB-C charging cable as well, so the actual gimbal will be charged using USB-C and I'll be showing you the port in a second, so this is basically covering you for whichever camera you have, and if you wanted to control your camera with the gimbal using the cable, then you can do that using the USB port on there, and pretty much you'll be covered with all of the options that come with it. And you also have various different screws and Allen keys to make sure you can connect everything to your camera.  So pretty solid design, it's not too heavy, it's actually the same weight as the AK2000, which is 1.2 kilograms, so it's a pretty lightweight, you can carry it quite easily, obviously I will be attaching the versatile arm to it as well, so you can see all of the different features that come with this, so let me go ahead and set this all up and give you a full overview. Alright guys, so just to give you an overview of the buttons on the actual gimbal and a bit of the design, so you have all of the axes here, they each come with a lock switch, so there's the pan-axis lock switch there, it's a little gray slider, you can see if I just zoom in a little bit, so I can lock it like this, unlock it like that, so that's for the tilt and the roll axis, they have it there as well, so I have them all unlocked at the moment. As I mentioned earlier, you have two USB ports here for the follow focus functionality, it comes with the adapter cable and then you also have the camera ports here, so if you want to use the gimbal connected via the USB cables provided, then you can use either camera one, which is USB-C or camera two, which is the traditional USB, so as you can see there. Now I've got the long plate already added onto the gimbal there, it's sliding at the moment, but I have the release plate on the bottom of my Sony a7 III already, so I can easily attach it onto the actual plate there and I can just tighten it and close it off right there from the top, so this has a little lock as well, so depending on how you're stabilizing it, you can actually just clip it in, put that into the lock position and tighten it up there. Now I'm going ahead showing you the buttons on the front, so you have your main joystick, you can move the gimbal around once you power it on. The first button here on the left is the mode button, a single tap will switch it between pan mode and lock mode, a double tap will make it into follow mode and a triple tap will make it into all follow mode. Then you have the shutter button there in the middle, if you press halfway you can focus it obviously that's only compatible with certain cameras and you can do a single tap to start the video recording or take a picture, if you long press this for five seconds it can do an automatic burst shot, but again you have to press it to quit out of that mode. The last one here on the right hand side is the function button, you can single tap to switch between video and photo modes. On the right hand side you have the USB-C charging port there and you also have the power button. Now for the power button you have to long press to turn this on and off, I'm just going to power it on now. You can also control the touchscreen with this power button, so a single tap will lock and also unlock the screen, so in case you wanted to take some videos or take some pictures you want to accidentally rub your fingers across it you can lock the screen, plus again the single tap it will unlock it so you can use that to change the different modes and go into the different settings. If you double tap it will go into standby mode, as you can see it's gone into standby mode there and if you single tap it will come out of standby mode. So very straightforward and then you also have the battery life there on the top right hand side. You can swipe and change the white balance and the ISO there, if you swipe to the other way you can go into inception mode, selfie mode, portrait mode etc and it also has a motion time lapse as well and then you can change it into selfie mode from here as well. Alright guys so these silver locks that I've shown you, the three different switches, if you manually wanted to lock any of the axes you can actually switch that on and then turn it into locks position and that part won't move. Now the trigger button at the back, if you press and hold this it will go into follow mode, if you double tap it will reset into the right position so if I just move this to there double tap the back it will reset as usual standard on most gimbals, if you triple tap it will reverse into selfie mode, as you can see here, double tap and it will reset back again. Now the multifunction knob here on the left hand side it controls all of the three different axes by pressing it once so right now you can see it's got the tilt axes, if I press it it will go up and down, press it again it will go side to side so you can change that accordingly. Now if you do long press it will control the focus and zoom motor so I don't have that set up because I don't have a zoom lens on this camera at the moment but you'll be able to do that using this as well. Now in terms of the settings if you swipe left you'll be able to see various different options, if you swipe twice then there's tons of options here brightness and different types of lighting conditions that you can change on the camera, if you swipe again to the right hand side you can go into multiple options and then you can go into the actual settings as well so from here you can do calibrations, language and so on, swiping left you can change the shooting modes, default, smooth, action, you can customize ones if you go back you can change the payload settings, knob settings, it can be controlled from here as well so there's multiple different things you can do so I'll leave a link in the description below on the full user guide so you guys will be able to go and check all of the different options that come with this gimbal and have a look and read of them yourselves.  Now it comes with different screws, there's three screws you have to do so there's two on the left hand side and there's one here on the right hand side so I'm going to go ahead and set this up with these screws that come in the box and I'll have to use the allen key as well to tighten them up and I will do that by turning off the gimbal first and then connecting it. Alright guys so the versatile arm is in place so I'll be able to hold it and start shooting video like so so this is very convenient it makes the grip very strong now the very last thing I want to do is I want to utilize the cable that came with this this is the USB-C to micro USB adapter now as I showed you earlier there is a camera one USB-C port in there I'm going to attach the USB-C one to this one and the other end will go into my micro USB port into my Sony camera here like so and the last thing I need to do is basically just turn it on and it should recognize it's giving you the message and it is ready to go so I can now control my camera using the on-screen buttons on the gimbal so now I'm ready to record so I can just hit the record button here it's now started recording I can maybe use a joystick just pan around quickly maybe use the multifunction knob and then let's just see how this looks so now that everything is set up it's ready to go so let's move on to some sample footage and take a look how smooth this is so there you have it this is probably been one of my favorite gadgets of 2019 it's an incredible upgrade to my AK2000 and I'll be using this throughout 2020 for sure so I hope you guys found that useful you saw some of the smooth footage I'm going to be showing you the full cinematic film from that same footage that's a cinematic Christmas short film which I completely recorded using this gimbal and I hope you guys like that and that will be linked down below as well I'll also leave a link in the description below of where you can purchase this it comes in around $450 at the time of this video and they do worldwide shipping as well so do check them out and have a read through all of the different specifications the tutorials and everything you need to know about this brand new gimbal if you have any questions drop them down below otherwise I hope you like this video I hope you subscribe for lots more camera gadget reviews in the future and I will catch you guys at the next one take care


Strange things seem to happen these days. What do you mean? You know, this morning I couldn't trust my eyes. Because of what? I saw a robot on the way to work. How do you know the robot was on the way to work?