Hello. OK, sorry. Hey, guys. I am Abhijit, presently working as a software engineer in the customer session team at Uber. So I'm going to talk about the new Async I interface in Linux, which was introduced in kernel version 5.1. Welcome, everyone. I hope that this talk is an enlightening session. And we get to learn through things about the interface that was introduced in the recent version of the kernel in the next slide. I'm trying to go to the next slide, although I'm not being able to. Yeah. So first, we'll talk about existing Io interfaces in Linux for doing file Io. Then we'll discuss about Io Uring, which is the main focus of this talk. We'll also talk about a libc library called libUring. We'll also discuss about system calls specifically that have been added in this interface, the examples, various options they provide. We'll discuss about the performance of the interface along with the observability and debugging aspects of Io Uring interface. Lastly, we'll converse about the internals and its adoption across the industry. So I'm having a bit of an issue in going to the next slide. Yeah. So yeah. So let's talk about the basics. So we have a basic system called read. That is, if you want to read any kind of file, be it a socket, be it a file on the disk, be it a pipe, we can issue this read system call, and we can get that file. So what happens here is that it is for regular files, that is files residing on the disk. If the requested pages of the files or the blocks are in page-gash, then it is fetched and returned. If not, it fetches the data from the disk and then blocks before returning to the user. So the user call to the disk is blocked until the data returns. If the file, of course, this is with exception that if the file is not opened with O non-block flags, let's say we have opened the file with O non-block flag, then depending on the file, this call might block or not. So for regular files, the O non-block flag does not work. So even with O non-block flag, it will block. But for sockets, spikes, and character devices, if we open the file via O non-block mode, then the read system call will give an error until that data is ready for reading. Next slide, please. So here we have an example of how the flow works for making a read call in a file in, let's say, an EXT45 system. So let's say we issue a read call. We go by this syscall interface. We issue the series of calls in the kernel. And what happens, ultimately, is that the user is kept waiting, or the application is kept waiting until the data is fetched from the disk, and then it is returned, trying to issue the animations here. So we see that the whole call chain is being called here. We issue the sysread call read. The kernel goes through all the series of these calls and then returns back data to the user while it is kept waiting for, let's say, a simple file that is being read from the disk. Now let's discuss if there are free pages available in the kernel or not. So if there are no free pages available in the kernel, this call might block. If there are free pages, then it will succeed without blocking. This is true even for O non-block flags. Files open with O non-block flag. So let's say if you have a socket or a character device or any kind of pipe, the write will block until the data is ready for writing. But if we use the O non-block flag, it won't block. And if you throw an error until the file descriptor is ready for writing. Next slide, please. So let's talk about what are the present interfaces existing in Linux or in general. So the POSIX async interface is something that we should discuss. This async interface is actually implemented in libc, glibc. And how it implements the async nature of read and write calls is that it maintains thread pulls. Thread pulls for offloading any kind of blocking calls. So let's say if you want to do a read operation over a buffer file residing on disk or let's say, residing on a block device, then we can use this library and it will basically offload the task to a background thread. And that's how it can become a non-blocking interface. So but we have issues with this interface. Its implementation is non-uniform across different architectures and its performance also can be improved. Next slide. So we have to talk about a flag called O direct. With this flag, if we use this flag when we are issuing system calls, let's say read or write, we can make this call non-blocking for files residing on disk. That is regular files. So the problems that we're having is mostly with regular files because as you have seen that the normal system call of read and write will be, you know, can be asynchronous if we use sockets or pipes or character devices. But if you are using the regular files or files residing on disk, which have to pass through the page cache, then we are having problems that the call is blocking. So if you use this flag, which means that we don't want to use the page cache at all. So if we use this flag and if we also use lib AIO interface, which we'll discuss later, then this call can be made non-blocking. So this flag helps us in issuing reads or writes via the lib AIO interface, which are non-blocking. So this also is one of the flags that we should be aware about apart from the existing interfaces. The next slide, please. So the current ways, how do we achieve, so we obviously have a lot of code, a lot of applications out there who are using, who are doing reads and write databases. Everything that we mostly use in Linux goes through these system calls. So how do we do it currently for asynchronous nature of this, to achieve asynchronous nature? So we use IO multiplex interfaces. So we have the select interface, ePoll or poll interface. What we can, how we leverage them is that. So these interfaces provide us system calls like ePoll, wait or like poll or like select. So what the system calls do basically is check if the file descriptor that we're interested in is ready for reading or writing. So they basically check that, okay, if the file descriptor is ready, then yes, your read can proceed without blocking. Or if your file descriptor is ready for writing, then your write can proceed before blocking. So with the help of these system calls, we can have asynchronous nature. We can check the readiness and then the readiness of the file descriptor and then issue our reads and writes call. So does it solve the issue of reading for buffered files? No, it doesn't because this files, let's say the files which reside on this are always considered readable and writable. So poll or select or any of these interfaces won't work. So if we operate using let's say any kind of interfaces like select ePoll or poll, these interfaces won't help us to achieve a synchrony because these files are always considered readable or writable because Linux in general considers disk as something in which result is guaranteed. So although there is latency but the result is guaranteed. So these type of files are always considered readable or writable because they're always ready for reading and writing. The readiness checks don't work. And that's why these system calls will say that your file descriptor is ready for reading or writing and hence your call can proceed. So but we still have to block because these files are residing on this and they have to be fetched from the disk but these are always considered ready because they are on disk. Other than that, if let's say if it's a file descriptor on a socket, then until the data comes on the socket or until the socket buffer is available for writing, the readiness checks won't be firing, let's say unless the event occurs, unless the readiness events occur. They don't, so the readiness events don't work nicely for awkward files. And that's why even using these interfaces, we can't achieve true asynchronous for regular files. Next slide please. So LibAIO interface. So again, this is an interface that is, that is again a native Linux interface. Basically what it consists of is essentially three system calls. So the first system call is IO setup. So what it does, it basically sets, it initializes the metadata, it initializes the unique instance for this IO operation. Then you submit the IO operations using IO submit system call and we can issue multiple read and write requests using the system call. And then finally we have the final one which is IO get events to synchronously wait for these set of requests. So all the operations that we, IO operations that we issued using IO submit system call, we will have to wait for all of them via the IO get events system call. And the IO get events system call is a blocking system call but the IO submit system call is actually a non-blocking call. So does it solve the problem of regular buffered file IO? Yes, it does solve the problem of regular unbuffered files but regular buffered files still blocked. So this is the interface which we are talking about before that if we open the file descriptors with O non-block flag and if we issue the IO request using this interface, it won't block because it will bypass the page cache and it will issue the calls asynchronously. But if we do not open the flag with that interface with, sorry with that, if we do not open the files with that flag then this interface will also block. So we're talking about IO submit calls, blocking for file descriptors upon which we want to issue, read or write operations. So still it's, I think it's a neat way of achieving batching of multiple IO requests. And as far as I learned about it, it is slated to be the default O6 IO backend in the future. Next slide, please. So now we come to the IO viewing interface. So this is talking about it quickly. This interface was developed by Jen, it's Xpo from Facebook. It was added in Lex 5.1. We have three system calls in this interface, IO ring register, IO ring enter, and sorry, IO ring, the last one is setup. I think so there's not correct data here. So IO ring setup is the third system call. Basically we have two ring buffers or circular arrays which are shared by both kernel and user. So these circular buffers or ring buffers are essentially used by user space to communicate with the kernel. So this is very unique as per what I have seen because normally when we issue a system call, we initialize the pointer to the data, we initialize the metadata and issue a system call. But here we are using ring buffers to actually communicate with the kernel. It's quite interesting and it is quite unique to this interface. So using this, using these circular buffers, we basically submit our iU request and we also look at completion of those iU requests. Next slide please. Yeah, so I believe if we click, we can see how these ring buffers are operating. It's an animation, so we have to click. Okay, so I have got a few questions here. Let me ask the questions if I can. So Praveen's question is that what about AFXDP which is a similar system call interface. So as far as my knowledge, AFXDP flag is being used is being used to do XDP networking via the VPF interface. So if we open sockets, if you have to open sockets, then we use that flag to open a raw socket and then we do the XDP networking. So I'm not sure if the VPF XDP interface, sorry, flag uses the similar interface. I do not think so. But yeah, I think so. I have a next question from Phong. What is your question? Can you stake please? Okay, so I think I'm not getting any answers from Phong. We can come back to it later. But let's see about this slide. Let's continue with the slide. So we have, we here see two ring buffers here. One is the submission queue and another is the completion queue. And we see seven submission queue entries here. So the user has initialized seven submission queue entries and the user has populated the submission queue with these entries. Now, after populating the submission queue, so we have some animation here. So basically with the queue, you consistently update the tail of the queue. So the user space app updates the submission queue and keeps on adding SQE to the circular ring buffer. After this has been done, then the user issues a IOU ring enter system call. So this is the main system call that indicates the kernel that, hey, we have a few entries in the submission queue ring and please read those entries and please complete the operations that are corresponding to those entries. So after the kernel knows about it, the kernel comes into picture and starts reading all the SQEs. So one by N, it will read the head of this ring buffer. So it keeps on reading the head of the circular array that is the submission queue and then it will issue the IO operation for each of the submission queue entry. And then finally, when the IO operation is complete, it will go to the completion queue and then start populating the completion queue ring buffer. So the completion queue ring buffer will then have all the completion queue events and then the user can start reading via, so the user can in the backgrounds keep reading the completion queue. So it's asynchronous in that the user can independently fill up the submission queue, issue a call, sorry, okay, yeah. I got a message from Swam, he just wanted to say hi to me, sorry, Swam, I was expecting a question, sorry, yeah. So I was telling that the user app can update, keep adding SQEs entries to the submission queue ring and then the kernel reads the entries from the submission queue ring and then the kernel when after the completion of each and every IO operation updates the completion queue ring, which the user reads. So these both submission and completion are decoupled, which results in asynchronous. So the user space application can independently keep filling the submission queue and asynchronously also reading the completion queue for the completion of the events, next slide please. So discussing about it in general, so this interface provides true asynchronous for all operation types. So if you're talking about storage networking, if you're talking about storage or networking, then any kind of operations are handled here. We also need to talk about the SQPole mode, which is very interesting because we do not have to enter and we do not have to make any system call if we use this plan. So we'll discuss about it later, but we'll just discuss briefly about it now that we can completely bypass the kernel and issue all our IO operations and which helps us in not making any context switches at all. So this can really transform the way that we interact with the kernel for making IO operations. Next slide please. So here I just presented example of the SQPole mode in which you can run IO-euring interface and in this mode, there are animations here, in this mode, the user space app just fills the submission queue entries in the submission queue ring buffer. They basically just update the tail and then the kernel creates a background thread and the background thread pulls for any entries that come in the submission queue and then reads all the entries and then basically performs the IO operations and then fills the completion queue with the completion events. The kernel is eagerly creating a background thread and then processing all the requests which are without having the user space application to make a system call. So that is the difference between this mode and the normal mode. So in this mode, there's no system call as we are seeing. It can operate without that. Next slide please. So discussing again about, so apart from read and write operations, it has support for send, receive, open, close, connect, fstack, if allocate. So all this system calls that we normally make that we somehow interact, somehow do an IO operation here. All these operations are supported. So let's say if you want to issue a connect operation and if you want to connect to a socket, we can do that. We can do a read, we can do a send, we can do a receive. So we do not, we normally we do not use the system calls that have been provided to us by this interface directly. We can though, but it's a helper library has been made available to us to use or to leverage the system calls. The helper library is liburing and this library provides us several functions that we can use to avoid the complexity of the ring buffer management and the ring buffer management of the submission and completion queue. So again, one thing that we should take note of is the both the submission and completion queue are shared by user space and the kernel. So those, the ring buffers that were used for communication between the user space and kernel are being shared by both of them. So we do not make any copies, we do not make any copies of data while we're transferring the request from user space to kernel. So there is significant amount of savings in copying data. Also, because the IO operations itself are unpredictable, so we cannot guarantee the order of submission and completion requests. So if let's say we issue some submission requests in order, the completion events for those requests might not be in order because they're all asynchronous. So depending on which one of them complete first, we'll have the completion queue events for each of those IO operations. Yeah, performance wise, lastly, yeah, performance wise, we have seen a good performance, we have seen much better performance from IO-euring interface than compared to previous interface phases. We'll discuss, we'll see, look at the performance numbers. Although there are quite some resources there which point that IO-euring is performing really well in at least in buffered reads, in normal use cases where we are doing reads and writes call, also in networking cases, let's say where we are trying to run a server. So, yeah, performance wise, it's also good. So let's look at design with similarity. Circular buffers, as we know, are an elegant design pattern, elegant architecture. Why? Because they have back pressure built in. So you do not, it's not like a linked list where you can keep adding entries. There is a fixed size to it. So your memory is bounded, your memory is reusable and your back pressure is built in. So let's say if your ring size is full, then you won't accept any entry from the user space application. So reverse is true for the kernel as well when kernel is filling the completion queue ring buffer. So when the completion queue ring buffer is full, the kernel will have to wait. So the back pressure is built in. Also, ring buffers are used ubiquitously. For example, in networking, we use ring buffers where the socket entries, socket buffers, then we create them in Linux after getting them through the networking driver. The memory is bounded. So it's a ring buffer which keeps on being reused for the sockets which are being allocated feed continuously as the networking layer processes those packets. Microsoft Windows uses IOCP as the traditional IO stack which is the IO completion ports which is similar in design to IO ring as it also uses ring buffers which is a generic asynchronous IO interface for Windows. So Windows has something similar which also provides asynchronous interface for making IO operations. Next slide, please. So the first set of calls that was wrongly mentioned in one of the previous slides is IO ring setup. So this is basically to create an IO instance, a unique IO instance, IO ring instance that we can, we'll be using in the system calls after this. So basically we provide the number of entries we want to make the number of entries that the number of operations we want to, IO operations we want to make and some parameters. So in these parameters, most importantly are the flags. Next slide, please. So most of the entries here, if you populate the flags most of the entries here will be populated by the kernel itself. So flags is something that we discussed SQL mode. Here we can configure the IO instance to start from SQL mode and we can also configure it from IO pole mode. So in the IO pole mode, the application has to pull the kernel for any kind of completion events. So normally the application just needs to read the completion queue events, but events from the completion queue. But if we use this flag then the application will have to pull the kernel for any kind of completions. We initialize struct of IO ring patterns and then using some checks, we can initialize the flags that if you want to use IO pole mode or if you want to use SQL mode. And if you do not use any of these modes it's also completely fine. We then issue the system call where we want to initialize IO ring instance for 100 entries and some parameters that we initialized. Next slide, please. Yeah, the submission. So let's talk about the SQL struct here. So this, so when we talk about submission queue ring buffers it is filled with instances of these structs. So each entry of the submission queue ring buffer is an instance of this struct. So if we look at the fields that are used here the opcode is what specifies which operation we want to perform. Do we want to perform a read or a write or a send or a receive, connect, close, accept, whatever. So flags is something that we'll discuss later on. IO priority or IO prio is basically to indicate if we can prioritize this request. File descriptor or FD is something that which, what is the FD that you want to do the operation upon. Other fields are also kind of metadata related to the submission entry. The user data field is something that we that the user provides. So let's say if we issue a read operation then we want some buffer upon which the kernel can write the response of that read request too. So the user data is actually that. So this is what is passed to the completion queue entry as well and the user can read this data completely. So we also see a few flags here that are for each operation. So you have some flags for F sync, you have some flags for accept, cancel. So these are per operation flags. So that is that, so let's move to the next slide. We will discuss how we initialize this previous idea. So this is a bit complex as from the code that we can see, not, I mean not a lot, but yeah, somewhat. So we basically, because it's a ring buffer, we know that there are lots of, there are head and tail. We know that we have some flags and we also keep some metadata around the dropped or submitted around the offset. So basically what we're doing is we are creating a shared memory space between the user and the kernel.  And finally we are after having all this metadata we are complete with our submission queue entry struct, next slide please. So now this is the completion queue entry. This is basically for the completion queue ring buffer we have each entry of that ring buffer is an instance of this struct. So in this struct we have three fields and it's pretty simple as compared to the submission queue entry struct where we have user data, which is again the same thing that I pointed before that is the address at which the kernel can populate data of populate the result data. So after the completion of any IO event, whatever the result was of that IO event the kernel can populate the result of that IO event in this field. Result, RES or result is used to indicate if the flag, sorry, it was to indicate if the IO operation was successful or not. So that is indicator of if this operation was successful. Flags is something that is used to signal that which buffer was used for submission. So we'll talk about again this flag later on. Let's move ahead. So here we see, here we try to initialize a completion queue entry. So here again it's very similar to submission queue entry where we do an M map and we just update a few metadata fields mostly head, tail and some metadata around the completion queue. Overall, what we're trying to do here is that we are trying to study the submission queue entry struct and initialize the metadata around the struct and the completion queue struct. Next slide please. The kernel that we have a few submission queue entries we want entering entry call. So this is the system call that is used by the application to issue and wait for an IO request. So here I have used the word wait for an IO request but it's not necessary. So normally if you indicate via the flags that has been used in the parameter. So this IO ring entry call has five parameters, the file descriptor first of them, the two submit second of them, min complete flags and sync. So in flags you have two flags that we want to talk about, IO ring enter get events and IO ring enter sqwaker. So if you use the first flag, it means that we want to wait for min complete number of events to be in the completion queue till this call is complete. So this call will block until there are min complete IO completions. So if let's say if we pass min complete as three and if we set this flag as two in the bit set, then it will wait for three IO operations to complete. And then only after that it will return. So if we do not use this flag, it's not necessary that it won't block and this will complete. So IO ring enter sqwaker is a flag that we use to signal the kernel to wake up and start polling again. So normally when we use IO ring in sqwale mode, the kernel creates a thread which continuously pulls the submission queue ring buffer for any metadata. So any kind of entry in the submission queue ring buffer will be pulled by the kernel thread. But let's say if the submission queue ring buffer is quiet and there are no entries there. So then the kernel thread will start sleeping and this flag is used by the user space application to indicate that the kernel can start the background thread again. Next slide please. So IO ring register is basically to pre-register set of buffers or files which the user space application can later on leverage for writing or reading data. So let's say if we have a file descriptor, the kernel will want to know the file reference of it. The kernel will want to know what is the buffer, what is the buffer that he wants to access to write data of the result too. So for those meta data or for those if the kernel wants to do that for each and every IO operation, the kernel will have to get the file reference. The kernel will issue get or output calls. The kernel will also try to copy the user data that is there in the submission queue entry and then copy it back to the completion queue entry. Now what these system call does is that it provides an option to the user application to pre-register those buffers. If they pre-register those buffers, then the kernel will not have to copy data while reading an IO while the, let's say if it's processing an IO operation. So it makes things cheaper. So when we have read all of this, we need to summarize ourselves with the complexity that are involved in this whole thing. That we didn't discuss about one important thing that when we are reading and writing to the ring buffer via the submission queue entry and the completion queue entries, we want to have some kind of synchronization between the kernel and the user space. And that synchronization is brought by memory barriers. So all of this initialization plus synchronization is a bit complex. And then the liburing library helps to alleviate all that complexity. Next slide, please. So if we use the liburing library that is provided by the glibs, that's provided by glibc. We just, we say here we see how we use this library. We first issue a IO ring queue I need to call to get a unique IO ring instance. Then we get a submission queue entry. Then we set the submission queue entry with some data that we want to provide. And then finally we submit this IO ring operation. So this is the submission part of it. Next slide, please. So then we take the ring instance that was given to us by the IO ring queue I need to call. And then with the completion queue entry that we have initialized it, we basically wait for that completion queue entry to be available. And then finally we get data from it once the IO ring wait CQE call returns. So using these set of calls, we have completely avoided the submission queue entry initialization, the ring buffer initialization, the synchronization between the kernel and user space, all that is completely avoided. So it becomes quite simple to use this IO interface. But yeah, if we wanted to, I think so for both control, we can still use that interface. People prefer more control, people prefer simplicity. I think so, people prefer it either ways. But yeah, this library helps it much, helps makes it much easier to use the interface. Next slide, please. So here we have an example of reading a file by a LibUring interface. So first we open the file using an open system call. Then we initialize the structs that we require. So you ring interface for IO ring struct, then SQE IO rings SQE struct, then IO data, IO ring SQE struct. So the IO data struct is something that we have initialized ourselves. This is not something that we have to do normally. So this we have initialized ourselves so that the kernel can understand it. So what we do is that we first issue an IO, using a QINit call, then we finally, then we get an SQE entry. Then we prep that SQE entry with the read request and then we set the SQE with the data, then we submit that request.  So we have made here five to six calls and we have completely ended the complexity that was associated with the previous, that was issued with the previous set of code. I'll discuss about these flags a bit later. Let's move to the next slide here. Yeah, so when we saw the submission Q entry struct we saw a few flags there. Let's discuss about those flags now. So this flag, the first flag, IO SQE IO ring flag, what it does is that it creates an ordering between the SQE's. So all the SQE's which we populate in the submission Q entry, it forces all of them to be ordered, not asynchronously each of them. So let's say one, two, three, four, five, six, seven, 10. Then we'll process first, first, second, later, third, fourth, fifth and not each of them asynchronously. So the second flag that is the IO SQE IO ring flag it forces that each entry of the submission Q entry submission Q ring buffer has to be processed first before the new ones have to be taken. So the new ones will be taken later and then after the existing ones have completed. So obviously this is quite a dream and this is actually a performance killer. So the SQE IO SQE Async flag, using this flag, we can the IO ring, the kernel will straight away offload this operation, the IO operation to a kernel thread right from the start. So right from the start, you have a background thread which takes care of all this and you don't, the kernel does not rely upon page callback mechanisms or page callback mechanism or interrupt driven behavior to signal IO completion. Next slide, please. So we have discussed these flags, IO pole and IO SQE pole flags. So using this, the second flag is quite interesting as I discussed that the user space application does not need to make any system call here. It can directly populate the submission Q ring buffer and the kernel will create a background thread and will read entries from the submission Q ring buffer and populate the results of them to the completion Q ring buffer. In the IO, using the IO pole flag, the application will have to pull for any completions using the IO ring entry system call. So normally this is done with applications to have low latency. So we have other flags which I'll skip but yeah, I think we can read about it but yeah, I think I'll skip here. So I have a few minutes left. So let's go to performance, the size which we discussed about performance. Yeah. So let's discuss about performance aspects of it. On AWS, in CEC2, I bench, I had, I made, I connected to performance benchmark of using FIO. So it was an M5B large instance with x86, 64. There was two, they were two, a VCQs, a Geo physical memory. It was in the storage was 75 GB of non-rotal memory. The kernel version was 5.4, 5.4.0. It was, it was, the operating system was going to do 20.04. We have IO ops and bandwidth benchmarks and the IOQ that use was 16. Although higher IOQ lens were also indicating same performance. So let's look at the numbers. Yeah. So the green one is for, is for blocks which are of four kilobytes, red from red, being red from the disk and the red ones are for the blocks which are of eight kilobytes being red from the disk. These are all, so one thing I forgot to mention was these are all random read operations being done for a file on the disk. So before any, before any benchmark result, we had to clean the, I had to flush the caches. So I know dentry, page caches, everything was flushed and normally we were doing a random read operation upon the file in disk. That file size was 4 GB as we saw in the previous slide. I have a few questions now that have come to me. I'll try to answer them later. So please be with me. Sorry. So in the previous slide, we saw that for direct IO, sorry, can we go back to the previous slide? Yeah. So the previous one again. Yeah. So we see in these benchmarks that for direct IO, for direct IO, if you want to bypass the page cache, the IO ring instances is comparable or better than performance amongst all the engines, but LibIO comparably fares well. So this is true for both block sizes, eight kilobytes and four kilobytes. And we have, we have compared POSIX, AIO and SYNC. SYNC is normally reads and writes system call, AP issue reads and writes system call. Then SYNC, we use SYNC for that in FIO. So this is for direct access to files, which means that we want to bypass the page cache. The second, let's go to the next slide. In the buffer, so this is basically, we are measuring IOs when we want to use this page cache. And we see that LibIO is clearly slower here because LibIO will prove, will actually not, will prove uses because LibIO is not asynchronous for buffered files. It is asynchronous only for files which are opened with four direct flag. So here we see IO ring clearly winning in IOPS than LibIO, POSIX or the SYNC interface. Next slide please. Bandwidth wise again, SYNC numbers are pretty similar that for direct IO, the numbers for IO ring and LibIO look similar, but POSIX and SYNC are still slower. So I have, it's saying that I have a couple of minutes, I'll just rush to the slides. Next slide please. So for buffered IO, we see that IO ring bandwidth is clearly better than all the interfaces. Next slide please. So observations here that IO ring is performing well. The instance that I used was bound by CPU, but not by the disk bandwidth because the NVMe would, NVMe not volatile memory storage could not be saturated, but the CPU utilization was exceeded 100%. So we needed more CPUs to test the two performance, but what results, what we can observe here is that even with limited CPU, IO ring pairs much better than other interfaces for bandwidth and IOPS. Next slide please. So debugging and observable. So we have a few static trace points that have been added in the interface. These static trace points, using these static trace points, what we can do is we can use a BCC or BPF trace or we can use modern debugging, next debugging tools to actually measure what is the latency or what is, if let's say if an IO operation is submitted but not complicated, if an IO operation is indeed submitted, we can measure all these things using these trace points that have been listed here. Next slide please. So here's an example of using BPF trace for measuring latency, where we are putting trace points using the submit SQT trace point and the completion SQT trace point. Next slide please. I wanted to talk about internal mechanics, but I think we are short of time. I'll, let's try to go to the adoption slide. Yeah, I think so this, I think so let's discuss about this. So yeah, ROXDB is using IO rings in its backend. There have been significant improvements in the multivit queries using IO ring as the backend. LibEV, which is a library for applications to leverage the event loop, might be using IO ring as the backend. So yeah, I think this is what I could have discussed in this time frame. Sorry for rushing through the slides. Big thank you to these two folks who reviewed all these slides and made several comments about the content of these slides and helped me improve on these slides. Gens also is the author of the interface. I'll try to answer questions now that have come from the people. So the first, so William has a question, William Lavender has a question. Will you be putting your slides on the ELC website? Yes. Answering question from Steve. What interval does the IO ring K thread use in SQL mode? Is the thread triggered by dirtying the off the submission Q page? Yes. The second part is true, which is let's say if the submission, if the submission Q ring buffer is, if there is an entry in the submission Q ring buffer, the kernel thread will start processing that entry. I'm not, I do not have knowledge of the interval, so I'll have to read more about that. Could send file use this IO ring?  Thank you for having me. Thank you for listening to this talk. Yeah, I'll answer all the questions in the, I synchronously. And please feel free to ask me any questions. I'll share this, I'll share the set of slides within the ELC website as well. Thank you, thank you guys.


Delicious. Hey y'all, it's Timmy. We're calling Belly Cooks and today we're making a no-bake Jello cheesecake with some fresh peaches. California peaches and I chopped up a couple of small Georgia peaches in here. The Georgia peaches have an orange flesh and the California peaches have a white flesh. The California peaches are real soft and the Georgia peaches are firm. So anyway we've got three quarters, we got two cups of peaches, three quarter cups of sugar and about three tablespoons of butter in here and I've got it on a high heat trying to cook it down to make a syrup. I also put in one little sprinkle of cinnamon. The next thing we're going to do is use our Jello no-bake cheesecake mix. I just thought this would be a good one for you guys if you're in a hurry and want to make something for the family. This is the crust and we're going to add, we're going to add four tablespoons of melted butter, two tablespoons. Lord have mercy. We're going to put in two tablespoons of sugar and we're going to mix this up and get it in our pie crust. So I'm just going to use a little spatula because it makes it easy to press the butter into the crust. So the crust mix, two tablespoons of butter, the crust, four tablespoons of butter, melted and two tablespoons of sugar and all the scissors from cracker crumbs.  So you may as well use it right instead of using one you've already got. Now we're going to press this into the bottom of this pie plate. Now when you, typically when you make your own pie crust, like what we're doing, it's pretty much the same thing, you bake it for a few minutes and they don't ask you to bake it so you know you skip that step. But it's probably a little more crumbly when you serve it than if we were to just put it in the oven and bake it a few minutes. That's totally up to you. We're not going to do that today for sake of time. I'm just going around my edges to make them flat and pretty. All right, there's our crust and this is coming along doing really well. It's got three minutes longer to cook while we mix up our filling. So that's going to be nice and syrupy for us. Peasy breezy. The cheesecake mix is just a jello top cheesecake. We're going to put it in here with one and one half cups of milk. It's powdery, isn't it? So we got one and a half cups of milk and you're supposed to beat this for three minutes. So I'm going to turn my timer on. You're going to mix it on low just for a few seconds. You can see they're nice and they've turned into a pretty dark syrup. That's what we wanted. And these are so pretty we probably don't even need these other peaches to decorate with. Well they look good, don't they? All right, let's get this party started. Because these are so hot, we're going to put them in this pan and put them in the freezer right quick to cool them down because you don't want to put hot peaches on top of of a nice cool cheesecake topping or it would ruin it. To chill this for minutes or so before we can put it on our cheesecake. And I use an eight inch pie plate instead of a nine inch because I wanted it to be nice and tall and one of the reasons they want you to whip it on high at first it splatters but once it starts to thicken the higher you go with your beating the more air it puts in the cheesecake and the prettier it's going to be in your pan. So I use the eight inch pan because I thought it would be really pretty and thick and tall. Now if you can't have a lot of sugar and you couldn't eat that peach topping you could always candy peaches like I did here and lay them on the top just you know for your friends and you and just lay them around on the top. But of course we're going to use our peach topping. I just thought I'd show you what you could do if you're diabetic and you are really watching your sugar. You wouldn't want to use what we're using on the top. But if you've got some friends that you want to make them and they'll bake and have them over I know it's some people are having to be real careful but if your friends haven't been out and you haven't been out might be pretty safe to get together have dessert so that you just don't go plumb crazy. Right look at our pretty fluffy cheesecake and here's our beautiful topping and it's cooled down enough to go on top of the cheesecake. I'm going to stir it up because part of it's cooler than the other and it looks really nice. Now the kids of course don't want to eat the peaches so I'm going to put it on half of the cheesecake at least. And of course you could leave it off and just serve it on the side and you can buy peach topping but not many people have it and this is way better than buying it. Let's dig into it now ran it it's not cool but you'll always like to see me taste something so that's what I'm gonna do I'm gonna cut it right here at the edge and after this sits in the refrigerator for a little while it'll cut easier. Fix a fish trap thing so let me give this a try man it looks good looks really good that is really good delicious I never make these I think I'll buy them. We have sliced then some Georgia peaches because the flesh is a little tougher and I sugared them really well put them in the air fryer at 380 degrees and they're in here cooking with a lot of sugar on them. Matter of fact I think I'm going to sugar them again so we sugared them once we're going to sugar them twice and we're just going to use these to decorate the top of our cheesecake. This is just a simple no-bake cheesecake anybody can make it out of a box. Those are ready for the top they look good so they're nice and candied so I'm just gonna sit this right here for now and we're going to use those peaches on the top of our pie to make it pretty. Thanks for watching Colored Valley Cooks where we cook like mama did get you some fresh peaches y'all make you one of these no-bake cheesecakes


New to Doctor Who? Let me fill you in. A time-travelling, grumpy grandfather who mellows over time thanks to his companions. The Cosmic Hobo who introduced Regeneration, the Sonic screwdriver, and Squatland! Confined to Earth, making gadgets, chopping the master, and looking damn good doing it. Goofy Sirius, one-of-a-kind but definitive, all somehow describe him. And you already own, or want to own, the scarf. A nice young chap who has served some of the hardest decisions any Doctor has ever faced, leading his companions to the brink of insanity, or an early grave. Starts out manic, but ends as a stern but unlovable figure. Colorful? Yes. Carrot juice? Disgusting. The world is his chessboard and everyone is his pawns. The magical Merlin who likes question marks just a little too much. Done literally everything multiple times. Some is hard to define, but goddamn, his voice is silky smooth. He's seen war but he's still chipper. Lover of Bananas? Hater of the Queen? What a king. You know him, you love him, the king of the modern show. Well, allergy? Avantia! Like he needs describing. More quirks than you can shake a Sonic at, and hair floppier than him when river's not round. This baby-faced old man is allergic to normal neckwear. The most perfect of castings, the magician with the attack eyebrows, he is the Doctor. She then remembers she can become anything, so she became another gender, full of childlike wonder that hides many secrets. So hear me out, it's Tenen, but again. I don't know what the hell is going on here, but I can't wait.


Hello people of the internet, my name is Johnny and welcome back to another Game Theory Reaction video. I'm a couple days late on this because my computer just would not turn on and I've been having so many technical issues with it, eventually we just had to wipe it, it kept all my postal files thankfully but I had to re-download everything so if OBS is a little bit weird I apologize, it should be good though. So I'm sorry that I'm pretty late, I think I'm only two days late to this which isn't too bad but you know I don't normally like to be this late. So this episode is a continuation of the previous one where we talked about the Fetch book and I'm pretty sure this is Matt talking about Psychic Friend Fredbear now and according to people on the Reddit this is a pretty interesting theory video so I'm not going to waste any more time and let's hop into it. Okay I think we're all ready, let's hop into it. The Body Snatchers. It's just the full song of Psychic Friend Fredbear now, I like that, oh it's not done. It's creepy. Is it the puppet of Golden Freddy or what? Now how long until someone remixes this? That's the question. Oh no, it's Psychic Friend Fredbear. Uh-oh. Uh-oh. A Game of Fury you could say. I think we need to solve the lore, yes. Am I doing it by asking really extreme over-the-top questions, looking into very specific details way too close and more often than not drawing connections between concepts that probably aren't connected at all? Absolutely, yes. Set effects of entering the theory zone may include a blown mind, a total and other confusion, a deeper appreciation of your favorite franchises, fun stories to share with friends, a head full of random factoids that will at no point aid you in life and disillusionment about human existence in our own place in the universe. Enter at your own risk. Disclaimers aside. Night terrors, night sweats. A good theory is one that has enough evidence to support it, but not so much evidence that it's useful in known fact. It requires extrapolation, thinking not to just the stuff that's so far past that that you're overreaching or making beastless claims. And all of that while still maintaining a satisfying story, a story that stays true to the franchise that you're talking about while not being afraid to take that known story in new, often riskier directions. In a lot of ways, theory crafting is like writing fan fiction, just, you know, with more evidence and less aggressive cuddling. Wait, where was this go? Aggressive cuddling than some of the fan fictions I've been exposed to over the years. Oh, yeah, yeah, yeah. Anyway, the reason I say all of this right now is that the theory is, how do I put it, it's one of the stretchier ones. Like, there are some theories that are solid. There's some really strong evidence in the game and it's supported by stuff that's said by the developers or creators. And there are others that require a bit more creative license, where the connections are there, but they might require you to squint a little bit or just, you know, have fun with it and go along for the ride. And this episode is one of those theories. Because today we're once again diving into the world of Fetch. Yes. Both of you are the Fazbear Fright series. You see, last time- Doesn't 135 come out in May? Was recently released and it is, um, not sure. I'm being honest, it feels much less connected to the series than the first one. The connections to the main lore are definitely a bit shakier. This one, meanwhile, introduces us to a lot of new things. A smartphone-powered animatronic dog named Fetch. Tiny, free-roaming versions of Freddy Fazbear called Lonely Freddy. That felt weird to me. Every story of the first book felt very connected to established characters. Okay. From putting bunny suits to shape-shifting baby to fun-time Freddy. So why was this one, the second book, doing so many new things? It felt like a pretty extreme tonal shift. Look. Look at the books to fill in things from the past. Yes, the plus-trap chasers.  The end of Skeleton? Oh boy. Oh. Okay. Um. It's definitely a stretch. Interesting. Okay. Yeah. Yeah. A suspect foul play. Who would have thought? I don't remember those things. Chuck E. Cheese. I never got anyone though. Yarg. Damn. Wow. Wow. Wow. Wow. Wow. Wow. Wow. Wow. Wow. Wow. Wow. Wow. Yarg. Oh. Wow. Wow. Nice. Hey boy. I got two toes. Because Alec's story strangely enough closely matches what we see happening in FNAF 4's Crying Child. Now, I don't think Alec is the Crying Child by any means. There are just way too many differences. He's definitely gonna have to sell me on this. Similar these two stories are because the parallels seem intentional. They seem to be inviting us to compare these two stories. So, let's do exactly that. Just like in FNAF 4, there's a countdown to a birthday party happening at Freddy's at the end of the week. Though in this case, it's a sister's party and not his. Speaking of the sister, Alec's younger sister Hazel is described as being cute and perfect with a lot of emphasis being placed on her green eyes and her curly yellow hair. An important physical description considering Elizabeth, the girl who would go on to possess baby and who we assume is Crying Child's sister and owner of this room in FNAF 4, is also blonde. If it was hard to go back to FNAF 4 with these theories, Very nostalgic. Hey. Alec and Hazel have green eyes and that lonely Freddy has blue eyes. Which, if you have an obsessive level of knowledge of the series, should ring the few alarm bells. In sister location, eye colour was a huge deal. So much so that I did a whole theory about it back when sister location came out. Yeah, so you're very nostalgic. Oh, God, right. I was thinking like, where is he going with this? Alec burned through his soul and he searched for answers of his own, but he only came away with more questions. Because the blue eyes of the bear had suddenly turned to light green. End quote. And again, in this quote, all Alec could see was the bear as its newly green eyes bore through him. So we've got a party, a sister. Why is there so much talk about eyes? It's weird. A sister and some eyes, but obviously there's got to be more, right? Alec, when his plan fails, runs off to hide in the storage room crying. Similar to the crying child being locked in the storage room in Fnap 4. And in general, Alec is just a sulky, moody kid who, similar to our crying child, both loves and hates Freddy Fazbear's. Alec, in the book, secretly loves the restaurant and its characters, but he also hates it because he only gets to go when it's for his sister. Which makes him bitter and angry to be there. Hmm. Okay. Yeah.  It's a small detail, but one that immediately made me think of the foxy that's in the crying child's head. He's missing his head, not the arm. Okay. Hey. That's no nice. Just like we see the older brother doing to torment the crying child in Fnap 4. Again, it's not exactly the same. Okay, but where's this going? Is that what the books are supposed to be? Oh. Aren't the books supposed to be alternate universes from the games? Maybe. Isn't that neat. Kind of weird. Kind of weird. Kind of weird. That's not nice. So that boy in the teaser looks really different from what he's supposed to look like. Sort. Forced it. Huh. Okay. Okay. Can't wait a second. Okay. That's kind of a big part of the book though. Okay. The man behind the slaughter. Damn it, Dale. We saw a psychic friend Fredbear for the first and only time in a non-8-bit setting.  So there you have it, friends. The official name of psychic friend Fredbear is Lonely Freddy. It's not quite as fun, but yes, it's easier for me to type, so that's a plus. I'm so confused about what this video is supposed to be. If you put this into a pot through a map app. Book plan. It's book time. Book time. God, I don't like those eyes. Yes, yes, Jesus. A girl who shares a name with the child that we know goes on to possess the puppet. Any descriptions? Wait, why would she be outside when she's throwing up? Go to the bathroom, man. It's book number four and five, coming out later this year that seem like they may hold the biggest lore reveals yet. It's been a while since I've read these descriptions. Yes, yes, yes, yes. I was the first. I have seen everything. Two brothers with divorced parents that have some blood curse on their families. You don't say. Yes. Yes. Yes. Yes. God, I hope there's more. Yeah. I feel like a sponsor's coming in. Wait. No! Okay, so I'm a little confused here. He didn't Matt say that this is just comparing the two? Comparing the similarities between the crying child story and Alec's story. But then he goes on to say that the crying child, or not crying child, my bad. That psychic friend Fredbear is called Lonely Freddy, right? So why? I'm just kind of confused about where this theory was heading. It is interesting. There are a lot of similarities between the two stories. But I'm just trying to figure out whether this was a comparing video or if it was a lore video piecing the two together and saying that Lonely Freddy is psychic friend Fredbear or if he is just comparing them. I'm not sure which one it is anymore. There you have it, friends. The official name of psychic friend Fredbear is Lonely Freddy. It's not... Okay, so there he's saying that psychic friend Fredbear is Lonely Freddy. But earlier on he was just comparing the two stories. So I'm just... Maybe I've missed something. Maybe I misheard him. But I'm... To truly understand why I think these two are connected. Connected, not like filling in the pieces of puzzle. I don't know. I don't know. I will admit this is an interesting video. He makes a lot of interesting points. But I'm just confused on the conclusion. Maybe I misheard something. But for right now, I'm not entirely sure what this theory was supposed to be because the books... Now correct me if I'm wrong because I'm almost positive you guys will. The books are supposed to be an alternate universe, right? From the games. That's what Scott originally said in that Steam post. They're not direct links, but they're similar. They're different stories from the universe, you know? So these two stories having similarities is the point of the story. Am I wrong? I don't know. It is... There are some interesting points, but I feel like... I just... I don't really know where Matt was going with this. So that's gonna be it for me today. Again, an interesting theory. It's just the conclusion is a little weird. But then again, maybe it's my fault. Maybe I misheard something. I'm not entirely sure. But this was game theory. FNAF for the body snatchers. Thanks for watching this episode. And I'm pretty sure 135am, the third Fazbear Fright book comes out in May. Actually let me double check. Yes, May 5th, 2020. So it's actually sooner than I thought. Oops. So yeah, we have that to look forward to. But for right now, thank you so much for watching another Reaction to Game Theory. And I'll see you all on the flip side. Goodbye.